diff --git a/CI_CD_PIPELINE_SETUP_GUIDE.md b/CI_CD_PIPELINE_SETUP_GUIDE.md index ac2b278..b3bb576 100644 --- a/CI_CD_PIPELINE_SETUP_GUIDE.md +++ b/CI_CD_PIPELINE_SETUP_GUIDE.md @@ -479,12 +479,18 @@ EOF #### 4.3 Create Authentication File ```bash +# Switch to SERVICE_USER (registry directory owner) +sudo su - SERVICE_USER + # Create htpasswd file for authentication (required for push operations) mkdir -p /opt/registry/auth htpasswd -Bbn push-user "$(openssl rand -base64 32)" > /opt/registry/auth.htpasswd # Create a read-only user (optional, for additional security) htpasswd -Bbn read-user "$(openssl rand -base64 32)" >> /opt/registry/auth.htpasswd + +# Exit SERVICE_USER shell +exit ``` **What this does**: Creates user credentials for registry authentication. @@ -493,9 +499,14 @@ htpasswd -Bbn read-user "$(openssl rand -base64 32)" >> /opt/registry/auth.htpas **Note**: Pull operations are public and don't require authentication, but push operations require these credentials. +**Security Note**: We switch to SERVICE_USER because the registry directory is owned by SERVICE_USER, maintaining proper file ownership and security. + #### 4.4 Create Docker Compose for Registry ```bash +# Switch to SERVICE_USER (registry directory owner) +sudo su - SERVICE_USER + cat > /opt/registry/docker-compose.yml << 'EOF' version: '3.8' @@ -532,6 +543,9 @@ networks: registry_network: driver: bridge EOF + +# Exit SERVICE_USER shell +exit ``` #### 4.5 Install Required Tools @@ -544,13 +558,22 @@ sudo apt install -y apache2-utils #### 4.6 Start Registry ```bash +# Switch to SERVICE_USER (registry directory owner) +sudo su - SERVICE_USER + cd /opt/registry docker-compose up -d + +# Exit SERVICE_USER shell +exit ``` #### 4.7 Test Registry Setup ```bash +# Switch to SERVICE_USER (registry directory owner) +sudo su - SERVICE_USER + # Check if containers are running cd /opt/registry docker-compose ps @@ -583,6 +606,9 @@ docker pull localhost:5000/test:latest # Clean up test image docker rmi localhost:5000/test:latest rm /tmp/test.Dockerfile + +# Exit SERVICE_USER shell +exit ``` **Expected Output**: @@ -1364,476 +1390,4 @@ tail -f /opt/APP_NAME/backup.log ```bash # If you created a local copy, use that instead (crontab -l 2>/dev/null; echo "0 2 * * * cd /opt/APP_NAME && ./backup-local.sh --type production --app-name APP_NAME >> /opt/APP_NAME/backup.log 2>&1") | crontab - -``` - -#### 18.7 Monitoring Script - -**Important**: The repository includes a pre-configured monitoring script in the `scripts/` directory that can be used for production monitoring. - -**Repository Script**: -- `scripts/monitor.sh` - Comprehensive monitoring script with support for both CI/CD and production environments - -**To use the repository monitoring script**: -```bash -# The script is already available in the cloned repository -cd /opt/APP_NAME - -# Make the script executable -chmod +x scripts/monitor.sh - -# Test production monitoring -./scripts/monitor.sh --type production --app-name APP_NAME -``` - -**Alternative: Create a local copy for convenience**: -```bash -# Copy the script to the application directory for easy access -cp scripts/monitor.sh /opt/APP_NAME/monitor-local.sh -chmod +x /opt/APP_NAME/monitor-local.sh - -# Test the local copy -cd /opt/APP_NAME -./monitor-local.sh --type production --app-name APP_NAME -``` - -**Note**: The repository script is more comprehensive and includes proper error handling, colored output, health checks, and automatic environment detection. It provides better monitoring information than a simple local script. - -#### 18.7.1 Set Up Automated Monitoring - -```bash -# Create a cron job to run monitoring every 5 minutes using the repository script -(crontab -l 2>/dev/null; echo "*/5 * * * * cd /opt/APP_NAME && ./scripts/monitor.sh --type production --app-name APP_NAME >> /opt/APP_NAME/monitor.log 2>&1") | crontab - - -# Verify the cron job was added -crontab -l -``` - -**What this does:** -- **Runs automatically**: The monitoring script runs every 5 minutes -- **Frequency**: Every 5 minutes to catch issues quickly -- **Logging**: All monitoring output is logged to `/opt/APP_NAME/monitor.log` -- **What it monitors**: Container status, recent logs, CPU/memory/disk usage, network connections, health checks - -**To test the monitoring manually:** -```bash -cd /opt/APP_NAME -./scripts/monitor.sh --type production --app-name APP_NAME -``` - -**To view monitoring logs:** -```bash -tail -f /opt/APP_NAME/monitor.log -``` - -### Step 19: Set Up SSH for CI/CD Communication - -#### 19.1 Generate SSH Key Pair - -```bash -ssh-keygen -t ed25519 -C "production-server" -f ~/.ssh/id_ed25519 -N "" -``` - -#### 19.2 Create SSH Config - -```bash -cat > ~/.ssh/config << 'EOF' -Host ci-cd - HostName YOUR_CI_CD_IP - User SERVICE_USER - IdentityFile ~/.ssh/id_ed25519 - StrictHostKeyChecking no - UserKnownHostsFile /dev/null -EOF - -chmod 600 ~/.ssh/config -``` - -### Step 20: Exchange SSH Keys - -#### 20.1 Get Your Public Key - -```bash -cat ~/.ssh/id_ed25519.pub -``` - -**Important**: Copy this public key - you'll need it for the CI/CD server setup. - -#### 20.2 Add CI/CD Server's Public Key - -```bash -echo "CI_CD_PUBLIC_KEY_HERE" >> ~/.ssh/authorized_keys -sed -i 's/YOUR_CI_CD_IP/YOUR_ACTUAL_CI_CD_IP/g' ~/.ssh/config -``` - -**Note**: Replace `CI_CD_PUBLIC_KEY_HERE` with the actual public key from your CI/CD server. - -### Step 21: Update Application Configuration for CI/CD - -#### 21.1 Update Environment Variables - -```bash -cd /opt/APP_NAME -nano .env -``` - -**Required changes**: -- Replace `YOUR_CI_CD_IP` with your actual CI/CD Linode IP address -- Replace `your_secure_password_here` with a strong database password -- Update `DATABASE_URL` with the same password - -### Step 22: Configure SSL Certificates (Optional - Domain Users Only) - -**Skip this step if you don't have a domain name.** - -#### 22.1 Install SSL Certificates - -```bash -sudo certbot --nginx -d your-domain.com -``` - -#### 22.2 Copy SSL Certificates - -```bash -sudo cp /etc/letsencrypt/live/your-domain.com/fullchain.pem /opt/APP_NAME/nginx/ssl/ -sudo cp /etc/letsencrypt/live/your-domain.com/privkey.pem /opt/APP_NAME/nginx/ssl/ -sudo chown SERVICE_USER:SERVICE_USER /opt/APP_NAME/nginx/ssl/* -``` - -### Step 23: Test CI/CD Integration - -#### 23.1 Test SSH Connection - -```bash -ssh ci-cd 'echo Connection successful' -``` - -**Expected output**: `Connection successful`. - -#### 23.2 Test Registry Connection - -```bash -curl http://YOUR_ACTUAL_CI_IP:5000/v2/_catalog -``` - -**Expected output**: `{"repositories":[]}` or a list of available images. - ---- - -## Part 3: Pipeline Configuration - -### Step 24: Configure Forgejo Repository Secrets - -Go to your Forgejo repository → Settings → Secrets and Variables → Actions, and add the following secrets: - -#### Required Secrets: - -- **`CI_HOST`**: Your CI/CD Linode IP address - - **Purpose**: Used by the workflow to connect to your private Docker registry - - **Example**: `192.168.1.100` - -- **`PROD_HOST`**: Your Production Linode IP address - - **Purpose**: Used by the deployment job to SSH into your production server - - **Example**: `192.168.1.101` - -- **`PROD_USER`**: SSH username for production server - - **Purpose**: Username for SSH connection to production server - - **Value**: Should be `DEPLOY_USER` (the deployment user you created) - - **Example**: `deploy` - -- **`PROD_SSH_KEY`**: SSH private key for deployment - - **Purpose**: Private key for SSH authentication to production server - - **Source**: Copy the private key from your CI/CD server - - **How to get**: On CI/CD server, run `cat ~/.ssh/id_ed25519` - - **Format**: Include the entire key including `-----BEGIN OPENSSH PRIVATE KEY-----` and `-----END OPENSSH PRIVATE KEY-----` - -#### Optional Secrets (for enhanced security and flexibility): - -- **`APP_NAME`**: Application name (used for directory, database, and image names) - - **Purpose**: Controls the application directory name and database names - - **Default**: `sharenet` (if not set) - - **Example**: `myapp`, `webapp`, `api` - - **Note**: This affects the deployment directory `/opt/APP_NAME` and database names - -- **`POSTGRES_USER`**: PostgreSQL username for the application database - - **Purpose**: Username for the application's PostgreSQL database - - **Default**: `sharenet` (if not set) - - **Example**: `appuser`, `webuser`, `apiuser` - - **Note**: Should match the user created in the PostgreSQL setup - -- **`POSTGRES_DB`**: PostgreSQL database name for the application - - **Purpose**: Name of the application's PostgreSQL database - - **Default**: `sharenet` (if not set) - - **Example**: `myapp`, `webapp`, `api` - - **Note**: Should match the database created in the PostgreSQL setup - -- **`POSTGRES_PASSWORD`**: Database password for production - - **Purpose**: Secure database password for production environment - - **Note**: If not set, the workflow will use a default password - -- **`REGISTRY_USERNAME`**: Username for Docker registry (if using authentication) - - **Purpose**: Username for private registry access - - **Note**: Only needed if your registry requires authentication - -- **`REGISTRY_PASSWORD`**: Password for Docker registry (if using authentication) - - **Purpose**: Password for private registry access - - **Note**: Only needed if your registry requires authentication - -#### How to Add Secrets: - -1. Go to your Forgejo repository -2. Navigate to **Settings** → **Secrets and Variables** → **Actions** -3. Click **New Secret** -4. Enter the secret name and value -5. Click **Add Secret** - -#### Security Notes: - -- **Never commit secrets to your repository** -- **Use strong, unique passwords** for each environment -- **Rotate secrets regularly** for enhanced security -- **Limit access** to repository settings to trusted team members only - -### Step 25: Test the Complete Pipeline - -#### 25.1 Push Code Changes - -Make a small change to your code and push to trigger the CI/CD pipeline: - -```bash -# In your local repository -echo "# Test deployment" >> README.md -git add README.md -git commit -m "Test CI/CD pipeline" -git push -``` - -#### 25.2 Monitor Pipeline - -1. Go to your Forgejo repository -2. Navigate to Actions tab -3. Monitor the workflow execution -4. Check for any errors or issues - -#### 25.3 Verify Deployment - -After successful pipeline execution: - -```bash -# Check application status -cd /opt/APP_NAME -docker-compose ps - -# Check application logs -docker-compose logs - -# Test application access -curl -I https://your-domain.com # or http://YOUR_PRODUCTION_IP -``` - ---- - -## Testing and Verification - -### Step 26: Test Application Access - -**If you have a domain:** -```bash -# Test HTTP redirect to HTTPS -curl -I http://your-domain.com - -# Test HTTPS access -curl -I https://your-domain.com - -# Test application health endpoint (checks backend services) -curl https://your-domain.com/health -``` - -**If you don't have a domain (IP access only):** -```bash -# Test HTTP access via IP -curl -I http://YOUR_PRODUCTION_IP - -# Test application health endpoint (checks backend services) -curl http://YOUR_PRODUCTION_IP/health -``` - -**Expected health endpoint response:** -```json -{ - "status": "healthy", - "service": "sharenet-api", - "timestamp": "2024-01-01T12:00:00Z" -} -``` - -**Note**: The `/health` endpoint now proxies to the backend service and returns actual service status. If the backend is not running, this endpoint will return an error, making it a true health check for the application. - -### Step 27: Test Monitoring - -```bash -# On CI/CD server -cd /opt/registry -./scripts/monitor.sh --type ci-cd - -# On Production server -cd /opt/APP_NAME -./scripts/monitor.sh --type production --app-name APP_NAME -``` - -### Step 28: Test Registry Access - -```bash -# Test registry API -curl http://YOUR_CI_CD_IP:5000/v2/_catalog - -# Test registry UI (optional) -curl -I http://YOUR_CI_CD_IP:8080 -``` - ---- - -## Troubleshooting - -### Common Issues - -1. **Docker permission denied**: - ```bash - sudo usermod -aG docker SERVICE_USER - newgrp docker - ``` - -2. **SSL certificate issues** (domain users only): - ```bash - sudo certbot certificates - sudo certbot renew --dry-run - ``` - -3. **Application not starting**: - ```bash - cd /opt/APP_NAME - docker-compose logs - ``` - -4. **SSH connection failed**: - ```bash - ssh -v ci-cd 'echo test' - ssh -v production 'echo test' - ``` - -5. **Registry connection failed**: - ```bash - curl -v http://YOUR_CI_HOST_IP:5000/v2/_catalog - ``` - -6. **Actions runner not starting**: - ```bash - sudo systemctl status forgejo-runner.service - sudo journalctl -u forgejo-runner.service -f - ``` - -### Useful Commands - -- **Check system resources**: `htop` -- **Check disk space**: `df -h` -- **Check memory usage**: `free -h` -- **Check network**: `ip addr show` -- **Check firewall**: `sudo ufw status` -- **Check logs**: `sudo journalctl -f` - -### Security Best Practices - -1. **Service Account**: Use dedicated `SERVICE_USER` user with limited privileges -2. **SSH Keys**: Use Ed25519 keys with proper permissions (600/700) -3. **Firewall**: Configure UFW to allow only necessary ports -4. **Fail2ban**: Protect against brute force attacks -5. **SSL/TLS**: Use Let's Encrypt certificates with automatic renewal (domain users only) -6. **Regular Backups**: Automated daily backups of database and configuration -7. **Container Isolation**: Applications run in isolated Docker containers -8. **Security Headers**: Nginx configured with security headers -9. **Registry Security**: Use secure authentication and HTTPS for registry access - -### Monitoring and Maintenance - -#### Daily Monitoring - -```bash -# On CI/CD server -cd /opt/registry -./scripts/monitor.sh --type ci-cd - -# On Production server -cd /opt/APP_NAME -./scripts/monitor.sh --type production --app-name APP_NAME -``` - -#### Weekly Maintenance - -1. **Check disk space**: `df -h` -2. **Review logs**: `docker-compose logs --tail=100` -3. **Update system**: `sudo apt update && sudo apt upgrade` -4. **Test backups**: Verify backup files exist and are recent - ```bash - # On Production server - cd /opt/APP_NAME - ./scripts/backup.sh --type production --app-name APP_NAME --dry-run - - # Check backup directory - ls -la backups/ - ``` - -#### Monthly Maintenance - -1. **Review security**: Check firewall rules and fail2ban status -2. **Update certificates**: Ensure SSL certificates are valid (domain users only) -3. **Clean up old images**: Run the cleanup script to remove unused Docker images - ```bash - # On CI/CD server - cd /opt/registry - ./scripts/cleanup.sh --type ci-cd - - # On Production server - cd /opt/APP_NAME - ./scripts/cleanup.sh --type production - ``` -4. **Review monitoring**: Check application performance and logs -5. **Verify registry access**: Test registry connectivity and authentication - ---- - -## Summary - -Your complete CI/CD pipeline is now ready! The setup includes: - -### CI/CD Linode Features -- **Forgejo Actions runner** for automated builds -- **Local Docker registry** with web UI for image management -- **Secure SSH communication** with production server -- **Monitoring and cleanup** scripts -- **Firewall protection** for security - -### Production Linode Features -- **Docker-based application deployment** -- **Nginx reverse proxy** with security headers -- **Automated backups and monitoring** scripts -- **Firewall and fail2ban protection** for security -- **Optional SSL/TLS certificate management** (if domain is provided) - -### Pipeline Features -- **Automated testing** on every code push -- **Automated image building** and registry push -- **Automated deployment** to production -- **Rollback capability** with image versioning -- **Health monitoring** and logging - -### Registry Integration -- **Private registry** on CI/CD Linode stores all production images -- **Images available** for manual deployment via `PRODUCTION_LINODE_MANUAL_SETUP.md` -- **Version control** with git commit SHA tags -- **Web UI** for image management at `http://YOUR_CI_CD_IP:8080` - -### Access Methods -- **Domain users**: Access via `https://your-domain.com` -- **IP-only users**: Access via `http://YOUR_PRODUCTION_IP` -- **Registry UI**: Access via `http://YOUR_CI_CD_IP:8080` - -For ongoing maintenance and troubleshooting, refer to the troubleshooting section and monitoring scripts provided in this guide. \ No newline at end of file +``` \ No newline at end of file