switch to SERVICE_USER when handling registry stuff
Some checks are pending
CI/CD Pipeline / Test Backend (push) Waiting to run
CI/CD Pipeline / Test Frontend (push) Waiting to run
CI/CD Pipeline / Build and Push Docker Images (push) Blocked by required conditions
CI/CD Pipeline / Deploy to Production (push) Blocked by required conditions

This commit is contained in:
continuist 2025-06-28 13:54:58 -04:00
parent e9ba3dc01e
commit 02569c8cf4

View file

@ -479,12 +479,18 @@ EOF
#### 4.3 Create Authentication File #### 4.3 Create Authentication File
```bash ```bash
# Switch to SERVICE_USER (registry directory owner)
sudo su - SERVICE_USER
# Create htpasswd file for authentication (required for push operations) # Create htpasswd file for authentication (required for push operations)
mkdir -p /opt/registry/auth mkdir -p /opt/registry/auth
htpasswd -Bbn push-user "$(openssl rand -base64 32)" > /opt/registry/auth.htpasswd htpasswd -Bbn push-user "$(openssl rand -base64 32)" > /opt/registry/auth.htpasswd
# Create a read-only user (optional, for additional security) # Create a read-only user (optional, for additional security)
htpasswd -Bbn read-user "$(openssl rand -base64 32)" >> /opt/registry/auth.htpasswd htpasswd -Bbn read-user "$(openssl rand -base64 32)" >> /opt/registry/auth.htpasswd
# Exit SERVICE_USER shell
exit
``` ```
**What this does**: Creates user credentials for registry authentication. **What this does**: Creates user credentials for registry authentication.
@ -493,9 +499,14 @@ htpasswd -Bbn read-user "$(openssl rand -base64 32)" >> /opt/registry/auth.htpas
**Note**: Pull operations are public and don't require authentication, but push operations require these credentials. **Note**: Pull operations are public and don't require authentication, but push operations require these credentials.
**Security Note**: We switch to SERVICE_USER because the registry directory is owned by SERVICE_USER, maintaining proper file ownership and security.
#### 4.4 Create Docker Compose for Registry #### 4.4 Create Docker Compose for Registry
```bash ```bash
# Switch to SERVICE_USER (registry directory owner)
sudo su - SERVICE_USER
cat > /opt/registry/docker-compose.yml << 'EOF' cat > /opt/registry/docker-compose.yml << 'EOF'
version: '3.8' version: '3.8'
@ -532,6 +543,9 @@ networks:
registry_network: registry_network:
driver: bridge driver: bridge
EOF EOF
# Exit SERVICE_USER shell
exit
``` ```
#### 4.5 Install Required Tools #### 4.5 Install Required Tools
@ -544,13 +558,22 @@ sudo apt install -y apache2-utils
#### 4.6 Start Registry #### 4.6 Start Registry
```bash ```bash
# Switch to SERVICE_USER (registry directory owner)
sudo su - SERVICE_USER
cd /opt/registry cd /opt/registry
docker-compose up -d docker-compose up -d
# Exit SERVICE_USER shell
exit
``` ```
#### 4.7 Test Registry Setup #### 4.7 Test Registry Setup
```bash ```bash
# Switch to SERVICE_USER (registry directory owner)
sudo su - SERVICE_USER
# Check if containers are running # Check if containers are running
cd /opt/registry cd /opt/registry
docker-compose ps docker-compose ps
@ -583,6 +606,9 @@ docker pull localhost:5000/test:latest
# Clean up test image # Clean up test image
docker rmi localhost:5000/test:latest docker rmi localhost:5000/test:latest
rm /tmp/test.Dockerfile rm /tmp/test.Dockerfile
# Exit SERVICE_USER shell
exit
``` ```
**Expected Output**: **Expected Output**:
@ -1364,476 +1390,4 @@ tail -f /opt/APP_NAME/backup.log
```bash ```bash
# If you created a local copy, use that instead # If you created a local copy, use that instead
(crontab -l 2>/dev/null; echo "0 2 * * * cd /opt/APP_NAME && ./backup-local.sh --type production --app-name APP_NAME >> /opt/APP_NAME/backup.log 2>&1") | crontab - (crontab -l 2>/dev/null; echo "0 2 * * * cd /opt/APP_NAME && ./backup-local.sh --type production --app-name APP_NAME >> /opt/APP_NAME/backup.log 2>&1") | crontab -
``` ```
#### 18.7 Monitoring Script
**Important**: The repository includes a pre-configured monitoring script in the `scripts/` directory that can be used for production monitoring.
**Repository Script**:
- `scripts/monitor.sh` - Comprehensive monitoring script with support for both CI/CD and production environments
**To use the repository monitoring script**:
```bash
# The script is already available in the cloned repository
cd /opt/APP_NAME
# Make the script executable
chmod +x scripts/monitor.sh
# Test production monitoring
./scripts/monitor.sh --type production --app-name APP_NAME
```
**Alternative: Create a local copy for convenience**:
```bash
# Copy the script to the application directory for easy access
cp scripts/monitor.sh /opt/APP_NAME/monitor-local.sh
chmod +x /opt/APP_NAME/monitor-local.sh
# Test the local copy
cd /opt/APP_NAME
./monitor-local.sh --type production --app-name APP_NAME
```
**Note**: The repository script is more comprehensive and includes proper error handling, colored output, health checks, and automatic environment detection. It provides better monitoring information than a simple local script.
#### 18.7.1 Set Up Automated Monitoring
```bash
# Create a cron job to run monitoring every 5 minutes using the repository script
(crontab -l 2>/dev/null; echo "*/5 * * * * cd /opt/APP_NAME && ./scripts/monitor.sh --type production --app-name APP_NAME >> /opt/APP_NAME/monitor.log 2>&1") | crontab -
# Verify the cron job was added
crontab -l
```
**What this does:**
- **Runs automatically**: The monitoring script runs every 5 minutes
- **Frequency**: Every 5 minutes to catch issues quickly
- **Logging**: All monitoring output is logged to `/opt/APP_NAME/monitor.log`
- **What it monitors**: Container status, recent logs, CPU/memory/disk usage, network connections, health checks
**To test the monitoring manually:**
```bash
cd /opt/APP_NAME
./scripts/monitor.sh --type production --app-name APP_NAME
```
**To view monitoring logs:**
```bash
tail -f /opt/APP_NAME/monitor.log
```
### Step 19: Set Up SSH for CI/CD Communication
#### 19.1 Generate SSH Key Pair
```bash
ssh-keygen -t ed25519 -C "production-server" -f ~/.ssh/id_ed25519 -N ""
```
#### 19.2 Create SSH Config
```bash
cat > ~/.ssh/config << 'EOF'
Host ci-cd
HostName YOUR_CI_CD_IP
User SERVICE_USER
IdentityFile ~/.ssh/id_ed25519
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
EOF
chmod 600 ~/.ssh/config
```
### Step 20: Exchange SSH Keys
#### 20.1 Get Your Public Key
```bash
cat ~/.ssh/id_ed25519.pub
```
**Important**: Copy this public key - you'll need it for the CI/CD server setup.
#### 20.2 Add CI/CD Server's Public Key
```bash
echo "CI_CD_PUBLIC_KEY_HERE" >> ~/.ssh/authorized_keys
sed -i 's/YOUR_CI_CD_IP/YOUR_ACTUAL_CI_CD_IP/g' ~/.ssh/config
```
**Note**: Replace `CI_CD_PUBLIC_KEY_HERE` with the actual public key from your CI/CD server.
### Step 21: Update Application Configuration for CI/CD
#### 21.1 Update Environment Variables
```bash
cd /opt/APP_NAME
nano .env
```
**Required changes**:
- Replace `YOUR_CI_CD_IP` with your actual CI/CD Linode IP address
- Replace `your_secure_password_here` with a strong database password
- Update `DATABASE_URL` with the same password
### Step 22: Configure SSL Certificates (Optional - Domain Users Only)
**Skip this step if you don't have a domain name.**
#### 22.1 Install SSL Certificates
```bash
sudo certbot --nginx -d your-domain.com
```
#### 22.2 Copy SSL Certificates
```bash
sudo cp /etc/letsencrypt/live/your-domain.com/fullchain.pem /opt/APP_NAME/nginx/ssl/
sudo cp /etc/letsencrypt/live/your-domain.com/privkey.pem /opt/APP_NAME/nginx/ssl/
sudo chown SERVICE_USER:SERVICE_USER /opt/APP_NAME/nginx/ssl/*
```
### Step 23: Test CI/CD Integration
#### 23.1 Test SSH Connection
```bash
ssh ci-cd 'echo Connection successful'
```
**Expected output**: `Connection successful`.
#### 23.2 Test Registry Connection
```bash
curl http://YOUR_ACTUAL_CI_IP:5000/v2/_catalog
```
**Expected output**: `{"repositories":[]}` or a list of available images.
---
## Part 3: Pipeline Configuration
### Step 24: Configure Forgejo Repository Secrets
Go to your Forgejo repository → Settings → Secrets and Variables → Actions, and add the following secrets:
#### Required Secrets:
- **`CI_HOST`**: Your CI/CD Linode IP address
- **Purpose**: Used by the workflow to connect to your private Docker registry
- **Example**: `192.168.1.100`
- **`PROD_HOST`**: Your Production Linode IP address
- **Purpose**: Used by the deployment job to SSH into your production server
- **Example**: `192.168.1.101`
- **`PROD_USER`**: SSH username for production server
- **Purpose**: Username for SSH connection to production server
- **Value**: Should be `DEPLOY_USER` (the deployment user you created)
- **Example**: `deploy`
- **`PROD_SSH_KEY`**: SSH private key for deployment
- **Purpose**: Private key for SSH authentication to production server
- **Source**: Copy the private key from your CI/CD server
- **How to get**: On CI/CD server, run `cat ~/.ssh/id_ed25519`
- **Format**: Include the entire key including `-----BEGIN OPENSSH PRIVATE KEY-----` and `-----END OPENSSH PRIVATE KEY-----`
#### Optional Secrets (for enhanced security and flexibility):
- **`APP_NAME`**: Application name (used for directory, database, and image names)
- **Purpose**: Controls the application directory name and database names
- **Default**: `sharenet` (if not set)
- **Example**: `myapp`, `webapp`, `api`
- **Note**: This affects the deployment directory `/opt/APP_NAME` and database names
- **`POSTGRES_USER`**: PostgreSQL username for the application database
- **Purpose**: Username for the application's PostgreSQL database
- **Default**: `sharenet` (if not set)
- **Example**: `appuser`, `webuser`, `apiuser`
- **Note**: Should match the user created in the PostgreSQL setup
- **`POSTGRES_DB`**: PostgreSQL database name for the application
- **Purpose**: Name of the application's PostgreSQL database
- **Default**: `sharenet` (if not set)
- **Example**: `myapp`, `webapp`, `api`
- **Note**: Should match the database created in the PostgreSQL setup
- **`POSTGRES_PASSWORD`**: Database password for production
- **Purpose**: Secure database password for production environment
- **Note**: If not set, the workflow will use a default password
- **`REGISTRY_USERNAME`**: Username for Docker registry (if using authentication)
- **Purpose**: Username for private registry access
- **Note**: Only needed if your registry requires authentication
- **`REGISTRY_PASSWORD`**: Password for Docker registry (if using authentication)
- **Purpose**: Password for private registry access
- **Note**: Only needed if your registry requires authentication
#### How to Add Secrets:
1. Go to your Forgejo repository
2. Navigate to **Settings****Secrets and Variables** → **Actions**
3. Click **New Secret**
4. Enter the secret name and value
5. Click **Add Secret**
#### Security Notes:
- **Never commit secrets to your repository**
- **Use strong, unique passwords** for each environment
- **Rotate secrets regularly** for enhanced security
- **Limit access** to repository settings to trusted team members only
### Step 25: Test the Complete Pipeline
#### 25.1 Push Code Changes
Make a small change to your code and push to trigger the CI/CD pipeline:
```bash
# In your local repository
echo "# Test deployment" >> README.md
git add README.md
git commit -m "Test CI/CD pipeline"
git push
```
#### 25.2 Monitor Pipeline
1. Go to your Forgejo repository
2. Navigate to Actions tab
3. Monitor the workflow execution
4. Check for any errors or issues
#### 25.3 Verify Deployment
After successful pipeline execution:
```bash
# Check application status
cd /opt/APP_NAME
docker-compose ps
# Check application logs
docker-compose logs
# Test application access
curl -I https://your-domain.com # or http://YOUR_PRODUCTION_IP
```
---
## Testing and Verification
### Step 26: Test Application Access
**If you have a domain:**
```bash
# Test HTTP redirect to HTTPS
curl -I http://your-domain.com
# Test HTTPS access
curl -I https://your-domain.com
# Test application health endpoint (checks backend services)
curl https://your-domain.com/health
```
**If you don't have a domain (IP access only):**
```bash
# Test HTTP access via IP
curl -I http://YOUR_PRODUCTION_IP
# Test application health endpoint (checks backend services)
curl http://YOUR_PRODUCTION_IP/health
```
**Expected health endpoint response:**
```json
{
"status": "healthy",
"service": "sharenet-api",
"timestamp": "2024-01-01T12:00:00Z"
}
```
**Note**: The `/health` endpoint now proxies to the backend service and returns actual service status. If the backend is not running, this endpoint will return an error, making it a true health check for the application.
### Step 27: Test Monitoring
```bash
# On CI/CD server
cd /opt/registry
./scripts/monitor.sh --type ci-cd
# On Production server
cd /opt/APP_NAME
./scripts/monitor.sh --type production --app-name APP_NAME
```
### Step 28: Test Registry Access
```bash
# Test registry API
curl http://YOUR_CI_CD_IP:5000/v2/_catalog
# Test registry UI (optional)
curl -I http://YOUR_CI_CD_IP:8080
```
---
## Troubleshooting
### Common Issues
1. **Docker permission denied**:
```bash
sudo usermod -aG docker SERVICE_USER
newgrp docker
```
2. **SSL certificate issues** (domain users only):
```bash
sudo certbot certificates
sudo certbot renew --dry-run
```
3. **Application not starting**:
```bash
cd /opt/APP_NAME
docker-compose logs
```
4. **SSH connection failed**:
```bash
ssh -v ci-cd 'echo test'
ssh -v production 'echo test'
```
5. **Registry connection failed**:
```bash
curl -v http://YOUR_CI_HOST_IP:5000/v2/_catalog
```
6. **Actions runner not starting**:
```bash
sudo systemctl status forgejo-runner.service
sudo journalctl -u forgejo-runner.service -f
```
### Useful Commands
- **Check system resources**: `htop`
- **Check disk space**: `df -h`
- **Check memory usage**: `free -h`
- **Check network**: `ip addr show`
- **Check firewall**: `sudo ufw status`
- **Check logs**: `sudo journalctl -f`
### Security Best Practices
1. **Service Account**: Use dedicated `SERVICE_USER` user with limited privileges
2. **SSH Keys**: Use Ed25519 keys with proper permissions (600/700)
3. **Firewall**: Configure UFW to allow only necessary ports
4. **Fail2ban**: Protect against brute force attacks
5. **SSL/TLS**: Use Let's Encrypt certificates with automatic renewal (domain users only)
6. **Regular Backups**: Automated daily backups of database and configuration
7. **Container Isolation**: Applications run in isolated Docker containers
8. **Security Headers**: Nginx configured with security headers
9. **Registry Security**: Use secure authentication and HTTPS for registry access
### Monitoring and Maintenance
#### Daily Monitoring
```bash
# On CI/CD server
cd /opt/registry
./scripts/monitor.sh --type ci-cd
# On Production server
cd /opt/APP_NAME
./scripts/monitor.sh --type production --app-name APP_NAME
```
#### Weekly Maintenance
1. **Check disk space**: `df -h`
2. **Review logs**: `docker-compose logs --tail=100`
3. **Update system**: `sudo apt update && sudo apt upgrade`
4. **Test backups**: Verify backup files exist and are recent
```bash
# On Production server
cd /opt/APP_NAME
./scripts/backup.sh --type production --app-name APP_NAME --dry-run
# Check backup directory
ls -la backups/
```
#### Monthly Maintenance
1. **Review security**: Check firewall rules and fail2ban status
2. **Update certificates**: Ensure SSL certificates are valid (domain users only)
3. **Clean up old images**: Run the cleanup script to remove unused Docker images
```bash
# On CI/CD server
cd /opt/registry
./scripts/cleanup.sh --type ci-cd
# On Production server
cd /opt/APP_NAME
./scripts/cleanup.sh --type production
```
4. **Review monitoring**: Check application performance and logs
5. **Verify registry access**: Test registry connectivity and authentication
---
## Summary
Your complete CI/CD pipeline is now ready! The setup includes:
### CI/CD Linode Features
- **Forgejo Actions runner** for automated builds
- **Local Docker registry** with web UI for image management
- **Secure SSH communication** with production server
- **Monitoring and cleanup** scripts
- **Firewall protection** for security
### Production Linode Features
- **Docker-based application deployment**
- **Nginx reverse proxy** with security headers
- **Automated backups and monitoring** scripts
- **Firewall and fail2ban protection** for security
- **Optional SSL/TLS certificate management** (if domain is provided)
### Pipeline Features
- **Automated testing** on every code push
- **Automated image building** and registry push
- **Automated deployment** to production
- **Rollback capability** with image versioning
- **Health monitoring** and logging
### Registry Integration
- **Private registry** on CI/CD Linode stores all production images
- **Images available** for manual deployment via `PRODUCTION_LINODE_MANUAL_SETUP.md`
- **Version control** with git commit SHA tags
- **Web UI** for image management at `http://YOUR_CI_CD_IP:8080`
### Access Methods
- **Domain users**: Access via `https://your-domain.com`
- **IP-only users**: Access via `http://YOUR_PRODUCTION_IP`
- **Registry UI**: Access via `http://YOUR_CI_CD_IP:8080`
For ongoing maintenance and troubleshooting, refer to the troubleshooting section and monitoring scripts provided in this guide.