switch to SERVICE_USER when handling registry stuff
Some checks are pending
Some checks are pending
This commit is contained in:
parent
e9ba3dc01e
commit
02569c8cf4
1 changed files with 27 additions and 473 deletions
|
@ -479,12 +479,18 @@ EOF
|
|||
#### 4.3 Create Authentication File
|
||||
|
||||
```bash
|
||||
# Switch to SERVICE_USER (registry directory owner)
|
||||
sudo su - SERVICE_USER
|
||||
|
||||
# Create htpasswd file for authentication (required for push operations)
|
||||
mkdir -p /opt/registry/auth
|
||||
htpasswd -Bbn push-user "$(openssl rand -base64 32)" > /opt/registry/auth.htpasswd
|
||||
|
||||
# Create a read-only user (optional, for additional security)
|
||||
htpasswd -Bbn read-user "$(openssl rand -base64 32)" >> /opt/registry/auth.htpasswd
|
||||
|
||||
# Exit SERVICE_USER shell
|
||||
exit
|
||||
```
|
||||
|
||||
**What this does**: Creates user credentials for registry authentication.
|
||||
|
@ -493,9 +499,14 @@ htpasswd -Bbn read-user "$(openssl rand -base64 32)" >> /opt/registry/auth.htpas
|
|||
|
||||
**Note**: Pull operations are public and don't require authentication, but push operations require these credentials.
|
||||
|
||||
**Security Note**: We switch to SERVICE_USER because the registry directory is owned by SERVICE_USER, maintaining proper file ownership and security.
|
||||
|
||||
#### 4.4 Create Docker Compose for Registry
|
||||
|
||||
```bash
|
||||
# Switch to SERVICE_USER (registry directory owner)
|
||||
sudo su - SERVICE_USER
|
||||
|
||||
cat > /opt/registry/docker-compose.yml << 'EOF'
|
||||
version: '3.8'
|
||||
|
||||
|
@ -532,6 +543,9 @@ networks:
|
|||
registry_network:
|
||||
driver: bridge
|
||||
EOF
|
||||
|
||||
# Exit SERVICE_USER shell
|
||||
exit
|
||||
```
|
||||
|
||||
#### 4.5 Install Required Tools
|
||||
|
@ -544,13 +558,22 @@ sudo apt install -y apache2-utils
|
|||
#### 4.6 Start Registry
|
||||
|
||||
```bash
|
||||
# Switch to SERVICE_USER (registry directory owner)
|
||||
sudo su - SERVICE_USER
|
||||
|
||||
cd /opt/registry
|
||||
docker-compose up -d
|
||||
|
||||
# Exit SERVICE_USER shell
|
||||
exit
|
||||
```
|
||||
|
||||
#### 4.7 Test Registry Setup
|
||||
|
||||
```bash
|
||||
# Switch to SERVICE_USER (registry directory owner)
|
||||
sudo su - SERVICE_USER
|
||||
|
||||
# Check if containers are running
|
||||
cd /opt/registry
|
||||
docker-compose ps
|
||||
|
@ -583,6 +606,9 @@ docker pull localhost:5000/test:latest
|
|||
# Clean up test image
|
||||
docker rmi localhost:5000/test:latest
|
||||
rm /tmp/test.Dockerfile
|
||||
|
||||
# Exit SERVICE_USER shell
|
||||
exit
|
||||
```
|
||||
|
||||
**Expected Output**:
|
||||
|
@ -1365,475 +1391,3 @@ tail -f /opt/APP_NAME/backup.log
|
|||
# If you created a local copy, use that instead
|
||||
(crontab -l 2>/dev/null; echo "0 2 * * * cd /opt/APP_NAME && ./backup-local.sh --type production --app-name APP_NAME >> /opt/APP_NAME/backup.log 2>&1") | crontab -
|
||||
```
|
||||
|
||||
#### 18.7 Monitoring Script
|
||||
|
||||
**Important**: The repository includes a pre-configured monitoring script in the `scripts/` directory that can be used for production monitoring.
|
||||
|
||||
**Repository Script**:
|
||||
- `scripts/monitor.sh` - Comprehensive monitoring script with support for both CI/CD and production environments
|
||||
|
||||
**To use the repository monitoring script**:
|
||||
```bash
|
||||
# The script is already available in the cloned repository
|
||||
cd /opt/APP_NAME
|
||||
|
||||
# Make the script executable
|
||||
chmod +x scripts/monitor.sh
|
||||
|
||||
# Test production monitoring
|
||||
./scripts/monitor.sh --type production --app-name APP_NAME
|
||||
```
|
||||
|
||||
**Alternative: Create a local copy for convenience**:
|
||||
```bash
|
||||
# Copy the script to the application directory for easy access
|
||||
cp scripts/monitor.sh /opt/APP_NAME/monitor-local.sh
|
||||
chmod +x /opt/APP_NAME/monitor-local.sh
|
||||
|
||||
# Test the local copy
|
||||
cd /opt/APP_NAME
|
||||
./monitor-local.sh --type production --app-name APP_NAME
|
||||
```
|
||||
|
||||
**Note**: The repository script is more comprehensive and includes proper error handling, colored output, health checks, and automatic environment detection. It provides better monitoring information than a simple local script.
|
||||
|
||||
#### 18.7.1 Set Up Automated Monitoring
|
||||
|
||||
```bash
|
||||
# Create a cron job to run monitoring every 5 minutes using the repository script
|
||||
(crontab -l 2>/dev/null; echo "*/5 * * * * cd /opt/APP_NAME && ./scripts/monitor.sh --type production --app-name APP_NAME >> /opt/APP_NAME/monitor.log 2>&1") | crontab -
|
||||
|
||||
# Verify the cron job was added
|
||||
crontab -l
|
||||
```
|
||||
|
||||
**What this does:**
|
||||
- **Runs automatically**: The monitoring script runs every 5 minutes
|
||||
- **Frequency**: Every 5 minutes to catch issues quickly
|
||||
- **Logging**: All monitoring output is logged to `/opt/APP_NAME/monitor.log`
|
||||
- **What it monitors**: Container status, recent logs, CPU/memory/disk usage, network connections, health checks
|
||||
|
||||
**To test the monitoring manually:**
|
||||
```bash
|
||||
cd /opt/APP_NAME
|
||||
./scripts/monitor.sh --type production --app-name APP_NAME
|
||||
```
|
||||
|
||||
**To view monitoring logs:**
|
||||
```bash
|
||||
tail -f /opt/APP_NAME/monitor.log
|
||||
```
|
||||
|
||||
### Step 19: Set Up SSH for CI/CD Communication
|
||||
|
||||
#### 19.1 Generate SSH Key Pair
|
||||
|
||||
```bash
|
||||
ssh-keygen -t ed25519 -C "production-server" -f ~/.ssh/id_ed25519 -N ""
|
||||
```
|
||||
|
||||
#### 19.2 Create SSH Config
|
||||
|
||||
```bash
|
||||
cat > ~/.ssh/config << 'EOF'
|
||||
Host ci-cd
|
||||
HostName YOUR_CI_CD_IP
|
||||
User SERVICE_USER
|
||||
IdentityFile ~/.ssh/id_ed25519
|
||||
StrictHostKeyChecking no
|
||||
UserKnownHostsFile /dev/null
|
||||
EOF
|
||||
|
||||
chmod 600 ~/.ssh/config
|
||||
```
|
||||
|
||||
### Step 20: Exchange SSH Keys
|
||||
|
||||
#### 20.1 Get Your Public Key
|
||||
|
||||
```bash
|
||||
cat ~/.ssh/id_ed25519.pub
|
||||
```
|
||||
|
||||
**Important**: Copy this public key - you'll need it for the CI/CD server setup.
|
||||
|
||||
#### 20.2 Add CI/CD Server's Public Key
|
||||
|
||||
```bash
|
||||
echo "CI_CD_PUBLIC_KEY_HERE" >> ~/.ssh/authorized_keys
|
||||
sed -i 's/YOUR_CI_CD_IP/YOUR_ACTUAL_CI_CD_IP/g' ~/.ssh/config
|
||||
```
|
||||
|
||||
**Note**: Replace `CI_CD_PUBLIC_KEY_HERE` with the actual public key from your CI/CD server.
|
||||
|
||||
### Step 21: Update Application Configuration for CI/CD
|
||||
|
||||
#### 21.1 Update Environment Variables
|
||||
|
||||
```bash
|
||||
cd /opt/APP_NAME
|
||||
nano .env
|
||||
```
|
||||
|
||||
**Required changes**:
|
||||
- Replace `YOUR_CI_CD_IP` with your actual CI/CD Linode IP address
|
||||
- Replace `your_secure_password_here` with a strong database password
|
||||
- Update `DATABASE_URL` with the same password
|
||||
|
||||
### Step 22: Configure SSL Certificates (Optional - Domain Users Only)
|
||||
|
||||
**Skip this step if you don't have a domain name.**
|
||||
|
||||
#### 22.1 Install SSL Certificates
|
||||
|
||||
```bash
|
||||
sudo certbot --nginx -d your-domain.com
|
||||
```
|
||||
|
||||
#### 22.2 Copy SSL Certificates
|
||||
|
||||
```bash
|
||||
sudo cp /etc/letsencrypt/live/your-domain.com/fullchain.pem /opt/APP_NAME/nginx/ssl/
|
||||
sudo cp /etc/letsencrypt/live/your-domain.com/privkey.pem /opt/APP_NAME/nginx/ssl/
|
||||
sudo chown SERVICE_USER:SERVICE_USER /opt/APP_NAME/nginx/ssl/*
|
||||
```
|
||||
|
||||
### Step 23: Test CI/CD Integration
|
||||
|
||||
#### 23.1 Test SSH Connection
|
||||
|
||||
```bash
|
||||
ssh ci-cd 'echo Connection successful'
|
||||
```
|
||||
|
||||
**Expected output**: `Connection successful`.
|
||||
|
||||
#### 23.2 Test Registry Connection
|
||||
|
||||
```bash
|
||||
curl http://YOUR_ACTUAL_CI_IP:5000/v2/_catalog
|
||||
```
|
||||
|
||||
**Expected output**: `{"repositories":[]}` or a list of available images.
|
||||
|
||||
---
|
||||
|
||||
## Part 3: Pipeline Configuration
|
||||
|
||||
### Step 24: Configure Forgejo Repository Secrets
|
||||
|
||||
Go to your Forgejo repository → Settings → Secrets and Variables → Actions, and add the following secrets:
|
||||
|
||||
#### Required Secrets:
|
||||
|
||||
- **`CI_HOST`**: Your CI/CD Linode IP address
|
||||
- **Purpose**: Used by the workflow to connect to your private Docker registry
|
||||
- **Example**: `192.168.1.100`
|
||||
|
||||
- **`PROD_HOST`**: Your Production Linode IP address
|
||||
- **Purpose**: Used by the deployment job to SSH into your production server
|
||||
- **Example**: `192.168.1.101`
|
||||
|
||||
- **`PROD_USER`**: SSH username for production server
|
||||
- **Purpose**: Username for SSH connection to production server
|
||||
- **Value**: Should be `DEPLOY_USER` (the deployment user you created)
|
||||
- **Example**: `deploy`
|
||||
|
||||
- **`PROD_SSH_KEY`**: SSH private key for deployment
|
||||
- **Purpose**: Private key for SSH authentication to production server
|
||||
- **Source**: Copy the private key from your CI/CD server
|
||||
- **How to get**: On CI/CD server, run `cat ~/.ssh/id_ed25519`
|
||||
- **Format**: Include the entire key including `-----BEGIN OPENSSH PRIVATE KEY-----` and `-----END OPENSSH PRIVATE KEY-----`
|
||||
|
||||
#### Optional Secrets (for enhanced security and flexibility):
|
||||
|
||||
- **`APP_NAME`**: Application name (used for directory, database, and image names)
|
||||
- **Purpose**: Controls the application directory name and database names
|
||||
- **Default**: `sharenet` (if not set)
|
||||
- **Example**: `myapp`, `webapp`, `api`
|
||||
- **Note**: This affects the deployment directory `/opt/APP_NAME` and database names
|
||||
|
||||
- **`POSTGRES_USER`**: PostgreSQL username for the application database
|
||||
- **Purpose**: Username for the application's PostgreSQL database
|
||||
- **Default**: `sharenet` (if not set)
|
||||
- **Example**: `appuser`, `webuser`, `apiuser`
|
||||
- **Note**: Should match the user created in the PostgreSQL setup
|
||||
|
||||
- **`POSTGRES_DB`**: PostgreSQL database name for the application
|
||||
- **Purpose**: Name of the application's PostgreSQL database
|
||||
- **Default**: `sharenet` (if not set)
|
||||
- **Example**: `myapp`, `webapp`, `api`
|
||||
- **Note**: Should match the database created in the PostgreSQL setup
|
||||
|
||||
- **`POSTGRES_PASSWORD`**: Database password for production
|
||||
- **Purpose**: Secure database password for production environment
|
||||
- **Note**: If not set, the workflow will use a default password
|
||||
|
||||
- **`REGISTRY_USERNAME`**: Username for Docker registry (if using authentication)
|
||||
- **Purpose**: Username for private registry access
|
||||
- **Note**: Only needed if your registry requires authentication
|
||||
|
||||
- **`REGISTRY_PASSWORD`**: Password for Docker registry (if using authentication)
|
||||
- **Purpose**: Password for private registry access
|
||||
- **Note**: Only needed if your registry requires authentication
|
||||
|
||||
#### How to Add Secrets:
|
||||
|
||||
1. Go to your Forgejo repository
|
||||
2. Navigate to **Settings** → **Secrets and Variables** → **Actions**
|
||||
3. Click **New Secret**
|
||||
4. Enter the secret name and value
|
||||
5. Click **Add Secret**
|
||||
|
||||
#### Security Notes:
|
||||
|
||||
- **Never commit secrets to your repository**
|
||||
- **Use strong, unique passwords** for each environment
|
||||
- **Rotate secrets regularly** for enhanced security
|
||||
- **Limit access** to repository settings to trusted team members only
|
||||
|
||||
### Step 25: Test the Complete Pipeline
|
||||
|
||||
#### 25.1 Push Code Changes
|
||||
|
||||
Make a small change to your code and push to trigger the CI/CD pipeline:
|
||||
|
||||
```bash
|
||||
# In your local repository
|
||||
echo "# Test deployment" >> README.md
|
||||
git add README.md
|
||||
git commit -m "Test CI/CD pipeline"
|
||||
git push
|
||||
```
|
||||
|
||||
#### 25.2 Monitor Pipeline
|
||||
|
||||
1. Go to your Forgejo repository
|
||||
2. Navigate to Actions tab
|
||||
3. Monitor the workflow execution
|
||||
4. Check for any errors or issues
|
||||
|
||||
#### 25.3 Verify Deployment
|
||||
|
||||
After successful pipeline execution:
|
||||
|
||||
```bash
|
||||
# Check application status
|
||||
cd /opt/APP_NAME
|
||||
docker-compose ps
|
||||
|
||||
# Check application logs
|
||||
docker-compose logs
|
||||
|
||||
# Test application access
|
||||
curl -I https://your-domain.com # or http://YOUR_PRODUCTION_IP
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing and Verification
|
||||
|
||||
### Step 26: Test Application Access
|
||||
|
||||
**If you have a domain:**
|
||||
```bash
|
||||
# Test HTTP redirect to HTTPS
|
||||
curl -I http://your-domain.com
|
||||
|
||||
# Test HTTPS access
|
||||
curl -I https://your-domain.com
|
||||
|
||||
# Test application health endpoint (checks backend services)
|
||||
curl https://your-domain.com/health
|
||||
```
|
||||
|
||||
**If you don't have a domain (IP access only):**
|
||||
```bash
|
||||
# Test HTTP access via IP
|
||||
curl -I http://YOUR_PRODUCTION_IP
|
||||
|
||||
# Test application health endpoint (checks backend services)
|
||||
curl http://YOUR_PRODUCTION_IP/health
|
||||
```
|
||||
|
||||
**Expected health endpoint response:**
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"service": "sharenet-api",
|
||||
"timestamp": "2024-01-01T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
**Note**: The `/health` endpoint now proxies to the backend service and returns actual service status. If the backend is not running, this endpoint will return an error, making it a true health check for the application.
|
||||
|
||||
### Step 27: Test Monitoring
|
||||
|
||||
```bash
|
||||
# On CI/CD server
|
||||
cd /opt/registry
|
||||
./scripts/monitor.sh --type ci-cd
|
||||
|
||||
# On Production server
|
||||
cd /opt/APP_NAME
|
||||
./scripts/monitor.sh --type production --app-name APP_NAME
|
||||
```
|
||||
|
||||
### Step 28: Test Registry Access
|
||||
|
||||
```bash
|
||||
# Test registry API
|
||||
curl http://YOUR_CI_CD_IP:5000/v2/_catalog
|
||||
|
||||
# Test registry UI (optional)
|
||||
curl -I http://YOUR_CI_CD_IP:8080
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Docker permission denied**:
|
||||
```bash
|
||||
sudo usermod -aG docker SERVICE_USER
|
||||
newgrp docker
|
||||
```
|
||||
|
||||
2. **SSL certificate issues** (domain users only):
|
||||
```bash
|
||||
sudo certbot certificates
|
||||
sudo certbot renew --dry-run
|
||||
```
|
||||
|
||||
3. **Application not starting**:
|
||||
```bash
|
||||
cd /opt/APP_NAME
|
||||
docker-compose logs
|
||||
```
|
||||
|
||||
4. **SSH connection failed**:
|
||||
```bash
|
||||
ssh -v ci-cd 'echo test'
|
||||
ssh -v production 'echo test'
|
||||
```
|
||||
|
||||
5. **Registry connection failed**:
|
||||
```bash
|
||||
curl -v http://YOUR_CI_HOST_IP:5000/v2/_catalog
|
||||
```
|
||||
|
||||
6. **Actions runner not starting**:
|
||||
```bash
|
||||
sudo systemctl status forgejo-runner.service
|
||||
sudo journalctl -u forgejo-runner.service -f
|
||||
```
|
||||
|
||||
### Useful Commands
|
||||
|
||||
- **Check system resources**: `htop`
|
||||
- **Check disk space**: `df -h`
|
||||
- **Check memory usage**: `free -h`
|
||||
- **Check network**: `ip addr show`
|
||||
- **Check firewall**: `sudo ufw status`
|
||||
- **Check logs**: `sudo journalctl -f`
|
||||
|
||||
### Security Best Practices
|
||||
|
||||
1. **Service Account**: Use dedicated `SERVICE_USER` user with limited privileges
|
||||
2. **SSH Keys**: Use Ed25519 keys with proper permissions (600/700)
|
||||
3. **Firewall**: Configure UFW to allow only necessary ports
|
||||
4. **Fail2ban**: Protect against brute force attacks
|
||||
5. **SSL/TLS**: Use Let's Encrypt certificates with automatic renewal (domain users only)
|
||||
6. **Regular Backups**: Automated daily backups of database and configuration
|
||||
7. **Container Isolation**: Applications run in isolated Docker containers
|
||||
8. **Security Headers**: Nginx configured with security headers
|
||||
9. **Registry Security**: Use secure authentication and HTTPS for registry access
|
||||
|
||||
### Monitoring and Maintenance
|
||||
|
||||
#### Daily Monitoring
|
||||
|
||||
```bash
|
||||
# On CI/CD server
|
||||
cd /opt/registry
|
||||
./scripts/monitor.sh --type ci-cd
|
||||
|
||||
# On Production server
|
||||
cd /opt/APP_NAME
|
||||
./scripts/monitor.sh --type production --app-name APP_NAME
|
||||
```
|
||||
|
||||
#### Weekly Maintenance
|
||||
|
||||
1. **Check disk space**: `df -h`
|
||||
2. **Review logs**: `docker-compose logs --tail=100`
|
||||
3. **Update system**: `sudo apt update && sudo apt upgrade`
|
||||
4. **Test backups**: Verify backup files exist and are recent
|
||||
```bash
|
||||
# On Production server
|
||||
cd /opt/APP_NAME
|
||||
./scripts/backup.sh --type production --app-name APP_NAME --dry-run
|
||||
|
||||
# Check backup directory
|
||||
ls -la backups/
|
||||
```
|
||||
|
||||
#### Monthly Maintenance
|
||||
|
||||
1. **Review security**: Check firewall rules and fail2ban status
|
||||
2. **Update certificates**: Ensure SSL certificates are valid (domain users only)
|
||||
3. **Clean up old images**: Run the cleanup script to remove unused Docker images
|
||||
```bash
|
||||
# On CI/CD server
|
||||
cd /opt/registry
|
||||
./scripts/cleanup.sh --type ci-cd
|
||||
|
||||
# On Production server
|
||||
cd /opt/APP_NAME
|
||||
./scripts/cleanup.sh --type production
|
||||
```
|
||||
4. **Review monitoring**: Check application performance and logs
|
||||
5. **Verify registry access**: Test registry connectivity and authentication
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Your complete CI/CD pipeline is now ready! The setup includes:
|
||||
|
||||
### CI/CD Linode Features
|
||||
- **Forgejo Actions runner** for automated builds
|
||||
- **Local Docker registry** with web UI for image management
|
||||
- **Secure SSH communication** with production server
|
||||
- **Monitoring and cleanup** scripts
|
||||
- **Firewall protection** for security
|
||||
|
||||
### Production Linode Features
|
||||
- **Docker-based application deployment**
|
||||
- **Nginx reverse proxy** with security headers
|
||||
- **Automated backups and monitoring** scripts
|
||||
- **Firewall and fail2ban protection** for security
|
||||
- **Optional SSL/TLS certificate management** (if domain is provided)
|
||||
|
||||
### Pipeline Features
|
||||
- **Automated testing** on every code push
|
||||
- **Automated image building** and registry push
|
||||
- **Automated deployment** to production
|
||||
- **Rollback capability** with image versioning
|
||||
- **Health monitoring** and logging
|
||||
|
||||
### Registry Integration
|
||||
- **Private registry** on CI/CD Linode stores all production images
|
||||
- **Images available** for manual deployment via `PRODUCTION_LINODE_MANUAL_SETUP.md`
|
||||
- **Version control** with git commit SHA tags
|
||||
- **Web UI** for image management at `http://YOUR_CI_CD_IP:8080`
|
||||
|
||||
### Access Methods
|
||||
- **Domain users**: Access via `https://your-domain.com`
|
||||
- **IP-only users**: Access via `http://YOUR_PRODUCTION_IP`
|
||||
- **Registry UI**: Access via `http://YOUR_CI_CD_IP:8080`
|
||||
|
||||
For ongoing maintenance and troubleshooting, refer to the troubleshooting section and monitoring scripts provided in this guide.
|
Loading…
Add table
Reference in a new issue