Clean up CI guide

This commit is contained in:
continuist 2025-09-06 10:35:23 -04:00
parent 6e0d66a200
commit b0234f13b5

View file

@ -1207,7 +1207,7 @@ podman exec ci-pip-local podman run --rm \
sh -c "cargo test --lib -- --test-threads=1"
# Run backend integration tests with real database
podman exec ci-pip podman run --rm \
podman exec ci-pip-local podman run --rm \
-v $(pwd)/backend:/workspace \
-w /workspace \
-e DATABASE_URL=postgres://testuser:testpassword@localhost:5432/testdb \
@ -1252,32 +1252,6 @@ The CI/CD pipeline uses ephemeral PiP containers with this secure workflow:
- 🛡️ **Network isolation**: PiP containers have no external network
- 🛡️ **Ephemeral execution**: Fresh environment every time
#### 7.2 Configure PiP for Forgejo Container Registry
```bash
# Navigate to the application directory
cd /opt/APP_NAME
# Copy client certificates to PiP container
# Forgejo Container Registry authentication is handled through environment variables
# Test Forgejo Container Registry connectivity from PiP
podman exec ci-pip podman pull alpine:latest
podman exec ci-pip podman tag alpine:latest your-forgejo-domain.com/your-username/APP_NAME/test:latest
# Log in to Forgejo Container Registry
podman exec ci-pip podman login git.gcdo.org -u YOUR_USERNAME -p YOUR_PERSONAL_ACCESS_TOKEN
# Push the image
podman exec ci-pip podman push your-forgejo-domain.com/your-username/APP_NAME/test:latest
# Test unauthenticated pull from standard port 443
podman exec ci-pip podman pull your-forgejo-domain.com/your-username/APP_NAME/test:latest
# Clean up test images
podman exec ci-pip podman rmi your-forgejo-domain.com/your-username/APP_NAME/test:latest
```
#### 7.3 Set Up Workspace Directory
**Important**: The CI workflow needs a workspace directory for code checkout. This directory will be used by the Forgejo Actions runner.
@ -1335,7 +1309,7 @@ The Forgejo Container Registry setup uses the built-in registry functionality, p
- **Tests connectivity**: Verifies DinD can pull, tag, and push images to Forgejo Container Registry
- **Validates setup**: Ensures the complete CI/CD pipeline will work
#### 6.4 CI/CD Workflow Architecture with Ephemeral PiP
#### 7.4 CI/CD Workflow Architecture with Ephemeral PiP
The CI/CD pipeline uses ephemeral Podman-in-Podman containers with a secure four-stage approach:
@ -1383,31 +1357,7 @@ The CI/CD pipeline uses ephemeral Podman-in-Podman containers with a secure four
- ✅ **Comprehensive Coverage**: Unit + integration tests
- ✅ **Isolated Execution**: Each test run completely independent
**Testing DinD Setup:**
```bash
# Test DinD functionality
docker exec ci-dind docker run --rm alpine:latest echo "DinD is working!"
# Test Forgejo Container Registry integration
docker exec ci-dind docker pull alpine:latest
docker exec ci-dind docker tag alpine:latest YOUR_CI_CD_IP:4443/APP_NAME/dind-test:latest
docker exec ci-dind docker push YOUR_CI_CD_IP:4443/APP_NAME/dind-test:latest
# Test unauthenticated pull
docker exec ci-dind docker pull YOUR_CI_CD_IP/APP_NAME/dind-test:latest
# Clean up test
docker exec ci-dind docker rmi YOUR_CI_CD_IP:4443/APP_NAME/dind-test:latest
docker exec ci-dind docker rmi YOUR_CI_CD_IP/APP_NAME/dind-test:latest
```
**Expected Output**:
- DinD container should be running and accessible
- Docker commands should work inside DinD
- Forgejo Container Registry push/pull should work from DinD
#### 6.5 Production Deployment Architecture
#### 7.5 Production Deployment Architecture
The production deployment uses a separate pod configuration (`prod-pod.yaml`) that pulls built images from the Forgejo Container Registry and deploys the complete application stack.
@ -1430,35 +1380,6 @@ The production deployment uses a separate pod configuration (`prod-pod.yaml`) th
- **🛡️ Rollback Capability**: Can easily rollback to previous image versions
- **📊 Health Monitoring**: Built-in health checks for all services
#### 6.6 Monitoring Script
**Important**: The repository includes a pre-configured monitoring script in the `scripts/` directory that can be used for both CI/CD and production monitoring.
**Repository Script**:
- `scripts/monitor.sh` - Comprehensive monitoring script with support for both CI/CD and production environments
**To use the repository monitoring script**:
```bash
# The repository is already cloned at /opt/APP_NAME/
cd /opt/APP_NAME
# Make the script executable
chmod +x scripts/monitor.sh
# Test CI/CD monitoring
./scripts/monitor.sh --type ci-cd
# Test production monitoring (if you have a production setup)
./scripts/monitor.sh --type production
```
**Note**: The repository script is more comprehensive and includes proper error handling, colored output, and support for both CI/CD and production environments. It automatically detects the environment and provides appropriate monitoring information.
**Security Model**:
- **Forgejo Registry**: Integrated authentication and authorization
- **SSH**: Restricted to your IP addresses
- **All other ports**: Blocked
### Step 8: Security Verification and Testing
#### 8.1 Security Audit - Verify NO Exposure
@ -1514,15 +1435,15 @@ podman exec ci-pip podman search alpine
## Part 2: Production Linode Setup
### Step 10: Initial System Setup
### Step 9: Initial System Setup
#### 10.1 Update the System
#### 9.1 Update the System
```bash
sudo apt update && sudo apt upgrade -y
```
#### 10.2 Configure Timezone
#### 9.2 Configure Timezone
```bash
# Configure timezone interactively
@ -1536,7 +1457,7 @@ date
**Expected output**: After selecting your timezone, the `date` command should show the current date and time in your selected timezone.
#### 10.3 Configure /etc/hosts
#### 9.3 Configure /etc/hosts
```bash
# Add localhost entries for both IPv4 and IPv6
@ -1557,7 +1478,7 @@ cat /etc/hosts
**Expected output**: The `/etc/hosts` file should show entries for `127.0.0.1`, `::1`, and your Linode's actual IP addresses all mapping to `localhost`.
#### 10.4 Install Essential Packages
#### 9.4 Install Essential Packages
```bash
sudo apt install -y \
@ -1575,7 +1496,7 @@ sudo apt install -y \
python3-certbot-nginx
```
#### 10.5 Secure SSH Configuration
#### 9.5 Secure SSH Configuration
**Critical Security Step**: After setting up SSH key authentication, you must disable password authentication and root login to secure your Production server.
@ -1722,9 +1643,9 @@ sudo sshd -t
sudo systemctl restart ssh
```
### Step 11: Create Users
### Step 10: Create Users
#### 11.1 Create the PROD_SERVICE_USER User
#### 10.1 Create the PROD_SERVICE_USER User
```bash
# Create dedicated group for the production service account
@ -1735,7 +1656,7 @@ sudo useradd -r -g PROD_SERVICE_USER -s /bin/bash -m -d /home/PROD_SERVICE_USER
echo "PROD_SERVICE_USER:$(openssl rand -base64 32)" | sudo chpasswd
```
#### 11.2 Verify Users
#### 10.2 Verify Users
```bash
sudo su - PROD_SERVICE_USER
@ -1749,9 +1670,9 @@ pwd
exit
```
### Step 12: Install Podman
### Step 11: Install Podman
#### 12.1 Install Podman
#### 11.1 Install Podman
```bash
# Install Podman and related tools
@ -1761,14 +1682,14 @@ sudo apt install -y podman
podman --version
```
#### 12.2 Configure Podman for Production Service Account
#### 11.2 Configure Podman for Production Service Account
```bash
# Podman runs rootless by default, no group membership needed
# The subuid/subgid ranges configured in Step 1.5 enable rootless operation
```
#### 12.4 Create Application Directory
#### 11.4 Create Application Directory
```bash
# Create application directory for deployment
@ -1785,33 +1706,11 @@ ls -la /opt/APP_NAME
- Sets proper ownership for the PROD_SERVICE_USER
- Ensures the directory exists before the CI workflow runs
### Step 13: Configure Podman for Forgejo Container Registry Access
**Important**: The Production Linode needs to be able to pull images from the Forgejo Container Registry. Authentication is handled through the CI/CD pipeline configuration.
```bash
# Change to the PROD_SERVICE_USER
sudo su - PROD_SERVICE_USER
# Test Forgejo Container Registry access will be verified during deployment
# Images are pulled from the configured REGISTRY_HOST during the CI/CD process
# Change back to PROD_DEPLOY_USER
exit
```
**Note**: Production deployments pull images from Forgejo Container Registry using the configured authentication.
**What this does**:
- **Tests Forgejo Container Registry access**: Registry access is verified during the deployment process
- **Integrated authentication**: Forgejo handles authentication automatically
- **Simple setup**: No complex certificate management required
### Step 14: Set Up Forgejo Runner for Production Deployment
### Step 12: Set Up Forgejo Runner for Production Deployment
**Important**: The Production Linode needs a Forgejo runner to execute the deployment job from the CI/CD workflow. This runner will pull images from Forgejo Container Registry and deploy using the production pod configuration.
#### 14.1 Download Runner
#### 12.1 Download Runner
**Important**: Run this step as the **PROD_DEPLOY_USER** (not root or PROD_SERVICE_USER). The PROD_DEPLOY_USER handles deployment tasks including downloading and installing the Forgejo runner.
@ -1848,14 +1747,14 @@ sudo mv forgejo-runner-${VERSION#v}-linux-amd64 /usr/bin/forgejo-runner
**Production Recommendation**: Use version pinning in production environments to ensure consistency and avoid unexpected breaking changes.
#### 14.2 Get Registration Token
#### 12.2 Get Registration Token
1. Go to your Forgejo repository
2. Navigate to **Settings → Actions → Runners**
3. Click **"New runner"**
4. Copy the registration token
#### 14.3 Register the Production Runner
#### 12.3 Register the Production Runner
**Step 1: Register the Runner**
@ -1903,7 +1802,7 @@ sudo chmod 600 /etc/forgejo-runner/.runner
- Registers the runner with your Forgejo instance
- Sets up the runner with appropriate labels for production deployment
#### 14.4 Create Systemd Service
#### 12.4 Create Systemd Service
```bash
# Create systemd service file
@ -1933,7 +1832,7 @@ sudo systemctl start forgejo-runner.service
sudo systemctl status forgejo-runner.service
```
#### 14.5 Test Runner Configuration
#### 12.5 Test Runner Configuration
```bash
# Check if the runner is running
@ -1967,7 +1866,7 @@ When the workflow runs, it will:
The production runner will automatically handle the deployment process when you push to the main branch.
#### 14.6 Understanding the Production Pod Setup
#### 12.6 Understanding the Production Pod Setup
The `prod-pod.yaml` file is specifically designed for production deployment and uses Kubernetes pod specifications:
@ -1992,9 +1891,9 @@ The `prod-pod.yaml` file is specifically designed for production deployment and
4. Waits for all services to be healthy
5. Verifies the deployment was successful
### Step 15: Configure Security
### Step 13: Configure Security
#### 15.1 Configure Firewall
#### 13.1 Configure Firewall
```bash
sudo ufw --force enable
@ -2007,7 +1906,7 @@ sudo ufw allow 443/tcp
**Security Note**: We only allow ports 80 and 443 for external access. The application services (backend on 3001, frontend on 3000) are only accessible through the Nginx reverse proxy, which provides better security and SSL termination.
#### 15.2 Configure Fail2ban
#### 13.2 Configure Fail2ban
**Fail2ban** is an intrusion prevention system that monitors logs and automatically blocks IP addresses showing malicious behavior.
@ -2081,6 +1980,7 @@ sudo tail -f /var/log/fail2ban.log
# Check all active jails
sudo fail2ban-client status
```
**Why This Matters for Production**:
- **Your server is exposed**: The Production Linode is accessible from the internet
@ -2088,15 +1988,15 @@ sudo fail2ban-client status
- **Resource protection**: Prevents attackers from consuming CPU/memory
- **Security layers**: Works with the firewall to provide defense in depth
### Step 16: Test Production Setup
### Step 14: Test Production Setup
#### 16.1 Test Podman Installation
#### 14.1 Test Podman Installation
```bash
podman --version
```
#### 16.2 Test Forgejo Container Registry Access
#### 14.2 Test Forgejo Container Registry Access
```bash
# Test pulling an image from Forgejo Container Registry
@ -2113,17 +2013,12 @@ podman pull YOUR_CI_CD_IP/APP_NAME/test:latest
## Part 3: Final Configuration and Testing
### Step 17: Configure Forgejo Repository Secrets
### Step 15: Configure Forgejo Repository Secrets
Go to your Forgejo repository and add these secrets in **Settings → Secrets and Variables → Actions**:
**Required Secrets:**
- `CI_HOST`: Your CI/CD Linode IP address (used for testing environment)
- `PRODUCTION_IP`: Your Production Linode IP address
- `PROD_DEPLOY_USER`: The production deployment user name (e.g., `prod-deploy`)
- `PROD_SERVICE_USER`: The production service user name (e.g., `prod-service`)
- `APP_NAME`: Your application name (e.g., `sharenet`)
- `POSTGRES_PASSWORD`: A strong password for the PostgreSQL database
- `REGISTRY_HOST`: Your Forgejo instance's registry URL
- `REGISTRY_USERNAME`: Your Forgejo username for registry authentication
- `REGISTRY_TOKEN`: Personal Access Token with `write:packages` scope for registry pushes
@ -2134,10 +2029,6 @@ Go to your Forgejo repository and add these secrets in **Settings → Secrets an
- `NODE_IMG_DIGEST`: Pinned Node.js image digest (e.g., `docker.io/library/node@sha256:...`)
- `POSTGRES_IMG_DIGEST`: Pinned PostgreSQL image digest (e.g., `docker.io/library/postgres@sha256:...`)
**Optional Secrets (for enhanced security):**
- `COSIGN_PRIVATE_KEY`: Private key for Cosign image signing
- `COSIGN_PASSWORD`: Password for Cosign private key
#### How to Obtain Each Secret (with Purpose and Commands):
**1. Image Digests (Security-Critical - Prevent Supply Chain Attacks):**
@ -2208,42 +2099,22 @@ Go to your Forgejo repository and add these secrets in **Settings → Secrets an
**4. Application Configuration:**
- **`CI_HOST`**: Your CI/CD Linode IP address
*Purpose: Testing environment configuration*
- **`PRODUCTION_IP`**: Your Production Linode IP address
*Purpose: Deployment target*
- **`PROD_DEPLOY_USER`**: Production deployment username (e.g., `prod-deploy`)
*Purpose: SSH user for deployment operations*
- **`PROD_SERVICE_USER`**: Production service username (e.g., `prod-service`)
*Purpose: User that runs application services*
- **`APP_NAME`**: Your application name (e.g., `sharenet`)
*Purpose: Image naming and directory structure*
- **`POSTGRES_PASSWORD`**: Strong password for PostgreSQL database
```bash
# Generate secure password
openssl rand -base64 32
```
**Security Note**: All secrets are managed by Forgejo and never exposed in logs or environment variables. The ephemeral PiP approach ensures secrets are only used during execution and never persist.
**Note**: This setup uses custom Dockerfiles for testing environments with base images. The CI pipeline automatically checks if base images exist in Forgejo Container Registry and pulls them from Docker Hub only when needed, eliminating rate limiting issues and providing better control over the testing environment.
### Step 18: Test Complete Pipeline
### Step 16: Test Complete Pipeline
#### 18.1 Trigger a Test Build
#### 16.1 Trigger a Test Build
1. **Make a small change** to your repository (e.g., update a comment or add a test file)
2. **Commit and push** the changes to trigger the CI/CD pipeline
3. **Monitor the build** in your Forgejo repository → Actions tab
#### 18.2 Verify Pipeline Steps
#### 16.2 Verify Pipeline Steps
The pipeline should execute these steps in order:
@ -2256,7 +2127,7 @@ The pipeline should execute these steps in order:
7. **Push to Registry**: Push images to Forgejo Container Registry from DinD
8. **Deploy to Production**: Deploy to production server
#### 18.3 Check Forgejo Container Registry
#### 16.3 Check Forgejo Container Registry
```bash
# On CI/CD Linode
@ -2277,7 +2148,7 @@ curl -k https://YOUR_CI_CD_IP:4443/v2/_catalog
# Expected: This should return authentication error without credentials
```
#### 18.4 Verify Production Deployment
#### 16.4 Verify Production Deployment
```bash
# On Production Linode
@ -2295,16 +2166,16 @@ podman logs sharenet-production-pod-backend
podman logs sharenet-production-pod-frontend
```
#### 18.5 Test Application Functionality
#### 16.5 Test Application Functionality
1. **Frontend**: Visit your production URL (IP address)
2. **Backend API**: Test API endpoints
3. **Database**: Verify database connections
4. **Logs**: Check for any errors in application logs
### Step 19: Final Verification
### Step 17: Final Verification
#### 19.1 Security Check
#### 17.1 Security Check
```bash
# Check firewall status
@ -2317,7 +2188,7 @@ sudo systemctl status fail2ban
sudo grep "PasswordAuthentication" /etc/ssh/sshd_config
```
#### 19.2 Performance Check
#### 17.2 Performance Check
```bash
# Check system resources
@ -2330,7 +2201,7 @@ df -h
docker system df
```
#### 19.3 Backup Verification
#### 17.3 Backup Verification
```bash
# Test backup script
@ -2341,16 +2212,16 @@ cd /opt/APP_NAME
./scripts/backup.sh
```
### Step 20: Documentation and Maintenance
### Step 18: Documentation and Maintenance
#### 20.1 Update Documentation
#### 18.1 Update Documentation
1. **Update README.md** with deployment information
2. **Document environment variables** and their purposes
3. **Create troubleshooting guide** for common issues
4. **Document backup and restore procedures**
#### 20.2 Set Up Monitoring Alerts
#### 18.2 Set Up Monitoring Alerts
```bash
# Set up monitoring cron job
@ -2360,7 +2231,7 @@ cd /opt/APP_NAME
tail -f /tmp/monitor.log
```
#### 20.3 Regular Maintenance Tasks
#### 18.3 Regular Maintenance Tasks
**Daily:**
- Check application logs for errors
@ -2392,15 +2263,20 @@ Before using this setup, ensure that:
### Repository Secrets
The CI pipeline expects the following secrets to be configured in your Forgejo repository:
The CI pipeline uses the following secrets. Only the required secrets are needed for current CI operations:
| Secret Name | Description | Example |
|-------------|-------------|---------|
| `REGISTRY_HOST` | Your Forgejo instance hostname | `forgejo.example.com` |
| `REGISTRY_USERNAME` | Bot or owner account username | `ci-bot` |
| `REGISTRY_TOKEN` | PAT with `write:packages` scope | `gto_...` |
| `COSIGN_PRIVATE_KEY` | (Optional) Private key for image signing | `-----BEGIN PRIVATE KEY-----...` |
| `COSIGN_PASSWORD` | (Optional) Password for Cosign private key | `your-password` |
| Secret Name | Description | Example | Required |
|-------------|-------------|---------|----------|
| `APP_NAME` | Your application name | `sharenet` | ✅ Yes |
| `REGISTRY_HOST` | Your Forgejo instance hostname | `forgejo.example.com` | ✅ Yes |
| `REGISTRY_USERNAME` | Bot or owner account username | `ci-bot` | ✅ Yes |
| `REGISTRY_TOKEN` | PAT with `write:packages` scope | `gto_...` | ✅ Yes |
| `SSH_PRIVATE_KEY` | SSH private key for deployment | `-----BEGIN OPENSSH PRIVATE KEY-----...` | ✅ Yes |
| `SSH_KNOWN_HOSTS` | SSH known_hosts entry | `production-ip ecdsa-sha2-nistp256...` | ✅ Yes |
| `PODMAN_CLIENT_IMG_DIGEST` | Pinned Podman client image digest | `quay.io/podman/stable@sha256:...` | ✅ Yes |
| `RUST_IMG_DIGEST` | Pinned Rust image digest | `docker.io/library/rust@sha256:...` | ✅ Yes |
| `NODE_IMG_DIGEST` | Pinned Node.js image digest | `docker.io/library/node@sha256:...` | ✅ Yes |
| `POSTGRES_IMG_DIGEST` | Pinned PostgreSQL image digest | `docker.io/library/postgres@sha256:...` | ✅ Yes |
### How It Works