devteam/sharenet
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Remove Let's Encrypt guide for now to focus on IP-only installs
Some checks are pending
CI/CD Pipeline (Fully Isolated DinD) / Run Tests (DinD) (push) Waiting to run
Details
CI/CD Pipeline (Fully Isolated DinD) / Build and Push Docker Images (DinD) (push) Blocked by required conditions
Details
CI/CD Pipeline (Fully Isolated DinD) / Deploy to Production (push) Blocked by required conditions
Details

Browse source
This commit is contained in:
continuist 2025-08-24 11:21:16 -04:00
parent 6c30dd20aa
commit 7525f936bf
1 changed files with 23 additions and 168 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

191
CI_CD_PIPELINE_SETUP_GUIDE.md
View file

@ -51,11 +51,10 @@ This guide covers setting up a complete Continuous Integration/Continuous Deploy
- Two Ubuntu 24.04 LTS Linodes with root access
- Basic familiarity with Linux commands and SSH
- Forgejo repository with Actions enabled
- **Optional**: Domain name for Production Linode (for SSL/TLS)
## Quick Start
1. **Set up CI/CD Linode** (Steps 0-9)
1. **Set up CI/CD Linode** (Steps 0-8)
2. **Set up Production Linode** (Steps 10-16)
3. **Set up Forgejo repository secrets** (Step 17)
4. **Test the complete pipeline** (Step 18)
@ -74,7 +73,6 @@ This guide covers setting up a complete Continuous Integration/Continuous Deploy
### Production Linode Features
- Podman-based application deployment
- **Optional SSL/TLS certificate management** (if domain is provided)
- Nginx reverse proxy with security headers
- Automated backups and monitoring
- Firewall and fail2ban protection
@ -136,7 +134,6 @@ This guide assumes you have already:
2. **Set root passwords** for both Linodes
3. **Have SSH client** installed on your local machine
4. **Have Forgejo repository** with Actions enabled
5. **Optional**: Domain name pointing to Production Linode's IP addresses
### Step 0: Initial SSH Access and Verification
@ -146,7 +143,7 @@ Before proceeding with the setup, you need to establish initial SSH access to bo
From your Linode dashboard, note the IP addresses for:
- **CI/CD Linode**: `YOUR_CI_CD_IP` (IP address only, no domain needed)
- **Production Linode**: `YOUR_PRODUCTION_IP` (IP address for SSH, domain for web access)
- **Production Linode**: `YOUR_PRODUCTION_IP` (IP address for SSH and web access)
#### 0.2 Test Initial SSH Access
@ -182,15 +179,14 @@ Before proceeding, decide on:
3. **Application Name**: Choose a name for your application (e.g., `sharenet`)
- Replace `APP_NAME` in this guide with your chosen name
4. **Domain Name** (Optional): If you have a domain, note it for SSL configuration
- Replace `your-domain.com` in this guide with your actual domain
**Example**:
- If you choose `ci-service` as CI service account, `ci-deploy` as CI deployment user, and `sharenet` as application name:
- Replace all `CI_SERVICE_USER` with `ci-service`
- Replace all `CI_DEPLOY_USER` with `ci-deploy`
- Replace all `APP_NAME` with `sharenet`
- If you have a domain `example.com`, replace `your-domain.com` with `example.com`
**Security Model**:
- **CI Service Account (`CI_SERVICE_USER`)**: Runs CI pipeline and Docker operations, no sudo access
@ -766,13 +762,7 @@ sudo chmod 755 /var/lib/registry/data # Registry data
#### 4.2.4 Generate TLS Certificate and Install for Podman
**Choose one of the following options based on whether you have a domain name:**
---
#### **Option A: Self-Signed Certificate (No Domain Required)**
**Perform all of these steps if you do NOT have a domain name:**
**Generate Self-Signed Certificate:**
```bash
# 1. Generate self-signed certificate with proper CA chain using FHS-compliant structure
@ -830,80 +820,11 @@ sudo update-ca-certificates
---
#### **Option B: Let's Encrypt Certificate (Domain Required)**
**Perform all of these steps if you DO have a domain name:**
```bash
# 1. Generate Let's Encrypt certificate
sudo apt update
sudo apt install -y certbot python3-certbot-nginx
sudo certbot certonly --standalone \
--email your-email@example.com \
--agree-tos \
--no-eff-email \
-d YOUR_DOMAIN_NAME
sudo certbot certificates
sudo cp /etc/letsencrypt/live/YOUR_DOMAIN_NAME/fullchain.pem /etc/registry/certs/registry.crt
sudo cp /etc/letsencrypt/live/YOUR_DOMAIN_NAME/privkey.pem /etc/registry/certs/registry.key
sudo chown CI_SERVICE_USER:CI_SERVICE_USER /etc/registry/certs/registry.crt /etc/registry/certs/registry.key
sudo chmod 644 /etc/registry/certs/registry.crt
sudo chmod 600 /etc/registry/certs/registry.key
# 2. Install certificate in system trust store (for curl, wget, etc.)
# Note: Let's Encrypt certificates are already trusted by the system, but we install for consistency
sudo cp /etc/registry/certs/registry.crt /usr/local/share/ca-certificates/registry-ca.crt
sudo update-ca-certificates
```
---
**Note:**
- For **Option A**: Replace `YOUR_ACTUAL_IP_ADDRESS` with your server's IP address.
- For **Option B**: Replace `YOUR_DOMAIN_NAME` with your domain name and `your-email@example.com` with your email address.
- Replace `YOUR_ACTUAL_IP_ADDRESS` with your server's IP address.
---
**After completing the steps for your chosen option, continue with Step 4.5 (Set Up Systemd Service for Docker Registry v2).**
#### 4.3 Install Certificate for Podman (Option B Only)
**Important**: This step adds the Let's Encrypt certificate to the system trust store. Since Let's Encrypt is a trusted CA, system tools will automatically trust this certificate.
```bash
# Install certificate in system trust store (for curl, wget, etc.)
sudo cp /etc/registry/certs/registry.crt /usr/local/share/ca-certificates/registry-ca.crt
sudo update-ca-certificates
# Verify certificate installation
if [ -f "/usr/local/share/ca-certificates/registry-ca.crt" ]; then
echo "✅ Let's Encrypt certificate installed in system trust store"
else
echo "❌ Failed to install certificate in system trust store"
exit 1
fi
echo "Certificate installation completed successfully!"
echo "System tools can now connect to the registry securely using your domain name"
```
#### 4.4 Set Up Automatic Certificate Renewal (Option B Only)
**Important**: Let's Encrypt certificates expire after 90 days, so we need to set up automatic renewal.
```bash
# Test automatic renewal
sudo certbot renew --dry-run
# Set up automatic renewal cron job
sudo crontab -e
# Add this line to renew certificates twice daily (Let's Encrypt allows renewal 30 days before expiry):
# 0 12,18 * * * /usr/bin/certbot renew --quiet --post-hook "cp /etc/letsencrypt/live/YOUR_DOMAIN_NAME/fullchain.pem /etc/registry/certs/registry.crt && cp /etc/letsencrypt/live/YOUR_DOMAIN_NAME/privkey.pem /etc/registry/certs/registry.key && chown CI_SERVICE_USER:CI_SERVICE_USER /etc/registry/certs/registry.* && chmod 644 /etc/registry/certs/registry.crt && chmod 600 /etc/registry/certs/registry.key && cp /etc/registry/certs/registry.crt /usr/local/share/ca-certificates/registry-ca.crt && update-ca-certificates && systemctl restart docker-registry.service"
echo "Automatic certificate renewal configured!"
echo "Certificates will be renewed automatically and the registry service will be restarted"
```
#### 4.5 Set Up Systemd Service for Docker Registry v2
```bash
@ -1026,52 +947,17 @@ env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/
exit
```
**For Option B (Let's Encrypt certificates):**
```bash
# Switch to CI_SERVICE_USER for testing (CI_SERVICE_USER runs CI pipeline and Docker operations)
sudo su - CI_SERVICE_USER
# Navigate to the application directory
cd /opt/APP_NAME
# Test Podman login and push (now using Let's Encrypt certificate with domain)
echo "your-secure-registry-password" | env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman login YOUR_DOMAIN_NAME -u registry-user --password-stdin
# Create and push test image
echo "FROM alpine:latest" > /tmp/test.Dockerfile
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman build -f /tmp/test.Dockerfile -t YOUR_DOMAIN_NAME/APP_NAME/test:latest /tmp
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman push YOUR_DOMAIN_NAME/APP_NAME/test:latest
# Test public pull (no authentication)
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman logout YOUR_DOMAIN_NAME
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman pull YOUR_DOMAIN_NAME/APP_NAME/test:latest
# Test that unauthorized push is blocked
echo "FROM alpine:latest" > /tmp/unauthorized.Dockerfile
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman build -f /tmp/unauthorized.Dockerfile -t YOUR_DOMAIN_NAME/APP_NAME/unauthorized:latest /tmp
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman push YOUR_DOMAIN_NAME/APP_NAME/unauthorized:latest
# Expected: This should fail with authentication error
# Clean up
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman rmi YOUR_DOMAIN_NAME/APP_NAME/test:latest
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman rmi YOUR_DOMAIN_NAME/APP_NAME/unauthorized:latest
exit
```
**Important**: For Option B, the registry configuration is handled via environment variables in the Docker Compose file, so no additional configuration changes are needed.
**Expected behavior**:
- ✅ Push requires authentication with `registry-user` credentials on port 4443
- ✅ Pull works without authentication (public read access) on port 443
- ✅ Unauthorized push is blocked on authenticated endpoint
- ✅ Registry accessible at `https://YOUR_ACTUAL_IP_ADDRESS:4443` for authenticated operations (Option A)
- ✅ Registry accessible at `https://YOUR_ACTUAL_IP_ADDRESS` for unauthenticated pulls (Option A)
- ✅ Registry accessible at `https://YOUR_DOMAIN_NAME` with valid Let's Encrypt certificate (Option B)
- ✅ Certificate automatically renews every 60 days (Option B only)
- ✅ Registry accessible at `https://YOUR_ACTUAL_IP_ADDRESS:4443` for authenticated operations
- ✅ Registry accessible at `https://YOUR_ACTUAL_IP_ADDRESS` for unauthenticated pulls
- ✅ Proper FHS-compliant certificate structure with secure permissions
**Troubleshooting TLS Errors (Option A only):**
**Troubleshooting TLS Errors:**
If you get a TLS error like `remote error: tls: internal error` when using self-signed certificates, verify the certificate installation and Docker configuration:
@ -1398,6 +1284,7 @@ podman exec ci-pip podman pull YOUR_CI_CD_IP/APP_NAME/test:latest
# Clean up test images
podman exec ci-pip podman rmi YOUR_CI_CD_IP:4443/APP_NAME/test:latest
podman exec ci-pip podman rmi YOUR_CI_CD_IP/APP_NAME/test:latest
```
#### 6.3 Set Up Workspace Directory
@ -1460,7 +1347,6 @@ The Docker Registry setup now follows the Filesystem Hierarchy Standard (FHS) fo
- **Log management**: Logs in `/var/log/nginx/` for centralized logging
- **Configuration separation**: App configs in app directory, system data in system directories
- **Policy enforcement**: Container policies for image signature verification
```
**What this does**:
- **Configures certificate trust**: Properly sets up Docker Registry certificate trust in DinD
@ -1869,11 +1755,10 @@ exit
```bash
# Install Podman and related tools
sudo apt install -y podman podman-compose
sudo apt install -y podman
# Verify installation
podman --version
podman-compose --version
```
#### 12.2 Configure Podman for Production Service Account
@ -2213,7 +2098,6 @@ sudo fail2ban-client status
```bash
podman --version
podman-compose --version
```
#### 16.2 Test Docker Registry v2 Access
@ -2227,7 +2111,7 @@ podman pull YOUR_CI_CD_IP/APP_NAME/test:latest
**Note**: Production uses unauthenticated pulls from the standard HTTPS port (443) for deployment operations.
**Note**: Application deployment testing will be done in Step 20 after the complete CI/CD pipeline is set up.
**Note**: Application deployment testing will be done in Step 19 after the complete CI/CD pipeline is set up.
---
@ -2249,9 +2133,7 @@ Go to your Forgejo repository and add these secrets in **Settings → Secrets an
- `REGISTRY_PUSH_URL`: Docker Registry v2 URL for authenticated pushes (e.g., `YOUR_CI_CD_IP:4443`)
- `REGISTRY_PULL_URL`: Docker Registry v2 URL for unauthenticated pulls (e.g., `YOUR_CI_CD_IP`)
**Optional Secrets (for domain users):**
- `DOMAIN`: Your domain name (e.g., `example.com`)
- `EMAIL`: Your email for SSL certificate notifications
**Note**: This setup uses custom Dockerfiles for testing environments with base images stored in Docker Registry. The CI pipeline automatically checks if base images exist in Docker Registry and pulls them from Docker Hub only when needed, eliminating rate limiting issues and providing better control over the testing environment.
@ -2317,40 +2199,14 @@ podman logs sharenet-production-pod-frontend
#### 18.5 Test Application Functionality
1. **Frontend**: Visit your production URL (IP or domain)
1. **Frontend**: Visit your production URL (IP address)
2. **Backend API**: Test API endpoints
3. **Database**: Verify database connections
4. **Logs**: Check for any errors in application logs
### Step 19: Set Up SSL/TLS (Optional - Domain Users)
### Step 19: Final Verification
#### 19.1 Install SSL Certificate
If you have a domain pointing to your Production Linode:
```bash
# On Production Linode
sudo certbot --nginx -d your-domain.com
# Verify certificate
sudo certbot certificates
```
#### 19.2 Configure Auto-Renewal
```bash
# Test auto-renewal
sudo certbot renew --dry-run
# Add to crontab for automatic renewal
sudo crontab -e
# Add this line:
# 0 12 * * * /usr/bin/certbot renew --quiet
```
### Step 20: Final Verification
#### 20.1 Security Check
#### 19.1 Security Check
```bash
# Check firewall status
@ -2363,7 +2219,7 @@ sudo systemctl status fail2ban
sudo grep "PasswordAuthentication" /etc/ssh/sshd_config
```
#### 20.2 Performance Check
#### 19.2 Performance Check
```bash
# Check system resources
@ -2376,7 +2232,7 @@ df -h
docker system df
```
#### 20.3 Backup Verification
#### 19.3 Backup Verification
```bash
# Test backup script
@ -2387,16 +2243,16 @@ cd /opt/APP_NAME
./scripts/backup.sh
```
### Step 21: Documentation and Maintenance
### Step 20: Documentation and Maintenance
#### 21.1 Update Documentation
#### 20.1 Update Documentation
1. **Update README.md** with deployment information
2. **Document environment variables** and their purposes
3. **Create troubleshooting guide** for common issues
4. **Document backup and restore procedures**
#### 21.2 Set Up Monitoring Alerts
#### 20.2 Set Up Monitoring Alerts
```bash
# Set up monitoring cron job
@ -2406,7 +2262,7 @@ cd /opt/APP_NAME
tail -f /tmp/monitor.log
```
#### 21.3 Regular Maintenance Tasks
#### 20.3 Regular Maintenance Tasks
**Daily:**
- Check application logs for errors
@ -2420,7 +2276,6 @@ tail -f /tmp/monitor.log
**Monthly:**
- Review and rotate logs
- Update SSL certificates
- Review and update documentation
---
@ -2435,7 +2290,7 @@ You have successfully set up a complete CI/CD pipeline with:
- ✅ **Health monitoring** and logging
- ✅ **Backup and cleanup** automation
- ✅ **Security hardening** with proper user separation
- ✅ **SSL/TLS support** for production (optional)
- ✅ **SSL/TLS support** with self-signed certificates
- ✅ **Zero resource contention** between CI/CD and Docker Registry
- ✅ **FHS-compliant directory structure** for better organization and security