devteam/sharenet
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Fix prod Forgejo runner install steps

Browse source
This commit is contained in:
continuist 2025-07-05 00:39:42 -04:00
parent f95359dda2
commit f37825f845
1 changed files with 170 additions and 75 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

245
CI_CD_PIPELINE_SETUP_GUIDE.md
View file

@ -55,8 +55,8 @@ This guide covers setting up a complete Continuous Integration/Continuous Deploy
## Quick Start ## Quick Start
1. **Set up CI/CD Linode** (Steps 1-9) 1. **Set up CI/CD Linode** (Steps 0-9)
2. **Set up Production Linode** (Steps 10-15) 2. **Set up Production Linode** (Steps 10-16)
3. **Set up Forgejo repository secrets** (Step 17) 3. **Set up Forgejo repository secrets** (Step 17)
4. **Test the complete pipeline** (Step 18) 4. **Test the complete pipeline** (Step 18)
@ -892,7 +892,7 @@ forgejo-runner register \
sudo mkdir -p /etc/forgejo-runner sudo mkdir -p /etc/forgejo-runner
# Copy the runner configuration to system location # Copy the runner configuration to system location
sudo cp /home/CI_DEPLOY_USER/.runner /etc/forgejo-runner/.runner sudo mv /home/CI_DEPLOY_USER/.runner /etc/forgejo-runner/.runner
# Set proper ownership and permissions # Set proper ownership and permissions
sudo chown CI_SERVICE_USER:CI_SERVICE_USER /etc/forgejo-runner/.runner sudo chown CI_SERVICE_USER:CI_SERVICE_USER /etc/forgejo-runner/.runner
@ -1380,60 +1380,99 @@ exit
**Important**: The Production Linode needs a Forgejo runner to execute the deployment job from the CI/CD workflow. This runner will pull images from Harbor and deploy using `docker-compose.prod.yml`. **Important**: The Production Linode needs a Forgejo runner to execute the deployment job from the CI/CD workflow. This runner will pull images from Harbor and deploy using `docker-compose.prod.yml`.
#### 14.1 Install Forgejo Runner #### 14.1 Download Runner
**Important**: Run this step as the **PROD_DEPLOY_USER** (not root or PROD_SERVICE_USER). The PROD_DEPLOY_USER handles deployment tasks including downloading and installing the Forgejo runner.
```bash ```bash
# Download the latest Forgejo runner cd ~
wget -O forgejo-runner https://codeberg.org/forgejo/runner/releases/download/v4.0.0/forgejo-runner-linux-amd64
# Make it executable # Get the latest version dynamically
chmod +x forgejo-runner LATEST_VERSION=$(curl -s https://code.forgejo.org/api/v1/repos/forgejo/runner/releases | jq -r '.[0].tag_name')
echo "Downloading Forgejo runner version: $LATEST_VERSION"
# Move to system location # Download the latest runner
sudo mv forgejo-runner /usr/bin/forgejo-runner wget https://code.forgejo.org/forgejo/runner/releases/download/${LATEST_VERSION}/forgejo-runner-${LATEST_VERSION#v}-linux-amd64
chmod +x forgejo-runner-${LATEST_VERSION#v}-linux-amd64
# Verify installation sudo mv forgejo-runner-${LATEST_VERSION#v}-linux-amd64 /usr/bin/forgejo-runner
forgejo-runner --version
``` ```
#### 14.2 Set Up Runner Directory for PROD_SERVICE_USER **Alternative: Pin to Specific Version (Recommended for Production)**
If you prefer to pin to a specific version for stability, replace the dynamic download with:
```bash ```bash
# Create runner directory owned by PROD_SERVICE_USER cd ~
sudo mkdir -p /opt/forgejo-runner VERSION="v6.3.1" # Pin to specific version
sudo chown PROD_SERVICE_USER:PROD_SERVICE_USER /opt/forgejo-runner wget https://code.forgejo.org/forgejo/runner/releases/download/${VERSION}/forgejo-runner-${VERSION#v}-linux-amd64
chmod +x forgejo-runner-${VERSION#v}-linux-amd64
sudo mv forgejo-runner-${VERSION#v}-linux-amd64 /usr/bin/forgejo-runner
``` ```
#### 14.3 Get Registration Token **What this does**:
- **Dynamic approach**: Downloads the latest stable Forgejo Actions runner
- **Version pinning**: Allows you to specify a known-good version for production
- **System installation**: Installs the binary system-wide in `/usr/bin/` for proper Linux structure
- **Makes the binary executable** and available system-wide
**Production Recommendation**: Use version pinning in production environments to ensure consistency and avoid unexpected breaking changes.
#### 14.2 Get Registration Token
1. Go to your Forgejo repository 1. Go to your Forgejo repository
2. Navigate to **Settings → Actions → Runners** 2. Navigate to **Settings → Actions → Runners**
3. Click **"New runner"** 3. Click **"New runner"**
4. Copy the registration token 4. Copy the registration token
#### 14.4 Register the Production Runner #### 14.3 Register the Production Runner
**Step 1: Register the Runner**
```bash ```bash
# Switch to PROD_SERVICE_USER # Switch to PROD_DEPLOY_USER to register the runner
sudo su - PROD_SERVICE_USER sudo su - PROD_DEPLOY_USER
# Register the runner with production label cd ~
# Register the runner with your Forgejo instance
forgejo-runner register \ forgejo-runner register \
--instance https://your-forgejo-instance \ --instance https://your-forgejo-instance \
--token YOUR_REGISTRATION_TOKEN \ --token YOUR_REGISTRATION_TOKEN \
--name "production-runner" \ --name "prod-runner" \
--labels "prod" \ --labels "prod" \
--no-interactive --no-interactive
# Copy configuration to system location
sudo cp /home/PROD_SERVICE_USER/.runner /opt/forgejo-runner/.runner
sudo chown PROD_SERVICE_USER:PROD_SERVICE_USER /opt/forgejo-runner/.runner
sudo chmod 600 /opt/forgejo-runner/.runner
``` ```
**Important**: Replace `your-forgejo-instance` with your actual Forgejo instance URL and `YOUR_REGISTRATION_TOKEN` with the token you copied from Step 14.3. **Important**: Replace `your-forgejo-instance` with your actual Forgejo instance URL and `YOUR_REGISTRATION_TOKEN` with the token you copied from Step 14.2. Also make sure it ends in a `/`.
#### 14.5 Create Systemd Service **Note**: The `your-forgejo-instance` should be the **base URL** of your Forgejo instance (e.g., `https://git.<your-domain>/`), not the full path to the repository. The runner registration process will handle connecting to the specific repository based on the token you provide.
**What this does**:
- Creates the required `.runner` configuration file in the PROD_DEPLOY_USER's home directory
- Registers the runner with your Forgejo instance
- Sets up the runner with appropriate labels for production deployment
**Step 2: Set Up System Configuration**
```bash
# Create system config directory for Forgejo runner
sudo mkdir -p /etc/forgejo-runner
# Copy the runner configuration to system location
sudo mv /home/PROD_DEPLOY_USER/.runner /etc/forgejo-runner/.runner
# Set proper ownership and permissions
sudo chown PROD_SERVICE_USER:PROD_SERVICE_USER /etc/forgejo-runner/.runner
sudo chmod 600 /etc/forgejo-runner/.runner
```
**What this does**:
- Copies the configuration to the system location (`/etc/forgejo-runner/.runner`)
- Sets proper ownership and permissions for PROD_SERVICE_USER to access the config
- Registers the runner with your Forgejo instance
- Sets up the runner with appropriate labels for production deployment
#### 14.4 Create Systemd Service
```bash ```bash
# Create systemd service file # Create systemd service file
@ -1445,11 +1484,10 @@ After=network.target docker.service
[Service] [Service]
Type=simple Type=simple
User=PROD_SERVICE_USER User=PROD_SERVICE_USER
WorkingDirectory=/opt/forgejo-runner WorkingDirectory=/etc/forgejo-runner
ExecStart=/usr/bin/forgejo-runner daemon ExecStart=/usr/bin/forgejo-runner daemon
Restart=always Restart=always
RestartSec=10 RestartSec=10
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[Install] [Install]
WantedBy=multi-user.target WantedBy=multi-user.target
@ -1464,7 +1502,7 @@ sudo systemctl start forgejo-runner.service
sudo systemctl status forgejo-runner.service sudo systemctl status forgejo-runner.service
``` ```
#### 14.6 Test Runner Configuration #### 14.5 Test Runner Configuration
```bash ```bash
# Check if the runner is running # Check if the runner is running
@ -1475,7 +1513,7 @@ sudo journalctl -u forgejo-runner.service -f --no-pager
# Verify runner appears in Forgejo # Verify runner appears in Forgejo
# Go to your Forgejo repository → Settings → Actions → Runners # Go to your Forgejo repository → Settings → Actions → Runners
# You should see your runner listed as "production-runner" with status "Online" # You should see your runner listed as "prod-runner" with status "Online"
``` ```
**Expected Output**: **Expected Output**:
@ -1491,7 +1529,7 @@ sudo journalctl -u forgejo-runner.service -f --no-pager
The production runner will automatically handle the deployment process when you push to the main branch. The production runner will automatically handle the deployment process when you push to the main branch.
#### 14.7 Understanding the Production Docker Compose Setup #### 14.6 Understanding the Production Docker Compose Setup
The `docker-compose.prod.yml` file is specifically designed for production deployment and differs from development setups: The `docker-compose.prod.yml` file is specifically designed for production deployment and differs from development setups:
@ -1527,17 +1565,92 @@ sudo ufw default allow outgoing
sudo ufw allow ssh sudo ufw allow ssh
sudo ufw allow 80/tcp sudo ufw allow 80/tcp
sudo ufw allow 443/tcp sudo ufw allow 443/tcp
sudo ufw allow 3000/tcp
sudo ufw allow 3001/tcp
``` ```
**Security Note**: We only allow ports 80 and 443 for external access. The application services (backend on 3001, frontend on 3000) are only accessible through the Nginx reverse proxy, which provides better security and SSL termination.
#### 15.2 Configure Fail2ban #### 15.2 Configure Fail2ban
**Fail2ban** is an intrusion prevention system that monitors logs and automatically blocks IP addresses showing malicious behavior.
```bash ```bash
# Install fail2ban (if not already installed)
sudo apt install -y fail2ban
# Create a custom jail configuration
sudo tee /etc/fail2ban/jail.local > /dev/null << 'EOF'
[DEFAULT]
# Ban time in seconds (24 hours)
bantime = 86400
# Find time in seconds (10 minutes)
findtime = 600
# Max retries before ban
maxretry = 3
# Ban action (use ufw since we're using ufw firewall)
banaction = ufw
# Log level
loglevel = INFO
# Log target
logtarget = /var/log/fail2ban.log
# SSH protection
[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
# Note: Nginx protection is handled by the firewall and application-level security
# Docker containers are isolated, and Nginx logs are not directly accessible to fail2ban
# Web attack protection is provided by:
# 1. UFW firewall (ports 80/443 only)
# 2. Nginx security headers and rate limiting
# 3. Application-level input validation
EOF
# Enable and start fail2ban
sudo systemctl enable fail2ban sudo systemctl enable fail2ban
sudo systemctl start fail2ban sudo systemctl start fail2ban
# Verify fail2ban is running
sudo systemctl status fail2ban
# Check current jails
sudo fail2ban-client status
``` ```
**What this does**:
- **SSH Protection**: Blocks IPs that fail SSH login 3 times in 10 minutes
- **24-hour bans**: Banned IPs are blocked for 24 hours
- **Automatic monitoring**: Continuously watches SSH logs
**Web Security Note**: Since Nginx runs in a Docker container, web attack protection is handled by:
- **UFW Firewall**: Only allows ports 80/443 (no direct access to app services)
- **Nginx Security**: Built-in rate limiting and security headers
- **Application Security**: Input validation in the backend/frontend code
**Monitoring Fail2ban**:
```bash
# Check banned IPs
sudo fail2ban-client status sshd
# Unban an IP if needed
sudo fail2ban-client set sshd unbanip IP_ADDRESS
# View fail2ban logs
sudo tail -f /var/log/fail2ban.log
# Check all active jails
sudo fail2ban-client status
```
**Why This Matters for Production**:
- **Your server is exposed**: The Production Linode is accessible from the internet
- **Automated attacks**: Bots constantly scan for vulnerable servers
- **Resource protection**: Prevents attackers from consuming CPU/memory
- **Security layers**: Works with the firewall to provide defense in depth
### Step 16: Test Production Setup ### Step 16: Test Production Setup
#### 16.1 Test Docker Installation #### 16.1 Test Docker Installation
@ -1551,36 +1664,18 @@ docker compose --version
```bash ```bash
# Test pulling an image from the CI/CD Harbor registry # Test pulling an image from the CI/CD Harbor registry
docker pull YOUR_CI_CD_IP:8080/public/backend:latest docker pull YOUR_CI_CD_IP/APP_NAME/test:latest
``` ```
**Important**: Replace `YOUR_CI_CD_IP` with your actual CI/CD Linode IP address. **Important**: Replace `YOUR_CI_CD_IP` with your actual CI/CD Linode IP address.
#### 16.3 Test Application Deployment **Note**: Application deployment testing will be done in Step 20 after the complete CI/CD pipeline is set up.
```bash
cd /opt/APP_NAME
docker compose up -d
```
#### 16.4 Verify Application Status
```bash
docker compose ps
curl http://localhost:3000
curl http://localhost:3001/health
```
**Expected Output**:
- All containers should be running
- Frontend should be accessible on port 3000
- Backend health check should return 200 OK
--- ---
## Part 3: Final Configuration and Testing ## Part 3: Final Configuration and Testing
### Step 19: Configure Forgejo Repository Secrets ### Step 17: Configure Forgejo Repository Secrets
Go to your Forgejo repository and add these secrets in **Settings → Secrets and Variables → Actions**: Go to your Forgejo repository and add these secrets in **Settings → Secrets and Variables → Actions**:
@ -1598,15 +1693,15 @@ Go to your Forgejo repository and add these secrets in **Settings → Secrets an
- `DOMAIN`: Your domain name (e.g., `example.com`) - `DOMAIN`: Your domain name (e.g., `example.com`)
- `EMAIL`: Your email for SSL certificate notifications - `EMAIL`: Your email for SSL certificate notifications
### Step 20: Test Complete Pipeline ### Step 18: Test Complete Pipeline
#### 20.1 Trigger a Test Build #### 18.1 Trigger a Test Build
1. **Make a small change** to your repository (e.g., update a comment or add a test file) 1. **Make a small change** to your repository (e.g., update a comment or add a test file)
2. **Commit and push** the changes to trigger the CI/CD pipeline 2. **Commit and push** the changes to trigger the CI/CD pipeline
3. **Monitor the build** in your Forgejo repository → Actions tab 3. **Monitor the build** in your Forgejo repository → Actions tab
#### 20.2 Verify Pipeline Steps #### 18.2 Verify Pipeline Steps
The pipeline should execute these steps in order: The pipeline should execute these steps in order:
@ -1619,7 +1714,7 @@ The pipeline should execute these steps in order:
7. **Push to Registry**: Push images to Harbor registry from DinD 7. **Push to Registry**: Push images to Harbor registry from DinD
8. **Deploy to Production**: Deploy to production server 8. **Deploy to Production**: Deploy to production server
#### 20.3 Check Harbor #### 18.3 Check Harbor
```bash ```bash
# On CI/CD Linode # On CI/CD Linode
@ -1633,7 +1728,7 @@ curl -k https://localhost:8080/v2/public/backend/tags/list
curl -k https://localhost:8080/v2/public/frontend/tags/list curl -k https://localhost:8080/v2/public/frontend/tags/list
``` ```
#### 20.4 Verify Production Deployment #### 18.4 Verify Production Deployment
```bash ```bash
# On Production Linode # On Production Linode
@ -1651,16 +1746,16 @@ docker compose logs backend
docker compose logs frontend docker compose logs frontend
``` ```
#### 20.5 Test Application Functionality #### 18.5 Test Application Functionality
1. **Frontend**: Visit your production URL (IP or domain) 1. **Frontend**: Visit your production URL (IP or domain)
2. **Backend API**: Test API endpoints 2. **Backend API**: Test API endpoints
3. **Database**: Verify database connections 3. **Database**: Verify database connections
4. **Logs**: Check for any errors in application logs 4. **Logs**: Check for any errors in application logs
### Step 21: Set Up SSL/TLS (Optional - Domain Users) ### Step 19: Set Up SSL/TLS (Optional - Domain Users)
#### 21.1 Install SSL Certificate #### 19.1 Install SSL Certificate
If you have a domain pointing to your Production Linode: If you have a domain pointing to your Production Linode:
@ -1672,7 +1767,7 @@ sudo certbot --nginx -d your-domain.com
sudo certbot certificates sudo certbot certificates
``` ```
#### 21.2 Configure Auto-Renewal #### 19.2 Configure Auto-Renewal
```bash ```bash
# Test auto-renewal # Test auto-renewal
@ -1684,9 +1779,9 @@ sudo crontab -e
# 0 12 * * * /usr/bin/certbot renew --quiet # 0 12 * * * /usr/bin/certbot renew --quiet
``` ```
### Step 22: Final Verification ### Step 20: Final Verification
#### 22.1 Security Check #### 20.1 Security Check
```bash ```bash
# Check firewall status # Check firewall status
@ -1699,7 +1794,7 @@ sudo systemctl status fail2ban
sudo grep "PasswordAuthentication" /etc/ssh/sshd_config sudo grep "PasswordAuthentication" /etc/ssh/sshd_config
``` ```
#### 22.2 Performance Check #### 20.2 Performance Check
```bash ```bash
# Check system resources # Check system resources
@ -1712,7 +1807,7 @@ df -h
docker system df docker system df
``` ```
#### 22.3 Backup Verification #### 20.3 Backup Verification
```bash ```bash
# Test backup script # Test backup script
@ -1723,16 +1818,16 @@ cd /opt/APP_NAME
./scripts/backup.sh ./scripts/backup.sh
``` ```
### Step 23: Documentation and Maintenance ### Step 21: Documentation and Maintenance
#### 23.1 Update Documentation #### 21.1 Update Documentation
1. **Update README.md** with deployment information 1. **Update README.md** with deployment information
2. **Document environment variables** and their purposes 2. **Document environment variables** and their purposes
3. **Create troubleshooting guide** for common issues 3. **Create troubleshooting guide** for common issues
4. **Document backup and restore procedures** 4. **Document backup and restore procedures**
#### 23.2 Set Up Monitoring Alerts #### 21.2 Set Up Monitoring Alerts
```bash ```bash
# Set up monitoring cron job # Set up monitoring cron job
@ -1742,7 +1837,7 @@ cd /opt/APP_NAME
tail -f /tmp/monitor.log tail -f /tmp/monitor.log
``` ```
#### 23.3 Regular Maintenance Tasks #### 21.3 Regular Maintenance Tasks
**Daily:** **Daily:**
- Check application logs for errors - Check application logs for errors