devteam/sharenet

Fork 0

continuist 2346c1b569

CI/CD Pipeline / Test Backend (push) Waiting to run

Details

CI/CD Pipeline / Test Frontend (push) Waiting to run

Details

CI/CD Pipeline / Build and Push Docker Images (push) Blocked by required conditions

Details

CI/CD Pipeline / Deploy to Production (push) Blocked by required conditions

Details

Add steps to change timezone and set hosts file

2025-06-27 23:52:50 -04:00

46 KiB

Raw Blame History

CI/CD Pipeline Setup Guide

This guide covers setting up a complete Continuous Integration/Continuous Deployment (CI/CD) pipeline with a CI/CD Linode and Production Linode for automated builds, testing, and deployments.

Architecture Overview

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Forgejo Host  │    │   CI/CD Linode  │    │ Production Linode│
│   (Repository)  │    │ (Actions Runner)│    │ (Docker Deploy) │
│                 │    │ + Docker Registry│   │                 │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         │                       │                       │
         └─────────── Push ──────┼───────────────────────┘
                                 │
                                 └─── Deploy ────────────┘

Pipeline Flow

Code Push: Developer pushes code to Forgejo repository
Automated Testing: CI/CD Linode runs tests on backend and frontend
Image Building: If tests pass, Docker images are built
Registry Push: Images are pushed to private registry on CI/CD Linode
Production Deployment: Production Linode pulls images and deploys
Health Check: Application is verified and accessible

Prerequisites

Two Ubuntu 24.04 LTS Linodes with root access
Basic familiarity with Linux commands and SSH
Forgejo repository with Actions enabled
Optional: Domain name for Production Linode (for SSL/TLS)

Quick Start

Set up CI/CD Linode (Steps 1-13)
Set up Production Linode (Steps 14-26)
Configure SSH key exchange (Step 27)
Set up Forgejo repository secrets (Step 28)
Test the complete pipeline (Step 29)

What's Included

CI/CD Linode Features

Forgejo Actions runner for automated builds
Local Docker registry for image storage
Registry web UI for image management
Automated cleanup of old images
Secure SSH communication with production

Production Linode Features

Docker-based application deployment
Optional SSL/TLS certificate management (if domain is provided)
Nginx reverse proxy with security headers
Automated backups and monitoring
Firewall and fail2ban protection

Pipeline Features

Automated testing on every code push
Automated image building and registry push
Automated deployment to production
Rollback capability with image versioning
Health monitoring and logging

Security Model and User Separation

This setup uses a principle of least privilege approach with separate users for different purposes:

User Roles

Root User
- Purpose: Initial system setup only
- SSH Access: Disabled after setup
- Privileges: Full system access (used only during initial configuration)
Deployment User (DEPLOY_USER)
- Purpose: SSH access, deployment tasks, system administration
- SSH Access: Enabled with key-based authentication
- Privileges: Sudo access for deployment and administrative tasks
- Examples: deploy, ci, admin
Service Account (SERVICE_USER)
- Purpose: Running application services (Docker containers, databases)
- SSH Access: None (no login shell)
- Privileges: No sudo access, minimal system access
- Examples: appuser, service, app

Security Benefits

No root SSH access: Eliminates the most common attack vector
Principle of least privilege: Each user has only the access they need
Separation of concerns: Deployment tasks vs. service execution are separate
Audit trail: Clear distinction between deployment and service activities
Reduced attack surface: Service account has minimal privileges

File Permissions

Application files: Owned by SERVICE_USER for security
Docker operations: Run by DEPLOY_USER with sudo (deployment only)
Service execution: Run by SERVICE_USER (no sudo needed)

Prerequisites and Initial Setup

What's Already Done (Assumptions)

This guide assumes you have already:

Created two Ubuntu 24.04 LTS Linodes with root access
Set root passwords for both Linodes
Have SSH client installed on your local machine
Have Forgejo repository with Actions enabled
Optional: Domain name pointing to Production Linode's IP addresses

Step 0: Initial SSH Access and Verification

Before proceeding with the setup, you need to establish initial SSH access to both Linodes.

0.1 Get Your Linode IP Addresses

From your Linode dashboard, note the IP addresses for:

CI/CD Linode: YOUR_CI_CD_IP (IP address only, no domain needed)
Production Linode: YOUR_PRODUCTION_IP (IP address for SSH, domain for web access)

0.2 Test Initial SSH Access

Test SSH access to both Linodes:

# Test CI/CD Linode (IP address only)
ssh root@YOUR_CI_CD_IP

# Test Production Linode (IP address only)
ssh root@YOUR_PRODUCTION_IP

Expected output: SSH login prompt asking for root password.

If something goes wrong:

Verify the IP addresses are correct
Check that SSH is enabled on the Linodes
Ensure your local machine can reach the Linodes (no firewall blocking)

0.3 Choose Your Names

Before proceeding, decide on:

Service Account Name: Choose a username for the service account (e.g., appuser, deploy, service)
- Replace SERVICE_USER in this guide with your chosen name
- This account runs the actual application services
Deployment User Name: Choose a username for deployment tasks (e.g., deploy, ci, admin)
- Replace DEPLOY_USER in this guide with your chosen name
- This account has sudo privileges for deployment tasks
Application Name: Choose a name for your application (e.g., myapp, webapp, api)
- Replace APP_NAME in this guide with your chosen name
Domain Name (Optional): If you have a domain, note it for SSL configuration
- Replace your-domain.com in this guide with your actual domain

Example:

If you choose appuser as service account, deploy as deployment user, and myapp as application name:
- Replace all SERVICE_USER with appuser
- Replace all DEPLOY_USER with deploy
- Replace all APP_NAME with myapp
- If you have a domain example.com, replace your-domain.com with example.com

Security Model:

Service Account (SERVICE_USER): Runs application services, no sudo access
Deployment User (DEPLOY_USER): Handles deployments via SSH, has sudo access
Root: Only used for initial setup, then disabled for SSH access

0.4 Set Up SSH Key Authentication for Local Development

Important: This step should be done on both Linodes to enable secure SSH access from your local development machine.

0.4.1 Generate SSH Key on Your Local Machine

On your local development machine, generate an SSH key pair:

# Generate SSH key pair (if you don't already have one)
ssh-keygen -t ed25519 -C "your-email@example.com" -f ~/.ssh/id_ed25519 -N ""

# Or use existing key if you have one
ls ~/.ssh/id_ed25519.pub

0.4.2 Add Your Public Key to Both Linodes

Copy your public key to both Linodes:

# Copy your public key to CI/CD Linode
ssh-copy-id root@YOUR_CI_CD_IP

# Copy your public key to Production Linode
ssh-copy-id root@YOUR_PRODUCTION_IP

Alternative method (if ssh-copy-id doesn't work):

# Copy your public key content
cat ~/.ssh/id_ed25519.pub

# Then manually add to each server
ssh root@YOUR_CI_CD_IP
echo "YOUR_PUBLIC_KEY_CONTENT" >> ~/.ssh/authorized_keys

ssh root@YOUR_PRODUCTION_IP
echo "YOUR_PUBLIC_KEY_CONTENT" >> ~/.ssh/authorized_keys

0.4.3 Test SSH Key Authentication

Test that you can access both servers without passwords:

# Test CI/CD Linode
ssh root@YOUR_CI_CD_IP 'echo "SSH key authentication works for CI/CD"'

# Test Production Linode
ssh root@YOUR_PRODUCTION_IP 'echo "SSH key authentication works for Production"'

Expected output: The echo messages should appear without password prompts.

0.4.4 Create Deployment Users

On both Linodes, create the deployment user with sudo privileges:

# Create deployment user
sudo useradd -m -s /bin/bash DEPLOY_USER
sudo usermod -aG sudo DEPLOY_USER

# Set a secure password (you won't need it for SSH key auth, but it's good practice)
echo "DEPLOY_USER:$(openssl rand -base64 32)" | sudo chpasswd

# Copy your SSH key to the deployment user
sudo mkdir -p /home/DEPLOY_USER/.ssh
sudo cp ~/.ssh/authorized_keys /home/DEPLOY_USER/.ssh/
sudo chown -R DEPLOY_USER:DEPLOY_USER /home/DEPLOY_USER/.ssh
sudo chmod 700 /home/DEPLOY_USER/.ssh
sudo chmod 600 /home/DEPLOY_USER/.ssh/authorized_keys

0.4.5 Disable Root SSH Access

On both Linodes, disable root SSH access for security:

# Edit SSH configuration
sudo nano /etc/ssh/sshd_config

Find and modify these lines:

PasswordAuthentication no
PermitRootLogin no
PubkeyAuthentication yes

Note: We disable root SSH access entirely and use the deployment user for all SSH operations.

Restart SSH service:

sudo systemctl restart ssh

Important: Test SSH access with the deployment user before closing your current session to ensure you don't get locked out.

0.4.6 Test Deployment User Access

Test that you can access both servers as the deployment user:

# Test CI/CD Linode
ssh DEPLOY_USER@YOUR_CI_CD_IP 'echo "Deployment user SSH access works for CI/CD"'

# Test Production Linode
ssh DEPLOY_USER@YOUR_PRODUCTION_IP 'echo "Deployment user SSH access works for Production"'

Expected output: The echo messages should appear without password prompts.

0.4.7 Create SSH Config for Easy Access

On your local machine, create an SSH config file for easy access:

# Create SSH config
cat > ~/.ssh/config << 'EOF'
Host ci-cd-dev
    HostName YOUR_CI_CD_IP
    User DEPLOY_USER
    IdentityFile ~/.ssh/id_ed25519
    StrictHostKeyChecking no

Host production-dev
    HostName YOUR_PRODUCTION_IP
    User DEPLOY_USER
    IdentityFile ~/.ssh/id_ed25519
    StrictHostKeyChecking no
EOF

chmod 600 ~/.ssh/config

Now you can access servers easily:

ssh ci-cd-dev
ssh production-dev

Part 1: CI/CD Linode Setup

Step 1: Initial System Setup

1.1 Update the System

sudo apt update && sudo apt upgrade -y

What this does: Updates package lists and upgrades all installed packages to their latest versions.

Expected output: A list of packages being updated, followed by completion messages.

1.2 Configure Timezone

# Configure timezone interactively
sudo dpkg-reconfigure tzdata

# Verify timezone setting
date

What this does: Opens an interactive dialog to select your timezone. Navigate through the menus to choose your preferred timezone (e.g., UTC, America/New_York, Europe/London, Asia/Tokyo).

Expected output: After selecting your timezone, the date command should show the current date and time in your selected timezone.

1.3 Configure /etc/hosts

# Add localhost entries for both IPv4 and IPv6
echo "127.0.0.1 localhost" | sudo tee -a /etc/hosts
echo "::1 localhost ip6-localhost ip6-loopback" | sudo tee -a /etc/hosts
echo "YOUR_CI_CD_IPV4_ADDRESS localhost" | sudo tee -a /etc/hosts
echo "YOUR_CI_CD_IPV6_ADDRESS localhost" | sudo tee -a /etc/hosts

# Verify the configuration
cat /etc/hosts

What this does:

Adds localhost entries for both IPv4 and IPv6 addresses to /etc/hosts
Ensures proper localhost resolution for both IPv4 and IPv6

Important: Replace YOUR_CI_CD_IPV4_ADDRESS and YOUR_CI_CD_IPV6_ADDRESS with the actual IPv4 and IPv6 addresses of your CI/CD Linode obtained from your Linode dashboard.

Expected output: The /etc/hosts file should show entries for 127.0.0.1, ::1, and your Linode's actual IP addresses all mapping to localhost.

1.4 Install Essential Packages

sudo apt install -y \
    curl \
    wget \
    git \
    build-essential \
    pkg-config \
    libssl-dev \
    ca-certificates \
    apt-transport-https \
    software-properties-common \
    apache2-utils

What this does: Installs development tools, SSL libraries, and utilities needed for Docker and application building.

Step 2: Create Users

2.1 Create Service Account

sudo useradd -r -s /bin/bash -m -d /home/SERVICE_USER SERVICE_USER
sudo usermod -aG sudo SERVICE_USER
echo "SERVICE_USER:$(openssl rand -base64 32)" | sudo chpasswd

2.2 Verify Users

sudo su - SERVICE_USER
whoami
pwd
exit

sudo su - DEPLOY_USER
whoami
pwd
exit

Step 3: Install Docker

3.1 Add Docker Repository

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update

3.2 Install Docker Packages

sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

3.3 Configure Docker for Service Account

sudo usermod -aG docker SERVICE_USER

Step 4: Set Up Docker Registry

4.1 Create Registry Directory

sudo mkdir -p /opt/registry
sudo chown SERVICE_USER:SERVICE_USER /opt/registry

4.2 Create Registry Configuration

cat > /opt/registry/config.yml << 'EOF'
version: 0.1
log:
  level: info
storage:
  filesystem:
    rootdirectory: /var/lib/registry
  delete:
    enabled: true
  cache:
    blobdescriptor: inmemory
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
    X-Frame-Options: [DENY]
    X-XSS-Protection: [1; mode=block]
  # Enable public read access
  secret: "your-secret-key-here"
  # Restrict write access to specific IPs
  auth:
    htpasswd:
      realm: basic-realm
      path: /etc/docker/registry/auth.htpasswd
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
EOF

4.3 Create Authentication File

# Create htpasswd file for authentication
mkdir -p /opt/registry/auth
htpasswd -Bbn push-user "$(openssl rand -base64 32)" > /opt/registry/auth.htpasswd

# Create a read-only user (optional, for additional security)
htpasswd -Bbn read-user "$(openssl rand -base64 32)" >> /opt/registry/auth.htpasswd

4.4 Create Docker Compose for Registry

cat > /opt/registry/docker-compose.yml << 'EOF'
version: '3.8'

services:
  registry:
    image: registry:2
    ports:
      - "5000:5000"
    volumes:
      - ./config.yml:/etc/docker/registry/config.yml:ro
      - ./auth.htpasswd:/etc/docker/registry/auth.htpasswd:ro
      - registry_data:/var/lib/registry
    restart: unless-stopped
    networks:
      - registry_network

  registry-ui:
    image: joxit/docker-registry-ui:latest
    ports:
      - "8080:80"
    environment:
      - REGISTRY_TITLE=APP_NAME Registry
      - REGISTRY_URL=http://registry:5000
    depends_on:
      - registry
    restart: unless-stopped
    networks:
      - registry_network

volumes:
  registry_data:

networks:
  registry_network:
    driver: bridge
EOF

4.5 Install Required Tools

# Install htpasswd utility
sudo apt install -y apache2-utils

4.6 Start Registry

cd /opt/registry
docker-compose up -d

4.7 Test Registry Setup

# Check if containers are running
cd /opt/registry
docker-compose ps

# Test registry API
curl http://localhost:5000/v2/_catalog

# Test registry UI (optional)
curl -I http://localhost:8080

# Test Docker push/pull (optional but recommended)
# Create a test image
echo "FROM alpine:latest" > /tmp/test.Dockerfile
echo "RUN echo 'Hello from test image'" >> /tmp/test.Dockerfile

# Build and tag test image
docker build -f /tmp/test.Dockerfile -t localhost:5000/test:latest /tmp

# Push to registry
docker push localhost:5000/test:latest

# Verify image is in registry
curl http://localhost:5000/v2/_catalog
curl http://localhost:5000/v2/test/tags/list

# Pull image back (verifies pull works)
docker rmi localhost:5000/test:latest
docker pull localhost:5000/test:latest

# Clean up test image
docker rmi localhost:5000/test:latest
rm /tmp/test.Dockerfile

Expected Output:

docker-compose ps should show both registry and registry-ui as "Up"
curl http://localhost:5000/v2/_catalog should return {"repositories":[]} (empty initially)
curl -I http://localhost:8080 should return HTTP 200
Push/pull test should complete successfully

If something goes wrong:

Check container logs: docker-compose logs
Verify ports are open: netstat -tlnp | grep :5000
Check Docker daemon config: cat /etc/docker/daemon.json
Restart registry: docker-compose restart

Step 5: Configure Docker for Registry Access

5.1 Configure Docker for Registry Access

# Get the push user credentials
PUSH_USER="push-user"
PUSH_PASSWORD=$(grep push-user /opt/registry/auth.htpasswd | cut -d: -f2)

sudo tee /etc/docker/daemon.json << EOF
{
  "insecure-registries": ["YOUR_CI_CD_IP:5000"],
  "registry-mirrors": [],
  "auths": {
    "YOUR_CI_CD_IP:5000": {
      "auth": "$(echo -n "${PUSH_USER}:${PUSH_PASSWORD}" | base64)"
    }
  }
}
EOF

5.2 Restart Docker

sudo systemctl restart docker

Public Registry Access Model

Your registry is now configured with the following access model:

Public Read Access

Anyone can pull images without authentication:

# From any machine (public access)
docker pull YOUR_CI_CD_IP:5000/APP_NAME/backend:latest
docker pull YOUR_CI_CD_IP:5000/APP_NAME/frontend:latest

Authenticated Write Access

Only the CI/CD Linode can push images (using credentials):

# From CI/CD Linode only (authenticated)
docker push YOUR_CI_CD_IP:5000/APP_NAME/backend:latest
docker push YOUR_CI_CD_IP:5000/APP_NAME/frontend:latest

Registry UI Access

Public web interface for browsing images:

http://YOUR_CI_CD_IP:8080

Client Configuration

For other machines to pull images, they only need:

# Add to /etc/docker/daemon.json on client machines
{
  "insecure-registries": ["YOUR_CI_CD_IP:5000"]
}
# No authentication needed for pulls

Step 6: Set Up SSH for Production Communication

6.1 Generate SSH Key Pair

ssh-keygen -t ed25519 -C "ci-cd-server" -f ~/.ssh/id_ed25519 -N ""

6.2 Create SSH Config

cat > ~/.ssh/config << 'EOF'
Host production
    HostName YOUR_PRODUCTION_IP
    User DEPLOY_USER
    IdentityFile ~/.ssh/id_ed25519
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
EOF

chmod 600 ~/.ssh/config

Step 7: Install Forgejo Actions Runner

7.1 Download Runner

cd ~
wget https://code.forgejo.org/forgejo/runner/releases/download/v0.2.11/forgejo-runner-0.2.11-linux-amd64
chmod +x forgejo-runner-0.2.11-linux-amd64
sudo mv forgejo-runner-0.2.11-linux-amd64 /usr/local/bin/forgejo-runner

7.2 Create Systemd Service

sudo tee /etc/systemd/system/forgejo-runner.service > /dev/null << 'EOF'
[Unit]
Description=Forgejo Actions Runner
After=network.target

[Service]
Type=simple
User=SERVICE_USER
WorkingDirectory=/home/SERVICE_USER
ExecStart=/usr/local/bin/forgejo-runner daemon
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

7.3 Enable Service

sudo systemctl daemon-reload
sudo systemctl enable forgejo-runner.service

7.4 Test Runner Configuration

# Check if the runner is running
sudo systemctl status forgejo-runner.service

# Check runner logs
sudo journalctl -u forgejo-runner.service -f --no-pager

# Test runner connectivity (in a separate terminal)
forgejo-runner list

# Verify runner appears in Forgejo
# Go to your Forgejo repository → Settings → Actions → Runners
# You should see your runner listed as "ci-cd-runner" with status "Online"

Expected Output:

systemctl status should show "active (running)"
forgejo-runner list should show your runner
Forgejo web interface should show the runner as online

If something goes wrong:

Check logs: sudo journalctl -u forgejo-runner.service -f
Verify token: Make sure the registration token is correct
Check network: Ensure the runner can reach your Forgejo instance
Restart service: sudo systemctl restart forgejo-runner.service

Step 8: Set Up Monitoring and Cleanup

8.1 Create Monitoring Script

cat > ~/monitor.sh << 'EOF'
#!/bin/bash

echo "=== CI/CD Server Status ==="
echo "Date: $(date)"
echo "Uptime: $(uptime)"
echo ""

echo "=== Docker Status ==="
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
echo ""

echo "=== Registry Status ==="
cd /opt/registry
docker-compose ps
echo ""

echo "=== Actions Runner Status ==="
sudo systemctl status forgejo-runner.service --no-pager
echo ""

echo "=== System Resources ==="
echo "CPU Usage:"
top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1
echo "Memory Usage:"
free -h | grep Mem
echo "Disk Usage:"
df -h /
echo ""

echo "=== Recent Logs ==="
docker-compose logs --tail=10
EOF

chmod +x ~/monitor.sh

8.2 Create Cleanup Script

cat > ~/cleanup.sh << 'EOF'
#!/bin/bash

echo "Cleaning up old Docker images..."

# Remove unused images
docker image prune -f

# Remove unused volumes
docker volume prune -f

# Remove unused networks
docker network prune -f

# Remove old registry images (keep last 10 tags per repository)
cd /opt/registry
docker-compose exec registry registry garbage-collect /etc/docker/registry/config.yml

echo "Cleanup complete!"
EOF

chmod +x ~/cleanup.sh

8.3 Test Cleanup Script

# Create some test images to clean up
docker pull alpine:latest
docker pull nginx:latest
docker tag alpine:latest test-cleanup:latest
docker tag nginx:latest test-cleanup2:latest

# Run the cleanup script
./cleanup.sh

# Verify cleanup worked
echo "Checking remaining images:"
docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}"

echo "Checking remaining volumes:"
docker volume ls

echo "Checking remaining networks:"
docker network ls

Expected Output:

Cleanup script should run without errors
Test images should be removed
System should report cleanup completion
Remaining images should be minimal (only actively used ones)

If something goes wrong:

Check script permissions: ls -la ~/cleanup.sh
Verify Docker access: docker ps
Check registry access: cd /opt/registry && docker-compose ps
Run manually: bash -x ~/cleanup.sh

8.4 Set Up Automated Cleanup

(crontab -l 2>/dev/null; echo "0 3 * * * /home/SERVICE_USER/cleanup.sh") | crontab -

Step 9: Configure Firewall

sudo ufw --force enable
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 5000/tcp  # Docker registry (public read access)
sudo ufw allow 8080/tcp  # Registry UI (public read access)

Security Model:

Port 5000 (Registry): Public read access, authenticated write access
Port 8080 (UI): Public read access for browsing images
SSH: Restricted to your IP addresses
All other ports: Blocked

Step 10: Test CI/CD Setup

10.1 Test Docker Installation

docker --version
docker-compose --version

10.2 Check Registry Status

cd /opt/registry
docker-compose ps

10.3 Test Registry Access

curl http://localhost:5000/v2/_catalog

10.4 Get Public Key for Production Server

cat ~/.ssh/id_ed25519.pub

Important: Copy this public key - you'll need it for the production server setup.

Step 11: Configure Forgejo Actions Runner

11.1 Get Runner Token

Go to your Forgejo repository
Navigate to Settings → Actions → Runners
Click "New runner"
Copy the registration token

11.2 Configure Runner

# Get the registration token from your Forgejo repository
# Go to Settings → Actions → Runners → New runner
# Copy the registration token

# Configure the runner
forgejo-runner register \
  --instance https://your-forgejo-instance \
  --token YOUR_TOKEN \
  --name "ci-cd-runner" \
  --labels "ubuntu-latest,docker" \
  --no-interactive

11.3 Start Runner

sudo systemctl start forgejo-runner.service
sudo systemctl status forgejo-runner.service

11.4 Test Runner Configuration

# Check if the runner is running
sudo systemctl status forgejo-runner.service

# Check runner logs
sudo journalctl -u forgejo-runner.service -f --no-pager

# Test runner connectivity (in a separate terminal)
forgejo-runner list

# Verify runner appears in Forgejo
# Go to your Forgejo repository → Settings → Actions → Runners
# You should see your runner listed as "ci-cd-runner" with status "Online"

Expected Output:

systemctl status should show "active (running)"
forgejo-runner list should show your runner
Forgejo web interface should show the runner as online

If something goes wrong:

Check logs: sudo journalctl -u forgejo-runner.service -f
Verify token: Make sure the registration token is correct
Check network: Ensure the runner can reach your Forgejo instance
Restart service: sudo systemctl restart forgejo-runner.service

Part 2: Production Linode Setup

Step 12: Initial System Setup

12.1 Update the System

sudo apt update && sudo apt upgrade -y

12.2 Configure Timezone

# Configure timezone interactively
sudo dpkg-reconfigure tzdata

# Verify timezone setting
date

What this does: Opens an interactive dialog to select your timezone. Navigate through the menus to choose your preferred timezone (e.g., UTC, America/New_York, Europe/London, Asia/Tokyo).

Expected output: After selecting your timezone, the date command should show the current date and time in your selected timezone.

12.3 Configure /etc/hosts

# Add localhost entries for both IPv4 and IPv6
echo "127.0.0.1 localhost" | sudo tee -a /etc/hosts
echo "::1 localhost ip6-localhost ip6-loopback" | sudo tee -a /etc/hosts
echo "YOUR_PRODUCTION_IPV4_ADDRESS localhost" | sudo tee -a /etc/hosts
echo "YOUR_PRODUCTION_IPV6_ADDRESS localhost" | sudo tee -a /etc/hosts

# Verify the configuration
cat /etc/hosts

What this does:

Adds localhost entries for both IPv4 and IPv6 addresses to /etc/hosts
Ensures proper localhost resolution for both IPv4 and IPv6

Important: Replace YOUR_PRODUCTION_IPV4_ADDRESS and YOUR_PRODUCTION_IPV6_ADDRESS with the actual IPv4 and IPv6 addresses of your Production Linode obtained from your Linode dashboard.

Expected output: The /etc/hosts file should show entries for 127.0.0.1, ::1, and your Linode's actual IP addresses all mapping to localhost.

12.4 Install Essential Packages

sudo apt install -y \
    curl \
    wget \
    git \
    ca-certificates \
    apt-transport-https \
    software-properties-common \
    ufw \
    fail2ban \
    htop \
    nginx \
    certbot \
    python3-certbot-nginx

Step 13: Create Users

13.1 Create the SERVICE_USER User

sudo useradd -r -s /bin/bash -m -d /home/SERVICE_USER SERVICE_USER
sudo usermod -aG sudo SERVICE_USER
echo "SERVICE_USER:$(openssl rand -base64 32)" | sudo chpasswd

13.2 Verify Users

sudo su - SERVICE_USER
whoami
pwd
exit

sudo su - DEPLOY_USER
whoami
pwd
exit

Step 14: Install Docker

14.1 Add Docker Repository

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update

14.2 Install Docker Packages

sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

14.3 Configure Docker for Service Account

sudo usermod -aG docker SERVICE_USER

Step 15: Install Docker Compose

sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

Step 16: Configure Security

16.1 Configure Firewall

sudo ufw --force enable
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 3000/tcp
sudo ufw allow 3001/tcp

16.2 Configure Fail2ban

sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Step 17: Create Application Directory

17.1 Create Directory Structure

sudo mkdir -p /opt/APP_NAME
sudo chown SERVICE_USER:SERVICE_USER /opt/APP_NAME

Note: Replace APP_NAME with your actual application name. This directory name can be controlled via the APP_NAME secret in your Forgejo repository settings. If you set the APP_NAME secret to myapp, the deployment directory will be /opt/myapp.

17.2 Create SSL Directory (Optional - for domain users)

sudo mkdir -p /opt/APP_NAME/nginx/ssl
sudo chown SERVICE_USER:SERVICE_USER /opt/APP_NAME/nginx/ssl

Step 18: Clone Repository and Set Up Application Files

18.1 Switch to SERVICE_USER User

sudo su - SERVICE_USER

18.2 Clone Repository

cd /opt/APP_NAME
git clone https://your-forgejo-instance/your-username/APP_NAME.git .

Important: The repository includes a pre-configured nginx/nginx.conf file that handles both SSL and non-SSL scenarios, with proper security headers, rate limiting, and CORS configuration. This file will be automatically used by the Docker Compose setup.

Important: The repository also includes a pre-configured .forgejo/workflows/ci.yml file that handles the complete CI/CD pipeline including testing, building, and deployment. This workflow is already set up to work with the private registry and production deployment.

Note: Replace your-forgejo-instance and your-username/APP_NAME with your actual Forgejo instance URL and repository path.

18.3 Create Environment File

The repository doesn't include a .env.example file for security reasons. The CI/CD pipeline will create the .env file dynamically during deployment. However, for manual testing or initial setup, you can create a basic .env file:

cat > /opt/APP_NAME/.env << 'EOF'
# Production Environment Variables
POSTGRES_PASSWORD=your_secure_password_here
REGISTRY=YOUR_CI_CD_IP:5000
IMAGE_NAME=APP_NAME
IMAGE_TAG=latest

# Database Configuration
DATABASE_URL=postgresql://SERVICE_USER:your_secure_password_here@postgres:5432/APP_NAME

# Application Configuration
NODE_ENV=production
RUST_LOG=info
EOF

Important: Replace YOUR_CI_CD_IP with your actual CI/CD Linode IP address.

Note: The database user and database name can be controlled via the POSTGRES_USER and POSTGRES_DB secrets in your Forgejo repository settings. If you set these secrets, they will override the default values used in this environment file.

18.4 Verify Repository Contents

# Check that the docker-compose.yml file is present
ls -la docker-compose.yml

# Check that the nginx configuration is present
ls -la nginx/nginx.conf

# Check that the CI/CD workflow is present
ls -la .forgejo/workflows/ci.yml

# Check the repository structure
ls -la

# Verify the docker-compose.yml content
head -20 docker-compose.yml

# Verify the nginx configuration content
head -20 nginx/nginx.conf

# Verify the CI/CD workflow content
head -20 .forgejo/workflows/ci.yml

Expected output: You should see the docker-compose.yml file, nginx/nginx.conf file, .forgejo/workflows/ci.yml file, and other project files from your repository.

18.5 Create Deployment Script

cat > /opt/APP_NAME/deploy.sh << 'EOF'
#!/bin/bash

# Deployment script for APP_NAME
set -e

echo "Deploying APP_NAME..."

# Pull latest code from repository
git pull origin main

# Pull latest images
docker-compose pull

# Stop existing containers
docker-compose down

# Start new containers
docker-compose up -d

# Clean up old images
docker image prune -f

# Verify deployment
echo "Verifying deployment..."
sleep 10  # Give containers time to start
if docker-compose ps | grep -q "Up"; then
    echo "Deployment successful! All containers are running."
    docker-compose ps
else
    echo "Deployment failed! Some containers are not running."
    docker-compose ps
    docker-compose logs --tail=20
    exit 1
fi

echo "Deployment complete!"
EOF

chmod +x /opt/APP_NAME/deploy.sh

18.6 Create Backup Script

cat > /opt/APP_NAME/backup.sh << 'EOF'
#!/bin/bash

# Backup script for APP_NAME
set -e

BACKUP_DIR="/opt/APP_NAME/backups"
DATE=$(date +%Y%m%d_%H%M%S)

mkdir -p $BACKUP_DIR

# Backup database
docker-compose exec -T postgres pg_dump -U ${POSTGRES_USER:-sharenet} ${POSTGRES_DB:-sharenet} > $BACKUP_DIR/db_backup_$DATE.sql

# Backup configuration files
tar -czf $BACKUP_DIR/config_backup_$DATE.tar.gz .env docker-compose.yml nginx/

# Keep only last 7 days of backups
find $BACKUP_DIR -name "*.sql" -mtime +7 -delete
find $BACKUP_DIR -name "*.tar.gz" -mtime +7 -delete

echo "Backup completed: $BACKUP_DIR"
EOF

chmod +x /opt/APP_NAME/backup.sh

18.6.1 Set Up Automated Backup Scheduling

# Create a cron job to run backups daily at 2 AM
(crontab -l 2>/dev/null; echo "0 2 * * * /opt/APP_NAME/backup.sh >> /opt/APP_NAME/backup.log 2>&1") | crontab -

# Verify the cron job was added
crontab -l

What this does:

Runs automatically: The backup script runs every day at 2:00 AM
Frequency: Daily backups to ensure minimal data loss
Logging: All backup output is logged to /opt/APP_NAME/backup.log
Retention: The script automatically keeps only the last 7 days of backups

To test the backup manually:

cd /opt/APP_NAME
./backup.sh

To view backup logs:

tail -f /opt/APP_NAME/backup.log

18.7 Create Monitoring Script

cat > /opt/APP_NAME/monitor.sh << 'EOF'
#!/bin/bash

# Monitoring script for APP_NAME
echo "=== APP_NAME Application Status ==="
echo

echo "Container Status:"
docker-compose ps
echo

echo "Recent Logs:"
docker-compose logs --tail=20
echo

echo "System Resources:"
echo "CPU Usage:"
top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1
echo

echo "Memory Usage:"
free -h
echo

echo "Disk Usage:"
df -h
echo

echo "Network Connections:"
netstat -tuln | grep -E ':(80|443|3000|3001)'
EOF

chmod +x /opt/APP_NAME/monitor.sh

18.7.1 Set Up Automated Monitoring

# Create a cron job to run monitoring every 5 minutes
(crontab -l 2>/dev/null; echo "*/5 * * * * /opt/APP_NAME/monitor.sh >> /opt/APP_NAME/monitor.log 2>&1") | crontab -

# Verify the cron job was added
crontab -l

What this does:

Runs automatically: The monitoring script runs every 5 minutes
Frequency: Every 5 minutes to catch issues quickly
Logging: All monitoring output is logged to /opt/APP_NAME/monitor.log
What it monitors: Container status, recent logs, CPU/memory/disk usage, network connections

To test the monitoring manually:

cd /opt/APP_NAME
./monitor.sh

To view monitoring logs:

tail -f /opt/APP_NAME/monitor.log

Step 19: Set Up SSH for CI/CD Communication

19.1 Generate SSH Key Pair

ssh-keygen -t ed25519 -C "production-server" -f ~/.ssh/id_ed25519 -N ""

19.2 Create SSH Config

cat > ~/.ssh/config << 'EOF'
Host ci-cd
    HostName YOUR_CI_CD_IP
    User SERVICE_USER
    IdentityFile ~/.ssh/id_ed25519
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
EOF

chmod 600 ~/.ssh/config

Step 20: Exchange SSH Keys

20.1 Get Your Public Key

cat ~/.ssh/id_ed25519.pub

Important: Copy this public key - you'll need it for the CI/CD server setup.

20.2 Add CI/CD Server's Public Key

echo "CI_CD_PUBLIC_KEY_HERE" >> ~/.ssh/authorized_keys
sed -i 's/YOUR_CI_CD_IP/YOUR_ACTUAL_CI_CD_IP/g' ~/.ssh/config

Note: Replace CI_CD_PUBLIC_KEY_HERE with the actual public key from your CI/CD server.

Step 21: Update Application Configuration for CI/CD

21.1 Update Environment Variables

cd /opt/APP_NAME
nano .env

Required changes:

Replace YOUR_CI_CD_IP with your actual CI/CD Linode IP address
Replace your_secure_password_here with a strong database password
Update DATABASE_URL with the same password

Step 22: Configure SSL Certificates (Optional - Domain Users Only)

Skip this step if you don't have a domain name.

22.1 Install SSL Certificates

sudo certbot --nginx -d your-domain.com

22.2 Copy SSL Certificates

sudo cp /etc/letsencrypt/live/your-domain.com/fullchain.pem /opt/APP_NAME/nginx/ssl/
sudo cp /etc/letsencrypt/live/your-domain.com/privkey.pem /opt/APP_NAME/nginx/ssl/
sudo chown SERVICE_USER:SERVICE_USER /opt/APP_NAME/nginx/ssl/*

Step 23: Test CI/CD Integration

23.1 Test SSH Connection

ssh ci-cd 'echo Connection successful'

Expected output: Connection successful.

23.2 Test Registry Connection

curl http://YOUR_ACTUAL_CI_IP:5000/v2/_catalog

Expected output: {"repositories":[]} or a list of available images.

Part 3: Pipeline Configuration

Step 24: Configure Forgejo Repository Secrets

Go to your Forgejo repository → Settings → Secrets and Variables → Actions, and add the following secrets:

Required Secrets:

CI_HOST: Your CI/CD Linode IP address
- Purpose: Used by the workflow to connect to your private Docker registry
- Example: 192.168.1.100
PROD_HOST: Your Production Linode IP address
- Purpose: Used by the deployment job to SSH into your production server
- Example: 192.168.1.101
PROD_USER: SSH username for production server
- Purpose: Username for SSH connection to production server
- Value: Should be DEPLOY_USER (the deployment user you created)
- Example: deploy
PROD_SSH_KEY: SSH private key for deployment
- Purpose: Private key for SSH authentication to production server
- Source: Copy the private key from your CI/CD server
- How to get: On CI/CD server, run cat ~/.ssh/id_ed25519
- Format: Include the entire key including -----BEGIN OPENSSH PRIVATE KEY----- and -----END OPENSSH PRIVATE KEY-----

Optional Secrets (for enhanced security and flexibility):

APP_NAME: Application name (used for directory, database, and image names)
- Purpose: Controls the application directory name and database names
- Default: sharenet (if not set)
- Example: myapp, webapp, api
- Note: This affects the deployment directory /opt/APP_NAME and database names
POSTGRES_USER: PostgreSQL username for the application database
- Purpose: Username for the application's PostgreSQL database
- Default: sharenet (if not set)
- Example: appuser, webuser, apiuser
- Note: Should match the user created in the PostgreSQL setup
POSTGRES_DB: PostgreSQL database name for the application
- Purpose: Name of the application's PostgreSQL database
- Default: sharenet (if not set)
- Example: myapp, webapp, api
- Note: Should match the database created in the PostgreSQL setup
POSTGRES_PASSWORD: Database password for production
- Purpose: Secure database password for production environment
- Note: If not set, the workflow will use a default password
REGISTRY_USERNAME: Username for Docker registry (if using authentication)
- Purpose: Username for private registry access
- Note: Only needed if your registry requires authentication
REGISTRY_PASSWORD: Password for Docker registry (if using authentication)
- Purpose: Password for private registry access
- Note: Only needed if your registry requires authentication

How to Add Secrets:

Go to your Forgejo repository
Navigate to Settings → Secrets and Variables → Actions
Click New Secret
Enter the secret name and value
Click Add Secret

Security Notes:

Never commit secrets to your repository
Use strong, unique passwords for each environment
Rotate secrets regularly for enhanced security
Limit access to repository settings to trusted team members only

Step 25: Test the Complete Pipeline

25.1 Push Code Changes

Make a small change to your code and push to trigger the CI/CD pipeline:

# In your local repository
echo "# Test deployment" >> README.md
git add README.md
git commit -m "Test CI/CD pipeline"
git push

25.2 Monitor Pipeline

Go to your Forgejo repository
Navigate to Actions tab
Monitor the workflow execution
Check for any errors or issues

25.3 Verify Deployment

After successful pipeline execution:

# Check application status
cd /opt/APP_NAME
docker-compose ps

# Check application logs
docker-compose logs

# Test application access
curl -I https://your-domain.com  # or http://YOUR_PRODUCTION_IP

Testing and Verification

Step 26: Test Application Access

If you have a domain:

# Test HTTP redirect to HTTPS
curl -I http://your-domain.com

# Test HTTPS access
curl -I https://your-domain.com

# Test application endpoints
curl -I https://your-domain.com/api/health

If you don't have a domain (IP access only):

# Test HTTP access via IP
curl -I http://YOUR_PRODUCTION_IP

# Test application endpoints
curl -I http://YOUR_PRODUCTION_IP/api/health

Step 27: Test Monitoring

# On CI/CD server
cd /opt/registry
./monitor.sh

# On Production server
cd /opt/APP_NAME
./monitor.sh

Step 28: Test Registry Access

# Test registry API
curl http://YOUR_CI_CD_IP:5000/v2/_catalog

# Test registry UI (optional)
curl -I http://YOUR_CI_CD_IP:8080

Troubleshooting

Common Issues

Docker permission denied:

sudo usermod -aG docker SERVICE_USER
newgrp docker

SSL certificate issues (domain users only):

sudo certbot certificates
sudo certbot renew --dry-run

Application not starting:
```
cd /opt/APP_NAME
docker-compose logs
```

SSH connection failed:

ssh -v ci-cd 'echo test'
ssh -v production 'echo test'

Registry connection failed:

curl -v http://YOUR_CI_HOST_IP:5000/v2/_catalog

Actions runner not starting:

sudo systemctl status forgejo-runner.service
sudo journalctl -u forgejo-runner.service -f

Useful Commands

Check system resources: htop
Check disk space: df -h
Check memory usage: free -h
Check network: ip addr show
Check firewall: sudo ufw status
Check logs: sudo journalctl -f

Security Best Practices

Service Account: Use dedicated SERVICE_USER user with limited privileges
SSH Keys: Use Ed25519 keys with proper permissions (600/700)
Firewall: Configure UFW to allow only necessary ports
Fail2ban: Protect against brute force attacks
SSL/TLS: Use Let's Encrypt certificates with automatic renewal (domain users only)
Regular Backups: Automated daily backups of database and configuration
Container Isolation: Applications run in isolated Docker containers
Security Headers: Nginx configured with security headers
Registry Security: Use secure authentication and HTTPS for registry access

Monitoring and Maintenance

Daily Monitoring

# On CI/CD server
cd /opt/registry
./monitor.sh

# On Production server
cd /opt/APP_NAME
./monitor.sh

Weekly Maintenance

Check disk space: df -h
Review logs: docker-compose logs --tail=100
Update system: sudo apt update && sudo apt upgrade
Test backups: Verify backup files exist and are recent

Monthly Maintenance

Review security: Check firewall rules and fail2ban status
Update certificates: Ensure SSL certificates are valid (domain users only)
Clean up old images: Remove unused Docker images
Review monitoring: Check application performance and logs
Verify registry access: Test registry connectivity and authentication

Summary

Your complete CI/CD pipeline is now ready! The setup includes:

CI/CD Linode Features

Forgejo Actions runner for automated builds
Local Docker registry with web UI for image management
Secure SSH communication with production server
Monitoring and cleanup scripts
Firewall protection for security

Production Linode Features

Docker-based application deployment
Nginx reverse proxy with security headers
Automated backups and monitoring scripts
Firewall and fail2ban protection for security
Optional SSL/TLS certificate management (if domain is provided)

Pipeline Features

Automated testing on every code push
Automated image building and registry push
Automated deployment to production
Rollback capability with image versioning
Health monitoring and logging

Registry Integration

Private registry on CI/CD Linode stores all production images
Images available for manual deployment via PRODUCTION_LINODE_MANUAL_SETUP.md
Version control with git commit SHA tags
Web UI for image management at http://YOUR_CI_CD_IP:8080

Access Methods

Domain users: Access via https://your-domain.com
IP-only users: Access via http://YOUR_PRODUCTION_IP
Registry UI: Access via http://YOUR_CI_CD_IP:8080

For ongoing maintenance and troubleshooting, refer to the troubleshooting section and monitoring scripts provided in this guide.

46 KiB Raw Blame History

CI/CD Pipeline Setup Guide

Architecture Overview

Pipeline Flow

Prerequisites

Quick Start

What's Included

CI/CD Linode Features

Production Linode Features

Pipeline Features

Security Model and User Separation

User Roles

Security Benefits

File Permissions

Prerequisites and Initial Setup

What's Already Done (Assumptions)

Step 0: Initial SSH Access and Verification

0.1 Get Your Linode IP Addresses

0.2 Test Initial SSH Access

0.3 Choose Your Names

0.4 Set Up SSH Key Authentication for Local Development

0.4.1 Generate SSH Key on Your Local Machine

0.4.2 Add Your Public Key to Both Linodes

0.4.3 Test SSH Key Authentication

0.4.4 Create Deployment Users

0.4.5 Disable Root SSH Access

0.4.6 Test Deployment User Access

0.4.7 Create SSH Config for Easy Access

Part 1: CI/CD Linode Setup

Step 1: Initial System Setup

1.1 Update the System

1.2 Configure Timezone

1.3 Configure /etc/hosts

1.4 Install Essential Packages

Step 2: Create Users

2.1 Create Service Account

2.2 Verify Users

Step 3: Install Docker

3.1 Add Docker Repository

3.2 Install Docker Packages

3.3 Configure Docker for Service Account

Step 4: Set Up Docker Registry

4.1 Create Registry Directory

4.2 Create Registry Configuration

4.3 Create Authentication File

4.4 Create Docker Compose for Registry

4.5 Install Required Tools

4.6 Start Registry

4.7 Test Registry Setup

Step 5: Configure Docker for Registry Access

5.1 Configure Docker for Registry Access

5.2 Restart Docker

Public Registry Access Model

Public Read Access

Authenticated Write Access

Registry UI Access

Client Configuration

Step 6: Set Up SSH for Production Communication

6.1 Generate SSH Key Pair

6.2 Create SSH Config

Step 7: Install Forgejo Actions Runner

7.1 Download Runner

7.2 Create Systemd Service

7.3 Enable Service

7.4 Test Runner Configuration

Step 8: Set Up Monitoring and Cleanup

8.1 Create Monitoring Script

8.2 Create Cleanup Script

8.3 Test Cleanup Script

8.4 Set Up Automated Cleanup

Step 9: Configure Firewall

Step 10: Test CI/CD Setup

10.1 Test Docker Installation

10.2 Check Registry Status

10.3 Test Registry Access

10.4 Get Public Key for Production Server

Step 11: Configure Forgejo Actions Runner

11.1 Get Runner Token

11.2 Configure Runner

11.3 Start Runner

46 KiB

Raw Blame History