sharenet/CI_CD_SETUP_GUIDE.md

CI/CD Linode Setup Guide

This guide covers setting up your CI/CD Linode with Forgejo Actions runner and a local Docker registry for automated deployments.

Architecture Overview

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Forgejo Host  │    │   CI/CD Linode  │    │ Production Linode│
│   (Repository)  │    │ (Actions Runner)│    │ (Docker Deploy) │
│                 │    │ + Docker Registry│   │                 │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         │                       │                       │
         └─────────── Push ──────┼───────────────────────┘
                                 │
                                 └─── Deploy ────────────┘

Prerequisites

• Ubuntu 24.04 LTS Linode with root access
• Basic familiarity with Linux commands and SSH
• Forgejo repository with Actions enabled
• Production Linode IP address (for SSH key exchange)

Quick Start

1. Follow this complete CI/CD setup guide
2. Set up SSH keys for secure communication with Production server
3. Configure Forgejo Actions runner
4. Test registry and runner functionality
5. Exchange SSH keys with Production server
6. Configure Forgejo repository secrets

What's Included

CI/CD Linode Features

• Forgejo Actions runner for automated builds
• Local Docker registry for image storage
• Registry web UI for image management
• Automated cleanup of old images
• Secure SSH communication with production

Prerequisites and Initial Setup

What's Already Done (Assumptions)

This guide assumes you have already:

1. Created an Ubuntu 24.04 LTS Linode with root access
2. Set root password for the Linode
3. Have SSH client installed on your local machine
4. Have Production Linode IP address ready for SSH key exchange

Note: The CI/CD Linode will be accessed via IP address only, as it's primarily used for internal services (Docker registry, Forgejo Actions runner) that don't require public web access.

Step 0: Initial SSH Access and Verification

Before proceeding with the setup, you need to establish initial SSH access to your CI/CD Linode.

0.1 Get Your Linode IP Address

From your Linode dashboard, note the IP address for:

• CI/CD Linode: YOUR_CI_CD_IP (IP address only, no domain needed)

0.2 Test Initial SSH Access

Test SSH access to your CI/CD Linode:

# Test CI/CD Linode (IP address only)
ssh root@YOUR_CI_CD_IP

Expected output: SSH login prompt asking for root password.

If something goes wrong:

• Verify the IP address is correct
• Check that SSH is enabled on the Linode
• Ensure your local machine can reach the Linode (no firewall blocking)

0.3 Choose Your Names

Before proceeding, decide on:

1. Service Account Name: Choose a username for the service account (e.g., appuser, deploy, service)

   • Replace SERVICE_USER in this guide with your chosen name

2. Application Name: Choose a name for your application (e.g., myapp, webapp, api)

   • Replace APP_NAME in this guide with your chosen name

Example:

• If you choose appuser as service account and myapp as application name:
  • Replace all SERVICE_USER with appuser
  • Replace all APP_NAME with myapp

---

CI/CD Linode Setup

Step 1: Initial System Setup

1.1 Update the System

sudo apt update && sudo apt upgrade -y

What this does: Updates package lists and upgrades all installed packages to their latest versions.

Expected output: A list of packages being updated, followed by completion messages.

If something goes wrong:

• Check your internet connection
• Try running sudo apt update first, then sudo apt upgrade -y separately
• If you get package conflicts, run sudo apt --fix-broken install

1.2 Install Essential Packages

sudo apt install -y \
    curl \
    wget \
    git \
    build-essential \
    pkg-config \
    libssl-dev \
    ca-certificates \
    apt-transport-https \
    software-properties-common \
    apache2-utils

What this does: Installs development tools, SSL libraries, and utilities needed for Docker and application building.

Expected output: Package installation progress, ending with completion messages.

If something goes wrong:

• Check if any packages failed to install
• Run sudo apt install -f to fix broken dependencies
• Ensure you have sufficient disk space: df -h

Step 2: Create Service Account

2.1 Create the SERVICE_USER User

sudo useradd -r -s /bin/bash -m -d /home/SERVICE_USER SERVICE_USER
sudo usermod -aG sudo SERVICE_USER
echo "SERVICE_USER:$(openssl rand -base64 32)" | sudo chpasswd

What this does:

• Creates a dedicated service account named SERVICE_USER
• Gives it sudo privileges for administrative tasks
• Generates a random 32-character password

Expected output: No output (successful user creation is silent).

If something goes wrong:

• If user already exists: sudo userdel -r SERVICE_USER then retry
• Check user creation: id SERVICE_USER
• Verify sudo access: sudo -u SERVICE_USER sudo -l

2.2 Verify Service Account

sudo su - SERVICE_USER
whoami
pwd

What this does: Switches to the SERVICE_USER user and verifies the setup.

Expected output:

SERVICE_USER
/home/SERVICE_USER

Step 3: Install Docker

3.1 Add Docker Repository

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update

What this does: Adds Docker's official repository to your system for the latest Docker versions.

Expected output: GPG key import confirmation and package list update.

3.2 Install Docker Packages

sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

What this does: Installs Docker Engine, CLI, container runtime, and Docker Compose.

Expected output: Package installation progress, ending with completion messages.

3.3 Configure Docker for Service Account

sudo usermod -aG docker SERVICE_USER

What this does: Adds the SERVICE_USER user to the docker group so it can run Docker commands without sudo.

Step 4: Install Docker Compose

sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

What this does: Downloads the latest Docker Compose binary and makes it executable.

Step 5: Configure Docker for Local Registry

echo '{"insecure-registries": ["localhost:5000"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker

What this does: Configures Docker to allow connections to the local registry without SSL verification.

Step 6: Install Development Tools

6.1 Install Rust

sudo su - SERVICE_USER
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source $HOME/.cargo/env

What this does: Installs Rust programming language and Cargo package manager for building the backend.

6.2 Install Node.js

curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt install -y nodejs

What this does: Installs Node.js version 20 for building the frontend.

Step 7: Set Up Docker Registry

7.1 Create Registry Directory

sudo mkdir -p /opt/registry
sudo chown SERVICE_USER:SERVICE_USER /opt/registry

What this does: Creates a directory for the Docker registry and sets ownership to the SERVICE_USER user.

7.2 Create Docker Compose File for Registry

cat > /opt/registry/docker-compose.yml << 'EOF'
version: '3.8'

services:
  registry:
    image: registry:2
    container_name: docker-registry
    restart: unless-stopped
    ports:
      - "5000:5000"
    environment:
      REGISTRY_STORAGE_DELETE_ENABLED: "true"
      REGISTRY_STORAGE_FILESYSTEM_MAXTHREADS: "100"
    volumes:
      - registry_data:/var/lib/registry
      - ./config.yml:/etc/docker/registry/config.yml:ro
    networks:
      - registry-network

  registry-ui:
    image: joxit/docker-registry-ui:latest
    container_name: registry-ui
    restart: unless-stopped
    ports:
      - "8080:80"
    environment:
      REGISTRY_URL: http://registry:5000
      DELETE_IMAGES: "true"
    depends_on:
      - registry
    networks:
      - registry-network

volumes:
  registry_data:
    driver: local

networks:
  registry-network:
    driver: bridge
EOF

7.3 Create Registry Configuration

cat > /opt/registry/config.yml << 'EOF'
version: 0.1
log:
  fields:
    service: registry
storage:
  delete:
    enabled: true
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
EOF

7.4 Start the Registry

cd /opt/registry
docker-compose up -d

What this does: Starts the Docker registry and web UI in detached mode.

Expected output: Container creation and startup messages.

Step 8: Set Up SSH Keys

8.1 Create SSH Directory

mkdir -p ~/.ssh
chmod 700 ~/.ssh

8.2 Generate SSH Key Pair

ssh-keygen -t ed25519 -C "ci-cd-server" -f ~/.ssh/id_ed25519 -N ""

What this does: Generates an Ed25519 SSH key pair for secure communication with the production server.

Expected output: Key generation messages and file creation confirmation.

8.3 Create SSH Config

cat > ~/.ssh/config << 'EOF'
Host production
    HostName YOUR_PRODUCTION_IP
    User SERVICE_USER
    IdentityFile ~/.ssh/id_ed25519
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
EOF

chmod 600 ~/.ssh/config

What this does: Creates SSH configuration for easy connection to the production server.

Note: You'll update YOUR_PRODUCTION_IP with the actual production server IP later.

Step 9: Install Forgejo Actions Runner

9.1 Download Runner

cd ~
wget https://github.com/actions/runner/releases/download/v2.311.0/actions-runner-linux-x64-2.311.0.tar.gz
tar -xzf actions-runner-linux-x64-2.311.0.tar.gz
rm actions-runner-linux-x64-2.311.0.tar.gz

What this does: Downloads and extracts the Forgejo Actions runner.

9.2 Create Systemd Service

sudo tee /etc/systemd/system/github-runner.service > /dev/null << 'EOF'
[Unit]
Description=GitHub Actions Runner
After=network.target

[Service]
Type=simple
User=SERVICE_USER
WorkingDirectory=/home/SERVICE_USER/actions-runner
ExecStart=/home/SERVICE_USER/actions-runner/run.sh
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

What this does: Creates a systemd service to automatically start and manage the Actions runner.

9.3 Enable Service

sudo systemctl daemon-reload
sudo systemctl enable github-runner.service

What this does: Enables the Actions runner service to start automatically on boot.

Step 10: Set Up Monitoring and Cleanup

10.1 Create Monitoring Script

cat > ~/monitor.sh << 'EOF'
#!/bin/bash

echo "=== CI/CD Server Status ==="
echo "Date: $(date)"
echo "Uptime: $(uptime)"
echo ""

echo "=== Docker Status ==="
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
echo ""

echo "=== Registry Status ==="
cd /opt/registry
docker-compose ps
echo ""

echo "=== Actions Runner Status ==="
sudo systemctl status github-runner.service --no-pager
echo ""

echo "=== System Resources ==="
echo "CPU Usage:"
top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1
echo "Memory Usage:"
free -h | grep Mem
echo "Disk Usage:"
df -h /
echo ""

echo "=== Recent Logs ==="
docker-compose logs --tail=10
EOF

chmod +x ~/monitor.sh

What this does: Creates a monitoring script to check the status of all CI/CD services.

10.2 Create Cleanup Script

cat > ~/cleanup.sh << 'EOF'
#!/bin/bash

echo "Cleaning up old Docker images..."

# Remove unused images
docker image prune -f

# Remove unused volumes
docker volume prune -f

# Remove unused networks
docker network prune -f

# Remove old registry images (keep last 10 tags per repository)
cd /opt/registry
docker-compose exec registry registry garbage-collect /etc/docker/registry/config.yml

echo "Cleanup complete!"
EOF

chmod +x ~/cleanup.sh

What this does: Creates a cleanup script to remove old Docker images and registry data.

10.3 Set Up Automated Cleanup

(crontab -l 2>/dev/null; echo "0 3 * * * /home/SERVICE_USER/cleanup.sh") | crontab -

What this does: Schedules daily cleanup at 3 AM.

Step 11: Configure Firewall

sudo ufw --force enable
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 5000/tcp  # Docker registry
sudo ufw allow 8080/tcp  # Registry UI

What this does: Configures firewall to allow only necessary ports.

Step 12: Test CI/CD Setup

12.1 Test Docker Installation

docker --version
docker-compose --version

Expected output: Version information for both Docker and Docker Compose.

12.2 Check Registry Status

cd /opt/registry
docker-compose ps

Expected output: Status of registry and registry-ui containers showing "Up".

12.3 Test Registry Access

curl http://localhost:5000/v2/_catalog

Expected output: {"repositories":[]} (empty initially).

12.4 Get Public Key for Production Server

cat ~/.ssh/id_ed25519.pub

Important: Copy this public key - you'll need it for the production server setup.

Step 13: Configure Forgejo Actions Runner

13.1 Get Runner Token

1. Go to your Forgejo repository
2. Navigate to Settings → Actions → Runners
3. Click "New runner"
4. Copy the registration token

13.2 Configure Runner

cd ~/actions-runner
./config.sh --url https://your-forgejo-instance/your-username/APP_NAME --token YOUR_TOKEN

13.3 Start Runner

sudo systemctl start github-runner.service
sudo systemctl status github-runner.service

---

SSH Key Exchange

After setting up both servers, you need to exchange SSH public keys for secure communication:

Step 1: Get Public Keys from Both Servers

# On CI/CD server (get your public key)
cat ~/.ssh/id_ed25519.pub

# On Production server (get their public key)
# You'll need to get this from the Production server

Step 2: Add Production Server's Public Key

# On CI/CD server (add Production's public key)
echo "PRODUCTION_PUBLIC_KEY_HERE" >> ~/.ssh/authorized_keys
sed -i 's/YOUR_PRODUCTION_IP/YOUR_ACTUAL_PRODUCTION_IP/g' ~/.ssh/config

Step 3: Test SSH Connection

# Test from CI/CD to Production
ssh production 'echo Connection successful'

Expected output: Connection successful.

---

Registry Configuration

The CI/CD registry is configured to allow connections from the production server.

Step 1: Verify Registry Configuration

# Check registry is running
cd /opt/registry
docker-compose ps

# Test registry API
curl http://localhost:5000/v2/_catalog

Expected output:

• Registry containers showing "Up"
• Registry API: {"repositories":[]} or list of images

Step 2: Verify Registry UI Access

You can access the registry web interface to manage images:

# Test registry UI
curl -I http://localhost:8080

Expected output: HTTP response (may be 404 initially, which is normal).

Note: The registry UI is accessible at http://YOUR_CI_CD_IP:8080 for administrative purposes.

---

Forgejo Configuration

Step 1: Add Repository Secrets

Go to your Forgejo repository → Settings → Secrets and Variables → Actions, and add:

• CI_HOST: Your CI/CD Linode IP address
• PROD_HOST: Your production Linode IP
• PROD_USER: SSH username for production server (should be SERVICE_USER)
• PROD_SSH_KEY: SSH private key for deployment

Step 2: Generate SSH Key for Deployment

The setup automatically generates SSH keys for the SERVICE_USER service account. For Forgejo Actions deployment, use the CI/CD server's private key:

# On CI/CD server
cat ~/.ssh/id_ed25519

Copy the entire private key content (including the BEGIN and END lines) for the PROD_SSH_KEY secret.

---

Testing and Verification

Step 1: Test Registry Functionality

# Test registry API
curl http://localhost:5000/v2/_catalog

# Test registry UI (optional)
curl -I http://localhost:8080

Expected output:

• Registry API: {"repositories":[]} (empty initially)
• Registry UI: HTTP response (may be 404 initially, which is normal)

Step 2: Test Actions Runner

# Check runner status
sudo systemctl status github-runner.service

# Check runner logs
sudo journalctl -u github-runner.service -f

Expected output: Runner service showing as active and running.

Step 3: Test Monitoring Script

./monitor.sh

Expected output: Status information for all CI/CD services.

Step 4: Test SSH Connection to Production

# Test from CI/CD to Production
ssh production 'echo Connection successful'

Expected output: Connection successful.

Step 5: Trigger First Deployment

5.1 Push Code Changes

Make a small change to your code and push to trigger the CI/CD pipeline:

# In your local repository
echo "# Test deployment" >> README.md
git add README.md
git commit -m "Test CI/CD pipeline"
git push

5.2 Monitor Pipeline

1. Go to your Forgejo repository
2. Navigate to Actions tab
3. Monitor the workflow execution
4. Check for any errors or issues

5.3 Verify Deployment

After successful pipeline execution:

# Check if images were pushed to registry
curl http://localhost:5000/v2/_catalog

# Check registry UI for new images
# Open http://YOUR_CI_CD_IP:8080 in browser

---

Troubleshooting

Common Issues

1. Docker permission denied:

   sudo usermod -aG docker SERVICE_USER
   newgrp docker
   

2. Registry not accessible:

   cd /opt/registry
   docker-compose logs
   

3. Actions runner not starting:

   sudo systemctl status github-runner.service
   sudo journalctl -u github-runner.service -f
   

4. SSH key issues:

   chmod 600 ~/.ssh/id_ed25519
   chmod 700 ~/.ssh
   

5. Registry connection failed:

   curl -v http://localhost:5000/v2/_catalog
   

Useful Commands

• Check system resources: htop
• Check disk space: df -h
• Check memory usage: free -h
• Check network: ip addr show
• Check firewall: sudo ufw status
• Check logs: sudo journalctl -f

Security Best Practices

1. Service Account: Use dedicated SERVICE_USER user with limited privileges
2. SSH Keys: Use Ed25519 keys with proper permissions (600/700)
3. Firewall: Configure UFW to allow only necessary ports
4. Container Isolation: Registry runs in isolated Docker containers
5. Regular Cleanup: Automated daily cleanup of old images

---

Summary

Your CI/CD Linode is now ready to handle automated builds and deployments! The setup includes:

• Forgejo Actions runner for automated builds
• Local Docker registry with web UI for image management
• Secure SSH communication with production server
• Monitoring and cleanup scripts
• Firewall protection for security

For ongoing maintenance and troubleshooting, refer to the troubleshooting section and monitoring scripts provided in this guide.