devteam/sharenet

Fork 0

continuist 69690f9391

CI/CD Pipeline / Test Backend (push) Waiting to run

Details

CI/CD Pipeline / Test Frontend (push) Waiting to run

Details

CI/CD Pipeline / Build and Push Docker Images (push) Blocked by required conditions

Details

CI/CD Pipeline / Deploy to Production (push) Blocked by required conditions

Details

Tell docker to allow insecure registry

2025-06-28 19:14:40 -04:00

43 KiB

Raw Blame History

CI/CD Pipeline Setup Guide

This guide covers setting up a complete Continuous Integration/Continuous Deployment (CI/CD) pipeline with a CI/CD Linode and Production Linode for automated builds, testing, and deployments.

Architecture Overview

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Forgejo Host  │    │   CI/CD Linode  │    │ Production Linode│
│   (Repository)  │    │ (Actions Runner)│    │ (Docker Deploy) │
│                 │    │ + Docker Registry│   │                 │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         │                       │                       │
         └─────────── Push ──────┼───────────────────────┘
                                 │
                                 └─── Deploy ────────────┘

Pipeline Flow

Code Push: Developer pushes code to Forgejo repository
Automated Testing: CI/CD Linode runs tests on backend and frontend
Image Building: If tests pass, Docker images are built
Registry Push: Images are pushed to private registry on CI/CD Linode
Production Deployment: Production Linode pulls images and deploys
Health Check: Application is verified and accessible

Prerequisites

Two Ubuntu 24.04 LTS Linodes with root access
Basic familiarity with Linux commands and SSH
Forgejo repository with Actions enabled
Optional: Domain name for Production Linode (for SSL/TLS)

Quick Start

Set up CI/CD Linode (Steps 1-13)
Set up Production Linode (Steps 14-26)
Configure SSH key exchange (Step 27)
Set up Forgejo repository secrets (Step 28)
Test the complete pipeline (Step 29)

What's Included

CI/CD Linode Features

Forgejo Actions runner for automated builds
Local Docker registry for image storage
Registry web UI for image management
Automated cleanup of old images
Secure SSH communication with production

Production Linode Features

Docker-based application deployment
Optional SSL/TLS certificate management (if domain is provided)
Nginx reverse proxy with security headers
Automated backups and monitoring
Firewall and fail2ban protection

Pipeline Features

Automated testing on every code push
Automated image building and registry push
Automated deployment to production
Rollback capability with image versioning
Health monitoring and logging

Security Model and User Separation

This setup uses a principle of least privilege approach with separate users for different purposes:

User Roles

Root User
- Purpose: Initial system setup only
- SSH Access: Disabled after setup
- Privileges: Full system access (used only during initial configuration)
Deployment User (DEPLOY_USER)
- Purpose: SSH access, deployment tasks, system administration
- SSH Access: Enabled with key-based authentication
- Privileges: Sudo access for deployment and administrative tasks
- Examples: deploy, ci, admin
Service Account (SERVICE_USER)
- Purpose: Running application services (Docker containers, databases)
- SSH Access: None (no login shell)
- Privileges: No sudo access, minimal system access
- Examples: appuser, service, app

Security Benefits

No root SSH access: Eliminates the most common attack vector
Principle of least privilege: Each user has only the access they need
Separation of concerns: Deployment tasks vs. service execution are separate
Audit trail: Clear distinction between deployment and service activities
Reduced attack surface: Service account has minimal privileges

File Permissions

Application files: Owned by SERVICE_USER for security
Docker operations: Run by DEPLOY_USER with sudo (deployment only)
Service execution: Run by SERVICE_USER (no sudo needed)

Prerequisites and Initial Setup

What's Already Done (Assumptions)

This guide assumes you have already:

Created two Ubuntu 24.04 LTS Linodes with root access
Set root passwords for both Linodes
Have SSH client installed on your local machine
Have Forgejo repository with Actions enabled
Optional: Domain name pointing to Production Linode's IP addresses

Step 0: Initial SSH Access and Verification

Before proceeding with the setup, you need to establish initial SSH access to both Linodes.

0.1 Get Your Linode IP Addresses

From your Linode dashboard, note the IP addresses for:

CI/CD Linode: YOUR_CI_CD_IP (IP address only, no domain needed)
Production Linode: YOUR_PRODUCTION_IP (IP address for SSH, domain for web access)

0.2 Test Initial SSH Access

Test SSH access to both Linodes:

# Test CI/CD Linode (IP address only)
ssh root@YOUR_CI_CD_IP

# Test Production Linode (IP address only)
ssh root@YOUR_PRODUCTION_IP

Expected output: SSH login prompt asking for root password.

If something goes wrong:

Verify the IP addresses are correct
Check that SSH is enabled on the Linodes
Ensure your local machine can reach the Linodes (no firewall blocking)

0.3 Choose Your Names

Before proceeding, decide on:

Service Account Name: Choose a username for the service account (e.g., appuser, deploy, service)
- Replace SERVICE_USER in this guide with your chosen name
- This account runs the actual application services
Deployment User Name: Choose a username for deployment tasks (e.g., deploy, ci, admin)
- Replace DEPLOY_USER in this guide with your chosen name
- This account has sudo privileges for deployment tasks
Application Name: Choose a name for your application (e.g., myapp, webapp, api)
- Replace APP_NAME in this guide with your chosen name
Domain Name (Optional): If you have a domain, note it for SSL configuration
- Replace your-domain.com in this guide with your actual domain

Example:

If you choose appuser as service account, deploy as deployment user, and myapp as application name:
- Replace all SERVICE_USER with appuser
- Replace all DEPLOY_USER with deploy
- Replace all APP_NAME with myapp
- If you have a domain example.com, replace your-domain.com with example.com

Security Model:

Service Account (SERVICE_USER): Runs application services, no sudo access
Deployment User (DEPLOY_USER): Handles deployments via SSH, has sudo access
Root: Only used for initial setup, then disabled for SSH access

0.4 Set Up SSH Key Authentication for Local Development

Important: This step should be done on both Linodes to enable secure SSH access from your local development machine.

0.4.1 Generate SSH Key on Your Local Machine

On your local development machine, generate an SSH key pair:

# Generate SSH key pair (if you don't already have one)
ssh-keygen -t ed25519 -C "your-email@example.com" -f ~/.ssh/id_ed25519 -N ""

# Or use existing key if you have one
ls ~/.ssh/id_ed25519.pub

0.4.2 Add Your Public Key to Both Linodes

Copy your public key to both Linodes:

# Copy your public key to CI/CD Linode
ssh-copy-id root@YOUR_CI_CD_IP

# Copy your public key to Production Linode
ssh-copy-id root@YOUR_PRODUCTION_IP

Alternative method (if ssh-copy-id doesn't work):

# Copy your public key content
cat ~/.ssh/id_ed25519.pub

# Then manually add to each server
ssh root@YOUR_CI_CD_IP
echo "YOUR_PUBLIC_KEY_CONTENT" >> ~/.ssh/authorized_keys

ssh root@YOUR_PRODUCTION_IP
echo "YOUR_PUBLIC_KEY_CONTENT" >> ~/.ssh/authorized_keys

0.4.3 Test SSH Key Authentication

Test that you can access both servers without passwords:

# Test CI/CD Linode
ssh root@YOUR_CI_CD_IP 'echo "SSH key authentication works for CI/CD"'

# Test Production Linode
ssh root@YOUR_PRODUCTION_IP 'echo "SSH key authentication works for Production"'

Expected output: The echo messages should appear without password prompts.

0.4.4 Create Deployment Users

On both Linodes, create the deployment user with sudo privileges:

# Create deployment user
sudo useradd -m -s /bin/bash DEPLOY_USER
sudo usermod -aG sudo DEPLOY_USER

# Set a secure password (for emergency access only)
echo "DEPLOY_USER:$(openssl rand -base64 32)" | sudo chpasswd

# Copy your SSH key to the deployment user
sudo mkdir -p /home/DEPLOY_USER/.ssh
sudo cp ~/.ssh/authorized_keys /home/DEPLOY_USER/.ssh/
sudo chown -R DEPLOY_USER:DEPLOY_USER /home/DEPLOY_USER/.ssh
sudo chmod 700 /home/DEPLOY_USER/.ssh
sudo chmod 600 /home/DEPLOY_USER/.ssh/authorized_keys

# Configure sudo to use SSH key authentication (most secure)
echo "DEPLOY_USER ALL=(ALL) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/DEPLOY_USER
sudo chmod 440 /etc/sudoers.d/DEPLOY_USER

Security Note: This configuration allows the DEPLOY_USER to use sudo without a password, which is more secure for CI/CD automation since there are no passwords to store or expose. The random password is set for emergency console access only.

0.4.5 Test Sudo Access

Test that the deployment user can use sudo without password prompts:

# Test sudo access
ssh DEPLOY_USER@YOUR_CI_CD_IP 'sudo whoami'
ssh DEPLOY_USER@YOUR_PRODUCTION_IP 'sudo whoami'

Expected output: Both commands should return root without prompting for a password.

0.4.6 Test Deployment User Access

Test that you can access both servers as the deployment user:

# Test CI/CD Linode
ssh DEPLOY_USER@YOUR_CI_CD_IP 'echo "Deployment user SSH access works for CI/CD"'

# Test Production Linode
ssh DEPLOY_USER@YOUR_PRODUCTION_IP 'echo "Deployment user SSH access works for Production"'

Expected output: The echo messages should appear without password prompts.

0.4.7 Create SSH Config for Easy Access

On your local machine, create an SSH config file for easy access:

# Create SSH config
cat > ~/.ssh/config << 'EOF'
Host ci-cd-dev
    HostName YOUR_CI_CD_IP
    User DEPLOY_USER
    IdentityFile ~/.ssh/id_ed25519
    StrictHostKeyChecking no

Host production-dev
    HostName YOUR_PRODUCTION_IP
    User DEPLOY_USER
    IdentityFile ~/.ssh/id_ed25519
    StrictHostKeyChecking no
EOF

chmod 600 ~/.ssh/config

Now you can access servers easily:

ssh ci-cd-dev
ssh production-dev

Part 1: CI/CD Linode Setup

Step 1: Initial System Setup

1.1 Update the System

sudo apt update && sudo apt upgrade -y

What this does: Updates package lists and upgrades all installed packages to their latest versions.

Expected output: A list of packages being updated, followed by completion messages.

1.2 Configure Timezone

# Configure timezone interactively
sudo dpkg-reconfigure tzdata

# Verify timezone setting
date

What this does: Opens an interactive dialog to select your timezone. Navigate through the menus to choose your preferred timezone (e.g., UTC, America/New_York, Europe/London, Asia/Tokyo).

Expected output: After selecting your timezone, the date command should show the current date and time in your selected timezone.

1.3 Configure /etc/hosts

# Add localhost entries for both IPv4 and IPv6
echo "127.0.0.1 localhost" | sudo tee -a /etc/hosts
echo "::1 localhost ip6-localhost ip6-loopback" | sudo tee -a /etc/hosts
echo "YOUR_CI_CD_IPV4_ADDRESS localhost" | sudo tee -a /etc/hosts
echo "YOUR_CI_CD_IPV6_ADDRESS localhost" | sudo tee -a /etc/hosts

# Verify the configuration
cat /etc/hosts

What this does:

Adds localhost entries for both IPv4 and IPv6 addresses to /etc/hosts
Ensures proper localhost resolution for both IPv4 and IPv6

Important: Replace YOUR_CI_CD_IPV4_ADDRESS and YOUR_CI_CD_IPV6_ADDRESS with the actual IPv4 and IPv6 addresses of your CI/CD Linode obtained from your Linode dashboard.

Expected output: The /etc/hosts file should show entries for 127.0.0.1, ::1, and your Linode's actual IP addresses all mapping to localhost.

1.4 Install Essential Packages

sudo apt install -y \
    curl \
    wget \
    git \
    build-essential \
    pkg-config \
    libssl-dev \
    ca-certificates \
    apt-transport-https \
    software-properties-common \
    apache2-utils

What this does: Installs development tools, SSL libraries, and utilities needed for Docker and application building.

Step 2: Create Users

2.1 Create Service Account

# Create dedicated group for the service account
sudo groupadd -r SERVICE_USER

# Create service account user with dedicated group
sudo useradd -r -g SERVICE_USER -s /bin/bash -m -d /home/SERVICE_USER SERVICE_USER
echo "SERVICE_USER:$(openssl rand -base64 32)" | sudo chpasswd

2.2 Verify Users

sudo su - SERVICE_USER
whoami
pwd
exit

sudo su - DEPLOY_USER
whoami
pwd
exit

Step 3: Install Docker

3.1 Add Docker Repository

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update

3.2 Install Docker Packages

sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

3.3 Configure Docker for Service Account

sudo usermod -aG docker SERVICE_USER

Step 4: Set Up Docker Registry

4.1 Create Registry Directory

sudo mkdir -p /opt/registry
sudo chown SERVICE_USER:SERVICE_USER /opt/registry

4.2 Create Registry Configuration

# Switch to SERVICE_USER (registry directory owner)
sudo su - SERVICE_USER

cat > /opt/registry/config.yml << 'EOF'
version: 0.1
log:
  level: info
storage:
  filesystem:
    rootdirectory: /var/lib/registry
  delete:
    enabled: true
  cache:
    blobdescriptor: inmemory
http:
  addr: :5000
  tls:
    certificate: /etc/docker/registry/ssl/registry.crt
    key: /etc/docker/registry/ssl/registry.key
  headers:
    X-Content-Type-Options: [nosniff]
    X-Frame-Options: [DENY]
    X-XSS-Protection: [1; mode=block]
    Access-Control-Allow-Origin: ["*"]
    Access-Control-Allow-Methods: ["HEAD", "GET", "OPTIONS", "DELETE"]
    Access-Control-Allow-Headers: ["Authorization", "Content-Type", "Accept", "Accept-Encoding", "Accept-Language", "Cache-Control", "Connection", "DNT", "Pragma", "User-Agent"]
  # Public read access, authentication required for push
  auth:
    htpasswd:
      realm: basic-realm
      path: /etc/docker/registry/auth/auth.htpasswd
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
EOF

# Exit SERVICE_USER shell
exit

What this configuration does:

HTTPS Enabled: Uses TLS certificates for secure communication
Public Read Access: Anyone can pull images without authentication
Authenticated Push: Only authenticated users can push images
Security Headers: Protects against common web vulnerabilities
CORS Headers: Allows the registry UI to access the registry API with all necessary headers
No Secret Key: The secret field was unnecessary and has been removed

Security Note: We switch to SERVICE_USER because the registry directory is owned by SERVICE_USER, maintaining proper file ownership and security.

4.2.1 Generate SSL Certificates

# Switch to SERVICE_USER (registry directory owner)
sudo su - SERVICE_USER

# Create SSL directory
mkdir -p /opt/registry/ssl

# Generate self-signed certificate
openssl req -x509 -newkey rsa:4096 -keyout /opt/registry/ssl/registry.key -out /opt/registry/ssl/registry.crt -days 365 -nodes -subj "/C=US/ST=State/L=City/O=Organization/CN=YOUR_CI_CD_IP"

# Set proper permissions
chmod 600 /opt/registry/ssl/registry.key
chmod 644 /opt/registry/ssl/registry.crt

# Exit SERVICE_USER shell
exit

Important: Replace YOUR_CI_CD_IP with your actual CI/CD Linode IP address in the certificate generation command.

4.3 Create Authentication File

# Switch to SERVICE_USER (registry directory owner)
sudo su - SERVICE_USER

# Create htpasswd file for authentication (required for push operations only)
mkdir -p /opt/registry/auth
htpasswd -Bbn push-user "$(openssl rand -base64 32)" > /opt/registry/auth/auth.htpasswd

# Exit SERVICE_USER shell
exit

What this does: Creates user credentials for registry authentication.

push-user: Can push images (used by CI/CD pipeline for deployments)

Note: Pull operations are public and don't require authentication, but push operations require these credentials.

4.4 Create Docker Compose for Registry

# Switch to SERVICE_USER (registry directory owner)
sudo su - SERVICE_USER

cat > /opt/registry/docker-compose.yml << 'EOF'
version: '3.8'

services:
  registry:
    image: registry:2
    ports:
      - "5000:5000"
    volumes:
      - ./config.yml:/etc/docker/registry/config.yml:ro
      - ./auth/auth.htpasswd:/etc/docker/registry/auth/auth.htpasswd:ro
      - ./ssl/registry.crt:/etc/docker/registry/ssl/registry.crt:ro
      - ./ssl/registry.key:/etc/docker/registry/ssl/registry.key:ro
      - registry_data:/var/lib/registry
    restart: unless-stopped
    networks:
      - registry_network
    healthcheck:
      test: ["CMD", "curl", "-k", "-f", "https://localhost:5000/v2/_catalog"]
      interval: 30s
      timeout: 10s
      retries: 3

  registry-ui:
    image: joxit/docker-registry-ui:latest
    expose:
      - "80"
    environment:
      - REGISTRY_TITLE=APP_NAME Registry
      - REGISTRY_URL=https://YOUR_CI_CD_IP:8080
    depends_on:
      registry:
        condition: service_healthy
    restart: unless-stopped
    networks:
      - registry_network

  nginx:
    image: nginx:alpine
    ports:
      - "8080:443"
    volumes:
      - ./ssl:/etc/nginx/ssl:ro
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - registry-ui
    restart: unless-stopped
    networks:
      - registry_network

volumes:
  registry_data:

networks:
  registry_network:
    driver: bridge
EOF

# Exit SERVICE_USER shell
exit

Important: Replace YOUR_CI_CD_IP with your actual CI/CD Linode IP address in the REGISTRY_URL environment variable.

Note: We updated the volume mounts to explicitly map individual certificate files to their expected locations in the registry container.

4.4.1 Create Nginx Configuration

# Switch to SERVICE_USER (registry directory owner)
sudo su - SERVICE_USER

cat > /opt/registry/nginx.conf << 'EOF'
events {
    worker_connections 1024;
}

http {
    upstream registry_ui {
        server registry-ui:80;
    }

    upstream registry_api {
        server registry:5000;
    }

    server {
        listen 443 ssl;
        server_name YOUR_CI_CD_IP;

        ssl_certificate /etc/nginx/ssl/registry.crt;
        ssl_certificate_key /etc/nginx/ssl/registry.key;
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers HIGH:!aNULL:!MD5;

        # Proxy registry API requests
        location /v2/ {
            proxy_pass http://registry_api;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_connect_timeout 30s;
            proxy_send_timeout 30s;
            proxy_read_timeout 30s;
        }

        # Proxy registry UI requests
        location / {
            proxy_pass http://registry_ui;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_connect_timeout 30s;
            proxy_send_timeout 30s;
            proxy_read_timeout 30s;
        }
    }
}
EOF

# Exit SERVICE_USER shell
exit

Important: Replace YOUR_CI_CD_IP with your actual CI/CD Linode IP address in the nginx configuration.

4.5 Install Required Tools

# Install htpasswd utility
sudo apt install -y apache2-utils

4.6 Start Registry

# Switch to SERVICE_USER (registry directory owner)
sudo su - SERVICE_USER

cd /opt/registry
docker compose up -d

# Exit SERVICE_USER shell
exit

4.6.1 Restart Registry with Updated Configuration

If you've already started the registry and then updated the REGISTRY_URL in the docker-compose.yml file, you need to restart the containers for the changes to take effect:

# Switch to SERVICE_USER (registry directory owner)
sudo su - SERVICE_USER

cd /opt/registry

# Stop and remove the existing containers
docker compose down

# Start the containers with the updated configuration
docker compose up -d

# Exit SERVICE_USER shell
exit

Note: This step is only needed if you've already started the registry and then updated the REGISTRY_URL. If you're starting fresh, Step 4.6 is sufficient.

4.6.2 Troubleshoot Connection Issues

If you get "Unable to Connect" when accessing https://YOUR_CI_CD_IP:8080, run these diagnostic commands:

# Switch to SERVICE_USER (registry directory owner)
sudo su - SERVICE_USER

cd /opt/registry

# Check if all containers are running
docker compose ps

# Check container logs for errors
docker compose logs nginx
docker compose logs registry-ui
docker compose logs registry

# Check if nginx is listening on port 8080
netstat -tlnp | grep :8080

# Test nginx directly
curl -k https://localhost:8080

# Exit SERVICE_USER shell
exit

Common Issues and Solutions:

Container not running: Run docker compose up -d to start containers
Port conflict: Check if port 8080 is already in use
SSL certificate issues: Verify the certificate files exist and have correct permissions
Firewall blocking: Ensure port 8080 is open in your firewall

4.6.3 Fix Container Restart Issues

If containers are restarting repeatedly, check the logs and fix the configuration:

# Switch to SERVICE_USER (registry directory owner)
sudo su - SERVICE_USER

cd /opt/registry

# Stop all containers
docker compose down

# Check if SSL certificates exist
ls -la ssl/

# If certificates don't exist, generate them
if [ ! -f ssl/registry.crt ]; then
    echo "Generating SSL certificates..."
    mkdir -p ssl
    openssl req -x509 -newkey rsa:4096 -keyout ssl/registry.key -out ssl/registry.crt -days 365 -nodes -subj "/C=US/ST=State/L=City/O=Organization/CN=YOUR_CI_CD_IP"
    chmod 600 ssl/registry.key
    chmod 644 ssl/registry.crt
fi

# Check if nginx.conf exists
ls -la nginx.conf

# If nginx.conf doesn't exist, create it
if [ ! -f nginx.conf ]; then
    echo "Creating nginx configuration..."
    cat > nginx.conf << 'EOF'
events {
    worker_connections 1024;
}

http {
    upstream registry_ui {
        server registry-ui:80;
    }

    upstream registry_api {
        server registry:5000;
    }

    server {
        listen 443 ssl;
        server_name YOUR_CI_CD_IP;

        ssl_certificate /etc/nginx/ssl/registry.crt;
        ssl_certificate_key /etc/nginx/ssl/registry.key;
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers HIGH:!aNULL:!MD5;

        # Proxy registry API requests
        location /v2/ {
            proxy_pass http://registry_api;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }

        # Proxy registry UI requests
        location / {
            proxy_pass http://registry_ui;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
}
EOF
fi

# Replace YOUR_CI_CD_IP with actual IP in nginx.conf
sed -i "s/YOUR_CI_CD_IP/YOUR_ACTUAL_CI_CD_IP/g" nginx.conf

# Start containers and check logs
docker compose up -d

# Wait a moment, then check logs
sleep 5
docker compose logs nginx
docker compose logs registry

# Exit SERVICE_USER shell
exit

Important: Replace YOUR_ACTUAL_CI_CD_IP with your actual CI/CD Linode IP address in the command above.

4.7 Test Registry Setup

# Switch to SERVICE_USER (registry directory owner)
sudo su - SERVICE_USER

# Check if containers are running
cd /opt/registry
docker compose ps

# Test registry API (HTTPS via nginx)
curl -k https://localhost:8080/v2/_catalog

# Test registry UI (HTTPS via nginx)
curl -I https://localhost:8080

# Test Docker push/pull (optional but recommended)
# Create a test image
echo "FROM alpine:latest" > /tmp/test.Dockerfile
echo "RUN echo 'Hello from test image'" >> /tmp/test.Dockerfile

# Build and tag test image
docker build -f /tmp/test.Dockerfile -t localhost:8080/test:latest /tmp

# Push to registry
docker push localhost:8080/test:latest

# Verify image is in registry
curl -k https://localhost:8080/v2/_catalog
curl -k https://localhost:8080/v2/test/tags/list

# Pull image back (verifies pull works)
docker rmi localhost:8080/test:latest
docker pull localhost:8080/test:latest

# Clean up test image completely
# Remove from local Docker
docker rmi localhost:8080/test:latest

# Clean up test file
rm /tmp/test.Dockerfile

# Clean up test repository using registry UI
# 1. Open your browser and go to: https://YOUR_CI_CD_IP:8080
# 2. You should see the 'test' repository listed
# 3. Click on the 'test' repository
# 4. Click the delete button (trash icon) next to the 'latest' tag
# 5. Confirm the deletion
# 6. The test repository should now be removed

# Verify registry is empty
curl -k https://localhost:8080/v2/_catalog

# Exit SERVICE_USER shell
exit

Important Notes:

Registry API: Uses HTTPS on port 5000 (secure)
Registry UI: Uses HTTPS on port 8080 (secure, via nginx reverse proxy)
Access URLs:
- Registry UI: https://YOUR_CI_CD_IP:8080 (use HTTPS)
- Registry API: https://YOUR_CI_CD_IP:5000
Browser Access: Both services now use HTTPS for secure communication

Expected Output:

docker-compose ps should show both registry and registry-ui as "Up"
curl -k https://localhost:5000/v2/_catalog should return {"repositories":[]} (empty initially)
curl -I https://localhost:8080 should return HTTP 200
Push/pull test should complete successfully

If something goes wrong:

Check container logs: docker compose logs
Verify ports are open: netstat -tlnp | grep :5000
Check Docker daemon config: cat /etc/docker/daemon.json
Restart registry: docker compose restart

Step 5: Configure Docker for Registry Access

5.1 Configure Docker for Registry Access

# Get the push user credentials
PUSH_USER="push-user"
PUSH_PASSWORD=$(grep push-user /opt/registry/auth/auth.htpasswd | cut -d: -f2)

# Copy the certificate to Docker's trusted certificates
sudo cp /opt/registry/ssl/registry.crt /usr/local/share/ca-certificates/registry.crt
sudo update-ca-certificates

sudo tee /etc/docker/daemon.json << EOF
{
  "insecure-registries": ["YOUR_CI_CD_IP:8080"],
  "registry-mirrors": [],
  "auths": {
    "YOUR_CI_CD_IP:8080": {
      "auth": "$(echo -n "${PUSH_USER}:${PUSH_PASSWORD}" | base64)"
    }
  }
}
EOF

Important: Replace YOUR_CI_CD_IP with your actual CI/CD Linode IP address.

5.2 Restart Docker

sudo systemctl restart docker

Public Registry Access Model

Your registry is now configured with the following access model:

Public Read Access

Anyone can pull images without authentication:

# From any machine (public access)
docker pull YOUR_CI_CD_IP:5000/APP_NAME/backend:latest
docker pull YOUR_CI_CD_IP:5000/APP_NAME/frontend:latest

Authenticated Write Access

Only the CI/CD Linode can push images (using credentials):

# From CI/CD Linode only (authenticated)
docker push YOUR_CI_CD_IP:5000/APP_NAME/backend:latest
docker push YOUR_CI_CD_IP:5000/APP_NAME/frontend:latest

Registry UI Access

Public web interface for browsing images:

https://YOUR_CI_CD_IP:8080

Client Configuration

For other machines to pull images, they only need:

# Add to /etc/docker/daemon.json on client machines
{
  "insecure-registries": ["YOUR_CI_CD_IP:5000"]
}
# No authentication needed for pulls

Step 6: Set Up SSH for Production Communication

6.1 Generate SSH Key Pair

ssh-keygen -t ed25519 -C "ci-cd-server" -f ~/.ssh/id_ed25519 -N ""

6.2 Create SSH Config

cat > ~/.ssh/config << 'EOF'
Host production
    HostName YOUR_PRODUCTION_IP
    User DEPLOY_USER
    IdentityFile ~/.ssh/id_ed25519
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
EOF

chmod 600 ~/.ssh/config

Step 7: Install Forgejo Actions Runner

7.1 Download Runner

cd ~
wget https://code.forgejo.org/forgejo/runner/releases/download/v0.2.11/forgejo-runner-0.2.11-linux-amd64
chmod +x forgejo-runner-0.2.11-linux-amd64
sudo mv forgejo-runner-0.2.11-linux-amd64 /usr/local/bin/forgejo-runner

7.2 Create Systemd Service

sudo tee /etc/systemd/system/forgejo-runner.service > /dev/null << 'EOF'
[Unit]
Description=Forgejo Actions Runner
After=network.target

[Service]
Type=simple
User=SERVICE_USER
WorkingDirectory=/home/SERVICE_USER
ExecStart=/usr/local/bin/forgejo-runner daemon
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

7.3 Enable Service

sudo systemctl daemon-reload
sudo systemctl enable forgejo-runner.service

7.4 Test Runner Configuration

# Check if the runner is running
sudo systemctl status forgejo-runner.service

# Check runner logs
sudo journalctl -u forgejo-runner.service -f --no-pager

# Test runner connectivity (in a separate terminal)
forgejo-runner list

# Verify runner appears in Forgejo
# Go to your Forgejo repository → Settings → Actions → Runners
# You should see your runner listed as "ci-cd-runner" with status "Online"

Expected Output:

systemctl status should show "active (running)"
forgejo-runner list should show your runner
Forgejo web interface should show the runner as online

If something goes wrong:

Check logs: sudo journalctl -u forgejo-runner.service -f
Verify token: Make sure the registration token is correct
Check network: Ensure the runner can reach your Forgejo instance
Restart service: sudo systemctl restart forgejo-runner.service

Step 8: Set Up Monitoring and Cleanup

8.1 Monitoring Script

Important: The repository includes a pre-configured monitoring script in the scripts/ directory that can be used for both CI/CD and production monitoring.

Repository Script:

scripts/monitor.sh - Comprehensive monitoring script with support for both CI/CD and production environments

To use the repository monitoring script:

# Clone the repository if not already done
git clone https://your-forgejo-instance/your-username/APP_NAME.git /tmp/monitoring-setup
cd /tmp/monitoring-setup

# Make the script executable
chmod +x scripts/monitor.sh

# Test CI/CD monitoring
./scripts/monitor.sh --type ci-cd

# Test production monitoring (if you have a production setup)
./scripts/monitor.sh --type production

# Clean up
cd /
rm -rf /tmp/monitoring-setup

Alternative: Create a local copy for convenience:

# Copy the script to your home directory for easy access
cp /tmp/monitoring-setup/scripts/monitor.sh ~/monitor.sh
chmod +x ~/monitor.sh

# Test the local copy
~/monitor.sh --type ci-cd

Note: The repository script is more comprehensive and includes proper error handling, colored output, and support for both CI/CD and production environments. It automatically detects the environment and provides appropriate monitoring information.

8.2 Cleanup Script

Important: The repository includes a pre-configured cleanup script in the scripts/ directory that can be used for both CI/CD and production cleanup operations.

Repository Script:

scripts/cleanup.sh - Comprehensive cleanup script with support for both CI/CD and production environments

To use the repository cleanup script:

# Clone the repository if not already done
git clone https://your-forgejo-instance/your-username/APP_NAME.git /tmp/cleanup-setup
cd /tmp/cleanup-setup

# Make the script executable
chmod +x scripts/cleanup.sh

# Test CI/CD cleanup (dry run first)
./scripts/cleanup.sh --type ci-cd --dry-run

# Run CI/CD cleanup
./scripts/cleanup.sh --type ci-cd

# Test production cleanup (dry run first)
./scripts/cleanup.sh --type production --dry-run

# Clean up
cd /
rm -rf /tmp/cleanup-setup

Alternative: Create a local copy for convenience:

# Copy the script to your home directory for easy access
cp /tmp/cleanup-setup/scripts/cleanup.sh ~/cleanup.sh
chmod +x ~/cleanup.sh

# Test the local copy (dry run)
~/cleanup.sh --type ci-cd --dry-run

Note: The repository script is more comprehensive and includes proper error handling, colored output, dry-run mode, and support for both CI/CD and production environments. It automatically detects the environment and provides appropriate cleanup operations.

8.3 Test Cleanup Script

# Create some test images to clean up
docker pull alpine:latest
docker pull nginx:latest
docker tag alpine:latest test-cleanup:latest
docker tag nginx:latest test-cleanup2:latest

# Test cleanup with dry run first
./scripts/cleanup.sh --type ci-cd --dry-run

# Run the cleanup script
./scripts/cleanup.sh --type ci-cd

# Verify cleanup worked
echo "Checking remaining images:"
docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}"

echo "Checking remaining volumes:"
docker volume ls

echo "Checking remaining networks:"
docker network ls

Expected Output:

Cleanup script should run without errors
Test images should be removed
System should report cleanup completion
Remaining images should be minimal (only actively used ones)

If something goes wrong:

Check script permissions: ls -la scripts/cleanup.sh
Verify Docker access: docker ps
Check registry access: cd /opt/registry && docker compose ps
Run manually: bash -x scripts/cleanup.sh

8.4 Set Up Automated Cleanup

# Create a cron job to run cleanup daily at 3 AM using the repository script
(crontab -l 2>/dev/null; echo "0 3 * * * cd /tmp/cleanup-setup && ./scripts/cleanup.sh --type ci-cd >> /tmp/cleanup.log 2>&1") | crontab -

# Verify the cron job was added
crontab -l

What this does:

Runs automatically: The cleanup script runs every day at 3:00 AM
Frequency: Daily cleanup to prevent disk space issues
Logging: All cleanup output is logged to /tmp/cleanup.log
What it cleans: Unused Docker images, volumes, networks, and registry images

Alternative: Use a local copy for automated cleanup:

# If you created a local copy, use that instead
(crontab -l 2>/dev/null; echo "0 3 * * * ~/cleanup.sh --type ci-cd >> ~/cleanup.log 2>&1") | crontab -

Step 9: Configure Firewall

sudo ufw --force enable
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 5000/tcp  # Docker registry (public read access)
sudo ufw allow 8080/tcp  # Registry UI (public read access)

Security Model:

Port 5000 (Registry): Public read access, authenticated write access
Port 8080 (UI): Public read access for browsing images
SSH: Restricted to your IP addresses
All other ports: Blocked

Step 10: Test CI/CD Setup

10.1 Test Docker Installation

docker --version
docker compose --version

10.2 Check Registry Status

cd /opt/registry
docker compose ps

10.3 Test Registry Access

curl http://localhost:5000/v2/_catalog

10.4 Get Public Key for Production Server

cat ~/.ssh/id_ed25519.pub

Important: Copy this public key - you'll need it for the production server setup.

Step 11: Configure Forgejo Actions Runner

11.1 Get Runner Token

Go to your Forgejo repository
Navigate to Settings → Actions → Runners
Click "New runner"
Copy the registration token

11.2 Configure Runner

# Get the registration token from your Forgejo repository
# Go to Settings → Actions → Runners → New runner
# Copy the registration token

# Configure the runner
forgejo-runner register \
  --instance https://your-forgejo-instance \
  --token YOUR_TOKEN \
  --name "ci-cd-runner" \
  --labels "ubuntu-latest,docker" \
  --no-interactive

11.3 Start Runner

sudo systemctl start forgejo-runner.service
sudo systemctl status forgejo-runner.service

11.4 Test Runner Configuration

# Check if the runner is running
sudo systemctl status forgejo-runner.service

# Check runner logs
sudo journalctl -u forgejo-runner.service -f --no-pager

# Test runner connectivity (in a separate terminal)
forgejo-runner list

# Verify runner appears in Forgejo
# Go to your Forgejo repository → Settings → Actions → Runners
# You should see your runner listed as "ci-cd-runner" with status "Online"

Expected Output:

systemctl status should show "active (running)"
forgejo-runner list should show your runner
Forgejo web interface should show the runner as online

If something goes wrong:

Check logs: sudo journalctl -u forgejo-runner.service -f
Verify token: Make sure the registration token is correct
Check network: Ensure the runner can reach your Forgejo instance
Restart service: sudo systemctl restart forgejo-runner.service

Part 2: Production Linode Setup

Step 12: Initial System Setup

12.1 Update the System

sudo apt update && sudo apt upgrade -y

12.2 Configure Timezone

# Configure timezone interactively
sudo dpkg-reconfigure tzdata

# Verify timezone setting
date

What this does: Opens an interactive dialog to select your timezone. Navigate through the menus to choose your preferred timezone (e.g., UTC, America/New_York, Europe/London, Asia/Tokyo).

Expected output: After selecting your timezone, the date command should show the current date and time in your selected timezone.

12.3 Configure /etc/hosts

# Add localhost entries for both IPv4 and IPv6
echo "127.0.0.1 localhost" | sudo tee -a /etc/hosts
echo "::1 localhost ip6-localhost ip6-loopback" | sudo tee -a /etc/hosts
echo "YOUR_PRODUCTION_IPV4_ADDRESS localhost" | sudo tee -a /etc/hosts
echo "YOUR_PRODUCTION_IPV6_ADDRESS localhost" | sudo tee -a /etc/hosts

# Verify the configuration
cat /etc/hosts

What this does:

Adds localhost entries for both IPv4 and IPv6 addresses to /etc/hosts
Ensures proper localhost resolution for both IPv4 and IPv6

Important: Replace YOUR_PRODUCTION_IPV4_ADDRESS and YOUR_PRODUCTION_IPV6_ADDRESS with the actual IPv4 and IPv6 addresses of your Production Linode obtained from your Linode dashboard.

Expected output: The /etc/hosts file should show entries for 127.0.0.1, ::1, and your Linode's actual IP addresses all mapping to localhost.

12.4 Install Essential Packages

sudo apt install -y \
    curl \
    wget \
    git \
    ca-certificates \
    apt-transport-https \
    software-properties-common \
    ufw \
    fail2ban \
    htop \
    nginx \
    certbot \
    python3-certbot-nginx

Step 13: Create Users

13.1 Create the SERVICE_USER User

# Create dedicated group for the service account
sudo groupadd -r SERVICE_USER

# Create service account user with dedicated group
sudo useradd -r -g SERVICE_USER -s /bin/bash -m -d /home/SERVICE_USER SERVICE_USER
echo "SERVICE_USER:$(openssl rand -base64 32)" | sudo chpasswd

13.2 Verify Users

sudo su - SERVICE_USER
whoami
pwd
exit

sudo su - DEPLOY_USER
whoami
pwd
exit

Step 14: Install Docker

14.1 Add Docker Repository

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update

14.2 Install Docker Packages

sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

14.3 Configure Docker for Service Account

sudo usermod -aG docker SERVICE_USER

Step 15: Install Docker Compose

sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

Step 16: Configure Security

16.1 Configure Firewall

sudo ufw --force enable
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 3000/tcp
sudo ufw allow 3001/tcp

16.2 Configure Fail2ban

sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Step 17: Create Application Directory

17.1 Create Directory Structure

sudo mkdir -p /opt/APP_NAME
sudo chown SERVICE_USER:SERVICE_USER /opt/APP_NAME

Note: Replace APP_NAME with your actual application name. This directory name can be controlled via the APP_NAME secret in your Forgejo repository settings. If you set the APP_NAME secret to myapp, the deployment directory will be /opt/myapp.

17.2 Create SSL Directory (Optional - for domain users)

sudo mkdir -p /opt/APP_NAME/nginx/ssl
sudo chown SERVICE_USER:SERVICE_USER /opt/APP_NAME/nginx/ssl

Step 18: Clone Repository and Set Up Application Files

18.1 Switch to SERVICE_USER User

sudo su - SERVICE_USER

18.2 Clone Repository

cd /opt/APP_NAME
git clone https://your-forgejo-instance/your-username/APP_NAME.git .

Important: The repository includes a pre-configured nginx/nginx.conf file that handles both SSL and non-SSL scenarios, with proper security headers, rate limiting, and CORS configuration. This file will be automatically used by the Docker Compose setup.

Important: The repository also includes a pre-configured .forgejo/workflows/ci.yml file that handles the complete CI/CD pipeline including testing, building, and deployment. This workflow is already set up to work with the private registry and production deployment.

Note: Replace your-forgejo-instance and your-username/APP_NAME with your actual Forgejo instance URL and repository path.

18.3 Create Environment File

The repository doesn't include a .env.example file for security reasons. The CI/CD pipeline will create the .env file dynamically during deployment. However, for manual testing or initial setup, you can create a basic .env file:

cat > /opt/APP_NAME/.env << 'EOF'
# Production Environment Variables
POSTGRES_PASSWORD=your_secure_password_here
REGISTRY=YOUR_CI_CD_IP:5000
IMAGE_NAME=APP_NAME
IMAGE_TAG=latest

# Database Configuration
POSTGRES_DB=sharenet
POSTGRES_USER=sharenet
DATABASE_URL=postgresql://sharenet:your_secure_password_here@postgres:5432/sharenet

# Application Configuration
NODE_ENV=production
RUST_LOG=info
RUST_BACKTRACE=1
EOF

Important: Replace YOUR_CI_CD_IP

43 KiB Raw Blame History

CI/CD Pipeline Setup Guide

Architecture Overview

Pipeline Flow

Prerequisites

Quick Start

What's Included

CI/CD Linode Features

Production Linode Features

Pipeline Features

Security Model and User Separation

User Roles

Security Benefits

File Permissions

Prerequisites and Initial Setup

What's Already Done (Assumptions)

Step 0: Initial SSH Access and Verification

0.1 Get Your Linode IP Addresses

0.2 Test Initial SSH Access

0.3 Choose Your Names

0.4 Set Up SSH Key Authentication for Local Development

0.4.1 Generate SSH Key on Your Local Machine

0.4.2 Add Your Public Key to Both Linodes

0.4.3 Test SSH Key Authentication

0.4.4 Create Deployment Users

0.4.5 Test Sudo Access

0.4.6 Test Deployment User Access

0.4.7 Create SSH Config for Easy Access

Part 1: CI/CD Linode Setup

Step 1: Initial System Setup

1.1 Update the System

1.2 Configure Timezone

1.3 Configure /etc/hosts

1.4 Install Essential Packages

Step 2: Create Users

2.1 Create Service Account

2.2 Verify Users

Step 3: Install Docker

3.1 Add Docker Repository

3.2 Install Docker Packages

3.3 Configure Docker for Service Account

Step 4: Set Up Docker Registry

4.1 Create Registry Directory

4.2 Create Registry Configuration

4.2.1 Generate SSL Certificates

4.3 Create Authentication File

4.4 Create Docker Compose for Registry

4.4.1 Create Nginx Configuration

4.5 Install Required Tools

4.6 Start Registry

4.6.1 Restart Registry with Updated Configuration

4.6.2 Troubleshoot Connection Issues

4.6.3 Fix Container Restart Issues

4.7 Test Registry Setup

Step 5: Configure Docker for Registry Access

5.1 Configure Docker for Registry Access

5.2 Restart Docker

Public Registry Access Model

Public Read Access

Authenticated Write Access

Registry UI Access

Client Configuration

Step 6: Set Up SSH for Production Communication

6.1 Generate SSH Key Pair

6.2 Create SSH Config

Step 7: Install Forgejo Actions Runner

7.1 Download Runner

7.2 Create Systemd Service

7.3 Enable Service

7.4 Test Runner Configuration

Step 8: Set Up Monitoring and Cleanup

8.1 Monitoring Script

8.2 Cleanup Script

8.3 Test Cleanup Script

8.4 Set Up Automated Cleanup

Step 9: Configure Firewall

Step 10: Test CI/CD Setup

10.1 Test Docker Installation

10.2 Check Registry Status

10.3 Test Registry Access

43 KiB

Raw Blame History