sharenet/CI_CD_SETUP_GUIDE.md

# CI/CD Linode Setup Guide

This guide covers setting up your CI/CD Linode with Forgejo Actions runner and a local Docker registry for automated deployments.

## Architecture Overview

```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Forgejo Host  │    │   CI/CD Linode  │    │ Production Linode│
│   (Repository)  │    │ (Actions Runner)│    │ (Docker Deploy) │
│                 │    │ + Docker Registry│   │                 │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         │                       │                       │
         └─────────── Push ──────┼───────────────────────┘
                                 │
                                 └─── Deploy ────────────┘
```

## Prerequisites

- Ubuntu 24.04 LTS Linode with root access
- Basic familiarity with Linux commands and SSH
- Forgejo repository with Actions enabled
- Production Linode IP address (for SSH key exchange)

## Quick Start

1. **Follow this complete CI/CD setup guide**
2. **Set up SSH keys** for secure communication with Production server
3. **Configure Forgejo Actions runner**
4. **Test registry and runner functionality**
5. **Exchange SSH keys** with Production server
6. **Configure Forgejo repository secrets**

## What's Included

### CI/CD Linode Features
- Forgejo Actions runner for automated builds
- Local Docker registry for image storage
- Registry web UI for image management
- Automated cleanup of old images
- Secure SSH communication with production

## Prerequisites and Initial Setup

### What's Already Done (Assumptions)

This guide assumes you have already:

1. **Created an Ubuntu 24.04 LTS Linode** with root access
2. **Set root password** for the Linode
3. **Have SSH client** installed on your local machine
4. **Have Production Linode IP address** ready for SSH key exchange

**Note**: The CI/CD Linode will be accessed via IP address only, as it's primarily used for internal services (Docker registry, Forgejo Actions runner) that don't require public web access.

### Step 0: Initial SSH Access and Verification

Before proceeding with the setup, you need to establish initial SSH access to your CI/CD Linode.

#### 0.1 Get Your Linode IP Address

From your Linode dashboard, note the IP address for:
- **CI/CD Linode**: `YOUR_CI_CD_IP` (IP address only, no domain needed)

#### 0.2 Test Initial SSH Access

Test SSH access to your CI/CD Linode:

```bash
# Test CI/CD Linode (IP address only)
ssh root@YOUR_CI_CD_IP
```

**Expected output**: SSH login prompt asking for root password.

**If something goes wrong**:
- Verify the IP address is correct
- Check that SSH is enabled on the Linode
- Ensure your local machine can reach the Linode (no firewall blocking)

#### 0.3 Choose Your Names

Before proceeding, decide on:

1. **Service Account Name**: Choose a username for the service account (e.g., `appuser`, `deploy`, `service`)
   - Replace `SERVICE_USER` in this guide with your chosen name
   
2. **Application Name**: Choose a name for your application (e.g., `myapp`, `webapp`, `api`)
   - Replace `APP_NAME` in this guide with your chosen name

**Example**:
- If you choose `appuser` as service account and `myapp` as application name:
  - Replace all `SERVICE_USER` with `appuser`
  - Replace all `APP_NAME` with `myapp`

---

## CI/CD Linode Setup

### Step 1: Initial System Setup

#### 1.1 Update the System

```bash
sudo apt update && sudo apt upgrade -y
```

**What this does**: Updates package lists and upgrades all installed packages to their latest versions.

**Expected output**: A list of packages being updated, followed by completion messages.

**If something goes wrong**: 
- Check your internet connection
- Try running `sudo apt update` first, then `sudo apt upgrade -y` separately
- If you get package conflicts, run `sudo apt --fix-broken install`

#### 1.2 Install Essential Packages

```bash
sudo apt install -y \
    curl \
    wget \
    git \
    build-essential \
    pkg-config \
    libssl-dev \
    ca-certificates \
    apt-transport-https \
    software-properties-common \
    apache2-utils
```

**What this does**: Installs development tools, SSL libraries, and utilities needed for Docker and application building.

**Expected output**: Package installation progress, ending with completion messages.

**If something goes wrong**: 
- Check if any packages failed to install
- Run `sudo apt install -f` to fix broken dependencies
- Ensure you have sufficient disk space: `df -h`

### Step 2: Create Service Account

#### 2.1 Create the SERVICE_USER User

```bash
sudo useradd -r -s /bin/bash -m -d /home/SERVICE_USER SERVICE_USER
sudo usermod -aG sudo SERVICE_USER
echo "SERVICE_USER:$(openssl rand -base64 32)" | sudo chpasswd
```

**What this does**: 
- Creates a dedicated service account named `SERVICE_USER`
- Gives it sudo privileges for administrative tasks
- Generates a random 32-character password

**Expected output**: No output (successful user creation is silent).

**If something goes wrong**:
- If user already exists: `sudo userdel -r SERVICE_USER` then retry
- Check user creation: `id SERVICE_USER`
- Verify sudo access: `sudo -u SERVICE_USER sudo -l`

#### 2.2 Verify Service Account

```bash
sudo su - SERVICE_USER
whoami
pwd
```

**What this does**: Switches to the SERVICE_USER user and verifies the setup.

**Expected output**:
```
SERVICE_USER
/home/SERVICE_USER
```

### Step 3: Install Docker

#### 3.1 Add Docker Repository

```bash
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
```

**What this does**: Adds Docker's official repository to your system for the latest Docker versions.

**Expected output**: GPG key import confirmation and package list update.

#### 3.2 Install Docker Packages

```bash
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
```

**What this does**: Installs Docker Engine, CLI, container runtime, and Docker Compose.

**Expected output**: Package installation progress, ending with completion messages.

#### 3.3 Configure Docker for Service Account

```bash
sudo usermod -aG docker SERVICE_USER
```

**What this does**: Adds the SERVICE_USER user to the docker group so it can run Docker commands without sudo.

### Step 4: Install Docker Compose

```bash
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
```

**What this does**: Downloads the latest Docker Compose binary and makes it executable.

### Step 5: Configure Docker for Local Registry

```bash
echo '{"insecure-registries": ["localhost:5000"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
```

**What this does**: Configures Docker to allow connections to the local registry without SSL verification.

### Step 6: Install Development Tools

#### 6.1 Install Rust

```bash
sudo su - SERVICE_USER
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source $HOME/.cargo/env
```

**What this does**: Installs Rust programming language and Cargo package manager for building the backend.

#### 6.2 Install Node.js

```bash
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt install -y nodejs
```

**What this does**: Installs Node.js version 20 for building the frontend.

### Step 7: Set Up Docker Registry

#### 7.1 Create Registry Directory

```bash
sudo mkdir -p /opt/registry
sudo chown SERVICE_USER:SERVICE_USER /opt/registry
```

**What this does**: Creates a directory for the Docker registry and sets ownership to the SERVICE_USER user.

#### 7.2 Create Docker Compose File for Registry

```bash
cat > /opt/registry/docker-compose.yml << 'EOF'
version: '3.8'

services:
  registry:
    image: registry:2
    container_name: docker-registry
    restart: unless-stopped
    ports:
      - "5000:5000"
    environment:
      REGISTRY_STORAGE_DELETE_ENABLED: "true"
      REGISTRY_STORAGE_FILESYSTEM_MAXTHREADS: "100"
    volumes:
      - registry_data:/var/lib/registry
      - ./config.yml:/etc/docker/registry/config.yml:ro
    networks:
      - registry-network

  registry-ui:
    image: joxit/docker-registry-ui:latest
    container_name: registry-ui
    restart: unless-stopped
    ports:
      - "8080:80"
    environment:
      REGISTRY_URL: http://registry:5000
      DELETE_IMAGES: "true"
    depends_on:
      - registry
    networks:
      - registry-network

volumes:
  registry_data:
    driver: local

networks:
  registry-network:
    driver: bridge
EOF
```

#### 7.3 Create Registry Configuration

```bash
cat > /opt/registry/config.yml << 'EOF'
version: 0.1
log:
  fields:
    service: registry
storage:
  delete:
    enabled: true
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
EOF
```

#### 7.4 Start the Registry

```bash
cd /opt/registry
docker-compose up -d
```

**What this does**: Starts the Docker registry and web UI in detached mode.

**Expected output**: Container creation and startup messages.

### Step 8: Set Up SSH Keys

#### 8.1 Create SSH Directory

```bash
mkdir -p ~/.ssh
chmod 700 ~/.ssh
```

#### 8.2 Generate SSH Key Pair

```bash
ssh-keygen -t ed25519 -C "ci-cd-server" -f ~/.ssh/id_ed25519 -N ""
```

**What this does**: Generates an Ed25519 SSH key pair for secure communication with the production server.

**Expected output**: Key generation messages and file creation confirmation.

#### 8.3 Create SSH Config

```bash
cat > ~/.ssh/config << 'EOF'
Host production
    HostName YOUR_PRODUCTION_IP
    User SERVICE_USER
    IdentityFile ~/.ssh/id_ed25519
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
EOF

chmod 600 ~/.ssh/config
```

**What this does**: Creates SSH configuration for easy connection to the production server.

**Note**: You'll update `YOUR_PRODUCTION_IP` with the actual production server IP later.

### Step 9: Install Forgejo Actions Runner

#### 9.1 Download Runner

```bash
cd ~
wget https://github.com/actions/runner/releases/download/v2.311.0/actions-runner-linux-x64-2.311.0.tar.gz
tar -xzf actions-runner-linux-x64-2.311.0.tar.gz
rm actions-runner-linux-x64-2.311.0.tar.gz
```

**What this does**: Downloads and extracts the Forgejo Actions runner.

#### 9.2 Create Systemd Service

```bash
sudo tee /etc/systemd/system/github-runner.service > /dev/null << 'EOF'
[Unit]
Description=GitHub Actions Runner
After=network.target

[Service]
Type=simple
User=SERVICE_USER
WorkingDirectory=/home/SERVICE_USER/actions-runner
ExecStart=/home/SERVICE_USER/actions-runner/run.sh
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF
```

**What this does**: Creates a systemd service to automatically start and manage the Actions runner.

#### 9.3 Enable Service

```bash
sudo systemctl daemon-reload
sudo systemctl enable github-runner.service
```

**What this does**: Enables the Actions runner service to start automatically on boot.

### Step 10: Set Up Monitoring and Cleanup

#### 10.1 Create Monitoring Script

```bash
cat > ~/monitor.sh << 'EOF'
#!/bin/bash

echo "=== CI/CD Server Status ==="
echo "Date: $(date)"
echo "Uptime: $(uptime)"
echo ""

echo "=== Docker Status ==="
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
echo ""

echo "=== Registry Status ==="
cd /opt/registry
docker-compose ps
echo ""

echo "=== Actions Runner Status ==="
sudo systemctl status github-runner.service --no-pager
echo ""

echo "=== System Resources ==="
echo "CPU Usage:"
top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1
echo "Memory Usage:"
free -h | grep Mem
echo "Disk Usage:"
df -h /
echo ""

echo "=== Recent Logs ==="
docker-compose logs --tail=10
EOF

chmod +x ~/monitor.sh
```

**What this does**: Creates a monitoring script to check the status of all CI/CD services.

#### 10.2 Create Cleanup Script

```bash
cat > ~/cleanup.sh << 'EOF'
#!/bin/bash

echo "Cleaning up old Docker images..."

# Remove unused images
docker image prune -f

# Remove unused volumes
docker volume prune -f

# Remove unused networks
docker network prune -f

# Remove old registry images (keep last 10 tags per repository)
cd /opt/registry
docker-compose exec registry registry garbage-collect /etc/docker/registry/config.yml

echo "Cleanup complete!"
EOF

chmod +x ~/cleanup.sh
```

**What this does**: Creates a cleanup script to remove old Docker images and registry data.

#### 10.3 Set Up Automated Cleanup

```bash
(crontab -l 2>/dev/null; echo "0 3 * * * /home/SERVICE_USER/cleanup.sh") | crontab -
```

**What this does**: Schedules daily cleanup at 3 AM.

### Step 11: Configure Firewall

```bash
sudo ufw --force enable
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 5000/tcp  # Docker registry
sudo ufw allow 8080/tcp  # Registry UI
```

**What this does**: Configures firewall to allow only necessary ports.

### Step 12: Test CI/CD Setup

#### 12.1 Test Docker Installation

```bash
docker --version
docker-compose --version
```

**Expected output**: Version information for both Docker and Docker Compose.

#### 12.2 Check Registry Status

```bash
cd /opt/registry
docker-compose ps
```

**Expected output**: Status of registry and registry-ui containers showing "Up".

#### 12.3 Test Registry Access

```bash
curl http://localhost:5000/v2/_catalog
```

**Expected output**: `{"repositories":[]}` (empty initially).

#### 12.4 Get Public Key for Production Server

```bash
cat ~/.ssh/id_ed25519.pub
```

**Important**: Copy this public key - you'll need it for the production server setup.

### Step 13: Configure Forgejo Actions Runner

#### 13.1 Get Runner Token

1. Go to your Forgejo repository
2. Navigate to Settings → Actions → Runners
3. Click "New runner"
4. Copy the registration token

#### 13.2 Configure Runner

```bash
cd ~/actions-runner
./config.sh --url https://your-forgejo-instance/your-username/APP_NAME --token YOUR_TOKEN
```

#### 13.3 Start Runner

```bash
sudo systemctl start github-runner.service
sudo systemctl status github-runner.service
```

---

## SSH Key Exchange

After setting up both servers, you need to exchange SSH public keys for secure communication:

### Step 1: Get Public Keys from Both Servers

```bash
# On CI/CD server (get your public key)
cat ~/.ssh/id_ed25519.pub

# On Production server (get their public key)
# You'll need to get this from the Production server
```

### Step 2: Add Production Server's Public Key

```bash
# On CI/CD server (add Production's public key)
echo "PRODUCTION_PUBLIC_KEY_HERE" >> ~/.ssh/authorized_keys
sed -i 's/YOUR_PRODUCTION_IP/YOUR_ACTUAL_PRODUCTION_IP/g' ~/.ssh/config
```

### Step 3: Test SSH Connection

```bash
# Test from CI/CD to Production
ssh production 'echo Connection successful'
```

**Expected output**: `Connection successful`.

---

## Registry Configuration

The CI/CD registry is configured to allow connections from the production server.

### Step 1: Verify Registry Configuration

```bash
# Check registry is running
cd /opt/registry
docker-compose ps

# Test registry API
curl http://localhost:5000/v2/_catalog
```

**Expected output**: 
- Registry containers showing "Up"
- Registry API: `{"repositories":[]}` or list of images

### Step 2: Verify Registry UI Access

You can access the registry web interface to manage images:

```bash
# Test registry UI
curl -I http://localhost:8080
```

**Expected output**: HTTP response (may be 404 initially, which is normal).

**Note**: The registry UI is accessible at `http://YOUR_CI_CD_IP:8080` for administrative purposes.

---

## Forgejo Configuration

### Step 1: Add Repository Secrets

Go to your Forgejo repository → Settings → Secrets and Variables → Actions, and add:

- `CI_HOST`: Your CI/CD Linode IP address
- `PROD_HOST`: Your production Linode IP
- `PROD_USER`: SSH username for production server (should be `SERVICE_USER`)
- `PROD_SSH_KEY`: SSH private key for deployment

### Step 2: Generate SSH Key for Deployment

The setup automatically generates SSH keys for the SERVICE_USER service account. For Forgejo Actions deployment, use the CI/CD server's private key:

```bash
# On CI/CD server
cat ~/.ssh/id_ed25519
```

Copy the entire private key content (including the BEGIN and END lines) for the `PROD_SSH_KEY` secret.

---

## Testing and Verification

### Step 1: Test Registry Functionality

```bash
# Test registry API
curl http://localhost:5000/v2/_catalog

# Test registry UI (optional)
curl -I http://localhost:8080
```

**Expected output**: 
- Registry API: `{"repositories":[]}` (empty initially)
- Registry UI: HTTP response (may be 404 initially, which is normal)

### Step 2: Test Actions Runner

```bash
# Check runner status
sudo systemctl status github-runner.service

# Check runner logs
sudo journalctl -u github-runner.service -f
```

**Expected output**: Runner service showing as active and running.

### Step 3: Test Monitoring Script

```bash
./monitor.sh
```

**Expected output**: Status information for all CI/CD services.

### Step 4: Test SSH Connection to Production

```bash
# Test from CI/CD to Production
ssh production 'echo Connection successful'
```

**Expected output**: `Connection successful`.

### Step 5: Trigger First Deployment

#### 5.1 Push Code Changes

Make a small change to your code and push to trigger the CI/CD pipeline:

```bash
# In your local repository
echo "# Test deployment" >> README.md
git add README.md
git commit -m "Test CI/CD pipeline"
git push
```

#### 5.2 Monitor Pipeline

1. Go to your Forgejo repository
2. Navigate to Actions tab
3. Monitor the workflow execution
4. Check for any errors or issues

#### 5.3 Verify Deployment

After successful pipeline execution:

```bash
# Check if images were pushed to registry
curl http://localhost:5000/v2/_catalog

# Check registry UI for new images
# Open http://YOUR_CI_CD_IP:8080 in browser
```

---

## Troubleshooting

### Common Issues

1. **Docker permission denied**:
   ```bash
   sudo usermod -aG docker SERVICE_USER
   newgrp docker
   ```

2. **Registry not accessible**:
   ```bash
   cd /opt/registry
   docker-compose logs
   ```

3. **Actions runner not starting**:
   ```bash
   sudo systemctl status github-runner.service
   sudo journalctl -u github-runner.service -f
   ```

4. **SSH key issues**:
   ```bash
   chmod 600 ~/.ssh/id_ed25519
   chmod 700 ~/.ssh
   ```

5. **Registry connection failed**:
   ```bash
   curl -v http://localhost:5000/v2/_catalog
   ```

### Useful Commands

- **Check system resources**: `htop`
- **Check disk space**: `df -h`
- **Check memory usage**: `free -h`
- **Check network**: `ip addr show`
- **Check firewall**: `sudo ufw status`
- **Check logs**: `sudo journalctl -f`

### Security Best Practices

1. **Service Account**: Use dedicated `SERVICE_USER` user with limited privileges
2. **SSH Keys**: Use Ed25519 keys with proper permissions (600/700)
3. **Firewall**: Configure UFW to allow only necessary ports
4. **Container Isolation**: Registry runs in isolated Docker containers
5. **Regular Cleanup**: Automated daily cleanup of old images

---

## Summary

Your CI/CD Linode is now ready to handle automated builds and deployments! The setup includes:

- **Forgejo Actions runner** for automated builds
- **Local Docker registry** with web UI for image management
- **Secure SSH communication** with production server
- **Monitoring and cleanup** scripts
- **Firewall protection** for security

For ongoing maintenance and troubleshooting, refer to the troubleshooting section and monitoring scripts provided in this guide.