# CI/CD Pipeline Setup Guide

This guide covers setting up a complete Continuous Integration/Continuous Deployment (CI/CD) pipeline with a CI/CD Linode and Production Linode for automated builds, testing, and deployments.

## Architecture Overview

```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Forgejo Host  │    │   CI/CD Linode  │    │ Production Linode│
│   (Repository)  │    │ (Actions Runner)│    │ (Docker Deploy) │
│                 │    │ + Docker Registry│   │                 │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         │                       │                       │
         └─────────── Push ──────┼───────────────────────┘
                                 │
                                 └─── Deploy ────────────┘
```

## Pipeline Flow

1. **Code Push**: Developer pushes code to Forgejo repository
2. **Automated Testing**: CI/CD Linode runs tests on backend and frontend
3. **Image Building**: If tests pass, Docker images are built
4. **Registry Push**: Images are pushed to private registry on CI/CD Linode
5. **Production Deployment**: Production Linode pulls images and deploys
6. **Health Check**: Application is verified and accessible

## Prerequisites

- Two Ubuntu 24.04 LTS Linodes with root access
- Basic familiarity with Linux commands and SSH
- Forgejo repository with Actions enabled
- **Optional**: Domain name for Production Linode (for SSL/TLS)

## Quick Start

1. **Set up CI/CD Linode** (Steps 1-13)
2. **Set up Production Linode** (Steps 14-26)
3. **Configure SSH key exchange** (Step 27)
4. **Set up Forgejo repository secrets** (Step 28)
5. **Test the complete pipeline** (Step 29)

## What's Included

### CI/CD Linode Features
- Forgejo Actions runner for automated builds
- Local Docker registry for image storage
- Registry web UI for image management
- Automated cleanup of old images
- Secure SSH communication with production

### Production Linode Features
- Docker-based application deployment
- **Optional SSL/TLS certificate management** (if domain is provided)
- Nginx reverse proxy with security headers
- Automated backups and monitoring
- Firewall and fail2ban protection

### Pipeline Features
- **Automated testing** on every code push
- **Automated image building** and registry push
- **Automated deployment** to production
- **Rollback capability** with image versioning
- **Health monitoring** and logging

## Security Model and User Separation

This setup uses a **principle of least privilege** approach with separate users for different purposes:

### User Roles

1. **Root User**
   - **Purpose**: Initial system setup only
   - **SSH Access**: Disabled after setup
   - **Privileges**: Full system access (used only during initial configuration)

2. **Deployment User (`DEPLOY_USER`)**
   - **Purpose**: SSH access, deployment tasks, system administration
   - **SSH Access**: Enabled with key-based authentication
   - **Privileges**: Sudo access for deployment and administrative tasks
   - **Examples**: `deploy`, `ci`, `admin`

3. **Service Account (`SERVICE_USER`)**
   - **Purpose**: Running application services (Docker containers, databases)
   - **SSH Access**: None (no login shell)
   - **Privileges**: No sudo access, minimal system access
   - **Examples**: `appuser`, `service`, `app`

### Security Benefits

- **No root SSH access**: Eliminates the most common attack vector
- **Principle of least privilege**: Each user has only the access they need
- **Separation of concerns**: Deployment tasks vs. service execution are separate
- **Audit trail**: Clear distinction between deployment and service activities
- **Reduced attack surface**: Service account has minimal privileges

### File Permissions

- **Application files**: Owned by `SERVICE_USER` for security
- **Docker operations**: Run by `DEPLOY_USER` with sudo (deployment only)
- **Service execution**: Run by `SERVICE_USER` (no sudo needed)

---

## Prerequisites and Initial Setup

### What's Already Done (Assumptions)

This guide assumes you have already:

1. **Created two Ubuntu 24.04 LTS Linodes** with root access
2. **Set root passwords** for both Linodes
3. **Have SSH client** installed on your local machine
4. **Have Forgejo repository** with Actions enabled
5. **Optional**: Domain name pointing to Production Linode's IP addresses

### Step 0: Initial SSH Access and Verification

Before proceeding with the setup, you need to establish initial SSH access to both Linodes.

#### 0.1 Get Your Linode IP Addresses

From your Linode dashboard, note the IP addresses for:
- **CI/CD Linode**: `YOUR_CI_CD_IP` (IP address only, no domain needed)
- **Production Linode**: `YOUR_PRODUCTION_IP` (IP address for SSH, domain for web access)

#### 0.2 Test Initial SSH Access

Test SSH access to both Linodes:

```bash
# Test CI/CD Linode (IP address only)
ssh root@YOUR_CI_CD_IP

# Test Production Linode (IP address only)
ssh root@YOUR_PRODUCTION_IP
```

**Expected output**: SSH login prompt asking for root password.

**If something goes wrong**:
- Verify the IP addresses are correct
- Check that SSH is enabled on the Linodes
- Ensure your local machine can reach the Linodes (no firewall blocking)

#### 0.3 Choose Your Names

Before proceeding, decide on:

1. **Service Account Name**: Choose a username for the service account (e.g., `appuser`, `deploy`, `service`)
   - Replace `SERVICE_USER` in this guide with your chosen name
   - This account runs the actual application services
   
2. **Deployment User Name**: Choose a username for deployment tasks (e.g., `deploy`, `ci`, `admin`)
   - Replace `DEPLOY_USER` in this guide with your chosen name
   - This account has sudo privileges for deployment tasks
   
3. **Application Name**: Choose a name for your application (e.g., `myapp`, `webapp`, `api`)
   - Replace `APP_NAME` in this guide with your chosen name

4. **Domain Name** (Optional): If you have a domain, note it for SSL configuration
   - Replace `your-domain.com` in this guide with your actual domain

**Example**:
- If you choose `appuser` as service account, `deploy` as deployment user, and `myapp` as application name:
  - Replace all `SERVICE_USER` with `appuser`
  - Replace all `DEPLOY_USER` with `deploy`
  - Replace all `APP_NAME` with `myapp`
  - If you have a domain `example.com`, replace `your-domain.com` with `example.com`

**Security Model**:
- **Service Account (`SERVICE_USER`)**: Runs application services, no sudo access
- **Deployment User (`DEPLOY_USER`)**: Handles deployments via SSH, has sudo access
- **Root**: Only used for initial setup, then disabled for SSH access

#### 0.4 Set Up SSH Key Authentication for Local Development

**Important**: This step should be done on both Linodes to enable secure SSH access from your local development machine.

##### 0.4.1 Generate SSH Key on Your Local Machine

On your local development machine, generate an SSH key pair:

```bash
# Generate SSH key pair (if you don't already have one)
ssh-keygen -t ed25519 -C "your-email@example.com" -f ~/.ssh/id_ed25519 -N ""

# Or use existing key if you have one
ls ~/.ssh/id_ed25519.pub
```

##### 0.4.2 Add Your Public Key to Both Linodes

Copy your public key to both Linodes:

```bash
# Copy your public key to CI/CD Linode
ssh-copy-id root@YOUR_CI_CD_IP

# Copy your public key to Production Linode
ssh-copy-id root@YOUR_PRODUCTION_IP
```

**Alternative method** (if ssh-copy-id doesn't work):
```bash
# Copy your public key content
cat ~/.ssh/id_ed25519.pub

# Then manually add to each server
ssh root@YOUR_CI_CD_IP
echo "YOUR_PUBLIC_KEY_CONTENT" >> ~/.ssh/authorized_keys

ssh root@YOUR_PRODUCTION_IP
echo "YOUR_PUBLIC_KEY_CONTENT" >> ~/.ssh/authorized_keys
```

##### 0.4.3 Test SSH Key Authentication

Test that you can access both servers without passwords:

```bash
# Test CI/CD Linode
ssh root@YOUR_CI_CD_IP 'echo "SSH key authentication works for CI/CD"'

# Test Production Linode
ssh root@YOUR_PRODUCTION_IP 'echo "SSH key authentication works for Production"'
```

**Expected output**: The echo messages should appear without password prompts.

##### 0.4.4 Create Deployment Users

On both Linodes, create the deployment user with sudo privileges:

```bash
# Create deployment user
sudo useradd -m -s /bin/bash DEPLOY_USER
sudo usermod -aG sudo DEPLOY_USER

# Set a secure password (for emergency access only)
echo "DEPLOY_USER:$(openssl rand -base64 32)" | sudo chpasswd

# Copy your SSH key to the deployment user
sudo mkdir -p /home/DEPLOY_USER/.ssh
sudo cp ~/.ssh/authorized_keys /home/DEPLOY_USER/.ssh/
sudo chown -R DEPLOY_USER:DEPLOY_USER /home/DEPLOY_USER/.ssh
sudo chmod 700 /home/DEPLOY_USER/.ssh
sudo chmod 600 /home/DEPLOY_USER/.ssh/authorized_keys

# Configure sudo to use SSH key authentication (most secure)
echo "DEPLOY_USER ALL=(ALL) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/DEPLOY_USER
sudo chmod 440 /etc/sudoers.d/DEPLOY_USER
```

**Security Note**: This configuration allows the DEPLOY_USER to use sudo without a password, which is more secure for CI/CD automation since there are no passwords to store or expose. The random password is set for emergency console access only.

##### 0.4.5 Test Sudo Access

Test that the deployment user can use sudo without password prompts:

```bash
# Test sudo access
ssh DEPLOY_USER@YOUR_CI_CD_IP 'sudo whoami'
ssh DEPLOY_USER@YOUR_PRODUCTION_IP 'sudo whoami'
```

**Expected output**: Both commands should return `root` without prompting for a password.

##### 0.4.6 Test Deployment User Access

Test that you can access both servers as the deployment user:

```bash
# Test CI/CD Linode
ssh DEPLOY_USER@YOUR_CI_CD_IP 'echo "Deployment user SSH access works for CI/CD"'

# Test Production Linode
ssh DEPLOY_USER@YOUR_PRODUCTION_IP 'echo "Deployment user SSH access works for Production"'
```

**Expected output**: The echo messages should appear without password prompts.

##### 0.4.7 Create SSH Config for Easy Access

On your local machine, create an SSH config file for easy access:

```bash
# Create SSH config
cat > ~/.ssh/config << 'EOF'
Host ci-cd-dev
    HostName YOUR_CI_CD_IP
    User DEPLOY_USER
    IdentityFile ~/.ssh/id_ed25519
    StrictHostKeyChecking no

Host production-dev
    HostName YOUR_PRODUCTION_IP
    User DEPLOY_USER
    IdentityFile ~/.ssh/id_ed25519
    StrictHostKeyChecking no
EOF

chmod 600 ~/.ssh/config
```

Now you can access servers easily:
```bash
ssh ci-cd-dev
ssh production-dev
```

---

## Part 1: CI/CD Linode Setup

### Step 1: Initial System Setup

#### 1.1 Update the System

```bash
sudo apt update && sudo apt upgrade -y
```

**What this does**: Updates package lists and upgrades all installed packages to their latest versions.

**Expected output**: A list of packages being updated, followed by completion messages.

#### 1.2 Configure Timezone

```bash
# Configure timezone interactively
sudo dpkg-reconfigure tzdata

# Verify timezone setting
date
```

**What this does**: Opens an interactive dialog to select your timezone. Navigate through the menus to choose your preferred timezone (e.g., UTC, America/New_York, Europe/London, Asia/Tokyo).

**Expected output**: After selecting your timezone, the `date` command should show the current date and time in your selected timezone.

#### 1.3 Configure /etc/hosts

```bash
# Add localhost entries for both IPv4 and IPv6
echo "127.0.0.1 localhost" | sudo tee -a /etc/hosts
echo "::1 localhost ip6-localhost ip6-loopback" | sudo tee -a /etc/hosts
echo "YOUR_CI_CD_IPV4_ADDRESS localhost" | sudo tee -a /etc/hosts
echo "YOUR_CI_CD_IPV6_ADDRESS localhost" | sudo tee -a /etc/hosts

# Verify the configuration
cat /etc/hosts
```

**What this does**: 
- Adds localhost entries for both IPv4 and IPv6 addresses to `/etc/hosts`
- Ensures proper localhost resolution for both IPv4 and IPv6

**Important**: Replace `YOUR_CI_CD_IPV4_ADDRESS` and `YOUR_CI_CD_IPV6_ADDRESS` with the actual IPv4 and IPv6 addresses of your CI/CD Linode obtained from your Linode dashboard.

**Expected output**: The `/etc/hosts` file should show entries for `127.0.0.1`, `::1`, and your Linode's actual IP addresses all mapping to `localhost`.

#### 1.4 Install Essential Packages

```bash
sudo apt install -y \
    curl \
    wget \
    git \
    build-essential \
    pkg-config \
    libssl-dev \
    ca-certificates \
    apt-transport-https \
    software-properties-common \
    apache2-utils
```

**What this does**: Installs development tools, SSL libraries, and utilities needed for Docker and application building.

### Step 2: Create Users

#### 2.1 Create Service Account

```bash
# Create dedicated group for the service account
sudo groupadd -r SERVICE_USER

# Create service account user with dedicated group
sudo useradd -r -g SERVICE_USER -s /bin/bash -m -d /home/SERVICE_USER SERVICE_USER
echo "SERVICE_USER:$(openssl rand -base64 32)" | sudo chpasswd
```

#### 2.2 Verify Users

```bash
sudo su - SERVICE_USER
whoami
pwd
exit

sudo su - DEPLOY_USER
whoami
pwd
exit
```

### Step 3: Clone Repository for Registry Configuration

```bash
# Switch to DEPLOY_USER (who has sudo access)
sudo su - DEPLOY_USER

# Create application directory and clone repository
sudo mkdir -p /opt/APP_NAME
sudo chown SERVICE_USER:SERVICE_USER /opt/APP_NAME
cd /opt
sudo git clone https://your-forgejo-instance/your-username/APP_NAME.git
sudo chown -R SERVICE_USER:SERVICE_USER APP_NAME/

# Verify the registry folder exists
ls -la /opt/APP_NAME/registry/
```

**Important**: Replace `your-forgejo-instance`, `your-username`, and `APP_NAME` with your actual Forgejo instance URL, username, and application name.

**What this does**: 
- DEPLOY_USER creates the directory structure and clones the repository
- SERVICE_USER owns all the files for security
- Registry configuration files are now available at `/opt/APP_NAME/registry/`

### Step 4: Install Docker

#### 4.1 Add Docker Repository

```bash
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
```

#### 4.2 Install Docker Packages

```bash
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
```

#### 4.3 Configure Docker for Service Account

```bash
sudo usermod -aG docker SERVICE_USER
```

### Step 5: Set Up Docker Registry

#### 5.1 Create Registry Directory

```bash
sudo mkdir -p /opt/registry
sudo chown SERVICE_USER:SERVICE_USER /opt/registry
```

#### 5.2 Generate SSL Certificates

```bash
# Create system SSL directory for registry certificates
sudo mkdir -p /etc/ssl/registry

# Get your actual IP address
YOUR_ACTUAL_IP=$(curl -4 -s ifconfig.me)
echo "Your IP address is: $YOUR_ACTUAL_IP"

# Generate self-signed certificate with actual IP in system directory
sudo openssl req -x509 -newkey rsa:4096 -keyout /etc/ssl/registry/registry.key -out /etc/ssl/registry/registry.crt -days 365 -nodes -subj "/C=US/ST=State/L=City/O=Organization/CN=$YOUR_ACTUAL_IP"

# Set proper permissions
sudo chmod 600 /etc/ssl/registry/registry.key
sudo chmod 644 /etc/ssl/registry/registry.crt
```

**Important**: The certificate is now generated in the system SSL directory `/etc/ssl/registry/` with your actual CI/CD Linode IP address automatically.

**Note**: The permissions are set to:
- `registry.key`: `600` (owner read/write only) - private key must be secure
- `registry.crt`: `644` (owner read/write, group/others read) - certificate can be read by services

#### 5.3 Create Authentication File

```bash
# Create system auth directory for registry authentication
sudo mkdir -p /etc/registry/auth

# Create htpasswd file for authentication (required for push operations only)
sudo htpasswd -Bbn push-user "$(openssl rand -base64 32)" > /tmp/auth.htpasswd
sudo mv /tmp/auth.htpasswd /etc/registry/auth/auth.htpasswd
```

**What this does**: Creates user credentials for registry authentication in the system auth directory.
- `push-user`: Can push images (used by CI/CD pipeline for deployments)

**Note**: Pull operations are public and don't require authentication, but push operations require these credentials.

#### 5.4 Update Configuration with Actual IP Address

```bash
# Switch to SERVICE_USER (registry directory owner)
sudo su - SERVICE_USER

cd /opt/APP_NAME/registry

# Get your actual IP address
YOUR_ACTUAL_IP=$(curl -4 -s ifconfig.me)
echo "Your IP address is: $YOUR_ACTUAL_IP"

# Replace placeholder IP addresses in configuration files
sed -i "s/YOUR_CI_CD_IP/$YOUR_ACTUAL_IP/g" docker-compose.yml
sed -i "s/YOUR_CI_CD_IP/$YOUR_ACTUAL_IP/g" nginx.conf

# Exit SERVICE_USER shell
exit
```

**Important**: This step replaces all instances of `YOUR_CI_CD_IP` with your actual CI/CD Linode IP address in both the docker-compose.yml and nginx.conf files in the repository.

#### 5.5 Install Required Tools

```bash
# Install htpasswd utility
sudo apt install -y apache2-utils
```

#### 5.6 Start Registry

```bash
# Switch to SERVICE_USER (registry directory owner)
sudo su - SERVICE_USER

cd /opt/APP_NAME/registry
docker compose up -d

# Exit SERVICE_USER shell
exit
```

#### 5.7 Test Registry Setup

```bash
# Check if containers are running
cd /opt/APP_NAME/registry
docker compose ps

# Test registry API (HTTPS via nginx)
curl -k https://localhost:8080/v2/_catalog

# Test registry UI (HTTPS via nginx)
curl -I https://localhost:8080

# Test Docker push/pull (optional but recommended)
# Create a test image
echo "FROM alpine:latest" > /tmp/test.Dockerfile
echo "RUN echo 'Hello from test image'" >> /tmp/test.Dockerfile

# Build and tag test image
docker build -f /tmp/test.Dockerfile -t localhost:8080/test:latest /tmp

# Push to registry
docker push localhost:8080/test:latest

# Verify image is in registry
curl -k https://localhost:8080/v2/_catalog
curl -k https://localhost:8080/v2/test/tags/list

# Pull image back (verifies pull works)
docker rmi localhost:8080/test:latest
docker pull localhost:8080/test:latest

# Clean up test image completely
# Remove from local Docker
docker rmi localhost:8080/test:latest

# Clean up test file
rm /tmp/test.Dockerfile

# Get the manifest digest for the 'latest' tag
curl -k -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
  https://localhost:8080/v2/test/manifests/latest

# Copy the "config.digest" value from the output above (starts with "sha256:")
# Then delete the tag using that digest:
curl -k -X DELETE https://localhost:8080/v2/test/manifests/<digest>

# Run garbage collection to remove orphaned blobs
docker compose exec registry /bin/registry garbage-collect /etc/docker/registry/config.yml --delete-untagged

# Remove the repository directory structure
docker compose exec registry rm -rf /var/lib/registry/docker/registry/v2/repositories/test

# Verify registry is empty
echo "Verifying registry is now empty..."
curl -k https://localhost:8080/v2/_catalog

# Exit SERVICE_USER shell
exit
```

**Important Notes:**
- **Registry API**: Uses HTTPS on port 5000 (secure)
- **Registry UI**: Uses HTTPS on port 8080 (secure, via nginx reverse proxy)
- **Access URLs**: 
  - Registry UI: `https://YOUR_CI_CD_IP:8080` (use HTTPS)
  - Registry API: `https://YOUR_CI_CD_IP:5000`
- **Browser Access**: Both services now use HTTPS for secure communication

**Expected Output**:
- `docker-compose ps` should show both `registry` and `registry-ui` as "Up"
- `curl -k https://localhost:5000/v2/_catalog` should return `{"repositories":[]}` (empty initially)
- `curl -I https://localhost:8080` should return HTTP 200
- Push/pull test should complete successfully

**If something goes wrong**:
- Check container logs: `docker compose logs`
- Verify ports are open: `netstat -tlnp | grep :5000`
- Check Docker daemon config: `cat /etc/docker/daemon.json`
- Restart registry: `docker compose restart`

### Step 6: Configure Docker for Registry Access

#### 6.1 Configure Docker for Registry Access

```bash
# Get the push user credentials
PUSH_USER="push-user"
PUSH_PASSWORD=$(grep push-user /etc/registry/auth/auth.htpasswd | cut -d: -f2)

# Copy the certificate to Docker's trusted certificates
sudo cp /etc/ssl/registry/registry.crt /usr/local/share/ca-certificates/registry.crt
sudo update-ca-certificates

sudo tee /etc/docker/daemon.json << EOF
{
  "insecure-registries": ["YOUR_CI_CD_IP:8080"],
  "registry-mirrors": [],
  "auths": {
    "YOUR_CI_CD_IP:8080": {
      "auth": "$(echo -n "${PUSH_USER}:${PUSH_PASSWORD}" | base64)"
    }
  }
}
EOF
```

**Important**: Replace `YOUR_CI_CD_IP` with your actual CI/CD Linode IP address.

#### 6.2 Restart Docker

```bash
sudo systemctl restart docker
```

### Public Registry Access Model

Your registry is now configured with the following access model:

#### **Public Read Access**
Anyone can pull images without authentication:
```bash
# From any machine (public access)
docker pull YOUR_CI_CD_IP:5000/APP_NAME/backend:latest
docker pull YOUR_CI_CD_IP:5000/APP_NAME/frontend:latest
```

#### **Authenticated Write Access**
Only the CI/CD Linode can push images (using credentials):
```bash
# From CI/CD Linode only (authenticated)
docker push YOUR_CI_CD_IP:5000/APP_NAME/backend:latest
docker push YOUR_CI_CD_IP:5000/APP_NAME/frontend:latest
```

#### **Registry UI Access**
Public web interface for browsing images:
```
https://YOUR_CI_CD_IP:8080
```

#### **Client Configuration**
For other machines to pull images, they only need:
```bash
# Add to /etc/docker/daemon.json on client machines
{
  "insecure-registries": ["YOUR_CI_CD_IP:5000"]
}
# No authentication needed for pulls
```

### Step 7: Set Up SSH for Production Communication

#### 7.1 Generate SSH Key Pair

```bash
ssh-keygen -t ed25519 -C "ci-cd-server" -f ~/.ssh/id_ed25519 -N ""
```

#### 7.2 Create SSH Config

```bash
cat > ~/.ssh/config << 'EOF'
Host production
    HostName YOUR_PRODUCTION_IP
    User DEPLOY_USER
    IdentityFile ~/.ssh/id_ed25519
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
EOF

chmod 600 ~/.ssh/config
```

### Step 8: Install Forgejo Actions Runner

#### 8.1 Download Runner

```bash
cd ~
wget https://code.forgejo.org/forgejo/runner/releases/download/v0.2.11/forgejo-runner-0.2.11-linux-amd64
chmod +x forgejo-runner-0.2.11-linux-amd64
sudo mv forgejo-runner-0.2.11-linux-amd64 /usr/local/bin/forgejo-runner
```

#### 8.2 Create Systemd Service

```bash
sudo tee /etc/systemd/system/forgejo-runner.service > /dev/null << 'EOF'
[Unit]
Description=Forgejo Actions Runner
After=network.target

[Service]
Type=simple
User=SERVICE_USER
WorkingDirectory=/home/SERVICE_USER
ExecStart=/usr/local/bin/forgejo-runner daemon
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF
```

#### 8.3 Enable Service

```bash
sudo systemctl daemon-reload
sudo systemctl enable forgejo-runner.service
```

#### 8.4 Test Runner Configuration

```bash
# Check if the runner is running
sudo systemctl status forgejo-runner.service

# Check runner logs
sudo journalctl -u forgejo-runner.service -f --no-pager

# Test runner connectivity (in a separate terminal)
forgejo-runner list

# Verify runner appears in Forgejo
# Go to your Forgejo repository → Settings → Actions → Runners
# You should see your runner listed as "ci-cd-runner" with status "Online"
```

**Expected Output**:
- `systemctl status` should show "active (running)"
- `forgejo-runner list` should show your runner
- Forgejo web interface should show the runner as online

**If something goes wrong**:
- Check logs: `sudo journalctl -u forgejo-runner.service -f`
- Verify token: Make sure the registration token is correct
- Check network: Ensure the runner can reach your Forgejo instance
- Restart service: `sudo systemctl restart forgejo-runner.service`

### Step 9: Set Up Monitoring and Cleanup

#### 9.1 Monitoring Script

**Important**: The repository includes a pre-configured monitoring script in the `scripts/` directory that can be used for both CI/CD and production monitoring.

**Repository Script**:
- `scripts/monitor.sh` - Comprehensive monitoring script with support for both CI/CD and production environments

**To use the repository monitoring script**:
```bash
# The repository is already cloned at /opt/APP_NAME/
cd /opt/APP_NAME

# Make the script executable
chmod +x scripts/monitor.sh

# Test CI/CD monitoring
./scripts/monitor.sh --type ci-cd

# Test production monitoring (if you have a production setup)
./scripts/monitor.sh --type production
```

**Note**: The repository script is more comprehensive and includes proper error handling, colored output, and support for both CI/CD and production environments. It automatically detects the environment and provides appropriate monitoring information.

#### 9.2 Cleanup Script

**Important**: The repository includes a pre-configured cleanup script in the `scripts/` directory that can be used for both CI/CD and production cleanup operations.

**Repository Script**:
- `scripts/cleanup.sh` - Comprehensive cleanup script with support for both CI/CD and production environments

**To use the repository cleanup script**:
```bash
# The repository is already cloned at /opt/APP_NAME/
cd /opt/APP_NAME

# Make the script executable
chmod +x scripts/cleanup.sh

# Test CI/CD cleanup (dry run first)
./scripts/cleanup.sh --type ci-cd --dry-run

# Run CI/CD cleanup
./scripts/cleanup.sh --type ci-cd

# Test production cleanup (dry run first)
./scripts/cleanup.sh --type production --dry-run
```

**Note**: The repository script is more comprehensive and includes proper error handling, colored output, dry-run mode, and support for both CI/CD and production environments. It automatically detects the environment and provides appropriate cleanup operations.

#### 9.3 Test Cleanup Script

```bash
# Create some test images to clean up
docker pull alpine:latest
docker pull nginx:latest
docker tag alpine:latest test-cleanup:latest
docker tag nginx:latest test-cleanup2:latest

# Test cleanup with dry run first
./scripts/cleanup.sh --type ci-cd --dry-run

# Run the cleanup script
./scripts/cleanup.sh --type ci-cd

# Verify cleanup worked
echo "Checking remaining images:"
docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}"

echo "Checking remaining volumes:"
docker volume ls

echo "Checking remaining networks:"
docker network ls
```

**Expected Output**:
- Cleanup script should run without errors
- Test images should be removed
- System should report cleanup completion
- Remaining images should be minimal (only actively used ones)

**If something goes wrong**:
- Check script permissions: `ls -la scripts/cleanup.sh`
- Verify Docker access: `docker ps`
- Check registry access: `cd /opt/APP_NAME/registry && docker compose ps`
- Run manually: `bash -x scripts/cleanup.sh`

#### 9.4 Set Up Automated Cleanup

```bash
# Create a cron job to run cleanup daily at 3 AM using the repository script
(crontab -l 2>/dev/null; echo "0 3 * * * cd /opt/APP_NAME && ./scripts/cleanup.sh --type ci-cd >> /tmp/cleanup.log 2>&1") | crontab -

# Verify the cron job was added
crontab -l
```

**What this does:**
- **Runs automatically**: The cleanup script runs every day at 3:00 AM
- **Frequency**: Daily cleanup to prevent disk space issues
- **Logging**: All cleanup output is logged to `/tmp/cleanup.log`
- **What it cleans**: Unused Docker images, volumes, networks, and registry images

### Step 10: Configure Firewall

```bash
sudo ufw --force enable
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 5000/tcp  # Docker registry (public read access)
sudo ufw allow 8080/tcp  # Registry UI (public read access)
```

**Security Model**:
- **Port 5000 (Registry)**: Public read access, authenticated write access
- **Port 8080 (UI)**: Public read access for browsing images
- **SSH**: Restricted to your IP addresses
- **All other ports**: Blocked

### Step 11: Test CI/CD Setup

#### 11.1 Test Docker Installation

```bash
docker --version
docker compose --version
```

#### 11.2 Check Registry Status

```bash
cd /opt/registry
docker compose ps
```

#### 11.3 Test Registry Access

```bash
curl http://localhost:5000/v2/_catalog
```

#### 11.4 Get Public Key for Production Server

```bash
cat ~/.ssh/id_ed25519.pub
```

**Important**: Copy this public key - you'll need it for the production server setup.

---

## Part 2: Production Linode Setup

### Step 12: Initial System Setup

#### 12.1 Update the System

```bash
sudo apt update && sudo apt upgrade -y
```

#### 12.2 Configure Timezone

```bash
# Configure timezone interactively
sudo dpkg-reconfigure tzdata

# Verify timezone setting
date
```

**What this does**: Opens an interactive dialog to select your timezone. Navigate through the menus to choose your preferred timezone (e.g., UTC, America/New_York, Europe/London, Asia/Tokyo).

**Expected output**: After selecting your timezone, the `date` command should show the current date and time in your selected timezone.

#### 12.3 Configure /etc/hosts

```bash
# Add localhost entries for both IPv4 and IPv6
echo "127.0.0.1 localhost" | sudo tee -a /etc/hosts
echo "::1 localhost ip6-localhost ip6-loopback" | sudo tee -a /etc/hosts
echo "YOUR_PRODUCTION_IPV4_ADDRESS localhost" | sudo tee -a /etc/hosts
echo "YOUR_PRODUCTION_IPV6_ADDRESS localhost" | sudo tee -a /etc/hosts

# Verify the configuration
cat /etc/hosts
```

**What this does**: 
- Adds localhost entries for both IPv4 and IPv6 addresses to `/etc/hosts`
- Ensures proper localhost resolution for both IPv4 and IPv6

**Important**: Replace `YOUR_PRODUCTION_IPV4_ADDRESS` and `YOUR_PRODUCTION_IPV6_ADDRESS` with the actual IPv4 and IPv6 addresses of your Production Linode obtained from your Linode dashboard.

**Expected output**: The `/etc/hosts` file should show entries for `127.0.0.1`, `::1`, and your Linode's actual IP addresses all mapping to `localhost`.

#### 12.4 Install Essential Packages

```bash
sudo apt install -y \
    curl \
    wget \
    git \
    ca-certificates \
    apt-transport-https \
    software-properties-common \
    ufw \
    fail2ban \
    htop \
    nginx \
    certbot \
    python3-certbot-nginx
```

### Step 13: Create Users

#### 13.1 Create the SERVICE_USER User

```bash
# Create dedicated group for the service account
sudo groupadd -r SERVICE_USER

# Create service account user with dedicated group
sudo useradd -r -g SERVICE_USER -s /bin/bash -m -d /home/SERVICE_USER SERVICE_USER
echo "SERVICE_USER:$(openssl rand -base64 32)" | sudo chpasswd
```

#### 13.2 Create the DEPLOY_USER User

```bash
# Create deployment user
sudo useradd -m -s /bin/bash DEPLOY_USER
sudo usermod -aG sudo DEPLOY_USER
echo "DEPLOY_USER:$(openssl rand -base64 32)" | sudo chpasswd
```

#### 13.3 Verify Users

```bash
sudo su - SERVICE_USER
whoami
pwd
exit

sudo su - DEPLOY_USER
whoami
pwd
exit
```

### Step 14: Install Docker

#### 14.1 Add Docker Repository

```bash
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
```

#### 14.2 Install Docker Packages

```bash
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
```

#### 14.3 Configure Docker for Service Account

```bash
sudo usermod -aG docker SERVICE_USER
```

### Step 15: Install Docker Compose

```bash
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
```

### Step 16: Configure Security

#### 16.1 Configure Firewall

```bash
sudo ufw --force enable
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 3000/tcp
sudo ufw allow 3001/tcp
```

#### 16.2 Configure Fail2ban

```bash
sudo systemctl enable fail2ban
sudo systemctl start fail2ban
```

### Step 17: Create Application Directory

#### 17.1 Create Directory Structure

```bash
sudo mkdir -p /opt/APP_NAME
sudo chown SERVICE_USER:SERVICE_USER /opt/APP_NAME
```

**Note**: Replace `APP_NAME` with your actual application name. This directory name can be controlled via the `APP_NAME` secret in your Forgejo repository settings. If you set the `APP_NAME` secret to `myapp`, the deployment directory will be `/opt/myapp`.

#### 17.2 Create SSL Directory (Optional - for domain users)

```bash
sudo mkdir -p /opt/APP_NAME/nginx/ssl
sudo chown SERVICE_USER:SERVICE_USER /opt/APP_NAME/nginx/ssl
```

### Step 18: Clone Repository and Set Up Application Files

#### 18.1 Switch to SERVICE_USER User

```bash
sudo su - SERVICE_USER
```

#### 18.2 Clone Repository

```bash
cd /opt/APP_NAME
git clone https://your-forgejo-instance/your-username/APP_NAME.git .
```

**Important**: The repository includes a pre-configured `nginx/nginx.conf` file that handles both SSL and non-SSL scenarios, with proper security headers, rate limiting, and CORS configuration. This file will be automatically used by the Docker Compose setup.

**Important**: The repository also includes a pre-configured `.forgejo/workflows/ci.yml` file that handles the complete CI/CD pipeline including testing, building, and deployment. This workflow is already set up to work with the private registry and production deployment.

**Note**: Replace `your-forgejo-instance` and `your-username/APP_NAME` with your actual Forgejo instance URL and repository path.

#### 18.3 Create Environment File

The repository doesn't include a `.env.example` file for security reasons. The CI/CD pipeline will create the `.env` file dynamically during deployment. However, for manual testing or initial setup, you can create a basic `.env` file:

```bash
cat > /opt/APP_NAME/.env << 'EOF'
# Production Environment Variables
POSTGRES_PASSWORD=your_secure_password_here
REGISTRY=YOUR_CI_CD_IP:5000
IMAGE_NAME=APP_NAME
IMAGE_TAG=latest

# Database Configuration
POSTGRES_DB=sharenet
POSTGRES_USER=sharenet
DATABASE_URL=postgresql://sharenet:your_secure_password_here@postgres:5432/sharenet

# Application Configuration
NODE_ENV=production
RUST_LOG=info
RUST_BACKTRACE=1
EOF
```

**Important**: Replace `YOUR_CI_CD_IP` with your actual CI/CD Linode IP address and `your_secure_password_here` with a strong password.

#### 18.4 Configure Docker for Registry Access

```bash
# Add the CI/CD registry to Docker's insecure registries
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json << EOF
{
  "insecure-registries": ["YOUR_CI_CD_IP:5000"]
}
EOF

# Restart Docker to apply changes
sudo systemctl restart docker
```

**Important**: Replace `YOUR_CI_CD_IP` with your actual CI/CD Linode IP address.

### Step 19: Set Up SSH Key Authentication

#### 19.1 Add CI/CD Public Key

```bash
# Create .ssh directory for SERVICE_USER
mkdir -p ~/.ssh
chmod 700 ~/.ssh

# Add the CI/CD public key (copy from CI/CD Linode)
echo "YOUR_CI_CD_PUBLIC_KEY" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
```

**Important**: Replace `YOUR_CI_CD_PUBLIC_KEY` with the public key from the CI/CD Linode (the output from `cat ~/.ssh/id_ed25519.pub` on the CI/CD Linode).

#### 19.2 Test SSH Connection

From the CI/CD Linode, test the SSH connection:

```bash
ssh production
```

**Expected output**: You should be able to SSH to the production server without a password prompt.

### Step 20: Test Production Setup

#### 20.1 Test Docker Installation

```bash
docker --version
docker compose --version
```

#### 20.2 Test Registry Access

```bash
# Test pulling an image from the CI/CD registry
docker pull YOUR_CI_CD_IP:5000/APP_NAME/backend:latest
```

**Important**: Replace `YOUR_CI_CD_IP` with your actual CI/CD Linode IP address.

#### 20.3 Test Application Deployment

```bash
cd /opt/APP_NAME
docker compose up -d
```

#### 20.4 Verify Application Status

```bash
docker compose ps
curl http://localhost:3000
curl http://localhost:3001/health
```

**Expected Output**:
- All containers should be running
- Frontend should be accessible on port 3000
- Backend health check should return 200 OK

---

## Part 3: Final Configuration and Testing

### Step 21: Configure Forgejo Repository Secrets

#### 21.1 Required Repository Secrets

Go to your Forgejo repository and add these secrets in **Settings → Secrets and Variables → Actions**:

**Required Secrets:**
- `CI_CD_IP`: Your CI/CD Linode IP address
- `PRODUCTION_IP`: Your Production Linode IP address
- `DEPLOY_USER`: The deployment user name (e.g., `deploy`, `ci`, `admin`)
- `SERVICE_USER`: The service user name (e.g., `appuser`, `service`, `app`)
- `APP_NAME`: Your application name (e.g., `sharenet`, `myapp`)
- `POSTGRES_PASSWORD`: A strong password for the PostgreSQL database

**Optional Secrets (for domain users):**
- `DOMAIN`: Your domain name (e.g., `example.com`)
- `EMAIL`: Your email for SSL certificate notifications

#### 21.2 Configure Forgejo Actions Runner

##### 21.2.1 Get Runner Token

1. Go to your Forgejo repository
2. Navigate to **Settings → Actions → Runners**
3. Click **"New runner"**
4. Copy the registration token

##### 21.2.2 Configure Runner

```bash
# Switch to DEPLOY_USER on CI/CD Linode
sudo su - DEPLOY_USER

# Get the registration token from your Forgejo repository
# Go to Settings → Actions → Runners → New runner
# Copy the registration token

# Configure the runner
forgejo-runner register \
  --instance https://your-forgejo-instance \
  --token YOUR_TOKEN \
  --name "ci-cd-runner" \
  --labels "ubuntu-latest,docker" \
  --no-interactive
```

##### 21.2.3 Start Runner

```bash
sudo systemctl start forgejo-runner.service
sudo systemctl status forgejo-runner.service
```

##### 21.2.4 Test Runner Configuration

```bash
# Check if the runner is running
sudo systemctl status forgejo-runner.service

# Check runner logs
sudo journalctl -u forgejo-runner.service -f --no-pager

# Test runner connectivity (in a separate terminal)
forgejo-runner list

# Verify runner appears in Forgejo
# Go to your Forgejo repository → Settings → Actions → Runners
# You should see your runner listed as "ci-cd-runner" with status "Online"
```

**Expected Output**:
- `systemctl status` should show "active (running)"
- `forgejo-runner list` should show your runner
- Forgejo web interface should show the runner as online

**If something goes wrong**:
- Check logs: `sudo journalctl -u forgejo-runner.service -f`
- Verify token: Make sure the registration token is correct
- Check network: Ensure the runner can reach your Forgejo instance
- Restart service: `sudo systemctl restart forgejo-runner.service`

### Step 22: Set Up Monitoring and Cleanup

#### 22.1 Monitoring Script

**Important**: The repository includes a pre-configured monitoring script in the `scripts/` directory that can be used for both CI/CD and production monitoring.

**Repository Script**:
- `scripts/monitor.sh` - Comprehensive monitoring script with support for both CI/CD and production environments

**To use the repository monitoring script**:
```bash
# The repository is already cloned at /opt/APP_NAME/
cd /opt/APP_NAME

# Make the script executable
chmod +x scripts/monitor.sh

# Test CI/CD monitoring
./scripts/monitor.sh --type ci-cd

# Test production monitoring (if you have a production setup)
./scripts/monitor.sh --type production
```

**Note**: The repository script is more comprehensive and includes proper error handling, colored output, and support for both CI/CD and production environments. It automatically detects the environment and provides appropriate monitoring information.

#### 22.2 Cleanup Script

**Important**: The repository includes a pre-configured cleanup script in the `scripts/` directory that can be used for both CI/CD and production cleanup operations.

**Repository Script**:
- `scripts/cleanup.sh` - Comprehensive cleanup script with support for both CI/CD and production environments

**To use the repository cleanup script**:
```bash
# The repository is already cloned at /opt/APP_NAME/
cd /opt/APP_NAME

# Make the script executable
chmod +x scripts/cleanup.sh

# Test CI/CD cleanup (dry run first)
./scripts/cleanup.sh --type ci-cd --dry-run

# Run CI/CD cleanup
./scripts/cleanup.sh --type ci-cd

# Test production cleanup (dry run first)
./scripts/cleanup.sh --type production --dry-run
```

**Note**: The repository script is more comprehensive and includes proper error handling, colored output, dry-run mode, and support for both CI/CD and production environments. It automatically detects the environment and provides appropriate cleanup operations.

#### 22.3 Test Cleanup Script

```bash
# Create some test images to clean up
docker pull alpine:latest
docker pull nginx:latest
docker tag alpine:latest test-cleanup:latest
docker tag nginx:latest test-cleanup2:latest

# Test cleanup with dry run first
./scripts/cleanup.sh --type ci-cd --dry-run

# Run the cleanup script
./scripts/cleanup.sh --type ci-cd

# Verify cleanup worked
echo "Checking remaining images:"
docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}"

echo "Checking remaining volumes:"
docker volume ls

echo "Checking remaining networks:"
docker network ls
```

**Expected Output**:
- Cleanup script should run without errors
- Test images should be removed
- System should report cleanup completion
- Remaining images should be minimal (only actively used ones)

**If something goes wrong**:
- Check script permissions: `ls -la scripts/cleanup.sh`
- Verify Docker access: `docker ps`
- Check registry access: `cd /opt/APP_NAME/registry && docker compose ps`
- Run manually: `bash -x scripts/cleanup.sh`

#### 22.4 Set Up Automated Cleanup

```bash
# Create a cron job to run cleanup daily at 3 AM using the repository script
(crontab -l 2>/dev/null; echo "0 3 * * * cd /opt/APP_NAME && ./scripts/cleanup.sh --type ci-cd >> /tmp/cleanup.log 2>&1") | crontab -

# Verify the cron job was added
crontab -l
```

**What this does:**
- **Runs automatically**: The cleanup script runs every day at 3:00 AM
- **Frequency**: Daily cleanup to prevent disk space issues
- **Logging**: All cleanup output is logged to `/tmp/cleanup.log`
- **What it cleans**: Unused Docker images, volumes, networks, and registry images

### Step 23: Test Complete Pipeline

#### 23.1 Trigger a Test Build

1. **Make a small change** to your repository (e.g., update a comment or add a test file)
2. **Commit and push** the changes to trigger the CI/CD pipeline
3. **Monitor the build** in your Forgejo repository → Actions tab

#### 23.2 Verify Pipeline Steps

The pipeline should execute these steps in order:

1. **Checkout**: Clone the repository
2. **Test Backend**: Run backend tests
3. **Test Frontend**: Run frontend tests
4. **Build Backend**: Build backend Docker image
5. **Build Frontend**: Build frontend Docker image
6. **Push to Registry**: Push images to your private registry
7. **Deploy to Production**: Deploy to production server

#### 23.3 Check Registry

```bash
# On CI/CD Linode
cd /opt/APP_NAME/registry

# Check if new images were pushed
curl -k https://localhost:8080/v2/_catalog

# Check specific repository tags
curl -k https://localhost:8080/v2/APP_NAME/backend/tags/list
curl -k https://localhost:8080/v2/APP_NAME/frontend/tags/list
```

#### 23.4 Verify Production Deployment

```bash
# On Production Linode
cd /opt/APP_NAME

# Check if containers are running with new images
docker compose ps

# Check application health
curl http://localhost:3000
curl http://localhost:3001/health

# Check container logs for any errors
docker compose logs backend
docker compose logs frontend
```

#### 23.5 Test Application Functionality

1. **Frontend**: Visit your production URL (IP or domain)
2. **Backend API**: Test API endpoints
3. **Database**: Verify database connections
4. **Logs**: Check for any errors in application logs

### Step 24: Set Up SSL/TLS (Optional - Domain Users)

#### 24.1 Install SSL Certificate

If you have a domain pointing to your Production Linode:

```bash
# On Production Linode
sudo certbot --nginx -d your-domain.com

# Verify certificate
sudo certbot certificates
```

#### 24.2 Configure Auto-Renewal

```bash
# Test auto-renewal
sudo certbot renew --dry-run

# Add to crontab for automatic renewal
sudo crontab -e
# Add this line:
# 0 12 * * * /usr/bin/certbot renew --quiet
```

### Step 25: Final Verification

#### 25.1 Security Check

```bash
# Check firewall status
sudo ufw status

# Check fail2ban status
sudo systemctl status fail2ban

# Check SSH access (should be key-based only)
sudo grep "PasswordAuthentication" /etc/ssh/sshd_config
```

#### 25.2 Performance Check

```bash
# Check system resources
htop

# Check disk usage
df -h

# Check Docker disk usage
docker system df
```

#### 25.3 Backup Verification

```bash
# Test backup script
cd /opt/APP_NAME
./scripts/backup.sh --dry-run

# Run actual backup
./scripts/backup.sh
```

### Step 26: Documentation and Maintenance

#### 26.1 Update Documentation

1. **Update README.md** with deployment information
2. **Document environment variables** and their purposes
3. **Create troubleshooting guide** for common issues
4. **Document backup and restore procedures**

#### 26.2 Set Up Monitoring Alerts

```bash
# Set up monitoring cron job
(crontab -l 2>/dev/null; echo "*/5 * * * * cd /opt/APP_NAME && ./scripts/monitor.sh --type production >> /tmp/monitor.log 2>&1") | crontab -

# Check monitoring logs
tail -f /tmp/monitor.log
```

#### 26.3 Regular Maintenance Tasks

**Daily:**
- Check application logs for errors
- Monitor system resources
- Verify backup completion

**Weekly:**
- Review security logs
- Update system packages
- Test backup restoration

**Monthly:**
- Review and rotate logs
- Update SSL certificates
- Review and update documentation

---

## 🎉 Congratulations!

You have successfully set up a complete CI/CD pipeline with:

- ✅ **Automated testing** on every code push
- ✅ **Docker image building** and registry storage
- ✅ **Automated deployment** to production
- ✅ **Health monitoring** and logging
- ✅ **Backup and cleanup** automation
- ✅ **Security hardening** with proper user separation
- ✅ **SSL/TLS support** for production (optional)

Your application is now ready for continuous deployment with proper security, monitoring, and maintenance procedures in place!