sharenet/PRODUCTION_SETUP_GUIDE.md

# Production Linode Setup Guide

This guide covers setting up your Production Linode for hosting the APP_NAME application with Docker, SSL/TLS, and automated deployments.

## Architecture Overview

```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Forgejo Host  │    │   CI/CD Linode  │    │ Production Linode│
│   (Repository)  │    │ (Actions Runner)│    │ (Docker Deploy) │
│                 │    │ + Docker Registry│   │                 │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         │                       │                       │
         └─────────── Push ──────┼───────────────────────┘
                                 │
                                 └─── Deploy ────────────┘
```

## Prerequisites

- Ubuntu 24.04 LTS Linode with root access
- Domain name pointing to the Production Linode's IP addresses
- Basic familiarity with Linux commands and SSH
- **For CI/CD Integration**: CI/CD Linode already set up and IP address ready
- **For Manual Deployment**: Docker registry access (local or remote)

## Quick Start

### Basic Production Setup (Manual Deployment)
1. **Follow the basic production setup guide**
2. **Set up SSL certificates** for secure HTTPS access
3. **Configure application deployment** with Docker Compose
4. **Test application functionality** and monitoring

### CI/CD Integration Setup (Automated Deployment)
1. **Complete basic production setup first**
2. **Exchange SSH keys** with CI/CD server
3. **Configure registry connection** to CI/CD server
4. **Set up Forgejo repository secrets**
5. **Test automated deployment pipeline**

## What's Included

### Production Linode Features
- Docker-based application deployment
- SSL/TLS certificate management
- Nginx reverse proxy with security headers
- Automated backups and monitoring
- Firewall and fail2ban protection

### CI/CD Integration Features (Optional)
- Secure SSH communication with CI/CD server
- Automated deployment from CI/CD pipeline
- Registry connection for image pulling
- Forgejo Actions integration

## Prerequisites and Initial Setup

### What's Already Done (Assumptions)

This guide assumes you have already:

1. **Created an Ubuntu 24.04 LTS Linode** with root access
2. **Set root password** for the Linode
3. **Assigned DNS entries** for your domain pointing to the Production Linode's IPv4 and IPv6 addresses
4. **Have SSH client** installed on your local machine
5. **For CI/CD Integration**: CI/CD Linode is set up and running with IP address ready
6. **For Manual Deployment**: Have access to Docker images (local registry, Docker Hub, or other registry)

**Note**: The Production Linode needs a domain name for SSL certificates and public web access.

### Step 0: Initial SSH Access and Verification

Before proceeding with the setup, you need to establish initial SSH access to your Production Linode.

#### 0.1 Get Your Linode IP Address

From your Linode dashboard, note the IP address for:
- **Production Linode**: `YOUR_PRODUCTION_IP` (IP address for SSH, domain for web access)

#### 0.2 Test Initial SSH Access

Test SSH access to your Production Linode:

```bash
# Test Production Linode (IP address only)
ssh root@YOUR_PRODUCTION_IP
```

**Expected output**: SSH login prompt asking for root password.

**If something goes wrong**:
- Verify the IP address is correct
- Check that SSH is enabled on the Linode
- Ensure your local machine can reach the Linode (no firewall blocking)

#### 0.3 Verify DNS Configuration

On the Production Linode, verify DNS is properly configured for your domain:

```bash
# Check if domain resolves to this server
nslookup your-domain.com

# Check both IPv4 and IPv6
dig your-domain.com A
dig your-domain.com AAAA
```

**Expected output**: DNS records showing your Production Linode's IP addresses.

**If something goes wrong**:
- Verify DNS propagation: `dig your-domain.com +trace`
- Check DNS settings in your domain registrar
- Wait for DNS propagation (can take up to 48 hours)

#### 0.4 Test Domain Accessibility

Test that your domain is accessible on the Production Linode:

```bash
# Test HTTP access (should show default nginx/apache page or connection refused)
curl -I http://your-domain.com

# Test HTTPS access (should fail initially, but domain should resolve)
curl -I https://your-domain.com
```

**Expected output**: 
- HTTP: Either a web page or "Connection refused" (both are normal)
- HTTPS: SSL error (expected before certificates are installed)

**If something goes wrong**:
- Verify DNS propagation is complete
- Check that port 80/443 is not blocked by firewall
- Ensure the domain is pointing to the correct IP

#### 0.5: Choose Your Names

Before proceeding, decide on:

1. **Service Account Name**: Choose a username for the service account (e.g., `appuser`, `deploy`, `service`)
   - Replace `SERVICE_USER` in this guide with your chosen name
   
2. **Application Name**: Choose a name for your application (e.g., `myapp`, `webapp`, `api`)
   - Replace `APP_NAME` in this guide with your chosen name

**Example**:
- If you choose `appuser` as service account and `myapp` as application name:
  - Replace all `SERVICE_USER` with `appuser`
  - Replace all `APP_NAME` with `myapp`

---

## Basic Production Setup

This section covers the essential setup for hosting your application. **Skip to the next section if you only want manual deployments.**

### Step 1: Initial System Setup

#### 1.1 Update the System

```bash
sudo apt update && sudo apt upgrade -y
```

**What this does**: Updates package lists and upgrades all installed packages to their latest versions.

**Expected output**: A list of packages being updated, followed by completion messages.

**If something goes wrong**: 
- Check your internet connection
- Try running `sudo apt update` first, then `sudo apt upgrade -y` separately
- If you get package conflicts, run `sudo apt --fix-broken install`

#### 1.2 Install Essential Packages

```bash
sudo apt install -y \
    curl \
    wget \
    git \
    ca-certificates \
    apt-transport-https \
    software-properties-common \
    ufw \
    fail2ban \
    htop \
    nginx \
    certbot \
    python3-certbot-nginx
```

**What this does**: Installs web server, SSL tools, security packages, and utilities needed for production deployment.

**Expected output**: Package installation progress, ending with completion messages.

**If something goes wrong**: 
- Check if any packages failed to install
- Run `sudo apt install -f` to fix broken dependencies
- Ensure you have sufficient disk space: `df -h`

### Step 2: Create Service Account

#### 2.1 Create the SERVICE_USER User

```bash
sudo useradd -r -s /bin/bash -m -d /home/SERVICE_USER SERVICE_USER
sudo usermod -aG sudo SERVICE_USER
echo "SERVICE_USER:$(openssl rand -base64 32)" | sudo chpasswd
```

**What this does**: 
- Creates a dedicated service account named `SERVICE_USER`
- Gives it sudo privileges for administrative tasks
- Generates a random 32-character password

**Expected output**: No output (successful user creation is silent).

**If something goes wrong**:
- If user already exists: `sudo userdel -r SERVICE_USER` then retry
- Check user creation: `id SERVICE_USER`
- Verify sudo access: `sudo -u SERVICE_USER sudo -l`

#### 2.2 Verify Service Account

```bash
sudo su - SERVICE_USER
whoami
pwd
```

**What this does**: Switches to the SERVICE_USER user and verifies the setup.

**Expected output**:
```
SERVICE_USER
/home/SERVICE_USER
```

### Step 3: Install Docker

#### 3.1 Add Docker Repository

```bash
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
```

**What this does**: Adds Docker's official repository to your system for the latest Docker versions.

**Expected output**: GPG key import confirmation and package list update.

#### 3.2 Install Docker Packages

```bash
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
```

**What this does**: Installs Docker Engine, CLI, container runtime, and Docker Compose.

**Expected output**: Package installation progress, ending with completion messages.

#### 3.3 Configure Docker for Service Account

```bash
sudo usermod -aG docker SERVICE_USER
```

**What this does**: Adds the SERVICE_USER user to the docker group so it can run Docker commands without sudo.

### Step 4: Install Docker Compose

```bash
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
```

**What this does**: Downloads the latest Docker Compose binary and makes it executable.

### Step 5: Configure Security

#### 5.1 Configure Firewall

```bash
sudo ufw --force enable
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 3000/tcp
sudo ufw allow 3001/tcp
```

**What this does**: Configures firewall to allow only necessary ports for web traffic and application.

#### 5.2 Configure Fail2ban

```bash
sudo systemctl enable fail2ban
sudo systemctl start fail2ban
```

**What this does**: Enables fail2ban to protect against brute force attacks.

### Step 6: Create Application Directory

#### 6.1 Create Directory Structure

```bash
sudo mkdir -p /opt/APP_NAME
sudo chown SERVICE_USER:SERVICE_USER /opt/APP_NAME
```

**What this does**: Creates a directory for the application and sets ownership to the SERVICE_USER user.

#### 6.2 Create SSL Directory

```bash
sudo mkdir -p /opt/APP_NAME/nginx/ssl
sudo chown SERVICE_USER:SERVICE_USER /opt/APP_NAME/nginx/ssl
```

**What this does**: Creates a directory for SSL certificates.

### Step 7: Set Up Application Files

#### 7.1 Switch to SERVICE_USER User

```bash
sudo su - SERVICE_USER
```

#### 7.2 Create Environment File

```bash
cat > /opt/APP_NAME/.env << 'EOF'
# Production Environment Variables
POSTGRES_PASSWORD=your_secure_password_here
REGISTRY=your-remote-registry.com:port
IMAGE_NAME=APP_NAME
IMAGE_TAG=latest

# Database Configuration
DATABASE_URL=postgresql://SERVICE_USER:your_secure_password_here@postgres:5432/APP_NAME

# Application Configuration
NODE_ENV=production
RUST_LOG=info
EOF
```

**What this does**: Creates environment variables for the application deployment.

**Note**: For manual deployments, you'll typically:
- **Pull images from a remote registry** (private registry, GitHub Container Registry, etc.)
- **Update `your-remote-registry.com:port`** with your actual registry URL
- **Configure docker-compose.yml** to reference images from your registry

**Examples for different registry types**:
```bash
# Private registry
REGISTRY=registry.company.com:5000

# GitHub Container Registry
REGISTRY=ghcr.io

# GitLab Container Registry
REGISTRY=registry.gitlab.com

# AWS ECR
REGISTRY=123456789012.dkr.ecr.us-east-1.amazonaws.com
```

**Examples for docker-compose.yml**:
```yaml
# For remote registry:
services:
  backend:
    image: your-remote-registry.com:port/APP_NAME-backend:latest
  frontend:
    image: your-remote-registry.com:port/APP_NAME-frontend:latest
```

#### 7.3 Create Deployment Script

```bash
cat > /opt/APP_NAME/deploy.sh << 'EOF'
#!/bin/bash

# Deployment script for APP_NAME
set -e

echo "Deploying APP_NAME..."

# Pull latest images
docker-compose pull

# Stop existing containers
docker-compose down

# Start new containers
docker-compose up -d

# Clean up old images
docker image prune -f

echo "Deployment complete!"
echo "Check status with: docker-compose ps"
EOF

chmod +x /opt/APP_NAME/deploy.sh
```

**What this does**: Creates a deployment script for easy application updates.

#### 7.4 Create Backup Script

```bash
cat > /opt/APP_NAME/backup.sh << 'EOF'
#!/bin/bash

# Backup script for APP_NAME
set -e

BACKUP_DIR="/opt/APP_NAME/backups"
DATE=$(date +%Y%m%d_%H%M%S)

mkdir -p $BACKUP_DIR

# Backup database
docker-compose exec -T postgres pg_dump -U SERVICE_USER APP_NAME > $BACKUP_DIR/db_backup_$DATE.sql

# Backup configuration files
tar -czf $BACKUP_DIR/config_backup_$DATE.tar.gz .env docker-compose.yml nginx/

# Keep only last 7 days of backups
find $BACKUP_DIR -name "*.sql" -mtime +7 -delete
find $BACKUP_DIR -name "*.tar.gz" -mtime +7 -delete

echo "Backup completed: $BACKUP_DIR"
EOF

chmod +x /opt/APP_NAME/backup.sh
```

**What this does**: Creates a backup script for database and configuration files.

#### 7.5 Create Monitoring Script

```bash
cat > /opt/APP_NAME/monitor.sh << 'EOF'
#!/bin/bash

echo "=== APP_NAME Application Status ==="
echo "Date: $(date)"
echo "Uptime: $(uptime)"
echo ""

echo "=== Docker Containers ==="
docker-compose ps
echo ""

echo "=== Application Logs (last 20 lines) ==="
docker-compose logs --tail=20
echo ""

echo "=== System Resources ==="
echo "CPU Usage:"
top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1
echo "Memory Usage:"
free -h | grep Mem
echo "Disk Usage:"
df -h /
echo ""

echo "=== Network Status ==="
netstat -tlnp | grep -E ':(80|443|3000|3001)'
echo ""

echo "=== SSL Certificate Status ==="
if [ -f "/opt/APP_NAME/nginx/ssl/cert.pem" ]; then
    openssl x509 -in /opt/APP_NAME/nginx/ssl/cert.pem -text -noout | grep -E "(Not Before|Not After)"
else
    echo "SSL certificate not found"
fi
EOF

chmod +x /opt/APP_NAME/monitor.sh
```

**What this does**: Creates a monitoring script to check application and system status.

### Step 8: Set Up Automated Backups

```bash
sudo -u SERVICE_USER bash -c '(crontab -l 2>/dev/null; echo "0 2 * * * /opt/APP_NAME/backup.sh") | crontab -'
```

**What this does**: Schedules daily backups at 2 AM.

### Step 9: Copy Application Files

```bash
sudo su - SERVICE_USER
cd /opt/APP_NAME
git clone https://your-forgejo-instance/your-username/APP_NAME.git .
```

**What this does**: Clones your application repository to the production server.

**Note**: Replace the URL with your actual repository URL.

### Step 10: Set Up SSL Certificates

#### 10.1 Install SSL Certificate

```bash
sudo certbot --nginx -d your-domain.com
```

**What this does**: Installs Let's Encrypt SSL certificate for your domain.

**Expected output**: Certificate installation confirmation and Nginx configuration updates.

#### 10.2 Copy Certificates

```bash
sudo cp /etc/letsencrypt/live/your-domain.com/fullchain.pem /opt/APP_NAME/nginx/ssl/cert.pem
sudo cp /etc/letsencrypt/live/your-domain.com/privkey.pem /opt/APP_NAME/nginx/ssl/key.pem
sudo chown SERVICE_USER:SERVICE_USER /opt/APP_NAME/nginx/ssl/*
```

**What this does**: Copies SSL certificates to the application directory for Docker use.

### Step 11: Test Basic Application

#### 11.1 Test Application Deployment

```bash
cd /opt/APP_NAME
./deploy.sh
```

**What this does**: Deploys the application using Docker Compose.

**Expected output**: Deployment progress messages and completion.

#### 11.2 Monitor Application

```bash
cd /opt/APP_NAME
./monitor.sh
```

**Expected output**: Application status, system resources, and recent logs.

---

## CI/CD Integration Setup (Optional)

**⚠️ SKIP THIS SECTION if you're only doing manual deployments!**

This section covers integrating with a CI/CD pipeline for automated deployments.

### Prerequisites for CI/CD Integration

- CI/CD Linode already set up and running
- CI/CD Linode IP address available
- SSH access to CI/CD server

### Step 1: Configure Docker for CI/CD Registry

#### 1.1 Update Docker Daemon Configuration

```bash
# Configure Docker to allow connections to CI/CD registry
echo '{"insecure-registries": ["YOUR_CI_HOST_IP:5000"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
```

**What this does**: Configures Docker to allow connections to your CI/CD registry without SSL verification.

**Note**: Replace `YOUR_CI_HOST_IP` with your actual CI/CD server IP.

### Step 2: Set Up SSH Keys for CI/CD Communication

#### 2.1 Create SSH Directory

```bash
mkdir -p ~/.ssh
chmod 700 ~/.ssh
```

#### 2.2 Generate SSH Key Pair

```bash
ssh-keygen -t ed25519 -C "production-server" -f ~/.ssh/id_ed25519 -N ""
```

**What this does**: Generates an Ed25519 SSH key pair for secure communication with the CI/CD server.

#### 2.3 Create SSH Config

```bash
cat > ~/.ssh/config << 'EOF'
Host ci-cd
    HostName YOUR_CI_CD_IP
    User SERVICE_USER
    IdentityFile ~/.ssh/id_ed25519
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
EOF

chmod 600 ~/.ssh/config
```

**What this does**: Creates SSH configuration for easy connection to the CI/CD server.

**Note**: Replace `YOUR_CI_CD_IP` with your actual CI/CD server IP.

### Step 3: Exchange SSH Keys

#### 3.1 Get Your Public Key

```bash
cat ~/.ssh/id_ed25519.pub
```

**Important**: Copy this public key - you'll need it for the CI/CD server setup.

#### 3.2 Add CI/CD Server's Public Key

```bash
echo "CI_CD_PUBLIC_KEY_HERE" >> ~/.ssh/authorized_keys
sed -i 's/YOUR_CI_CD_IP/YOUR_ACTUAL_CI_CD_IP/g' ~/.ssh/config
```

**What this does**: Adds the CI/CD server's public key to allow secure communication.

**Note**: Replace `CI_CD_PUBLIC_KEY_HERE` with the actual public key from your CI/CD server.

### Step 4: Update Application Configuration for CI/CD

#### 4.1 Update Environment Variables

```bash
cd /opt/APP_NAME
nano .env
```

**Required changes**:
- Replace `your-registry-url:port` with `YOUR_CI_HOST_IP:5000`
- Replace `your_secure_password_here` with a strong database password
- Update `DATABASE_URL` with the same password

**Example**:
```bash
# Change this line:
REGISTRY=your-registry-url:port

# To this (replace with your actual CI/CD IP):
REGISTRY=192.168.1.100:5000
```

### Step 5: Test CI/CD Integration

#### 5.1 Test SSH Connection

```bash
ssh ci-cd 'echo Connection successful'
```

**Expected output**: `Connection successful`.

#### 5.2 Test Registry Connection

```bash
curl http://YOUR_ACTUAL_CI_IP:5000/v2/_catalog
```

**Expected output**: `{"repositories":[]}` or a list of available images.

### Step 6: Configure Forgejo Repository Secrets

Go to your Forgejo repository → Settings → Secrets and Variables → Actions, and add:

- `CI_HOST`: Your CI/CD Linode IP address
- `PROD_HOST`: Your production Linode IP
- `PROD_USER`: SSH username for production server (should be `SERVICE_USER`)
- `PROD_SSH_KEY`: SSH private key for deployment (from CI/CD server)

### Step 7: Test Automated Deployment

#### 7.1 Push Code Changes

Make a small change to your code and push to trigger the CI/CD pipeline:

```bash
# In your local repository
echo "# Test deployment" >> README.md
git add README.md
git commit -m "Test CI/CD pipeline"
git push
```

#### 7.2 Monitor Pipeline

1. Go to your Forgejo repository
2. Navigate to Actions tab
3. Monitor the workflow execution
4. Check for any errors or issues

#### 7.3 Verify Deployment

After successful pipeline execution:

```bash
# Check application status
cd /opt/APP_NAME
docker-compose ps

# Check application logs
docker-compose logs

# Test application access
curl -I https://your-domain.com
```

---

## Testing and Verification

### Step 1: Test Application Access

```bash
# Test HTTP redirect to HTTPS
curl -I http://your-domain.com

# Test HTTPS access
curl -I https://your-domain.com

# Test application endpoints
curl -I https://your-domain.com/api/health
```

**Expected output**: 
- HTTP: 301 redirect to HTTPS
- HTTPS: 200 OK response
- API: Application-specific response

### Step 2: Test Monitoring

```bash
cd /opt/APP_NAME
./monitor.sh
```

**Expected output**: Application status, system resources, and recent logs.

### Step 3: Test Backup

```bash
cd /opt/APP_NAME
./backup.sh
```

**Expected output**: Backup completion message with file locations.

### Step 4: Test Manual Deployment (if not using CI/CD)

```bash
cd /opt/APP_NAME
./pull-and-deploy.sh
```

**Expected output**: Pull progress messages, image download, and deployment completion.

**Alternative manual process**:
```bash
# Pull images manually
docker pull your-remote-registry.com:port/APP_NAME-backend:latest
docker pull your-remote-registry.com:port/APP_NAME-frontend:latest

# Deploy application
cd /opt/APP_NAME
./deploy.sh
```

### Step 5: Manual Image Management (for non-CI/CD deployments)

If you're not using CI/CD, you'll need to pull and manage Docker images from your remote registry:

#### 5.1 Configure Registry Authentication

```bash
# Login to your remote registry
docker login your-remote-registry.com:port

# For GitHub Container Registry
docker login ghcr.io -u YOUR_GITHUB_USERNAME -p YOUR_GITHUB_TOKEN

# For GitLab Container Registry
docker login registry.gitlab.com -u YOUR_GITLAB_USERNAME -p YOUR_GITLAB_TOKEN

# For AWS ECR
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 123456789012.dkr.ecr.us-east-1.amazonaws.com
```

#### 5.2 Pull Images from Remote Registry

```bash
# Pull backend image
docker pull your-remote-registry.com:port/APP_NAME-backend:latest

# Pull frontend image
docker pull your-remote-registry.com:port/APP_NAME-frontend:latest

# Verify images were pulled
docker images | grep APP_NAME
```

#### 5.3 Update Application Configuration

```bash
# Update docker-compose.yml to use your remote registry images
cd /opt/APP_NAME
nano docker-compose.yml

# Example configuration:
# image: your-remote-registry.com:port/APP_NAME-backend:latest
# image: your-remote-registry.com:port/APP_NAME-frontend:latest
```

#### 5.4 Create Manual Pull and Deploy Script

```bash
cat > /opt/APP_NAME/pull-and-deploy.sh << 'EOF'
#!/bin/bash

# Manual pull and deploy script
set -e

echo "Pulling APP_NAME images from remote registry..."

# Pull backend
docker pull your-remote-registry.com:port/APP_NAME-backend:latest

# Pull frontend
docker pull your-remote-registry.com:port/APP_NAME-frontend:latest

# Show pulled images
echo "Pulled images:"
docker images | grep APP_NAME

# Deploy application
cd /opt/APP_NAME
./deploy.sh

echo "Pull and deploy complete!"
EOF

chmod +x /opt/APP_NAME/pull-and-deploy.sh
```

**What this does**: Creates a script to pull images from your remote registry and deploy the application.

#### 5.5 Create Image Update Script

```bash
cat > /opt/APP_NAME/update-images.sh << 'EOF'
#!/bin/bash

# Update images from remote registry
set -e

echo "Updating images from remote registry..."

# Pull latest images
docker-compose pull

# Show updated images
echo "Updated images:"
docker images | grep APP_NAME

echo "Update complete!"
echo "Run './deploy.sh' to deploy with new images"
EOF

chmod +x /opt/APP_NAME/update-images.sh
```

**What this does**: Creates a script to update images from your remote registry.

---

## Troubleshooting

### Common Issues

1. **Docker permission denied**:
   ```bash
   sudo usermod -aG docker SERVICE_USER
   newgrp docker
   ```

2. **SSL certificate issues**:
   ```bash
   sudo certbot certificates
   sudo certbot renew --dry-run
   ```

3. **Application not starting**:
   ```bash
   cd /opt/APP_NAME
   docker-compose logs
   ```

4. **SSH connection failed** (CI/CD integration only):
   ```bash
   ssh -v ci-cd 'echo test'
   ```

5. **Registry connection failed** (CI/CD integration only):
   ```bash
   curl -v http://YOUR_CI_HOST_IP:5000/v2/_catalog
   ```

### Manual Deployment Issues

6. **Registry authentication fails**:
   ```bash
   # Check if you're logged in
   docker login your-remote-registry.com:port
   
   # For GitHub Container Registry
   docker login ghcr.io -u YOUR_GITHUB_USERNAME -p YOUR_GITHUB_TOKEN
   
   # For GitLab Container Registry
   docker login registry.gitlab.com -u YOUR_GITLAB_USERNAME -p YOUR_GITLAB_TOKEN
   
   # Check authentication status
   cat ~/.docker/config.json
   ```

7. **Images not found in remote registry**:
   ```bash
   # Check if images exist in your registry
   docker pull your-remote-registry.com:port/APP_NAME-backend:latest
   
   # List local images
   docker images | grep APP_NAME
   
   # Check registry connectivity
   curl -v https://your-remote-registry.com:port/v2/_catalog
   
   # For GitHub Container Registry
   curl -H "Authorization: Bearer YOUR_GITHUB_TOKEN" https://ghcr.io/v2/_catalog
   ```

8. **Network connectivity issues**:
   ```bash
   # Test basic connectivity
   ping your-remote-registry.com
   
   # Test port connectivity
   telnet your-remote-registry.com port
   
   # Check DNS resolution
   nslookup your-remote-registry.com
   
   # Test HTTPS connectivity
   curl -I https://your-remote-registry.com:port/v2/
   ```

9. **Application configuration issues**:
   ```bash
   # Check environment variables
   cd /opt/APP_NAME
   cat .env
   
   # Validate docker-compose configuration
   docker-compose config
   
   # Check application logs
   docker-compose logs --tail=50
   ```

10. **Manual pull and deploy process issues**:
    ```bash
    # Check if pull script exists
    ls -la /opt/APP_NAME/pull-and-deploy.sh
    
    # Run pull with verbose output
    bash -x /opt/APP_NAME/pull-and-deploy.sh
    
    # Check for pull errors
    docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.CreatedAt}}"
    
    # Verify docker-compose.yml references correct registry
    grep -A 5 "image:" /opt/APP_NAME/docker-compose.yml
    ```

### Useful Commands

- **Check system resources**: `htop`
- **Check disk space**: `df -h`
- **Check memory usage**: `free -h`
- **Check network**: `ip addr show`
- **Check firewall**: `sudo ufw status`
- **Check logs**: `sudo journalctl -f`

### Security Best Practices

1. **Service Account**: Use dedicated `SERVICE_USER` user with limited privileges
2. **SSH Keys**: Use Ed25519 keys with proper permissions (600/700)
3. **Firewall**: Configure UFW to allow only necessary ports
4. **Fail2ban**: Protect against brute force attacks
5. **SSL/TLS**: Use Let's Encrypt certificates with automatic renewal
6. **Regular Backups**: Automated daily backups of database and configuration
7. **Container Isolation**: Applications run in isolated Docker containers
8. **Security Headers**: Nginx configured with security headers

### Monitoring and Maintenance

#### Daily Monitoring

```bash
# On Production server
cd /opt/APP_NAME
./monitor.sh
```

#### Weekly Maintenance

1. **Check disk space**: `df -h`
2. **Review logs**: `docker-compose logs --tail=100`
3. **Update system**: `sudo apt update && sudo apt upgrade`
4. **Test backups**: Verify backup files exist and are recent

#### Monthly Maintenance

1. **Review security**: Check firewall rules and fail2ban status
2. **Update certificates**: Ensure SSL certificates are valid
3. **Clean up old images**: Remove unused Docker images
4. **Review monitoring**: Check application performance and logs

---

## Summary

Your Production Linode is now ready to host your application! The setup includes:

### Basic Features (All Deployments)
- **Docker-based application deployment** with SSL/TLS
- **Nginx reverse proxy** with security headers
- **Automated backups and monitoring** scripts
- **Firewall and fail2ban protection** for security
- **Manual image management** tools and scripts

### CI/CD Integration Features (Optional)
- **Secure SSH communication** with CI/CD server
- **Automated deployment** from CI/CD pipeline
- **Registry connection** for image pulling
- **Forgejo Actions integration**

For ongoing maintenance and troubleshooting, refer to the troubleshooting section and monitoring scripts provided in this guide.