sharenet/PRODUCTION_LINODE_MANUAL_SETUP.md

# Production Linode Setup Guide - Manual Deployment

This guide covers setting up your Production Linode for hosting the APP_NAME application with manual deployment by pulling Docker images from a private registry.

## Architecture Overview

```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Private Registry│    │ Production Linode│    │   Your Domain   │
│ (Docker Images) │    │ (Docker Deploy) │    │ (Optional SSL)  │
│                 │    │                 │    │                 │
│                 │    │                 │    │                 │
│                 │    │                 │    │                 │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         │                       │                       │
         └─────────── Pull ──────┼───────────────────────┘
                                 │
                                 └─── Serve ─────────────┘
```

## Prerequisites

- Ubuntu 24.04 LTS Linode with root access
- Basic familiarity with Linux commands and SSH
- **Access to a private Harbor registry** with authentication credentials
- Docker images already built and pushed to your registry
- **Optional**: Domain name pointing to the Production Linode's IP addresses (for SSL/TLS)

## Quick Start

1. **Complete basic production setup** (Steps 1-7)
2. **Configure registry authentication** (Step 8)
3. **Set up application deployment** (Step 9)
4. **Configure SSL certificates** (Step 10 - Optional)
5. **Test manual deployment** (Step 11)

## What's Included

### Production Linode Features
- Docker-based application deployment
- **Optional SSL/TLS certificate management** (if domain is provided)
- Nginx reverse proxy with security headers
- Automated backups and monitoring
- Firewall and fail2ban protection

### Manual Deployment Features
- Private registry authentication
- Manual image pulling and deployment
- Image update and rollback capabilities
- Deployment verification and testing

## Prerequisites and Initial Setup

### What's Already Done (Assumptions)

This guide assumes you have already:

1. **Created an Ubuntu 24.04 LTS Linode** with root access
2. **Set root password** for the Linode
3. **Have SSH client** installed on your local machine
4. **Have Docker images built and pushed** to your private registry
5. **Have private registry credentials** ready for authentication
6. **Optional**: Assigned DNS entries for your domain pointing to the Production Linode's IP addresses

**Note**: A domain name is optional but recommended for SSL/TLS certificates and professional web access. If you don't have a domain, you can access the application via IP address.

### Step 0: Initial SSH Access and Verification

Before proceeding with the setup, you need to establish initial SSH access to your Production Linode.

#### 0.1 Get Your Linode IP Address

From your Linode dashboard, note the IP address for:
- **Production Linode**: `YOUR_PRODUCTION_IP` (IP address for SSH and web access)

#### 0.2 Test Initial SSH Access

Test SSH access to your Production Linode:

```bash
# Test Production Linode (IP address only)
ssh root@YOUR_PRODUCTION_IP
```

**Expected output**: SSH login prompt asking for root password.

**If something goes wrong**:
- Verify the IP address is correct
- Check that SSH is enabled on the Linode
- Ensure your local machine can reach the Linode (no firewall blocking)

#### 0.3 Optional: Verify DNS Configuration (If Using Domain)

If you have a domain name, verify DNS is properly configured for your domain:

```bash
# Check if domain resolves to this server
nslookup your-domain.com

# Check both IPv4 and IPv6
dig your-domain.com A
dig your-domain.com AAAA
```

**Expected output**: DNS records showing your Production Linode's IP addresses.

**If something goes wrong**:
- Verify DNS propagation: `dig your-domain.com +trace`
- Check DNS settings in your domain registrar
- Wait for DNS propagation (can take up to 48 hours)

#### 0.4 Optional: Test Domain Accessibility (If Using Domain)

If you have a domain name, test that your domain is accessible on the Production Linode:

```bash
# Test HTTP access (should show default nginx/apache page or connection refused)
curl -I http://your-domain.com

# Test HTTPS access (should fail initially, but domain should resolve)
curl -I https://your-domain.com
```

**Expected output**: 
- HTTP: Either a web page or "Connection refused" (both are normal)
- HTTPS: SSL error (expected before certificates are installed)

**If something goes wrong**:
- Verify DNS propagation is complete
- Check that port 80/443 is not blocked by firewall
- Ensure the domain is pointing to the correct IP

#### 0.5: Choose Your Names

Before proceeding, decide on:

1. **Service Account Name**: Choose a username for the service account (e.g., `appuser`, `deploy`, `service`)
   - Replace `SERVICE_USER` in this guide with your chosen name
   
2. **Application Name**: Choose a name for your application (e.g., `myapp`, `webapp`, `api`)
   - Replace `APP_NAME` in this guide with your chosen name

3. **Domain Name** (Optional): If you have a domain, note it for SSL configuration
   - Replace `your-domain.com` in this guide with your actual domain

4. **Registry Information**: Note your private registry details
   - Registry URL: `your-registry.com:port` (replace with your actual registry)
   - Username: `YOUR_REGISTRY_USERNAME` (replace with your actual username)
   - Password/Token: `YOUR_REGISTRY_PASSWORD` (replace with your actual password/token)

**Example**:
- If you choose `appuser` as service account and `myapp` as application name:
  - Replace all `SERVICE_USER` with `appuser`
  - Replace all `APP_NAME` with `myapp`
  - If you have a domain `example.com`, replace `your-domain.com` with `example.com`
  - If your registry is `registry.company.com:5000`, replace `your-registry.com:port` with `registry.company.com:5000`

---

## Basic Production Setup

This section covers the essential setup for hosting your application with manual deployment.

### Step 1: Initial System Setup

#### 1.1 Update the System

```bash
sudo apt update && sudo apt upgrade -y
```

**What this does**: Updates package lists and upgrades all installed packages to their latest versions.

**Expected output**: A list of packages being updated, followed by completion messages.

**If something goes wrong**: 
- Check your internet connection
- Try running `sudo apt update` first, then `sudo apt upgrade -y` separately
- If you get package conflicts, run `sudo apt --fix-broken install`

#### 1.2 Install Essential Packages

```bash
sudo apt install -y \
    curl \
    wget \
    git \
    ca-certificates \
    apt-transport-https \
    software-properties-common \
    ufw \
    fail2ban \
    htop \
    nginx \
    certbot \
    python3-certbot-nginx
```

**What this does**: Installs web server, SSL tools, security packages, and utilities needed for production deployment.

**Expected output**: Package installation progress, ending with completion messages.

**If something goes wrong**: 
- Check if any packages failed to install
- Run `sudo apt install -f` to fix broken dependencies
- Ensure you have sufficient disk space: `df -h`

### Step 2: Create Service Account

#### 2.1 Create the SERVICE_USER User

```bash
# Create dedicated group for the service account
sudo groupadd -r SERVICE_USER

# Create service account user with dedicated group
sudo useradd -r -g SERVICE_USER -s /bin/bash -m -d /home/SERVICE_USER SERVICE_USER
echo "SERVICE_USER:$(openssl rand -base64 32)" | sudo chpasswd
```

**What this does**: 
- Creates a dedicated group for the service account
- Creates a dedicated service account named `SERVICE_USER`
- Generates a random 32-character password
- **Note**: This user has no sudo privileges for security (principle of least privilege)

**Expected output**: No output (successful user creation is silent).

**If something goes wrong**:
- If user already exists: `sudo userdel -r SERVICE_USER` then retry
- Check user creation: `id SERVICE_USER`
- Verify user exists: `getent passwd SERVICE_USER`

#### 2.2 Verify Service Account

```bash
sudo su - SERVICE_USER
whoami
pwd
```

**What this does**: Switches to the SERVICE_USER user and verifies the setup.

**Expected output**:
```
SERVICE_USER
/home/SERVICE_USER
```

### Step 3: Install Docker

#### 3.1 Add Docker Repository

```bash
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
```

**What this does**: Adds Docker's official repository to your system for the latest Docker versions.

**Expected output**: GPG key import confirmation and package list update.

#### 3.2 Install Docker Packages

```bash
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
```

**What this does**: Installs Docker Engine, CLI, container runtime, and Docker Compose.

**Expected output**: Package installation progress, ending with completion messages.

#### 3.3 Configure Docker for Service Account

```bash
sudo usermod -aG docker SERVICE_USER
```

**What this does**: Adds the SERVICE_USER user to the docker group so it can run Docker commands without sudo.

### Step 4: Install Docker Compose

```bash
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
```

**What this does**: Downloads the latest Docker Compose binary and makes it executable.

### Step 5: Configure Security

#### 5.1 Configure Firewall

```bash
sudo ufw --force enable
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 3000/tcp
sudo ufw allow 3001/tcp
```

**What this does**: Configures firewall to allow only necessary ports for web traffic and application.

#### 5.2 Configure Fail2ban

```bash
sudo systemctl enable fail2ban
sudo systemctl start fail2ban
```

**What this does**: Enables fail2ban to protect against brute force attacks.

### Step 6: Create Application Directory

#### 6.1 Create Directory Structure

```bash
sudo mkdir -p /opt/APP_NAME
sudo chown SERVICE_USER:SERVICE_USER /opt/APP_NAME
```

**What this does**: Creates a directory for the application and sets ownership to the SERVICE_USER user.

#### 6.2 Create SSL Directory (Optional - for domain users)

```bash
sudo mkdir -p /opt/APP_NAME/nginx/ssl
sudo chown SERVICE_USER:SERVICE_USER /opt/APP_NAME/nginx/ssl
```

**What this does**: Creates a directory for SSL certificates (only needed if using a domain).

### Step 7: Set Up Application Files

#### 7.1 Switch to SERVICE_USER User

```bash
sudo su - SERVICE_USER
```

#### 7.2 Create Environment File

```bash
cat > /opt/APP_NAME/.env << 'EOF'
# Production Environment Variables
POSTGRES_PASSWORD=your_secure_password_here
REGISTRY=your-registry.com:port
IMAGE_NAME=APP_NAME
IMAGE_TAG=latest

# Database Configuration
DATABASE_URL=postgresql://SERVICE_USER:your_secure_password_here@postgres:5432/APP_NAME

# Application Configuration
NODE_ENV=production
RUST_LOG=info
EOF
```

**What this does**: Creates environment variables for the application deployment.

**Note**: You'll need to update this with your actual registry information:
- Replace `your-registry.com:port` with your actual private registry URL
- Replace `your_secure_password_here` with a strong database password

**Registry URL Examples**:
```bash
# Private registry with custom port
REGISTRY=registry.company.com:5000

# Private registry with standard HTTPS port
REGISTRY=registry.company.com

# Private registry with custom domain
REGISTRY=docker.company.internal
```

#### 7.3 Create Deployment Script

```bash
cat > /opt/APP_NAME/deploy.sh << 'EOF'
#!/bin/bash

# Deployment script for APP_NAME
set -e

echo "Deploying APP_NAME..."

# Pull latest images
docker-compose pull

# Stop existing containers
docker-compose down

# Start new containers
docker-compose up -d

# Clean up old images
docker image prune -f

echo "Deployment complete!"
echo "Check status with: docker-compose ps"
EOF

chmod +x /opt/APP_NAME/deploy.sh
```

**What this does**: Creates a deployment script for easy application updates.

#### 7.4 Create Backup Script

```bash
cat > /opt/APP_NAME/backup.sh << 'EOF'
#!/bin/bash

# Backup script for APP_NAME
set -e

BACKUP_DIR="/opt/APP_NAME/backups"
DATE=$(date +%Y%m%d_%H%M%S)

mkdir -p $BACKUP_DIR

# Backup database
docker-compose exec -T postgres pg_dump -U SERVICE_USER APP_NAME > $BACKUP_DIR/db_backup_$DATE.sql

# Backup configuration files
tar -czf $BACKUP_DIR/config_backup_$DATE.tar.gz .env docker-compose.yml nginx/

# Keep only last 7 days of backups
find $BACKUP_DIR -name "*.sql" -mtime +7 -delete
find $BACKUP_DIR -name "*.tar.gz" -mtime +7 -delete

echo "Backup completed: $BACKUP_DIR"
EOF

chmod +x /opt/APP_NAME/backup.sh
```

**What this does**: Creates a backup script for database and configuration files.

#### 7.5 Create Monitoring Script

```bash
cat > /opt/APP_NAME/monitor.sh << 'EOF'
#!/bin/bash

# Monitoring script for APP_NAME
echo "=== APP_NAME Application Status ==="
echo

echo "Container Status:"
docker-compose ps
echo

echo "Recent Logs:"
docker-compose logs --tail=20
echo

echo "System Resources:"
echo "CPU Usage:"
top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1
echo

echo "Memory Usage:"
free -h
echo

echo "Disk Usage:"
df -h
echo

echo "Network Connections:"
netstat -tuln | grep -E ':(80|443|3000|3001)'
EOF

chmod +x /opt/APP_NAME/monitor.sh
```

**What this does**: Creates a monitoring script to check application and system status.

---

## Manual Deployment Setup

This section covers setting up the Production Linode for manual deployment from a private registry.

### Step 8: Configure Registry Authentication

#### 8.1 Login to Your Private Registry

```bash
docker login your-registry.com:port -u YOUR_REGISTRY_USERNAME -p YOUR_REGISTRY_PASSWORD
```

**What this does**: Authenticates with your private Harbor registry to allow pulling images.

**Expected output**: `Login Succeeded` message.

**If something goes wrong**:
- Verify your credentials are correct
- Check network connectivity to the registry
- Ensure your registry account has access to the images
- Verify the registry URL and port are correct

#### 8.2 Verify Authentication

```bash
# Check if you're logged in
cat ~/.docker/config.json

# Test registry connectivity
curl -v https://your-registry.com:port/v2/_catalog
```

**What this does**: Verifies that authentication was successful and registry is accessible.

**Note**: If your registry uses HTTP instead of HTTPS, use `http://` instead of `https://` in the curl command.

### Step 9: Set Up Application Deployment

#### 9.1 Create Docker Compose File

```bash
cat > /opt/APP_NAME/docker-compose.yml << 'EOF'
version: '3.8'

services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_DB: APP_NAME
      POSTGRES_USER: SERVICE_USER
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    restart: unless-stopped
    networks:
      - app_network

  backend:
    image: ${REGISTRY}/APP_NAME-backend:${IMAGE_TAG}
    environment:
      DATABASE_URL: ${DATABASE_URL}
      RUST_LOG: ${RUST_LOG}
    depends_on:
      - postgres
    restart: unless-stopped
    networks:
      - app_network

  frontend:
    image: ${REGISTRY}/APP_NAME-frontend:${IMAGE_TAG}
    environment:
      NODE_ENV: ${NODE_ENV}
    depends_on:
      - backend
    restart: unless-stopped
    networks:
      - app_network

  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./nginx/ssl:/etc/nginx/ssl:ro
    depends_on:
      - frontend
      - backend
    restart: unless-stopped
    networks:
      - app_network

volumes:
  postgres_data:

networks:
  app_network:
    driver: bridge
EOF
```

**What this does**: Creates a Docker Compose configuration for your application.

**Note**: Update the image names to match your actual registry and image names.

#### 9.2 Create Nginx Configuration

```bash
mkdir -p /opt/APP_NAME/nginx

# Choose configuration based on whether you have a domain
if [ -n "$DOMAIN_NAME" ]; then
    # Configuration with SSL for domain users
    cat > /opt/APP_NAME/nginx/nginx.conf << 'EOF'
events {
    worker_connections 1024;
}

http {
    upstream backend {
        server backend:3001;
    }

    upstream frontend {
        server frontend:3000;
    }

    # Redirect HTTP to HTTPS
    server {
        listen 80;
        server_name your-domain.com;
        return 301 https://$server_name$request_uri;
    }

    # HTTPS server
    server {
        listen 443 ssl http2;
        server_name your-domain.com;

        # SSL configuration
        ssl_certificate /etc/nginx/ssl/fullchain.pem;
        ssl_certificate_key /etc/nginx/ssl/privkey.pem;
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384;
        ssl_prefer_server_ciphers off;

        # Security headers
        add_header X-Frame-Options DENY;
        add_header X-Content-Type-Options nosniff;
        add_header X-XSS-Protection "1; mode=block";
        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";

        # API routes
        location /api/ {
            proxy_pass http://backend/;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }

        # Frontend routes
        location / {
            proxy_pass http://frontend/;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
}
EOF
else
    # Configuration without SSL for IP-only access
    cat > /opt/APP_NAME/nginx/nginx.conf << 'EOF'
events {
    worker_connections 1024;
}

http {
    upstream backend {
        server backend:3001;
    }

    upstream frontend {
        server frontend:3000;
    }

    # HTTP server (no SSL)
    server {
        listen 80;
        server_name _;

        # Security headers
        add_header X-Frame-Options DENY;
        add_header X-Content-Type-Options nosniff;
        add_header X-XSS-Protection "1; mode=block";

        # API routes
        location /api/ {
            proxy_pass http://backend/;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }

        # Frontend routes
        location / {
            proxy_pass http://frontend/;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
}
EOF
fi
```

**What this does**: Creates Nginx configuration for reverse proxy. If you have a domain, it includes SSL configuration. If not, it serves HTTP only.

**Note**: If you have a domain, replace `your-domain.com` with your actual domain name.

### Step 10: Configure SSL Certificates (Optional - Domain Users Only)

**Skip this step if you don't have a domain name.**

#### 10.1 Install SSL Certificates

```bash
sudo certbot --nginx -d your-domain.com
```

**What this does**: Obtains and installs SSL certificates from Let's Encrypt.

**Expected output**: Certificate installation confirmation and Nginx configuration update.

**If something goes wrong**:
- Verify DNS is pointing to this server
- Check that port 80 is accessible
- Ensure domain is not already using HTTPS

#### 10.2 Copy SSL Certificates

```bash
sudo cp /etc/letsencrypt/live/your-domain.com/fullchain.pem /opt/APP_NAME/nginx/ssl/
sudo cp /etc/letsencrypt/live/your-domain.com/privkey.pem /opt/APP_NAME/nginx/ssl/
sudo chown SERVICE_USER:SERVICE_USER /opt/APP_NAME/nginx/ssl/*
```

**What this does**: Copies SSL certificates to the application directory for Docker access.

### Step 11: Manual Image Management

#### 11.1 Pull Images from Private Registry

```bash
# Pull backend image
docker pull your-registry.com:port/APP_NAME-backend:latest

# Pull frontend image
docker pull your-registry.com:port/APP_NAME-frontend:latest

# Verify images were pulled
docker images | grep APP_NAME
```

**What this does**: Downloads the latest images from your private registry.

#### 11.2 Create Manual Pull and Deploy Script

```bash
cat > /opt/APP_NAME/pull-and-deploy.sh << 'EOF'
#!/bin/bash

# Manual pull and deploy script
set -e

echo "Pulling APP_NAME images from private registry..."

# Pull backend
docker pull your-registry.com:port/APP_NAME-backend:latest

# Pull frontend
docker pull your-registry.com:port/APP_NAME-frontend:latest

# Show pulled images
echo "Pulled images:"
docker images | grep APP_NAME

# Deploy application
cd /opt/APP_NAME
./deploy.sh

echo "Pull and deploy complete!"
EOF

chmod +x /opt/APP_NAME/pull-and-deploy.sh
```

**What this does**: Creates a script to pull images from your private registry and deploy the application.

#### 11.3 Create Image Update Script

```bash
cat > /opt/APP_NAME/update-images.sh << 'EOF'
#!/bin/bash

# Update images from private registry
set -e

echo "Updating images from private registry..."

# Pull latest images
docker-compose pull

# Show updated images
echo "Updated images:"
docker images | grep APP_NAME

echo "Update complete!"
echo "Run './deploy.sh' to deploy with new images"
EOF

chmod +x /opt/APP_NAME/update-images.sh
```

**What this does**: Creates a script to update images from your private registry.

---

## Testing and Verification

### Step 1: Test Application Access

**If you have a domain:**
```bash
# Test HTTP redirect to HTTPS
curl -I http://your-domain.com

# Test HTTPS access
curl -I https://your-domain.com

# Test application endpoints
curl -I https://your-domain.com/api/health
```

**If you don't have a domain (IP access only):**
```bash
# Test HTTP access via IP
curl -I http://YOUR_PRODUCTION_IP

# Test application endpoints
curl -I http://YOUR_PRODUCTION_IP/api/health
```

**Expected output**: 
- Domain users: HTTP redirect to HTTPS, HTTPS 200 OK response
- IP users: HTTP 200 OK response
- API: Application-specific response

### Step 2: Test Monitoring

```bash
cd /opt/APP_NAME
./monitor.sh
```

**Expected output**: Application status, system resources, and recent logs.

### Step 3: Test Backup

```bash
cd /opt/APP_NAME
./backup.sh
```

**Expected output**: Backup completion message with file locations.

### Step 4: Test Manual Deployment

```bash
cd /opt/APP_NAME
./pull-and-deploy.sh
```

**Expected output**: Pull progress messages, image download, and deployment completion.

---

## Troubleshooting

### Common Issues

1. **Docker permission denied**:
   ```bash
   sudo usermod -aG docker SERVICE_USER
   newgrp docker
   ```

2. **SSL certificate issues** (domain users only):
   ```bash
   sudo certbot certificates
   sudo certbot renew --dry-run
   ```

3. **Application not starting**:
   ```bash
   cd /opt/APP_NAME
   docker-compose logs
   ```

### Private Registry Issues

4. **Registry authentication fails**:
   ```bash
   # Check if you're logged in
   docker login your-registry.com:port -u YOUR_REGISTRY_USERNAME -p YOUR_REGISTRY_PASSWORD
   
   # Check authentication status
   cat ~/.docker/config.json
   
   # Test registry connectivity
   curl -v https://your-registry.com:port/v2/_catalog
   ```

5. **Images not found in private registry**:
   ```bash
   # Check if images exist in your registry
   docker pull your-registry.com:port/APP_NAME-backend:latest
   
   # List local images
   docker images | grep APP_NAME
   
   # Check registry connectivity
   curl -v https://your-registry.com:port/v2/_catalog
   
   # List available repositories
   curl -u YOUR_REGISTRY_USERNAME:YOUR_REGISTRY_PASSWORD https://your-registry.com:port/v2/_catalog
   ```

6. **Network connectivity issues**:
   ```bash
   # Test basic connectivity
   ping your-registry.com
   
   # Test port connectivity
   telnet your-registry.com port
   
   # Check DNS resolution
   nslookup your-registry.com
   
   # Test HTTPS connectivity
   curl -I https://your-registry.com:port/v2/
   
   # Test HTTP connectivity (if registry uses HTTP)
   curl -I http://your-registry.com:port/v2/
   ```

7. **Application configuration issues**:
   ```bash
   # Check environment variables
   cd /opt/APP_NAME
   cat .env
   
   # Validate docker-compose configuration
   docker-compose config
   
   # Check application logs
   docker-compose logs --tail=50
   ```

8. **Manual pull and deploy process issues**:
   ```bash
   # Check if pull script exists
   ls -la /opt/APP_NAME/pull-and-deploy.sh
   
   # Run pull with verbose output
   bash -x /opt/APP_NAME/pull-and-deploy.sh
   
   # Check for pull errors
   docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.CreatedAt}}"
   
   # Verify docker-compose.yml references correct registry
   grep -A 5 "image:" /opt/APP_NAME/docker-compose.yml
   ```

### Useful Commands

- **Check system resources**: `htop`
- **Check disk space**: `df -h`
- **Check memory usage**: `free -h`
- **Check network**: `ip addr show`
- **Check firewall**: `sudo ufw status`
- **Check logs**: `sudo journalctl -f`

### Security Best Practices

1. **Service Account**: Use dedicated `SERVICE_USER` user with limited privileges
2. **SSH Keys**: Use Ed25519 keys with proper permissions (600/700)
3. **Firewall**: Configure UFW to allow only necessary ports
4. **Fail2ban**: Protect against brute force attacks
5. **SSL/TLS**: Use Let's Encrypt certificates with automatic renewal (domain users only)
6. **Regular Backups**: Automated daily backups of database and configuration
7. **Container Isolation**: Applications run in isolated Docker containers
8. **Security Headers**: Nginx configured with security headers
9. **Registry Security**: Use secure authentication and HTTPS for private registry access

### Monitoring and Maintenance

#### Daily Monitoring

```bash
# On Production server
cd /opt/APP_NAME
./monitor.sh
```

#### Weekly Maintenance

1. **Check disk space**: `df -h`
2. **Review logs**: `docker-compose logs --tail=100`
3. **Update system**: `sudo apt update && sudo apt upgrade`
4. **Test backups**: Verify backup files exist and are recent

#### Monthly Maintenance

1. **Review security**: Check firewall rules and fail2ban status
2. **Update certificates**: Ensure SSL certificates are valid (domain users only)
3. **Clean up old images**: Remove unused Docker images
4. **Review monitoring**: Check application performance and logs
5. **Verify registry access**: Test private registry connectivity and authentication

---

## Summary

Your Production Linode is now ready for manual deployment from a private registry! The setup includes:

### Basic Features
- **Docker-based application deployment**
- **Nginx reverse proxy** with security headers
- **Automated backups and monitoring** scripts
- **Firewall and fail2ban protection** for security

### Optional Features (Domain Users)
- **SSL/TLS certificate management** with Let's Encrypt
- **HTTPS redirect** for secure web access

### Private Registry Features
- **Secure registry authentication** with username/password
- **Manual image pulling and deployment** tools
- **Image update and rollback** capabilities
- **Deployment verification and testing** scripts

### Access Methods
- **Domain users**: Access via `https://your-domain.com`
- **IP-only users**: Access via `http://YOUR_PRODUCTION_IP`

For ongoing maintenance and troubleshooting, refer to the troubleshooting section and monitoring scripts provided in this guide.