From 9103f536739d649ed6392379e317c3b747fbc837 Mon Sep 17 00:00:00 2001 From: continuist Date: Sun, 24 Aug 2025 12:38:52 -0400 Subject: [PATCH] Split out docker registry install steps into separate document --- CI_CD_PIPELINE_SETUP_GUIDE.md | 445 ++++--------------------- Docker_Registry_Install_Guide.md | 553 +++++++++++++++++++++++++++++++ 2 files changed, 615 insertions(+), 383 deletions(-) create mode 100644 Docker_Registry_Install_Guide.md diff --git a/CI_CD_PIPELINE_SETUP_GUIDE.md b/CI_CD_PIPELINE_SETUP_GUIDE.md index dd4492f..312a37c 100644 --- a/CI_CD_PIPELINE_SETUP_GUIDE.md +++ b/CI_CD_PIPELINE_SETUP_GUIDE.md @@ -64,12 +64,13 @@ This guide covers setting up a complete Continuous Integration/Continuous Deploy ### CI/CD Linode Features - Forgejo Actions runner for automated builds - **Podman-in-Podman (PiP) container** for isolated CI operations -- Docker Registry v2 with nginx reverse proxy for image storage +- **Rootless Docker Registry v2** with host TLS reverse proxy for image storage - **FHS-compliant directory structure** for data, certificates, and logs -- Unauthenticated pulls, authenticated pushes -- Automatic HTTPS with nginx +- **Port 443 allows unauthenticated Docker Registry v2 pulls; Port 4443 is for authenticated pushes (mTLS)** +- Automatic HTTPS with nginx reverse proxy - Secure SSH communication with production - **Simplified cleanup** - just restart PiP container +- **Systemd user manager** for robust rootless Podman services ### Production Linode Features - Podman-based application deployment @@ -84,6 +85,7 @@ This guide covers setting up a complete Continuous Integration/Continuous Deploy - **Rollback capability** with image versioning - **Health monitoring** and logging - **Zero resource contention** between CI/CD and Docker Registry v2 +- **Robust rootless services** via systemd user manager ## Security Model and User Separation @@ -575,6 +577,8 @@ sudo apt install -y \ #### 1.5 Install Podman +**Note**: For detailed Podman installation and configuration, see the [Docker Registry Install Guide](Docker_Registry_Install_Guide.md#step-1-install-podman-if-not-already-installed). + ```bash # Install Podman and related tools sudo apt install -y podman @@ -646,356 +650,21 @@ ls -la /opt/APP_NAME/registry/ - CI_SERVICE_USER owns all the files for security - Registry configuration files are now available at `/opt/APP_NAME/registry/` -### Step 4: Set Up Docker Registry v2 with nginx +### Step 4: Install Docker Registry -We'll set up a basic Docker Registry v2 with nginx as a reverse proxy, configured to allow unauthenticated pulls but require authentication for pushes. +**Note**: For complete Docker Registry installation and configuration, see the [Docker Registry Install Guide](Docker_Registry_Install_Guide.md). This guide covers: -#### 4.1 Configure FHS-Compliant Registry Directories +- Rootless Docker Registry v2 with host TLS reverse proxy +- mTLS authentication for secure push operations +- Unauthenticated pulls for public read access +- FHS-compliant directory structure +- Systemd user manager for robust rootless services -```bash -# Create FHS-compliant directories for registry data and certificates -sudo mkdir -p /var/lib/registry -sudo mkdir -p /etc/registry/certs -sudo mkdir -p /var/log/registry -sudo chown CI_SERVICE_USER:CI_SERVICE_USER /var/lib/registry -sudo chown CI_SERVICE_USER:CI_SERVICE_USER /etc/registry/certs -sudo chown CI_SERVICE_USER:CI_SERVICE_USER /var/log/registry -sudo chmod 755 /var/lib/registry -sudo chmod 755 /etc/registry/certs -sudo chmod 755 /var/log/registry -``` +**Quick Reference**: The Docker Registry will be accessible at: +- **Port 443**: Unauthenticated pulls (public read access) +- **Port 4443**: Authenticated pushes (mTLS client certificate required) -#### 4.2 Create Docker Registry v2 Pod Setup - -```bash -# Navigate to the cloned application directory -cd /opt/APP_NAME/registry - -# Update openssl.conf with your actual IP address and registry name -sudo sed -i "s/YOUR_CI_CD_IP/YOUR_ACTUAL_IP_ADDRESS/g" /opt/APP_NAME/registry/openssl.conf -sudo sed -i "s/YOUR_REGISTRY_NAME/APP_NAME-Registry/g" /opt/APP_NAME/registry/openssl.conf -``` - - - -#### 4.2.1 Security Features Applied - -The Docker Registry v2 setup includes comprehensive security hardening: - -**Container Security:** -- ✅ Rootless containers (runAsUser=1000, runAsGroup=1000) -- ✅ All Linux capabilities dropped -- ✅ Privilege escalation disabled -- ✅ Read-only root filesystem with tmpfs for /tmp -- ✅ Image deletion disabled (REGISTRY_STORAGE_DELETE_ENABLED=false) - -**Network Security:** -- ✅ TLS 1.2/1.3 only with modern ciphers -- ✅ HSTS headers enabled -- ✅ Rate limiting (10r/s for reads, 5r/s for writes) -- ✅ Client max body size limits (2GB) -- ✅ Registry listens only internally (no host-published port) - -**Resource Limits:** -- ✅ CPU limits: 1000m for registry, 500m for nginx -- ✅ Memory limits: 1Gi for registry, 512Mi for nginx -- ✅ File descriptor limits via ulimits - -**Authentication & Authorization:** -- ✅ Basic auth with htpasswd for write operations -- ✅ Container policy enforcement via containers-policy.json -- ✅ Volume mounts with read-only where possible - -#### 4.2.2 Set Up Authentication and Permissions - -```bash -# Create FHS-compliant authentication directory -sudo mkdir -p /etc/registry/auth -sudo chown CI_SERVICE_USER:CI_SERVICE_USER /etc/registry/auth -sudo chmod 755 /etc/registry/auth - -# Create htpasswd file for nginx authentication -# Save this password somewhere safe -REGISTRY_PASSWORD="your-secure-registry-password" - -# Create htpasswd file in FHS-compliant location -sudo htpasswd -cb /etc/registry/auth/.htpasswd registry-user "$REGISTRY_PASSWORD" - -# Set secure permissions on htpasswd file (owner read/write only) -sudo chown CI_SERVICE_USER:CI_SERVICE_USER /etc/registry/auth/.htpasswd -sudo chmod 600 /etc/registry/auth/.htpasswd - -# Set proper permissions for configuration files -sudo chown CI_SERVICE_USER:CI_SERVICE_USER /opt/APP_NAME/registry/nginx.conf -sudo chown CI_SERVICE_USER:CI_SERVICE_USER /opt/APP_NAME/registry/openssl.conf -sudo chown CI_SERVICE_USER:CI_SERVICE_USER /opt/APP_NAME/registry/registry-pod.yaml -sudo chown CI_SERVICE_USER:CI_SERVICE_USER /opt/APP_NAME/registry/containers-policy.json -sudo chmod 644 /opt/APP_NAME/registry/nginx.conf -sudo chmod 644 /opt/APP_NAME/registry/openssl.conf -sudo chmod 644 /opt/APP_NAME/registry/registry-pod.yaml -sudo chmod 644 /opt/APP_NAME/registry/containers-policy.json -``` - -#### 4.2.3 Create FHS-Compliant Directory Structure - -```bash -# Create FHS-compliant certificate directory structure -sudo mkdir -p /etc/registry/certs/private -sudo mkdir -p /etc/registry/certs/requests -sudo mkdir -p /etc/registry/certs/ca -sudo mkdir -p /var/lib/registry/data - -# Set proper ownership for registry data directory -sudo chown -R CI_SERVICE_USER:CI_SERVICE_USER /var/lib/registry/data - -# Set proper ownership and permissions for certificate subdirectories -sudo chown -R CI_SERVICE_USER:CI_SERVICE_USER /etc/registry/certs/private -sudo chown -R CI_SERVICE_USER:CI_SERVICE_USER /etc/registry/certs/requests -sudo chown -R CI_SERVICE_USER:CI_SERVICE_USER /etc/registry/certs/ca -sudo chmod 700 /etc/registry/certs/private # Private keys - restricted access -sudo chmod 755 /etc/registry/certs/requests # Certificate requests -sudo chmod 755 /etc/registry/certs/ca # CA certificates -sudo chmod 755 /var/lib/registry/data # Registry data - - -``` - -#### 4.2.4 Generate TLS Certificate and Install for Podman - -**Generate Self-Signed Certificate:** - -```bash -# 1. Generate self-signed certificate with proper CA chain using FHS-compliant structure -cd /etc/registry/certs - -# Generate CA private key in private subdirectory -sudo -u CI_SERVICE_USER openssl genrsa -out private/ca.key 4096 - -# Generate CA certificate in ca subdirectory -sudo -u CI_SERVICE_USER openssl req -new -x509 -key private/ca.key \ - -out ca/ca.crt \ - -days 365 \ - -subj "/O=YOUR_ORGANIZATION/CN=APP_NAME-Registry-CA" - -# Generate server private key in private subdirectory -sudo -u CI_SERVICE_USER openssl genrsa -out private/registry.key 4096 - -# Copy and use the project's OpenSSL configuration file -sudo cp /opt/APP_NAME/registry/openssl.conf /etc/registry/certs/requests/ -sudo chown CI_SERVICE_USER:CI_SERVICE_USER /etc/registry/certs/requests/openssl.conf - -# Generate server certificate signing request in requests subdirectory -sudo -u CI_SERVICE_USER openssl req -new -key private/registry.key \ - -out requests/registry.csr \ - -config requests/openssl.conf - -# Sign server certificate with CA -sudo -u CI_SERVICE_USER openssl x509 -req -in requests/registry.csr \ - -CA ca/ca.crt -CAkey private/ca.key -CAcreateserial \ - -out registry.crt \ - -days 365 \ - -extensions req_ext \ - -extfile requests/openssl.conf - -# Set proper FHS-compliant permissions -sudo chmod 600 private/ca.key private/registry.key # Private keys - owner read/write only -sudo chmod 644 ca/ca.crt registry.crt # Certificates - world readable -sudo chmod 644 requests/registry.csr requests/openssl.conf # Requests - world readable - -# Verify certificate creation -sudo -u CI_SERVICE_USER openssl x509 -in /etc/registry/certs/registry.crt -text -noout | grep -E "(Subject:|DNS:|IP Address:)" - -# Create required directories for containers -sudo mkdir -p /var/log/nginx -sudo mkdir -p /tmp/registry-tmp /tmp/nginx-tmp -sudo chown CI_SERVICE_USER:CI_SERVICE_USER /var/log/nginx /tmp/registry-tmp /tmp/nginx-tmp -sudo chmod 755 /var/log/nginx /tmp/registry-tmp /tmp/nginx-tmp - -# 2. Install CA certificate in system trust store (for curl, wget, etc.) -sudo cp /etc/registry/certs/ca/ca.crt /usr/local/share/ca-certificates/registry-ca.crt - -# This step should complete with "1 added, 0 removed". If it does not, there could be a problem with the certificate you generated, or you might already have a certificate in the trust store -sudo update-ca-certificates -``` - ---- - -**Note:** -- Replace `YOUR_ACTUAL_IP_ADDRESS` with your server's IP address. - ---- - -#### 4.5 Set Up Systemd Service for Docker Registry v2 - -```bash -# Create system-wide Podman configuration -sudo mkdir -p /etc/containers -sudo tee /etc/containers/registries.conf > /dev/null << 'EOF' -unqualified-search-registries = ["docker.io"] -EOF - -# Set proper permissions for system-wide Podman config -sudo chown CI_SERVICE_USER:CI_SERVICE_USER /etc/containers/registries.conf -sudo chmod 644 /etc/containers/registries.conf - -# Enable lingering for CI_SERVICE_USER to allow systemd services -sudo loginctl enable-linger CI_SERVICE_USER - -# Create Podman rootless directories outside home -sudo mkdir -p /var/tmp/podman-$(id -u CI_SERVICE_USER)/{root,run,tmp,xdg-data,xdg-config} -sudo chown -R CI_SERVICE_USER:CI_SERVICE_USER /var/tmp/podman-$(id -u CI_SERVICE_USER) -sudo chmod 755 /var/tmp/podman-$(id -u CI_SERVICE_USER) - -# Create runtime directory for user -sudo mkdir -p /run/user/$(id -u CI_SERVICE_USER)/podman-run -sudo chown CI_SERVICE_USER:CI_SERVICE_USER /run/user/$(id -u CI_SERVICE_USER)/podman-run -sudo chmod 755 /run/user/$(id -u CI_SERVICE_USER)/podman-run - -# Initialize Podman with rootless configuration (no home directory access) -sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman system migrate" -sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman info" - -# Install systemd service and configuration files from repository -sudo cp /opt/APP_NAME/registry/docker-registry.service /etc/systemd/system/docker-registry.service -sudo cp /opt/APP_NAME/registry/containers-policy.json /etc/containers/policy.json - -# Set proper permissions for policy file -sudo chown CI_SERVICE_USER:CI_SERVICE_USER /etc/containers/policy.json -sudo chmod 644 /etc/containers/policy.json - -# Replace APP_NAME placeholder with actual application name -sudo sed -i "s/APP_NAME/YOUR_ACTUAL_APP_NAME/g" /etc/systemd/system/docker-registry.service - -# Replace CI_SERVICE_USER placeholder with actual CI service user name -sudo sed -i "s/CI_SERVICE_USER/YOUR_ACTUAL_CI_SERVICE_USER/g" /etc/systemd/system/docker-registry.service - -# Note: The service is configured to use rootless Podman with all state outside home directory -# - PODMAN_ROOT=/var/tmp/podman-%u/root -# - PODMAN_RUNROOT=/run/user/%u/podman-run -# - PODMAN_TMPDIR=/var/tmp/podman-%u/tmp -# - XDG_DATA_HOME=/var/tmp/podman-%u/xdg-data -# - XDG_CONFIG_HOME=/var/tmp/podman-%u/xdg-config - -# Configure firewall for Docker Registry v2 ports -sudo ufw allow 443/tcp # Docker Registry via nginx (public read access) -sudo ufw allow 4443/tcp # Docker Registry via nginx (authenticated operations) - -# Enable and start Docker Registry v2 service -sudo systemctl daemon-reload -sudo systemctl enable docker-registry.service -sudo systemctl start docker-registry.service - -# Verify services are running -sudo systemctl status docker-registry.service - -# Check service logs for any issues -sudo journalctl -u docker-registry.service -f --no-pager -n 50 - -# Verify Podman is using non-home paths -sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman info --format '{{.Store.GraphRoot}} {{.Store.RunRoot}}'" -``` - -#### 4.6 Verify Docker Registry v2 Service - -```bash -# Check that the service is running properly -sudo systemctl status docker-registry.service - -# Check that pods are running -sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman pod ps" - -# Check nginx logs -sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman logs registry-pod-nginx" - -# Check Registry logs -sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman logs registry-pod-registry" -``` - -#### 4.7 Test Registry Setup - -**For Option A (Self-signed certificates):** - -```bash -# Switch to CI_SERVICE_USER for testing (CI_SERVICE_USER runs CI pipeline and Podman operations) -sudo su - CI_SERVICE_USER - -# Navigate to the application directory -cd /opt/APP_NAME - -# Test authenticated push using the project's registry configuration (port 4443) -echo "your-secure-registry-password" | env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman login YOUR_ACTUAL_IP_ADDRESS:4443 -u registry-user --password-stdin - -# Create and push test image to authenticated endpoint -echo "FROM alpine:latest" > /tmp/test.Dockerfile -env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman build -f /tmp/test.Dockerfile -t YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest /tmp -env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman push YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest - -# Test unauthenticated pull from standard HTTPS endpoint (port 443) -env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman logout YOUR_ACTUAL_IP_ADDRESS:4443 -env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman pull YOUR_ACTUAL_IP_ADDRESS/APP_NAME/test:latest - -# Test that unauthorized push to authenticated endpoint is blocked -echo "FROM alpine:latest" > /tmp/unauthorized.Dockerfile -env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman build -f /tmp/unauthorized.Dockerfile -t YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/unauthorized:latest /tmp -env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman push YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/unauthorized:latest -# Expected: This should fail with authentication error - -# Clean up -env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman rmi YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest 2>/dev/null || true -env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman rmi YOUR_ACTUAL_IP_ADDRESS/APP_NAME/test:latest 2>/dev/null || true -env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman rmi YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/unauthorized:latest 2>/dev/null || true -exit -``` - - - -**Expected behavior**: -- ✅ Push requires authentication with `registry-user` credentials on port 4443 -- ✅ Pull works without authentication (public read access) on port 443 -- ✅ Unauthorized push is blocked on authenticated endpoint -- ✅ Registry accessible at `https://YOUR_ACTUAL_IP_ADDRESS:4443` for authenticated operations -- ✅ Registry accessible at `https://YOUR_ACTUAL_IP_ADDRESS` for unauthenticated pulls -- ✅ Proper FHS-compliant certificate structure with secure permissions - -**Troubleshooting TLS Errors:** - -If you get a TLS error like `remote error: tls: internal error` when using self-signed certificates, verify the certificate installation and Docker configuration: - -```bash -# Verify the certificate was installed correctly in system trust store -ls -la /usr/local/share/ca-certificates/registry-ca.crt - -# Verify certificate chain is valid -openssl verify -CAfile /etc/registry/certs/ca/ca.crt /etc/registry/certs/registry.crt - -# Test the certificate connection -openssl s_client -connect YOUR_ACTUAL_IP_ADDRESS:4443 -servername YOUR_ACTUAL_IP_ADDRESS < /dev/null - -# Verify nginx is using the correct certificates -sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman exec registry-pod-nginx ls -la /etc/registry/certs/" - -# If issues persist, restart the Docker Registry v2 service to reload certificates -sudo systemctl restart docker-registry.service - -# Wait for Docker Registry to restart, then test again -sleep 10 - -# Test Podman login to authenticated endpoint -echo "your-secure-registry-password" | env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman login YOUR_ACTUAL_IP_ADDRESS:4443 -u registry-user --password-stdin -``` - -**Certificate Structure Summary:** - -The project uses a two-port configuration: -- **Port 443**: Unauthenticated pulls (public read access) -- **Port 4443**: Authenticated pushes (registry-user credentials required) - -**FHS-Compliant Certificate Locations:** -- **Private Keys**: `/etc/registry/certs/private/` (mode 600) -- **CA Certificates**: `/etc/registry/certs/ca/` (mode 644) -- **Certificate Requests**: `/etc/registry/certs/requests/` (mode 644) -- **Server Certificates**: `/etc/registry/certs/` (mode 644) -- **System Trust Store**: `/usr/local/share/ca-certificates/registry-ca.crt` +**Port 443 allows unauthenticated Docker Registry v2 pulls; Port 4443 is for authenticated pushes (mTLS).** ### Step 5: Install Forgejo Actions Runner @@ -1162,29 +831,27 @@ sudo chmod 600 /etc/forgejo-runner/.runner - Registers the runner with your Forgejo instance - Sets up the runner with appropriate labels for Ubuntu and Docker environments -**Step 5: Create and Enable Systemd Service** +**Step 5: Create and Enable Systemd User Service** ```bash -sudo tee /etc/systemd/system/forgejo-runner.service > /dev/null << 'EOF' +sudo tee /etc/systemd/user/forgejo-runner.service > /dev/null << 'EOF' [Unit] Description=Forgejo Actions Runner After=network.target [Service] Type=simple -User=CI_SERVICE_USER WorkingDirectory=/etc/forgejo-runner ExecStart=/usr/bin/forgejo-runner daemon Restart=always RestartSec=10 [Install] -WantedBy=multi-user.target +WantedBy=default.target EOF -# Enable the service -sudo systemctl daemon-reload -sudo systemctl enable forgejo-runner.service +# Enable the service via user manager +sudo systemctl --global enable forgejo-runner.service ``` **What this does**: @@ -1270,10 +937,22 @@ podman exec ci-pip podman version # Navigate to the application directory cd /opt/APP_NAME -# Login to Docker Registry v2 from within PiP (using authenticated port 4443) -echo "your-registry-password" | podman exec -i ci-pip podman login YOUR_CI_CD_IP:4443 -u registry-user --password-stdin +# Configure PiP to use client certificates for mTLS authentication +podman exec ci-pip mkdir -p /etc/containers +podman exec ci-pip tee /etc/containers/registries.conf > /dev/null << 'EOF' +[[registry]] +location = "YOUR_CI_CD_IP:4443" +client_cert = "/etc/registry/certs/clients/client.crt" +client_key = "/etc/registry/certs/private/client.key" +EOF -# Test Docker Registry v2 connectivity from PiP +# Copy client certificates to PiP container +podman cp /etc/registry/certs/clients/client.crt ci-pip:/etc/registry/certs/clients/ +podman cp /etc/registry/certs/private/client.key ci-pip:/etc/registry/certs/private/ +podman exec ci-pip chmod 600 /etc/registry/certs/private/client.key +podman exec ci-pip chmod 644 /etc/registry/certs/clients/client.crt + +# Test Docker Registry v2 connectivity from PiP (using mTLS on port 4443) podman exec ci-pip podman pull alpine:latest podman exec ci-pip podman tag alpine:latest YOUR_CI_CD_IP:4443/APP_NAME/test:latest podman exec ci-pip podman push YOUR_CI_CD_IP:4443/APP_NAME/test:latest @@ -1318,32 +997,31 @@ ls -la /tmp/ci-workspace ### FHS-Compliant Directory Structure -The Docker Registry setup now follows the Filesystem Hierarchy Standard (FHS) for better organization and security: +The Docker Registry setup follows the Filesystem Hierarchy Standard (FHS) for better organization and security. For complete details, see the [Docker Registry Install Guide](Docker_Registry_Install_Guide.md). **Application Files** (in `/opt/APP_NAME/registry/`): -- `registry-pod.yaml` - Kubernetes Pod manifest for Docker Registry v2 and nginx -- `nginx.conf` - nginx reverse proxy configuration from project repository -- `openssl.conf` - OpenSSL configuration for certificate generation from project repository - `containers-policy.json` - Container policy for image signature verification -- `docker-registry.service` - Systemd service file for Docker Registry v2 +- `openssl.conf` - OpenSSL configuration for certificate generation from project repository **System Files** (FHS-compliant locations): -- `/var/lib/registry/data/` - Registry data storage +- `/var/lib/registry/` - Registry data storage - `/etc/registry/certs/` - SSL/TLS certificate hierarchy: - `/etc/registry/certs/private/` - Private keys (mode 600) - `/etc/registry/certs/ca/` - CA certificates (mode 644) + - `/etc/registry/certs/clients/` - Client CA certificates (mode 640) - `/etc/registry/certs/requests/` - Certificate requests and configs (mode 644) - `/etc/registry/certs/registry.crt` - Server certificate (mode 644) -- `/etc/registry/auth/.htpasswd` - nginx authentication file (mode 600) -- `/etc/systemd/system/docker-registry.service` - Systemd service configuration -- `/var/log/registry/` - Registry and nginx logs +- `/etc/registry/nginx.conf` - nginx reverse proxy configuration (mode 644) +- `/etc/systemd/user/registry.service` - Systemd user service for rootless registry +- `/etc/systemd/system/registry-proxy.service` - Systemd system service for TLS proxy +- `/var/log/nginx/` - nginx proxy logs **Benefits of FHS Compliance**: -- **Data persistence**: Registry data stored in `/var/lib/registry/data/` survives container restarts +- **Data persistence**: Registry data stored in `/var/lib/registry/` survives container restarts - **Certificate security**: Hierarchical certificate structure with proper permissions -- **Authentication security**: htpasswd file stored in `/etc/registry/auth/` with restrictive permissions (600) -- **Service management**: Systemd service for proper startup, shutdown, and monitoring -- **Separation of concerns**: Private keys isolated from public certificates, auth isolated from configs +- **mTLS authentication**: Client certificates for secure push operations +- **Service management**: Systemd services for proper startup, shutdown, and monitoring +- **Separation of concerns**: Private keys isolated from public certificates - **Log management**: Logs in `/var/log/nginx/` for centralized logging - **Configuration separation**: App configs in app directory, system data in system directories - **Policy enforcement**: Container policies for image signature verification @@ -2128,8 +1806,8 @@ Go to your Forgejo repository and add these secrets in **Settings → Secrets an - `PROD_SERVICE_USER`: The production service user name (e.g., `prod-service`) - `APP_NAME`: Your application name (e.g., `sharenet`) - `POSTGRES_PASSWORD`: A strong password for the PostgreSQL database -- `REGISTRY_USER`: Docker Registry v2 username for CI operations (e.g., `registry-user`) -- `REGISTRY_PASSWORD`: Docker Registry v2 password for CI operations (the password you set in the nginx configuration, default: `your-secure-registry-password`) +- `REGISTRY_CLIENT_CERT`: Path to client certificate for mTLS authentication (e.g., `/etc/registry/certs/clients/client.crt`) +- `REGISTRY_CLIENT_KEY`: Path to client private key for mTLS authentication (e.g., `/etc/registry/certs/private/client.key`) - `REGISTRY_PUSH_URL`: Docker Registry v2 URL for authenticated pushes (e.g., `YOUR_CI_CD_IP:4443`) - `REGISTRY_PULL_URL`: Docker Registry v2 URL for unauthenticated pulls (e.g., `YOUR_CI_CD_IP`) @@ -2290,9 +1968,11 @@ You have successfully set up a complete CI/CD pipeline with: - ✅ **Health monitoring** and logging - ✅ **Backup and cleanup** automation - ✅ **Security hardening** with proper user separation -- ✅ **SSL/TLS support** with self-signed certificates +- ✅ **SSL/TLS support** with self-signed certificates and mTLS authentication - ✅ **Zero resource contention** between CI/CD and Docker Registry - ✅ **FHS-compliant directory structure** for better organization and security +- ✅ **Robust rootless services** via systemd user manager +- ✅ **Host TLS reverse proxy** with rootless registry isolation Your application is now ready for continuous deployment with proper security, monitoring, and maintenance procedures in place! @@ -2306,19 +1986,18 @@ sudo rm -rf /opt/APP_NAME/registry/openssl.conf sudo rm -rf /opt/APP_NAME/registry/certs/requests/openssl.conf # Note: DO NOT remove these files as they are needed for operation: -# - /opt/APP_NAME/registry/registry-pod.yaml -# - /opt/APP_NAME/registry/nginx.conf # - /opt/APP_NAME/registry/containers-policy.json -# - /opt/APP_NAME/registry/docker-registry.service -# - /etc/registry/auth/.htpasswd (contains the actual secrets) -# - /etc/systemd/system/docker-registry.service +# - /etc/registry/certs/ (contains all certificates and keys) +# - /etc/systemd/user/registry.service +# - /etc/systemd/system/registry-proxy.service +# - /etc/registry/nginx.conf ``` -**Security Note**: The `.htpasswd` file in `/etc/registry/auth/.htpasswd` contains sensitive authentication data and should be: +**Security Note**: The certificate files in `/etc/registry/certs/` contain sensitive authentication data and should be: - **Backed up securely** if needed for disaster recovery - **Never committed to version control** -- **Protected with proper permissions** (600 - owner read/write only) -- **Rotated regularly** by updating the password and regenerating the htpasswd file +- **Protected with proper permissions** (600 for private keys, 640 for client CA) +- **Rotated regularly** by regenerating certificates and updating client configurations ### Step 7.4 CI/CD Workflow Summary Table diff --git a/Docker_Registry_Install_Guide.md b/Docker_Registry_Install_Guide.md new file mode 100644 index 0000000..95f761c --- /dev/null +++ b/Docker_Registry_Install_Guide.md @@ -0,0 +1,553 @@ +# Docker Registry Install Guide + +This guide covers setting up a rootless Docker Registry v2 with host TLS reverse proxy for secure image storage. The registry runs rootless via Podman with loopback-only access, while a host nginx reverse proxy provides TLS termination and authentication. + +## Architecture Overview + +``` +┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ +│ External │ │ Host TLS │ │ Rootless │ +│ Clients │ │ Reverse Proxy │ │ Registry │ +│ │ │ (Nginx) │ │ (Podman) │ +│ │ │ │ │ │ +└─────────────────┘ └─────────────────┘ └─────────────────┘ + │ │ │ + │ │ │ + └─── HTTPS :443 ────────┼───────────────────────┘ + │ + └─── HTTPS :4443 ───────┘ + (mTLS Authentication) +``` + +## Security Model + +- **Rootless Registry**: Runs as `CI_SERVICE_USER` via systemd user manager +- **Host TLS Proxy**: Runs as `registry-proxy` with minimal capabilities +- **Loopback Isolation**: Registry only accessible via `127.0.0.1:5000` +- **mTLS Authentication**: Client certificates required for push operations +- **No $HOME I/O**: All Podman state outside user home directory + +## Prerequisites + +- Ubuntu 24.04 LTS with root access +- Podman installed and configured for rootless operation +- `CI_SERVICE_USER` created and configured +- Basic familiarity with Linux commands and SSH + +## Step 1: Install Podman (if not already installed) + +### 1.5 Install Podman + +```bash +# Install Podman and related tools +sudo apt install -y podman + +# Verify installation +podman --version + +# Configure Podman for rootless operation (optional but recommended) +echo 'kernel.unprivileged_userns_clone=1' | sudo tee -a /etc/sysctl.conf +sudo sysctl -p + +# Configure subuid/subgid for CI_SERVICE_USER +sudo usermod --add-subuids 100000-165535 CI_SERVICE_USER +sudo usermod --add-subgids 100000-165535 CI_SERVICE_USER +``` + +## Step 2: Set Up Rootless Docker Registry v2 with Host TLS Reverse Proxy + +### 2.1 Create System-wide Podman Configuration + +```bash +# Create system-wide Podman configuration +sudo mkdir -p /etc/containers +sudo tee /etc/containers/registries.conf > /dev/null << 'EOF' +unqualified-search-registries = ["docker.io"] +EOF + +# Set proper permissions for system-wide Podman config (root-owned) +sudo chown root:root /etc/containers/registries.conf +sudo chmod 644 /etc/containers/registries.conf +``` + +### 2.2 Enable User Manager and Create Directories + +```bash +# Enable lingering for CI_SERVICE_USER to allow systemd user manager +sudo loginctl enable-linger CI_SERVICE_USER + +# Create Podman rootless directories outside home +sudo mkdir -p /var/tmp/podman-$(id -u CI_SERVICE_USER)/{root,run,tmp,xdg-data,xdg-config} +sudo chown -R CI_SERVICE_USER:CI_SERVICE_USER /var/tmp/podman-$(id -u CI_SERVICE_USER) +sudo chmod 755 /var/tmp/podman-$(id -u CI_SERVICE_USER) + +# Create runtime directory for user +sudo mkdir -p /run/user/$(id -u CI_SERVICE_USER)/podman-run +sudo chown CI_SERVICE_USER:CI_SERVICE_USER /run/user/$(id -u CI_SERVICE_USER)/podman-run +sudo chmod 755 /run/user/$(id -u CI_SERVICE_USER)/podman-run + +# Initialize Podman with rootless configuration (no home directory access) +sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman system migrate" +sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman info" +``` + +### 2.3 Create Registry User and Configuration Directories + +```bash +# Create registry-proxy user for TLS reverse proxy +sudo useradd -r -s /bin/false registry-proxy + +# Create registry configuration directories +sudo mkdir -p /etc/registry/certs/private /etc/registry/certs/clients +sudo chown root:root /etc/registry/certs/private /etc/registry/certs/clients +sudo chmod 750 /etc/registry/certs/private +sudo chmod 755 /etc/registry/certs/clients +``` + +### 2.4 Install Systemd Services + +#### 2.4.1 Rootless Registry Service + +```bash +# Install systemd user service for rootless registry +sudo tee /etc/systemd/user/registry.service > /dev/null << 'EOF' +[Unit] +Description=Rootless Docker Registry v2 +After=network.target + +[Service] +Type=simple +Environment=PODMAN_ROOT=/var/tmp/podman-%U/root +Environment=PODMAN_RUNROOT=/run/user/%U/podman-run +Environment=PODMAN_TMPDIR=/var/tmp/podman-%U/tmp +Environment=XDG_DATA_HOME=/var/tmp/podman-%U/xdg-data +Environment=XDG_CONFIG_HOME=/var/tmp/podman-%U/xdg-config +Environment=REGISTRY_HTTP_ADDR=0.0.0.0:5000 +Environment=REGISTRY_STORAGE_DELETE_ENABLED=false +ExecStart=/usr/bin/podman run --rm --name registry \ + --root=${PODMAN_ROOT} --runroot=${PODMAN_RUNROOT} --tmpdir=${PODMAN_TMPDIR} --events-backend=file \ + -p 127.0.0.1:5000:5000 \ + --cap-drop=ALL --read-only --tmpfs /tmp:size=64m --security-opt=no-new-privileges \ + -v /var/lib/registry:/var/lib/registry:Z \ + registry:2 +ExecStop=/usr/bin/podman stop --root=${PODMAN_ROOT} --runroot=${PODMAN_RUNROOT} --tmpdir=${PODMAN_TMPDIR} --events-backend=file registry +Restart=always +RestartSec=10 + +[Install] +WantedBy=default.target +EOF +``` + +#### 2.4.2 TLS Reverse Proxy Service + +```bash +# Install systemd system service for TLS reverse proxy +sudo tee /etc/systemd/system/registry-proxy.service > /dev/null << 'EOF' +[Unit] +Description=Docker Registry TLS Reverse Proxy +After=network.target registry.service +Requires=registry.service + +[Service] +Type=simple +User=registry-proxy +Group=registry-proxy +AmbientCapabilities=CAP_NET_BIND_SERVICE +CapabilityBoundingSet=CAP_NET_BIND_SERVICE +NoNewPrivileges=yes +PrivateTmp=yes +ProtectSystem=strict +ProtectHome=yes +RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 +IPAddressDeny=any +IPAddressAllow=127.0.0.1/8 ::1 +ExecStart=/usr/sbin/nginx -c /etc/registry/nginx.conf +ExecReload=/usr/sbin/nginx -c /etc/registry/nginx.conf -s reload +Restart=always +RestartSec=10 + +[Install] +WantedBy=multi-user.target +EOF +``` + +### 2.5 Create Nginx Configuration + +```bash +# Create nginx configuration for TLS reverse proxy +sudo tee /etc/registry/nginx.conf > /dev/null << 'EOF' +events { + worker_connections 1024; +} + +http { + include /etc/nginx/mime.types; + default_type application/octet-stream; + + # Rate limiting + limit_req_zone $binary_remote_addr zone=read_limit:10m rate=10r/s; + limit_req_zone $binary_remote_addr zone=write_limit:10m rate=5r/s; + + # Upstream registry + upstream registry { + server 127.0.0.1:5000; + } + + # Port 443: Unauthenticated pulls only + server { + listen 443 ssl http2; + server_name _; + + # TLS configuration + ssl_protocols TLSv1.2 TLSv1.3; + ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384; + ssl_prefer_server_ciphers off; + ssl_session_cache shared:SSL:10m; + ssl_session_timeout 10m; + + # Security headers + add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; + add_header X-Frame-Options DENY always; + add_header X-Content-Type-Options nosniff always; + + # Certificate files + ssl_certificate /etc/registry/certs/registry.crt; + ssl_certificate_key /etc/registry/certs/private/registry.key; + + # Block write methods (PUT, PATCH, POST, DELETE) + if ($request_method ~ ^(PUT|PATCH|POST|DELETE)$) { + return 403; + } + + # Rate limiting for reads + limit_req zone=read_limit burst=20 nodelay; + + # Proxy to registry + location / { + proxy_pass http://registry; + proxy_set_header Host $http_host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } + } + + # Port 4443: Authenticated pushes only (mTLS) + server { + listen 4443 ssl http2; + server_name _; + + # TLS configuration + ssl_protocols TLSv1.2 TLSv1.3; + ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384; + ssl_prefer_server_ciphers off; + ssl_session_cache shared:SSL:10m; + ssl_session_timeout 10m; + + # mTLS configuration + ssl_client_certificate /etc/registry/certs/clients/ca.crt; + ssl_verify_client on; + + # Security headers + add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; + add_header X-Frame-Options DENY always; + add_header X-Content-Type-Options nosniff always; + + # Certificate files + ssl_certificate /etc/registry/certs/registry.crt; + ssl_certificate_key /etc/registry/certs/private/registry.key; + + # Rate limiting for writes + limit_req zone=write_limit burst=10 nodelay; + + # Proxy to registry (full API access) + location / { + proxy_pass http://registry; + proxy_set_header Host $http_host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } + } +} +EOF + +# Set proper permissions for nginx config (root-owned) +sudo chown root:root /etc/registry/nginx.conf +sudo chmod 644 /etc/registry/nginx.conf +``` + +### 2.6 Install Container Policy + +```bash +# Install container policy file (root-owned) +sudo cp /opt/APP_NAME/registry/containers-policy.json /etc/containers/policy.json +sudo chown root:root /etc/containers/policy.json +sudo chmod 644 /etc/containers/policy.json +``` + +## Step 3: Generate TLS Certificates + +### 3.1 Generate Server and Client CA Certificates + +```bash +# 1. Generate server CA and certificates with proper FHS-compliant structure +cd /etc/registry/certs + +# Generate server CA private key in private subdirectory +sudo -u CI_SERVICE_USER openssl genrsa -out private/ca.key 4096 + +# Generate server CA certificate in ca subdirectory +sudo -u CI_SERVICE_USER openssl req -new -x509 -key private/ca.key \ + -out ca/ca.crt \ + -days 365 \ + -subj "/O=YOUR_ORGANIZATION/CN=APP_NAME-Registry-CA" + +# Generate server private key in private subdirectory +sudo -u CI_SERVICE_USER openssl genrsa -out private/registry.key 4096 + +# Copy and use the project's OpenSSL configuration file +sudo cp /opt/APP_NAME/registry/openssl.conf /etc/registry/certs/requests/ +sudo chown CI_SERVICE_USER:CI_SERVICE_USER /etc/registry/certs/requests/openssl.conf + +# Generate server certificate signing request in requests subdirectory +sudo -u CI_SERVICE_USER openssl req -new -key private/registry.key \ + -out requests/registry.csr \ + -config requests/openssl.conf + +# Sign server certificate with CA +sudo -u CI_SERVICE_USER openssl x509 -req -in requests/registry.csr \ + -CA ca/ca.crt -CAkey private/ca.key -CAcreateserial \ + -out registry.crt \ + -days 365 \ + -extensions req_ext \ + -extfile requests/openssl.conf + +# 2. Generate client CA for mTLS authentication +# Generate client CA private key +sudo -u CI_SERVICE_USER openssl genrsa -out private/client-ca.key 4096 + +# Generate client CA certificate +sudo -u CI_SERVICE_USER openssl req -new -x509 -key private/client-ca.key \ + -out clients/ca.crt \ + -days 365 \ + -subj "/O=YOUR_ORGANIZATION/CN=APP_NAME-Client-CA" + +# Generate client certificate for CI operations +sudo -u CI_SERVICE_USER openssl genrsa -out private/client.key 4096 + +# Generate client certificate signing request +sudo -u CI_SERVICE_USER openssl req -new -key private/client.key \ + -out requests/client.csr \ + -subj "/O=YOUR_ORGANIZATION/CN=APP_NAME-CI-Client" + +# Sign client certificate with client CA +sudo -u CI_SERVICE_USER openssl x509 -req -in requests/client.csr \ + -CA clients/ca.crt -CAkey private/client-ca.key -CAcreateserial \ + -out clients/client.crt \ + -days 365 + +# Set proper FHS-compliant permissions +sudo chmod 600 private/ca.key private/registry.key private/client-ca.key private/client.key # Private keys - owner read/write only +sudo chmod 644 ca/ca.crt registry.crt clients/ca.crt clients/client.crt # Certificates - world readable +sudo chmod 644 requests/registry.csr requests/client.csr requests/openssl.conf # Requests - world readable + +# Make client CA readable by registry-proxy user for nginx +sudo chown root:registry-proxy /etc/registry/certs/clients/ca.crt +sudo chmod 640 /etc/registry/certs/clients/ca.crt + +# Verify certificate creation +sudo -u CI_SERVICE_USER openssl x509 -in /etc/registry/certs/registry.crt -text -noout | grep -E "(Subject:|DNS:|IP Address:)" + +# Create required directories for containers +sudo mkdir -p /var/log/nginx +sudo mkdir -p /tmp/registry-tmp /tmp/nginx-tmp +sudo chown CI_SERVICE_USER:CI_SERVICE_USER /var/log/nginx /tmp/registry-tmp /tmp/nginx-tmp +sudo chmod 755 /var/log/nginx /tmp/registry-tmp /tmp/nginx-tmp + +# 3. Install server CA certificate in system trust store (for curl, wget, etc.) +sudo cp /etc/registry/certs/ca/ca.crt /usr/local/share/ca-certificates/registry-ca.crt + +# This step should complete with "1 added, 0 removed". If it does not, there could be a problem with the certificate you generated, or you might already have a certificate in the trust store +sudo update-ca-certificates +``` + +## Step 4: Configure Firewall and Start Services + +### 4.1 Configure Firewall + +```bash +# Configure firewall for Docker Registry v2 ports +sudo ufw allow 443/tcp # Docker Registry via nginx (unauthenticated pulls) +sudo ufw allow 4443/tcp # Docker Registry via nginx (authenticated pushes with mTLS) +# Note: Port 5000 is NOT opened - registry runs loopback-only +``` + +### 4.2 Enable and Start Services + +```bash +# Enable and start services +sudo systemctl --global enable registry.service +sudo systemctl enable registry-proxy.service +sudo -u CI_SERVICE_USER -H sh -lc 'systemctl --user daemon-reload; systemctl --user enable --now registry.service' +sudo systemctl start registry-proxy.service +``` + +## Step 5: Verify Installation + +### 5.1 Check Service Status + +```bash +# Check that the services are running properly +sudo -u CI_SERVICE_USER -H sh -lc 'systemctl --user status registry.service' +sudo systemctl status registry-proxy.service + +# Check that registry container is running +sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman ps" + +# Check registry logs +sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman logs registry" + +# Check nginx proxy logs +sudo journalctl -u registry-proxy.service -f --no-pager -n 50 + +# Verify Podman is using non-home paths +sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman info --format '{{.Store.GraphRoot}} {{.Store.RunRoot}}'" +``` + +### 5.2 Test Registry Functionality + +#### 5.2.1 Test mTLS Authentication + +```bash +# Switch to CI_SERVICE_USER for testing (CI_SERVICE_USER runs CI pipeline and Podman operations) +sudo su - CI_SERVICE_USER + +# Navigate to the application directory +cd /opt/APP_NAME + +# Configure Podman to use client certificates for mTLS authentication +mkdir -p ~/.config/containers +cat > ~/.config/containers/registries.conf << EOF +[[registry]] +location = "YOUR_ACTUAL_IP_ADDRESS:4443" +client_cert = "/etc/registry/certs/clients/client.crt" +client_key = "/etc/registry/certs/private/client.key" +EOF + +# Test authenticated push using mTLS (port 4443) +echo "FROM alpine:latest" > /tmp/test.Dockerfile +env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman build -f /tmp/test.Dockerfile -t YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest /tmp +env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman push YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest + +# Test unauthenticated pull from standard HTTPS endpoint (port 443) +env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman pull YOUR_ACTUAL_IP_ADDRESS/APP_NAME/test:latest + +# Test that unauthorized push to port 443 is blocked +echo "FROM alpine:latest" > /tmp/unauthorized.Dockerfile +env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman build -f /tmp/unauthorized.Dockerfile -t YOUR_ACTUAL_IP_ADDRESS/APP_NAME/unauthorized:latest /tmp +env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman push YOUR_ACTUAL_IP_ADDRESS/APP_NAME/unauthorized:latest +# Expected: This should fail with 403 Forbidden + +# Clean up +env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman rmi YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest 2>/dev/null || true +env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman rmi YOUR_ACTUAL_IP_ADDRESS/APP_NAME/test:latest 2>/dev/null || true +env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman rmi YOUR_ACTUAL_IP_ADDRESS/APP_NAME/unauthorized:latest 2>/dev/null || true +exit +``` + +**Expected behavior**: +- ✅ Push requires mTLS client certificate authentication on port 4443 +- ✅ Pull works without authentication (public read access) on port 443 +- ✅ Unauthorized push is blocked on port 443 (403 Forbidden) +- ✅ Registry accessible at `https://YOUR_ACTUAL_IP_ADDRESS:4443` for authenticated operations (mTLS) +- ✅ Registry accessible at `https://YOUR_ACTUAL_IP_ADDRESS` for unauthenticated pulls +- ✅ Proper FHS-compliant certificate structure with secure permissions +- ✅ Port 443 allows unauthenticated Docker Registry v2 pulls; Port 4443 is for authenticated pushes (mTLS) + +### 5.3 Troubleshooting TLS Errors + +If you get a TLS error like `remote error: tls: internal error` when using self-signed certificates, verify the certificate installation and configuration: + +```bash +# Verify the certificate was installed correctly in system trust store +ls -la /usr/local/share/ca-certificates/registry-ca.crt + +# Verify certificate chain is valid +openssl verify -CAfile /etc/registry/certs/ca/ca.crt /etc/registry/certs/registry.crt + +# Test the certificate connection +openssl s_client -connect YOUR_ACTUAL_IP_ADDRESS:4443 -servername YOUR_ACTUAL_IP_ADDRESS < /dev/null + +# Test mTLS connection with client certificate +openssl s_client -connect YOUR_ACTUAL_IP_ADDRESS:4443 -servername YOUR_ACTUAL_IP_ADDRESS \ + -cert /etc/registry/certs/clients/client.crt \ + -key /etc/registry/certs/private/client.key < /dev/null + +# Verify nginx is using the correct certificates +sudo ls -la /etc/registry/certs/registry.crt /etc/registry/certs/private/registry.key /etc/registry/certs/clients/ca.crt + +# If issues persist, restart the services to reload certificates +sudo -u CI_SERVICE_USER -H sh -lc 'systemctl --user restart registry.service' +sudo systemctl restart registry-proxy.service + +# Wait for services to restart, then test again +sleep 10 + +# Test mTLS authentication +env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman pull YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest +``` + +## Certificate Structure Summary + +The project uses a two-port configuration: +- **Port 443**: Unauthenticated pulls (public read access) +- **Port 4443**: Authenticated pushes (mTLS client certificate required) + +**FHS-Compliant Certificate Locations:** +- **Private Keys**: `/etc/registry/certs/private/` (mode 600) +- **CA Certificates**: `/etc/registry/certs/ca/` (mode 644) +- **Client CA**: `/etc/registry/certs/clients/` (mode 640, readable by registry-proxy) +- **Certificate Requests**: `/etc/registry/certs/requests/` (mode 644) +- **Server Certificates**: `/etc/registry/certs/` (mode 644) +- **System Trust Store**: `/usr/local/share/ca-certificates/registry-ca.crt` + +## Security Notes + +**Certificate Security**: The certificate files in `/etc/registry/certs/` contain sensitive authentication data and should be: +- **Backed up securely** if needed for disaster recovery +- **Never committed to version control** +- **Protected with proper permissions** (600 for private keys, 640 for client CA) +- **Rotated regularly** by regenerating certificates and updating client configurations + +**Files to Preserve**: DO NOT remove these files as they are needed for operation: +- `/opt/APP_NAME/registry/containers-policy.json` +- `/etc/registry/certs/` (contains all certificates and keys) +- `/etc/systemd/user/registry.service` +- `/etc/systemd/system/registry-proxy.service` +- `/etc/registry/nginx.conf` + +## Architecture Benefits + +1. **🔒 Enhanced Security**: Keys stay on host, minimal capabilities, strict isolation +2. **🛡️ mTLS Authentication**: Stronger than basic auth for push operations +3. **🚫 No Resource Contention**: Registry isolated from other operations +4. **📁 No $HOME I/O**: All state outside user home directory +5. **🔧 Robust Boot**: User manager guaranteed to exist at boot time +6. **🎯 Clear Separation**: Unauthenticated pulls vs authenticated pushes +7. **⚡ Low-Port Binding**: Via minimal capability (`CAP_NET_BIND_SERVICE`) + +## 🎉 Congratulations! + +You have successfully set up a rootless Docker Registry v2 with host TLS reverse proxy featuring: + +- ✅ **Rootless registry** via systemd user manager +- ✅ **Host TLS reverse proxy** with minimal capabilities +- ✅ **mTLS authentication** for secure push operations +- ✅ **Unauthenticated pulls** for public read access +- ✅ **FHS-compliant directory structure** for better organization and security +- ✅ **No $HOME I/O** policy with all state outside user home directory +- ✅ **Port 443 allows unauthenticated Docker Registry v2 pulls; Port 4443 is for authenticated pushes (mTLS)** + +Your Docker Registry is now ready for secure image storage with proper authentication and isolation!