devteam/sharenet
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Split out docker registry install steps into separate document
Some checks are pending
CI/CD Pipeline (Fully Isolated DinD) / Run Tests (DinD) (push) Waiting to run
Details
CI/CD Pipeline (Fully Isolated DinD) / Build and Push Docker Images (DinD) (push) Blocked by required conditions
Details
CI/CD Pipeline (Fully Isolated DinD) / Deploy to Production (push) Blocked by required conditions
Details

Browse source
This commit is contained in:
continuist 2025-08-24 12:38:52 -04:00
parent 7525f936bf
commit 9103f53673
2 changed files with 615 additions and 383 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

445
CI_CD_PIPELINE_SETUP_GUIDE.md
View file

@ -64,12 +64,13 @@ This guide covers setting up a complete Continuous Integration/Continuous Deploy
### CI/CD Linode Features ### CI/CD Linode Features
- Forgejo Actions runner for automated builds - Forgejo Actions runner for automated builds
- **Podman-in-Podman (PiP) container** for isolated CI operations - **Podman-in-Podman (PiP) container** for isolated CI operations
- Docker Registry v2 with nginx reverse proxy for image storage - **Rootless Docker Registry v2** with host TLS reverse proxy for image storage
- **FHS-compliant directory structure** for data, certificates, and logs - **FHS-compliant directory structure** for data, certificates, and logs
- Unauthenticated pulls, authenticated pushes - **Port 443 allows unauthenticated Docker Registry v2 pulls; Port 4443 is for authenticated pushes (mTLS)**
- Automatic HTTPS with nginx - Automatic HTTPS with nginx reverse proxy
- Secure SSH communication with production - Secure SSH communication with production
- **Simplified cleanup** - just restart PiP container - **Simplified cleanup** - just restart PiP container
- **Systemd user manager** for robust rootless Podman services
### Production Linode Features ### Production Linode Features
- Podman-based application deployment - Podman-based application deployment
@ -84,6 +85,7 @@ This guide covers setting up a complete Continuous Integration/Continuous Deploy
- **Rollback capability** with image versioning - **Rollback capability** with image versioning
- **Health monitoring** and logging - **Health monitoring** and logging
- **Zero resource contention** between CI/CD and Docker Registry v2 - **Zero resource contention** between CI/CD and Docker Registry v2
- **Robust rootless services** via systemd user manager
## Security Model and User Separation ## Security Model and User Separation
@ -575,6 +577,8 @@ sudo apt install -y \
#### 1.5 Install Podman #### 1.5 Install Podman
**Note**: For detailed Podman installation and configuration, see the [Docker Registry Install Guide](Docker_Registry_Install_Guide.md#step-1-install-podman-if-not-already-installed).
```bash ```bash
# Install Podman and related tools # Install Podman and related tools
sudo apt install -y podman sudo apt install -y podman
@ -646,356 +650,21 @@ ls -la /opt/APP_NAME/registry/
- CI_SERVICE_USER owns all the files for security - CI_SERVICE_USER owns all the files for security
- Registry configuration files are now available at `/opt/APP_NAME/registry/` - Registry configuration files are now available at `/opt/APP_NAME/registry/`
### Step 4: Set Up Docker Registry v2 with nginx ### Step 4: Install Docker Registry
We'll set up a basic Docker Registry v2 with nginx as a reverse proxy, configured to allow unauthenticated pulls but require authentication for pushes. **Note**: For complete Docker Registry installation and configuration, see the [Docker Registry Install Guide](Docker_Registry_Install_Guide.md). This guide covers:
#### 4.1 Configure FHS-Compliant Registry Directories - Rootless Docker Registry v2 with host TLS reverse proxy
- mTLS authentication for secure push operations
- Unauthenticated pulls for public read access
- FHS-compliant directory structure
- Systemd user manager for robust rootless services
```bash **Quick Reference**: The Docker Registry will be accessible at:
# Create FHS-compliant directories for registry data and certificates - **Port 443**: Unauthenticated pulls (public read access)
sudo mkdir -p /var/lib/registry - **Port 4443**: Authenticated pushes (mTLS client certificate required)
sudo mkdir -p /etc/registry/certs
sudo mkdir -p /var/log/registry
sudo chown CI_SERVICE_USER:CI_SERVICE_USER /var/lib/registry
sudo chown CI_SERVICE_USER:CI_SERVICE_USER /etc/registry/certs
sudo chown CI_SERVICE_USER:CI_SERVICE_USER /var/log/registry
sudo chmod 755 /var/lib/registry
sudo chmod 755 /etc/registry/certs
sudo chmod 755 /var/log/registry
```
#### 4.2 Create Docker Registry v2 Pod Setup **Port 443 allows unauthenticated Docker Registry v2 pulls; Port 4443 is for authenticated pushes (mTLS).**
```bash
# Navigate to the cloned application directory
cd /opt/APP_NAME/registry
# Update openssl.conf with your actual IP address and registry name
sudo sed -i "s/YOUR_CI_CD_IP/YOUR_ACTUAL_IP_ADDRESS/g" /opt/APP_NAME/registry/openssl.conf
sudo sed -i "s/YOUR_REGISTRY_NAME/APP_NAME-Registry/g" /opt/APP_NAME/registry/openssl.conf
```
#### 4.2.1 Security Features Applied
The Docker Registry v2 setup includes comprehensive security hardening:
**Container Security:**
- ✅ Rootless containers (runAsUser=1000, runAsGroup=1000)
- ✅ All Linux capabilities dropped
- ✅ Privilege escalation disabled
- ✅ Read-only root filesystem with tmpfs for /tmp
- ✅ Image deletion disabled (REGISTRY_STORAGE_DELETE_ENABLED=false)
**Network Security:**
- ✅ TLS 1.2/1.3 only with modern ciphers
- ✅ HSTS headers enabled
- ✅ Rate limiting (10r/s for reads, 5r/s for writes)
- ✅ Client max body size limits (2GB)
- ✅ Registry listens only internally (no host-published port)
**Resource Limits:**
- ✅ CPU limits: 1000m for registry, 500m for nginx
- ✅ Memory limits: 1Gi for registry, 512Mi for nginx
- ✅ File descriptor limits via ulimits
**Authentication & Authorization:**
- ✅ Basic auth with htpasswd for write operations
- ✅ Container policy enforcement via containers-policy.json
- ✅ Volume mounts with read-only where possible
#### 4.2.2 Set Up Authentication and Permissions
```bash
# Create FHS-compliant authentication directory
sudo mkdir -p /etc/registry/auth
sudo chown CI_SERVICE_USER:CI_SERVICE_USER /etc/registry/auth
sudo chmod 755 /etc/registry/auth
# Create htpasswd file for nginx authentication
# Save this password somewhere safe
REGISTRY_PASSWORD="your-secure-registry-password"
# Create htpasswd file in FHS-compliant location
sudo htpasswd -cb /etc/registry/auth/.htpasswd registry-user "$REGISTRY_PASSWORD"
# Set secure permissions on htpasswd file (owner read/write only)
sudo chown CI_SERVICE_USER:CI_SERVICE_USER /etc/registry/auth/.htpasswd
sudo chmod 600 /etc/registry/auth/.htpasswd
# Set proper permissions for configuration files
sudo chown CI_SERVICE_USER:CI_SERVICE_USER /opt/APP_NAME/registry/nginx.conf
sudo chown CI_SERVICE_USER:CI_SERVICE_USER /opt/APP_NAME/registry/openssl.conf
sudo chown CI_SERVICE_USER:CI_SERVICE_USER /opt/APP_NAME/registry/registry-pod.yaml
sudo chown CI_SERVICE_USER:CI_SERVICE_USER /opt/APP_NAME/registry/containers-policy.json
sudo chmod 644 /opt/APP_NAME/registry/nginx.conf
sudo chmod 644 /opt/APP_NAME/registry/openssl.conf
sudo chmod 644 /opt/APP_NAME/registry/registry-pod.yaml
sudo chmod 644 /opt/APP_NAME/registry/containers-policy.json
```
#### 4.2.3 Create FHS-Compliant Directory Structure
```bash
# Create FHS-compliant certificate directory structure
sudo mkdir -p /etc/registry/certs/private
sudo mkdir -p /etc/registry/certs/requests
sudo mkdir -p /etc/registry/certs/ca
sudo mkdir -p /var/lib/registry/data
# Set proper ownership for registry data directory
sudo chown -R CI_SERVICE_USER:CI_SERVICE_USER /var/lib/registry/data
# Set proper ownership and permissions for certificate subdirectories
sudo chown -R CI_SERVICE_USER:CI_SERVICE_USER /etc/registry/certs/private
sudo chown -R CI_SERVICE_USER:CI_SERVICE_USER /etc/registry/certs/requests
sudo chown -R CI_SERVICE_USER:CI_SERVICE_USER /etc/registry/certs/ca
sudo chmod 700 /etc/registry/certs/private # Private keys - restricted access
sudo chmod 755 /etc/registry/certs/requests # Certificate requests
sudo chmod 755 /etc/registry/certs/ca # CA certificates
sudo chmod 755 /var/lib/registry/data # Registry data
```
#### 4.2.4 Generate TLS Certificate and Install for Podman
**Generate Self-Signed Certificate:**
```bash
# 1. Generate self-signed certificate with proper CA chain using FHS-compliant structure
cd /etc/registry/certs
# Generate CA private key in private subdirectory
sudo -u CI_SERVICE_USER openssl genrsa -out private/ca.key 4096
# Generate CA certificate in ca subdirectory
sudo -u CI_SERVICE_USER openssl req -new -x509 -key private/ca.key \
-out ca/ca.crt \
-days 365 \
-subj "/O=YOUR_ORGANIZATION/CN=APP_NAME-Registry-CA"
# Generate server private key in private subdirectory
sudo -u CI_SERVICE_USER openssl genrsa -out private/registry.key 4096
# Copy and use the project's OpenSSL configuration file
sudo cp /opt/APP_NAME/registry/openssl.conf /etc/registry/certs/requests/
sudo chown CI_SERVICE_USER:CI_SERVICE_USER /etc/registry/certs/requests/openssl.conf
# Generate server certificate signing request in requests subdirectory
sudo -u CI_SERVICE_USER openssl req -new -key private/registry.key \
-out requests/registry.csr \
-config requests/openssl.conf
# Sign server certificate with CA
sudo -u CI_SERVICE_USER openssl x509 -req -in requests/registry.csr \
-CA ca/ca.crt -CAkey private/ca.key -CAcreateserial \
-out registry.crt \
-days 365 \
-extensions req_ext \
-extfile requests/openssl.conf
# Set proper FHS-compliant permissions
sudo chmod 600 private/ca.key private/registry.key # Private keys - owner read/write only
sudo chmod 644 ca/ca.crt registry.crt # Certificates - world readable
sudo chmod 644 requests/registry.csr requests/openssl.conf # Requests - world readable
# Verify certificate creation
sudo -u CI_SERVICE_USER openssl x509 -in /etc/registry/certs/registry.crt -text -noout | grep -E "(Subject:|DNS:|IP Address:)"
# Create required directories for containers
sudo mkdir -p /var/log/nginx
sudo mkdir -p /tmp/registry-tmp /tmp/nginx-tmp
sudo chown CI_SERVICE_USER:CI_SERVICE_USER /var/log/nginx /tmp/registry-tmp /tmp/nginx-tmp
sudo chmod 755 /var/log/nginx /tmp/registry-tmp /tmp/nginx-tmp
# 2. Install CA certificate in system trust store (for curl, wget, etc.)
sudo cp /etc/registry/certs/ca/ca.crt /usr/local/share/ca-certificates/registry-ca.crt
# This step should complete with "1 added, 0 removed". If it does not, there could be a problem with the certificate you generated, or you might already have a certificate in the trust store
sudo update-ca-certificates
```
---
**Note:**
- Replace `YOUR_ACTUAL_IP_ADDRESS` with your server's IP address.
---
#### 4.5 Set Up Systemd Service for Docker Registry v2
```bash
# Create system-wide Podman configuration
sudo mkdir -p /etc/containers
sudo tee /etc/containers/registries.conf > /dev/null << 'EOF'
unqualified-search-registries = ["docker.io"]
EOF
# Set proper permissions for system-wide Podman config
sudo chown CI_SERVICE_USER:CI_SERVICE_USER /etc/containers/registries.conf
sudo chmod 644 /etc/containers/registries.conf
# Enable lingering for CI_SERVICE_USER to allow systemd services
sudo loginctl enable-linger CI_SERVICE_USER
# Create Podman rootless directories outside home
sudo mkdir -p /var/tmp/podman-$(id -u CI_SERVICE_USER)/{root,run,tmp,xdg-data,xdg-config}
sudo chown -R CI_SERVICE_USER:CI_SERVICE_USER /var/tmp/podman-$(id -u CI_SERVICE_USER)
sudo chmod 755 /var/tmp/podman-$(id -u CI_SERVICE_USER)
# Create runtime directory for user
sudo mkdir -p /run/user/$(id -u CI_SERVICE_USER)/podman-run
sudo chown CI_SERVICE_USER:CI_SERVICE_USER /run/user/$(id -u CI_SERVICE_USER)/podman-run
sudo chmod 755 /run/user/$(id -u CI_SERVICE_USER)/podman-run
# Initialize Podman with rootless configuration (no home directory access)
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman system migrate"
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman info"
# Install systemd service and configuration files from repository
sudo cp /opt/APP_NAME/registry/docker-registry.service /etc/systemd/system/docker-registry.service
sudo cp /opt/APP_NAME/registry/containers-policy.json /etc/containers/policy.json
# Set proper permissions for policy file
sudo chown CI_SERVICE_USER:CI_SERVICE_USER /etc/containers/policy.json
sudo chmod 644 /etc/containers/policy.json
# Replace APP_NAME placeholder with actual application name
sudo sed -i "s/APP_NAME/YOUR_ACTUAL_APP_NAME/g" /etc/systemd/system/docker-registry.service
# Replace CI_SERVICE_USER placeholder with actual CI service user name
sudo sed -i "s/CI_SERVICE_USER/YOUR_ACTUAL_CI_SERVICE_USER/g" /etc/systemd/system/docker-registry.service
# Note: The service is configured to use rootless Podman with all state outside home directory
# - PODMAN_ROOT=/var/tmp/podman-%u/root
# - PODMAN_RUNROOT=/run/user/%u/podman-run
# - PODMAN_TMPDIR=/var/tmp/podman-%u/tmp
# - XDG_DATA_HOME=/var/tmp/podman-%u/xdg-data
# - XDG_CONFIG_HOME=/var/tmp/podman-%u/xdg-config
# Configure firewall for Docker Registry v2 ports
sudo ufw allow 443/tcp # Docker Registry via nginx (public read access)
sudo ufw allow 4443/tcp # Docker Registry via nginx (authenticated operations)
# Enable and start Docker Registry v2 service
sudo systemctl daemon-reload
sudo systemctl enable docker-registry.service
sudo systemctl start docker-registry.service
# Verify services are running
sudo systemctl status docker-registry.service
# Check service logs for any issues
sudo journalctl -u docker-registry.service -f --no-pager -n 50
# Verify Podman is using non-home paths
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman info --format '{{.Store.GraphRoot}} {{.Store.RunRoot}}'"
```
#### 4.6 Verify Docker Registry v2 Service
```bash
# Check that the service is running properly
sudo systemctl status docker-registry.service
# Check that pods are running
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman pod ps"
# Check nginx logs
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman logs registry-pod-nginx"
# Check Registry logs
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman logs registry-pod-registry"
```
#### 4.7 Test Registry Setup
**For Option A (Self-signed certificates):**
```bash
# Switch to CI_SERVICE_USER for testing (CI_SERVICE_USER runs CI pipeline and Podman operations)
sudo su - CI_SERVICE_USER
# Navigate to the application directory
cd /opt/APP_NAME
# Test authenticated push using the project's registry configuration (port 4443)
echo "your-secure-registry-password" | env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman login YOUR_ACTUAL_IP_ADDRESS:4443 -u registry-user --password-stdin
# Create and push test image to authenticated endpoint
echo "FROM alpine:latest" > /tmp/test.Dockerfile
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman build -f /tmp/test.Dockerfile -t YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest /tmp
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman push YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest
# Test unauthenticated pull from standard HTTPS endpoint (port 443)
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman logout YOUR_ACTUAL_IP_ADDRESS:4443
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman pull YOUR_ACTUAL_IP_ADDRESS/APP_NAME/test:latest
# Test that unauthorized push to authenticated endpoint is blocked
echo "FROM alpine:latest" > /tmp/unauthorized.Dockerfile
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman build -f /tmp/unauthorized.Dockerfile -t YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/unauthorized:latest /tmp
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman push YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/unauthorized:latest
# Expected: This should fail with authentication error
# Clean up
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman rmi YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest 2>/dev/null || true
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman rmi YOUR_ACTUAL_IP_ADDRESS/APP_NAME/test:latest 2>/dev/null || true
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman rmi YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/unauthorized:latest 2>/dev/null || true
exit
```
**Expected behavior**:
- ✅ Push requires authentication with `registry-user` credentials on port 4443
- ✅ Pull works without authentication (public read access) on port 443
- ✅ Unauthorized push is blocked on authenticated endpoint
- ✅ Registry accessible at `https://YOUR_ACTUAL_IP_ADDRESS:4443` for authenticated operations
- ✅ Registry accessible at `https://YOUR_ACTUAL_IP_ADDRESS` for unauthenticated pulls
- ✅ Proper FHS-compliant certificate structure with secure permissions
**Troubleshooting TLS Errors:**
If you get a TLS error like `remote error: tls: internal error` when using self-signed certificates, verify the certificate installation and Docker configuration:
```bash
# Verify the certificate was installed correctly in system trust store
ls -la /usr/local/share/ca-certificates/registry-ca.crt
# Verify certificate chain is valid
openssl verify -CAfile /etc/registry/certs/ca/ca.crt /etc/registry/certs/registry.crt
# Test the certificate connection
openssl s_client -connect YOUR_ACTUAL_IP_ADDRESS:4443 -servername YOUR_ACTUAL_IP_ADDRESS < /dev/null
# Verify nginx is using the correct certificates
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman exec registry-pod-nginx ls -la /etc/registry/certs/"
# If issues persist, restart the Docker Registry v2 service to reload certificates
sudo systemctl restart docker-registry.service
# Wait for Docker Registry to restart, then test again
sleep 10
# Test Podman login to authenticated endpoint
echo "your-secure-registry-password" | env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman login YOUR_ACTUAL_IP_ADDRESS:4443 -u registry-user --password-stdin
```
**Certificate Structure Summary:**
The project uses a two-port configuration:
- **Port 443**: Unauthenticated pulls (public read access)
- **Port 4443**: Authenticated pushes (registry-user credentials required)
**FHS-Compliant Certificate Locations:**
- **Private Keys**: `/etc/registry/certs/private/` (mode 600)
- **CA Certificates**: `/etc/registry/certs/ca/` (mode 644)
- **Certificate Requests**: `/etc/registry/certs/requests/` (mode 644)
- **Server Certificates**: `/etc/registry/certs/` (mode 644)
- **System Trust Store**: `/usr/local/share/ca-certificates/registry-ca.crt`
### Step 5: Install Forgejo Actions Runner ### Step 5: Install Forgejo Actions Runner
@ -1162,29 +831,27 @@ sudo chmod 600 /etc/forgejo-runner/.runner
- Registers the runner with your Forgejo instance - Registers the runner with your Forgejo instance
- Sets up the runner with appropriate labels for Ubuntu and Docker environments - Sets up the runner with appropriate labels for Ubuntu and Docker environments
**Step 5: Create and Enable Systemd Service** **Step 5: Create and Enable Systemd User Service**
```bash ```bash
sudo tee /etc/systemd/system/forgejo-runner.service > /dev/null << 'EOF' sudo tee /etc/systemd/user/forgejo-runner.service > /dev/null << 'EOF'
[Unit] [Unit]
Description=Forgejo Actions Runner Description=Forgejo Actions Runner
After=network.target After=network.target
[Service] [Service]
Type=simple Type=simple
User=CI_SERVICE_USER
WorkingDirectory=/etc/forgejo-runner WorkingDirectory=/etc/forgejo-runner
ExecStart=/usr/bin/forgejo-runner daemon ExecStart=/usr/bin/forgejo-runner daemon
Restart=always Restart=always
RestartSec=10 RestartSec=10
[Install] [Install]
WantedBy=multi-user.target WantedBy=default.target
EOF EOF
# Enable the service # Enable the service via user manager
sudo systemctl daemon-reload sudo systemctl --global enable forgejo-runner.service
sudo systemctl enable forgejo-runner.service
``` ```
**What this does**: **What this does**:
@ -1270,10 +937,22 @@ podman exec ci-pip podman version
# Navigate to the application directory # Navigate to the application directory
cd /opt/APP_NAME cd /opt/APP_NAME
# Login to Docker Registry v2 from within PiP (using authenticated port 4443) # Configure PiP to use client certificates for mTLS authentication
echo "your-registry-password" | podman exec -i ci-pip podman login YOUR_CI_CD_IP:4443 -u registry-user --password-stdin podman exec ci-pip mkdir -p /etc/containers
podman exec ci-pip tee /etc/containers/registries.conf > /dev/null << 'EOF'
[[registry]]
location = "YOUR_CI_CD_IP:4443"
client_cert = "/etc/registry/certs/clients/client.crt"
client_key = "/etc/registry/certs/private/client.key"
EOF
# Test Docker Registry v2 connectivity from PiP # Copy client certificates to PiP container
podman cp /etc/registry/certs/clients/client.crt ci-pip:/etc/registry/certs/clients/
podman cp /etc/registry/certs/private/client.key ci-pip:/etc/registry/certs/private/
podman exec ci-pip chmod 600 /etc/registry/certs/private/client.key
podman exec ci-pip chmod 644 /etc/registry/certs/clients/client.crt
# Test Docker Registry v2 connectivity from PiP (using mTLS on port 4443)
podman exec ci-pip podman pull alpine:latest podman exec ci-pip podman pull alpine:latest
podman exec ci-pip podman tag alpine:latest YOUR_CI_CD_IP:4443/APP_NAME/test:latest podman exec ci-pip podman tag alpine:latest YOUR_CI_CD_IP:4443/APP_NAME/test:latest
podman exec ci-pip podman push YOUR_CI_CD_IP:4443/APP_NAME/test:latest podman exec ci-pip podman push YOUR_CI_CD_IP:4443/APP_NAME/test:latest
@ -1318,32 +997,31 @@ ls -la /tmp/ci-workspace
### FHS-Compliant Directory Structure ### FHS-Compliant Directory Structure
The Docker Registry setup now follows the Filesystem Hierarchy Standard (FHS) for better organization and security: The Docker Registry setup follows the Filesystem Hierarchy Standard (FHS) for better organization and security. For complete details, see the [Docker Registry Install Guide](Docker_Registry_Install_Guide.md).
**Application Files** (in `/opt/APP_NAME/registry/`): **Application Files** (in `/opt/APP_NAME/registry/`):
- `registry-pod.yaml` - Kubernetes Pod manifest for Docker Registry v2 and nginx
- `nginx.conf` - nginx reverse proxy configuration from project repository
- `openssl.conf` - OpenSSL configuration for certificate generation from project repository
- `containers-policy.json` - Container policy for image signature verification - `containers-policy.json` - Container policy for image signature verification
- `docker-registry.service` - Systemd service file for Docker Registry v2 - `openssl.conf` - OpenSSL configuration for certificate generation from project repository
**System Files** (FHS-compliant locations): **System Files** (FHS-compliant locations):
- `/var/lib/registry/data/` - Registry data storage - `/var/lib/registry/` - Registry data storage
- `/etc/registry/certs/` - SSL/TLS certificate hierarchy: - `/etc/registry/certs/` - SSL/TLS certificate hierarchy:
- `/etc/registry/certs/private/` - Private keys (mode 600) - `/etc/registry/certs/private/` - Private keys (mode 600)
- `/etc/registry/certs/ca/` - CA certificates (mode 644) - `/etc/registry/certs/ca/` - CA certificates (mode 644)
- `/etc/registry/certs/clients/` - Client CA certificates (mode 640)
- `/etc/registry/certs/requests/` - Certificate requests and configs (mode 644) - `/etc/registry/certs/requests/` - Certificate requests and configs (mode 644)
- `/etc/registry/certs/registry.crt` - Server certificate (mode 644) - `/etc/registry/certs/registry.crt` - Server certificate (mode 644)
- `/etc/registry/auth/.htpasswd` - nginx authentication file (mode 600) - `/etc/registry/nginx.conf` - nginx reverse proxy configuration (mode 644)
- `/etc/systemd/system/docker-registry.service` - Systemd service configuration - `/etc/systemd/user/registry.service` - Systemd user service for rootless registry
- `/var/log/registry/` - Registry and nginx logs - `/etc/systemd/system/registry-proxy.service` - Systemd system service for TLS proxy
- `/var/log/nginx/` - nginx proxy logs
**Benefits of FHS Compliance**: **Benefits of FHS Compliance**:
- **Data persistence**: Registry data stored in `/var/lib/registry/data/` survives container restarts - **Data persistence**: Registry data stored in `/var/lib/registry/` survives container restarts
- **Certificate security**: Hierarchical certificate structure with proper permissions - **Certificate security**: Hierarchical certificate structure with proper permissions
- **Authentication security**: htpasswd file stored in `/etc/registry/auth/` with restrictive permissions (600) - **mTLS authentication**: Client certificates for secure push operations
- **Service management**: Systemd service for proper startup, shutdown, and monitoring - **Service management**: Systemd services for proper startup, shutdown, and monitoring
- **Separation of concerns**: Private keys isolated from public certificates, auth isolated from configs - **Separation of concerns**: Private keys isolated from public certificates
- **Log management**: Logs in `/var/log/nginx/` for centralized logging - **Log management**: Logs in `/var/log/nginx/` for centralized logging
- **Configuration separation**: App configs in app directory, system data in system directories - **Configuration separation**: App configs in app directory, system data in system directories
- **Policy enforcement**: Container policies for image signature verification - **Policy enforcement**: Container policies for image signature verification
@ -2128,8 +1806,8 @@ Go to your Forgejo repository and add these secrets in **Settings → Secrets an
- `PROD_SERVICE_USER`: The production service user name (e.g., `prod-service`) - `PROD_SERVICE_USER`: The production service user name (e.g., `prod-service`)
- `APP_NAME`: Your application name (e.g., `sharenet`) - `APP_NAME`: Your application name (e.g., `sharenet`)
- `POSTGRES_PASSWORD`: A strong password for the PostgreSQL database - `POSTGRES_PASSWORD`: A strong password for the PostgreSQL database
- `REGISTRY_USER`: Docker Registry v2 username for CI operations (e.g., `registry-user`) - `REGISTRY_CLIENT_CERT`: Path to client certificate for mTLS authentication (e.g., `/etc/registry/certs/clients/client.crt`)
- `REGISTRY_PASSWORD`: Docker Registry v2 password for CI operations (the password you set in the nginx configuration, default: `your-secure-registry-password`) - `REGISTRY_CLIENT_KEY`: Path to client private key for mTLS authentication (e.g., `/etc/registry/certs/private/client.key`)
- `REGISTRY_PUSH_URL`: Docker Registry v2 URL for authenticated pushes (e.g., `YOUR_CI_CD_IP:4443`) - `REGISTRY_PUSH_URL`: Docker Registry v2 URL for authenticated pushes (e.g., `YOUR_CI_CD_IP:4443`)
- `REGISTRY_PULL_URL`: Docker Registry v2 URL for unauthenticated pulls (e.g., `YOUR_CI_CD_IP`) - `REGISTRY_PULL_URL`: Docker Registry v2 URL for unauthenticated pulls (e.g., `YOUR_CI_CD_IP`)
@ -2290,9 +1968,11 @@ You have successfully set up a complete CI/CD pipeline with:
- ✅ **Health monitoring** and logging - ✅ **Health monitoring** and logging
- ✅ **Backup and cleanup** automation - ✅ **Backup and cleanup** automation
- ✅ **Security hardening** with proper user separation - ✅ **Security hardening** with proper user separation
- ✅ **SSL/TLS support** with self-signed certificates - ✅ **SSL/TLS support** with self-signed certificates and mTLS authentication
- ✅ **Zero resource contention** between CI/CD and Docker Registry - ✅ **Zero resource contention** between CI/CD and Docker Registry
- ✅ **FHS-compliant directory structure** for better organization and security - ✅ **FHS-compliant directory structure** for better organization and security
- ✅ **Robust rootless services** via systemd user manager
- ✅ **Host TLS reverse proxy** with rootless registry isolation
Your application is now ready for continuous deployment with proper security, monitoring, and maintenance procedures in place! Your application is now ready for continuous deployment with proper security, monitoring, and maintenance procedures in place!
@ -2306,19 +1986,18 @@ sudo rm -rf /opt/APP_NAME/registry/openssl.conf
sudo rm -rf /opt/APP_NAME/registry/certs/requests/openssl.conf sudo rm -rf /opt/APP_NAME/registry/certs/requests/openssl.conf
# Note: DO NOT remove these files as they are needed for operation: # Note: DO NOT remove these files as they are needed for operation:
# - /opt/APP_NAME/registry/registry-pod.yaml
# - /opt/APP_NAME/registry/nginx.conf
# - /opt/APP_NAME/registry/containers-policy.json # - /opt/APP_NAME/registry/containers-policy.json
# - /opt/APP_NAME/registry/docker-registry.service # - /etc/registry/certs/ (contains all certificates and keys)
# - /etc/registry/auth/.htpasswd (contains the actual secrets) # - /etc/systemd/user/registry.service
# - /etc/systemd/system/docker-registry.service # - /etc/systemd/system/registry-proxy.service
# - /etc/registry/nginx.conf
``` ```
**Security Note**: The `.htpasswd` file in `/etc/registry/auth/.htpasswd` contains sensitive authentication data and should be: **Security Note**: The certificate files in `/etc/registry/certs/` contain sensitive authentication data and should be:
- **Backed up securely** if needed for disaster recovery - **Backed up securely** if needed for disaster recovery
- **Never committed to version control** - **Never committed to version control**
- **Protected with proper permissions** (600 - owner read/write only) - **Protected with proper permissions** (600 for private keys, 640 for client CA)
- **Rotated regularly** by updating the password and regenerating the htpasswd file - **Rotated regularly** by regenerating certificates and updating client configurations
### Step 7.4 CI/CD Workflow Summary Table ### Step 7.4 CI/CD Workflow Summary Table

553
Docker_Registry_Install_Guide.md Normal file
View file

@ -0,0 +1,553 @@
# Docker Registry Install Guide
This guide covers setting up a rootless Docker Registry v2 with host TLS reverse proxy for secure image storage. The registry runs rootless via Podman with loopback-only access, while a host nginx reverse proxy provides TLS termination and authentication.
## Architecture Overview
```
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ External │ │ Host TLS │ │ Rootless │
│ Clients │ │ Reverse Proxy │ │ Registry │
│ │ │ (Nginx) │ │ (Podman) │
│ │ │ │ │ │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
│ │ │
└─── HTTPS :443 ────────┼───────────────────────┘
│
└─── HTTPS :4443 ───────┘
(mTLS Authentication)
```
## Security Model
- **Rootless Registry**: Runs as `CI_SERVICE_USER` via systemd user manager
- **Host TLS Proxy**: Runs as `registry-proxy` with minimal capabilities
- **Loopback Isolation**: Registry only accessible via `127.0.0.1:5000`
- **mTLS Authentication**: Client certificates required for push operations
- **No $HOME I/O**: All Podman state outside user home directory
## Prerequisites
- Ubuntu 24.04 LTS with root access
- Podman installed and configured for rootless operation
- `CI_SERVICE_USER` created and configured
- Basic familiarity with Linux commands and SSH
## Step 1: Install Podman (if not already installed)
### 1.5 Install Podman
```bash
# Install Podman and related tools
sudo apt install -y podman
# Verify installation
podman --version
# Configure Podman for rootless operation (optional but recommended)
echo 'kernel.unprivileged_userns_clone=1' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
# Configure subuid/subgid for CI_SERVICE_USER
sudo usermod --add-subuids 100000-165535 CI_SERVICE_USER
sudo usermod --add-subgids 100000-165535 CI_SERVICE_USER
```
## Step 2: Set Up Rootless Docker Registry v2 with Host TLS Reverse Proxy
### 2.1 Create System-wide Podman Configuration
```bash
# Create system-wide Podman configuration
sudo mkdir -p /etc/containers
sudo tee /etc/containers/registries.conf > /dev/null << 'EOF'
unqualified-search-registries = ["docker.io"]
EOF
# Set proper permissions for system-wide Podman config (root-owned)
sudo chown root:root /etc/containers/registries.conf
sudo chmod 644 /etc/containers/registries.conf
```
### 2.2 Enable User Manager and Create Directories
```bash
# Enable lingering for CI_SERVICE_USER to allow systemd user manager
sudo loginctl enable-linger CI_SERVICE_USER
# Create Podman rootless directories outside home
sudo mkdir -p /var/tmp/podman-$(id -u CI_SERVICE_USER)/{root,run,tmp,xdg-data,xdg-config}
sudo chown -R CI_SERVICE_USER:CI_SERVICE_USER /var/tmp/podman-$(id -u CI_SERVICE_USER)
sudo chmod 755 /var/tmp/podman-$(id -u CI_SERVICE_USER)
# Create runtime directory for user
sudo mkdir -p /run/user/$(id -u CI_SERVICE_USER)/podman-run
sudo chown CI_SERVICE_USER:CI_SERVICE_USER /run/user/$(id -u CI_SERVICE_USER)/podman-run
sudo chmod 755 /run/user/$(id -u CI_SERVICE_USER)/podman-run
# Initialize Podman with rootless configuration (no home directory access)
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman system migrate"
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman info"
```
### 2.3 Create Registry User and Configuration Directories
```bash
# Create registry-proxy user for TLS reverse proxy
sudo useradd -r -s /bin/false registry-proxy
# Create registry configuration directories
sudo mkdir -p /etc/registry/certs/private /etc/registry/certs/clients
sudo chown root:root /etc/registry/certs/private /etc/registry/certs/clients
sudo chmod 750 /etc/registry/certs/private
sudo chmod 755 /etc/registry/certs/clients
```
### 2.4 Install Systemd Services
#### 2.4.1 Rootless Registry Service
```bash
# Install systemd user service for rootless registry
sudo tee /etc/systemd/user/registry.service > /dev/null << 'EOF'
[Unit]
Description=Rootless Docker Registry v2
After=network.target
[Service]
Type=simple
Environment=PODMAN_ROOT=/var/tmp/podman-%U/root
Environment=PODMAN_RUNROOT=/run/user/%U/podman-run
Environment=PODMAN_TMPDIR=/var/tmp/podman-%U/tmp
Environment=XDG_DATA_HOME=/var/tmp/podman-%U/xdg-data
Environment=XDG_CONFIG_HOME=/var/tmp/podman-%U/xdg-config
Environment=REGISTRY_HTTP_ADDR=0.0.0.0:5000
Environment=REGISTRY_STORAGE_DELETE_ENABLED=false
ExecStart=/usr/bin/podman run --rm --name registry \
--root=${PODMAN_ROOT} --runroot=${PODMAN_RUNROOT} --tmpdir=${PODMAN_TMPDIR} --events-backend=file \
-p 127.0.0.1:5000:5000 \
--cap-drop=ALL --read-only --tmpfs /tmp:size=64m --security-opt=no-new-privileges \
-v /var/lib/registry:/var/lib/registry:Z \
registry:2
ExecStop=/usr/bin/podman stop --root=${PODMAN_ROOT} --runroot=${PODMAN_RUNROOT} --tmpdir=${PODMAN_TMPDIR} --events-backend=file registry
Restart=always
RestartSec=10
[Install]
WantedBy=default.target
EOF
```
#### 2.4.2 TLS Reverse Proxy Service
```bash
# Install systemd system service for TLS reverse proxy
sudo tee /etc/systemd/system/registry-proxy.service > /dev/null << 'EOF'
[Unit]
Description=Docker Registry TLS Reverse Proxy
After=network.target registry.service
Requires=registry.service
[Service]
Type=simple
User=registry-proxy
Group=registry-proxy
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
IPAddressDeny=any
IPAddressAllow=127.0.0.1/8 ::1
ExecStart=/usr/sbin/nginx -c /etc/registry/nginx.conf
ExecReload=/usr/sbin/nginx -c /etc/registry/nginx.conf -s reload
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF
```
### 2.5 Create Nginx Configuration
```bash
# Create nginx configuration for TLS reverse proxy
sudo tee /etc/registry/nginx.conf > /dev/null << 'EOF'
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Rate limiting
limit_req_zone $binary_remote_addr zone=read_limit:10m rate=10r/s;
limit_req_zone $binary_remote_addr zone=write_limit:10m rate=5r/s;
# Upstream registry
upstream registry {
server 127.0.0.1:5000;
}
# Port 443: Unauthenticated pulls only
server {
listen 443 ssl http2;
server_name _;
# TLS configuration
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
# Security headers
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
add_header X-Frame-Options DENY always;
add_header X-Content-Type-Options nosniff always;
# Certificate files
ssl_certificate /etc/registry/certs/registry.crt;
ssl_certificate_key /etc/registry/certs/private/registry.key;
# Block write methods (PUT, PATCH, POST, DELETE)
if ($request_method ~ ^(PUT|PATCH|POST|DELETE)$) {
return 403;
}
# Rate limiting for reads
limit_req zone=read_limit burst=20 nodelay;
# Proxy to registry
location / {
proxy_pass http://registry;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
# Port 4443: Authenticated pushes only (mTLS)
server {
listen 4443 ssl http2;
server_name _;
# TLS configuration
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
# mTLS configuration
ssl_client_certificate /etc/registry/certs/clients/ca.crt;
ssl_verify_client on;
# Security headers
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
add_header X-Frame-Options DENY always;
add_header X-Content-Type-Options nosniff always;
# Certificate files
ssl_certificate /etc/registry/certs/registry.crt;
ssl_certificate_key /etc/registry/certs/private/registry.key;
# Rate limiting for writes
limit_req zone=write_limit burst=10 nodelay;
# Proxy to registry (full API access)
location / {
proxy_pass http://registry;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
}
EOF
# Set proper permissions for nginx config (root-owned)
sudo chown root:root /etc/registry/nginx.conf
sudo chmod 644 /etc/registry/nginx.conf
```
### 2.6 Install Container Policy
```bash
# Install container policy file (root-owned)
sudo cp /opt/APP_NAME/registry/containers-policy.json /etc/containers/policy.json
sudo chown root:root /etc/containers/policy.json
sudo chmod 644 /etc/containers/policy.json
```
## Step 3: Generate TLS Certificates
### 3.1 Generate Server and Client CA Certificates
```bash
# 1. Generate server CA and certificates with proper FHS-compliant structure
cd /etc/registry/certs
# Generate server CA private key in private subdirectory
sudo -u CI_SERVICE_USER openssl genrsa -out private/ca.key 4096
# Generate server CA certificate in ca subdirectory
sudo -u CI_SERVICE_USER openssl req -new -x509 -key private/ca.key \
-out ca/ca.crt \
-days 365 \
-subj "/O=YOUR_ORGANIZATION/CN=APP_NAME-Registry-CA"
# Generate server private key in private subdirectory
sudo -u CI_SERVICE_USER openssl genrsa -out private/registry.key 4096
# Copy and use the project's OpenSSL configuration file
sudo cp /opt/APP_NAME/registry/openssl.conf /etc/registry/certs/requests/
sudo chown CI_SERVICE_USER:CI_SERVICE_USER /etc/registry/certs/requests/openssl.conf
# Generate server certificate signing request in requests subdirectory
sudo -u CI_SERVICE_USER openssl req -new -key private/registry.key \
-out requests/registry.csr \
-config requests/openssl.conf
# Sign server certificate with CA
sudo -u CI_SERVICE_USER openssl x509 -req -in requests/registry.csr \
-CA ca/ca.crt -CAkey private/ca.key -CAcreateserial \
-out registry.crt \
-days 365 \
-extensions req_ext \
-extfile requests/openssl.conf
# 2. Generate client CA for mTLS authentication
# Generate client CA private key
sudo -u CI_SERVICE_USER openssl genrsa -out private/client-ca.key 4096
# Generate client CA certificate
sudo -u CI_SERVICE_USER openssl req -new -x509 -key private/client-ca.key \
-out clients/ca.crt \
-days 365 \
-subj "/O=YOUR_ORGANIZATION/CN=APP_NAME-Client-CA"
# Generate client certificate for CI operations
sudo -u CI_SERVICE_USER openssl genrsa -out private/client.key 4096
# Generate client certificate signing request
sudo -u CI_SERVICE_USER openssl req -new -key private/client.key \
-out requests/client.csr \
-subj "/O=YOUR_ORGANIZATION/CN=APP_NAME-CI-Client"
# Sign client certificate with client CA
sudo -u CI_SERVICE_USER openssl x509 -req -in requests/client.csr \
-CA clients/ca.crt -CAkey private/client-ca.key -CAcreateserial \
-out clients/client.crt \
-days 365
# Set proper FHS-compliant permissions
sudo chmod 600 private/ca.key private/registry.key private/client-ca.key private/client.key # Private keys - owner read/write only
sudo chmod 644 ca/ca.crt registry.crt clients/ca.crt clients/client.crt # Certificates - world readable
sudo chmod 644 requests/registry.csr requests/client.csr requests/openssl.conf # Requests - world readable
# Make client CA readable by registry-proxy user for nginx
sudo chown root:registry-proxy /etc/registry/certs/clients/ca.crt
sudo chmod 640 /etc/registry/certs/clients/ca.crt
# Verify certificate creation
sudo -u CI_SERVICE_USER openssl x509 -in /etc/registry/certs/registry.crt -text -noout | grep -E "(Subject:|DNS:|IP Address:)"
# Create required directories for containers
sudo mkdir -p /var/log/nginx
sudo mkdir -p /tmp/registry-tmp /tmp/nginx-tmp
sudo chown CI_SERVICE_USER:CI_SERVICE_USER /var/log/nginx /tmp/registry-tmp /tmp/nginx-tmp
sudo chmod 755 /var/log/nginx /tmp/registry-tmp /tmp/nginx-tmp
# 3. Install server CA certificate in system trust store (for curl, wget, etc.)
sudo cp /etc/registry/certs/ca/ca.crt /usr/local/share/ca-certificates/registry-ca.crt
# This step should complete with "1 added, 0 removed". If it does not, there could be a problem with the certificate you generated, or you might already have a certificate in the trust store
sudo update-ca-certificates
```
## Step 4: Configure Firewall and Start Services
### 4.1 Configure Firewall
```bash
# Configure firewall for Docker Registry v2 ports
sudo ufw allow 443/tcp # Docker Registry via nginx (unauthenticated pulls)
sudo ufw allow 4443/tcp # Docker Registry via nginx (authenticated pushes with mTLS)
# Note: Port 5000 is NOT opened - registry runs loopback-only
```
### 4.2 Enable and Start Services
```bash
# Enable and start services
sudo systemctl --global enable registry.service
sudo systemctl enable registry-proxy.service
sudo -u CI_SERVICE_USER -H sh -lc 'systemctl --user daemon-reload; systemctl --user enable --now registry.service'
sudo systemctl start registry-proxy.service
```
## Step 5: Verify Installation
### 5.1 Check Service Status
```bash
# Check that the services are running properly
sudo -u CI_SERVICE_USER -H sh -lc 'systemctl --user status registry.service'
sudo systemctl status registry-proxy.service
# Check that registry container is running
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman ps"
# Check registry logs
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman logs registry"
# Check nginx proxy logs
sudo journalctl -u registry-proxy.service -f --no-pager -n 50
# Verify Podman is using non-home paths
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman info --format '{{.Store.GraphRoot}} {{.Store.RunRoot}}'"
```
### 5.2 Test Registry Functionality
#### 5.2.1 Test mTLS Authentication
```bash
# Switch to CI_SERVICE_USER for testing (CI_SERVICE_USER runs CI pipeline and Podman operations)
sudo su - CI_SERVICE_USER
# Navigate to the application directory
cd /opt/APP_NAME
# Configure Podman to use client certificates for mTLS authentication
mkdir -p ~/.config/containers
cat > ~/.config/containers/registries.conf << EOF
[[registry]]
location = "YOUR_ACTUAL_IP_ADDRESS:4443"
client_cert = "/etc/registry/certs/clients/client.crt"
client_key = "/etc/registry/certs/private/client.key"
EOF
# Test authenticated push using mTLS (port 4443)
echo "FROM alpine:latest" > /tmp/test.Dockerfile
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman build -f /tmp/test.Dockerfile -t YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest /tmp
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman push YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest
# Test unauthenticated pull from standard HTTPS endpoint (port 443)
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman pull YOUR_ACTUAL_IP_ADDRESS/APP_NAME/test:latest
# Test that unauthorized push to port 443 is blocked
echo "FROM alpine:latest" > /tmp/unauthorized.Dockerfile
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman build -f /tmp/unauthorized.Dockerfile -t YOUR_ACTUAL_IP_ADDRESS/APP_NAME/unauthorized:latest /tmp
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman push YOUR_ACTUAL_IP_ADDRESS/APP_NAME/unauthorized:latest
# Expected: This should fail with 403 Forbidden
# Clean up
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman rmi YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest 2>/dev/null || true
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman rmi YOUR_ACTUAL_IP_ADDRESS/APP_NAME/test:latest 2>/dev/null || true
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman rmi YOUR_ACTUAL_IP_ADDRESS/APP_NAME/unauthorized:latest 2>/dev/null || true
exit
```
**Expected behavior**:
- ✅ Push requires mTLS client certificate authentication on port 4443
- ✅ Pull works without authentication (public read access) on port 443
- ✅ Unauthorized push is blocked on port 443 (403 Forbidden)
- ✅ Registry accessible at `https://YOUR_ACTUAL_IP_ADDRESS:4443` for authenticated operations (mTLS)
- ✅ Registry accessible at `https://YOUR_ACTUAL_IP_ADDRESS` for unauthenticated pulls
- ✅ Proper FHS-compliant certificate structure with secure permissions
- ✅ Port 443 allows unauthenticated Docker Registry v2 pulls; Port 4443 is for authenticated pushes (mTLS)
### 5.3 Troubleshooting TLS Errors
If you get a TLS error like `remote error: tls: internal error` when using self-signed certificates, verify the certificate installation and configuration:
```bash
# Verify the certificate was installed correctly in system trust store
ls -la /usr/local/share/ca-certificates/registry-ca.crt
# Verify certificate chain is valid
openssl verify -CAfile /etc/registry/certs/ca/ca.crt /etc/registry/certs/registry.crt
# Test the certificate connection
openssl s_client -connect YOUR_ACTUAL_IP_ADDRESS:4443 -servername YOUR_ACTUAL_IP_ADDRESS < /dev/null
# Test mTLS connection with client certificate
openssl s_client -connect YOUR_ACTUAL_IP_ADDRESS:4443 -servername YOUR_ACTUAL_IP_ADDRESS \
-cert /etc/registry/certs/clients/client.crt \
-key /etc/registry/certs/private/client.key < /dev/null
# Verify nginx is using the correct certificates
sudo ls -la /etc/registry/certs/registry.crt /etc/registry/certs/private/registry.key /etc/registry/certs/clients/ca.crt
# If issues persist, restart the services to reload certificates
sudo -u CI_SERVICE_USER -H sh -lc 'systemctl --user restart registry.service'
sudo systemctl restart registry-proxy.service
# Wait for services to restart, then test again
sleep 10
# Test mTLS authentication
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman pull YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest
```
## Certificate Structure Summary
The project uses a two-port configuration:
- **Port 443**: Unauthenticated pulls (public read access)
- **Port 4443**: Authenticated pushes (mTLS client certificate required)
**FHS-Compliant Certificate Locations:**
- **Private Keys**: `/etc/registry/certs/private/` (mode 600)
- **CA Certificates**: `/etc/registry/certs/ca/` (mode 644)
- **Client CA**: `/etc/registry/certs/clients/` (mode 640, readable by registry-proxy)
- **Certificate Requests**: `/etc/registry/certs/requests/` (mode 644)
- **Server Certificates**: `/etc/registry/certs/` (mode 644)
- **System Trust Store**: `/usr/local/share/ca-certificates/registry-ca.crt`
## Security Notes
**Certificate Security**: The certificate files in `/etc/registry/certs/` contain sensitive authentication data and should be:
- **Backed up securely** if needed for disaster recovery
- **Never committed to version control**
- **Protected with proper permissions** (600 for private keys, 640 for client CA)
- **Rotated regularly** by regenerating certificates and updating client configurations
**Files to Preserve**: DO NOT remove these files as they are needed for operation:
- `/opt/APP_NAME/registry/containers-policy.json`
- `/etc/registry/certs/` (contains all certificates and keys)
- `/etc/systemd/user/registry.service`
- `/etc/systemd/system/registry-proxy.service`
- `/etc/registry/nginx.conf`
## Architecture Benefits
1. **🔒 Enhanced Security**: Keys stay on host, minimal capabilities, strict isolation
2. **🛡️ mTLS Authentication**: Stronger than basic auth for push operations
3. **🚫 No Resource Contention**: Registry isolated from other operations
4. **📁 No $HOME I/O**: All state outside user home directory
5. **🔧 Robust Boot**: User manager guaranteed to exist at boot time
6. **🎯 Clear Separation**: Unauthenticated pulls vs authenticated pushes
7. **⚡ Low-Port Binding**: Via minimal capability (`CAP_NET_BIND_SERVICE`)
## 🎉 Congratulations!
You have successfully set up a rootless Docker Registry v2 with host TLS reverse proxy featuring:
- ✅ **Rootless registry** via systemd user manager
- ✅ **Host TLS reverse proxy** with minimal capabilities
- ✅ **mTLS authentication** for secure push operations
- ✅ **Unauthenticated pulls** for public read access
- ✅ **FHS-compliant directory structure** for better organization and security
- ✅ **No $HOME I/O** policy with all state outside user home directory
- ✅ **Port 443 allows unauthenticated Docker Registry v2 pulls; Port 4443 is for authenticated pushes (mTLS)**
Your Docker Registry is now ready for secure image storage with proper authentication and isolation!