sharenet/Docker_Registry_Install_Guide.md

Docker Registry Install Guide

This guide covers setting up a rootless Docker Registry v2 with host TLS reverse proxy for secure image storage. The registry runs rootless via Podman with loopback-only access, while a host nginx reverse proxy provides TLS termination and mTLS authentication.

Architecture Overview

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   External      │    │   Host TLS      │    │   Rootless      │
│   Clients       │    │   Reverse Proxy │    │   Registry      │
│                 │    │   (Nginx)       │    │   (Podman)      │
│                 │    │                 │    │                 │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         │                       │                       │
         └─── HTTPS :443 ────────┼───────────────────────┘
         (Unauthenticated pulls) │
                                 └─── HTTPS :4443 ───────┘
                                 (mTLS Authentication)

Security Model

• Rootless Registry: Runs as CI_SERVICE_USER via systemd user manager
• Host TLS Proxy: Runs as registry-proxy with minimal capabilities
• Loopback Isolation: Registry only accessible via 127.0.0.1:5000
• mTLS Authentication: Client certificates required for push operations
• No $HOME I/O: All Podman state outside user home directory
• Port 443: Unauthenticated pulls only (GET requests)
• Port 4443: Authenticated pushes only (mTLS required)

Prerequisites

• Ubuntu 24.04 LTS with root access
• Podman installed and configured for rootless operation
• CI_SERVICE_USER created and configured
• Basic familiarity with Linux commands and SSH

Step 1: Install Podman (if not already installed)

1.1 Install Podman

# Install Podman and related tools
sudo apt install -y podman slirp4netns fuse-overlayfs nginx

# Disable stock nginx.service to avoid conflicts with hardened registry-proxy.service
sudo systemctl disable --now nginx.service

# Verify installation
podman --version

# Configure Podman for rootless operation (optional but recommended)
echo 'kernel.unprivileged_userns_clone=1' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# Configure subuid/subgid for CI_SERVICE_USER
sudo usermod --add-subuids 100000-165535 CI_SERVICE_USER
sudo usermod --add-subgids 100000-165535 CI_SERVICE_USER

Step 2: Set Up Rootless Docker Registry v2

2.1 Create System-wide Podman Configuration

# Create system-wide Podman configuration
sudo mkdir -p /etc/containers
sudo tee /etc/containers/registries.conf > /dev/null << 'EOF'
unqualified-search-registries = ["docker.io"]
short-name-mode = "enforcing"
EOF

# Set proper permissions for system-wide Podman config (root-owned)
sudo chown root:root /etc/containers/registries.conf
sudo chmod 644 /etc/containers/registries.conf

2.2 Enable User Manager and Create Directories

# Enable lingering for CI_SERVICE_USER to allow systemd user manager
sudo loginctl enable-linger CI_SERVICE_USER

# Create Podman rootless directories outside home
sudo mkdir -p /var/tmp/podman-$(id -u CI_SERVICE_USER)/{root,run,tmp,xdg-data,xdg-config}
sudo chown -R CI_SERVICE_USER:CI_SERVICE_USER /var/tmp/podman-$(id -u CI_SERVICE_USER)
sudo chmod 750 /var/tmp/podman-$(id -u CI_SERVICE_USER)

# Initialize Podman with rootless configuration (no home directory access)
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman system migrate"
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman info"

2.3 Create Registry User and Configuration Directories

# Create registry-proxy user for TLS reverse proxy
sudo useradd -r -s /bin/false registry-proxy

# Create registry configuration directories
sudo mkdir -p /etc/registry/certs/{private,clients,ca,requests}
sudo chown -R root:root /etc/registry/certs
sudo chmod 750 /etc/registry/certs/private
sudo chmod 755 /etc/registry/certs/{clients,ca,requests}

# Create registry data directory
sudo mkdir -p /var/lib/registry
sudo chown CI_SERVICE_USER:CI_SERVICE_USER /var/lib/registry
sudo chmod 750 /var/lib/registry

# Create log directory for nginx proxy
sudo install -d -o registry-proxy -g registry-proxy /var/log/registry-proxy

# Create logrotate configuration for registry proxy logs
sudo tee /etc/logrotate.d/registry-proxy > /dev/null << 'EOF'
/var/log/registry-proxy/*.log {
    daily
    rotate 14
    compress
    delaycompress
    copytruncate
    missingok
    notifempty
    create 640 registry-proxy registry-proxy
}
EOF

2.4 Install Systemd Services

2.4.1 Rootless Registry Service

# Install systemd user service for rootless registry
sudo tee /etc/systemd/user/registry.service > /dev/null << 'EOF'
[Unit]
Description=Rootless Docker Registry v2 (loopback only)
After=network-online.target
Wants=network-online.target

[Service]
Environment=PODMAN_ROOT=/var/tmp/podman-%U/root
Environment=PODMAN_RUNROOT=/run/user/%U/podman-run
Environment=PODMAN_TMPDIR=/var/tmp/podman-%U/tmp
Environment=XDG_DATA_HOME=/var/tmp/podman-%U/xdg-data
Environment=XDG_CONFIG_HOME=/var/tmp/podman-%U/xdg-config
ExecStart=/usr/bin/podman --root=${PODMAN_ROOT} --runroot=${PODMAN_RUNROOT} --tmpdir=${PODMAN_TMPDIR} --events-backend=file \
  run --rm --name registry \
  -p 127.0.0.1:5000:5000 \
  --read-only --tmpfs /tmp:size=64m --cap-drop=ALL --security-opt=no-new-privileges \
  -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \
  -e REGISTRY_STORAGE_DELETE_ENABLED=false \
  -v /var/lib/registry:/var/lib/registry:U,z \
  docker.io/library/registry@sha256:8be26f81ffea54106bae012c6f349df70f4d5e7e2ec01b143c46e2c03b9e551d
ExecStop=/usr/bin/podman --root=${PODMAN_ROOT} --runroot=${PODMAN_RUNROOT} --tmpdir=${PODMAN_TMPDIR} stop -t 10 registry
Restart=on-failure
MemoryMax=1G
CPUQuota=100%

[Install]
WantedBy=default.target
EOF

2.4.2 TLS Reverse Proxy Service

# Install systemd system service for TLS reverse proxy
sudo tee /etc/systemd/system/registry-proxy.service > /dev/null << 'EOF'
[Unit]
Description=TLS proxy for Docker Registry (443 pulls / 4443 pushes)
After=network-online.target
Wants=network-online.target

[Service]
User=registry-proxy
Group=registry-proxy
UMask=0077
RuntimeDirectory=registry-proxy
LogsDirectory=registry-proxy
ReadWritePaths=/run/registry-proxy /var/log/registry-proxy
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ProtectClock=yes
ProtectHostname=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
RestrictSUIDSGID=yes
RemoveIPC=yes
ProtectProc=invisible
ProcSubset=pid

# Allow loopback for upstream to registry and allow the server's public IP(s) for binds:
IPAddressDeny=any
IPAddressAllow=127.0.0.1
IPAddressAllow=<SERVER_IPV4>
# If using IPv6, uncomment and set:
# IPAddressAllow=<SERVER_IPV6>
LimitNOFILE=65536
ExecStartPre=/usr/sbin/nginx -t -c /etc/registry/nginx.conf
ExecStart=/usr/sbin/nginx -g 'daemon off;' -c /etc/registry/nginx.conf
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

2.5 Create Nginx Configuration

# Create nginx configuration for TLS reverse proxy
sudo tee /etc/registry/nginx.conf > /dev/null << 'EOF'
worker_processes auto;
events { worker_connections 1024; }

pid /run/registry-proxy/nginx.pid;
access_log /var/log/registry-proxy/access.log;
error_log  /var/log/registry-proxy/error.log;

http {
  server_tokens off;

  # Rate/connection limits (tune for CI bursts as needed)
  limit_req_zone $binary_remote_addr zone=reg_read:10m rate=10r/s;
  limit_req_zone $binary_remote_addr zone=reg_write:10m rate=5r/s;
  limit_conn_zone $binary_remote_addr zone=perip:10m;

  client_max_body_size 2g;

  # TLS hardening
  ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:\
ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:\
ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256';
  ssl_ecdh_curve X25519:P-256;
  ssl_prefer_server_ciphers on;
  ssl_verify_depth 2;
  ssl_session_cache shared:SSL:10m;
  ssl_session_timeout 10m;
  ssl_session_tickets off;

  # OCSP stapling is intentionally DISABLED for private CA deployments.
  # (Clients trust via installed CA cert; no public OCSP endpoint exists.)

  # Proxy settings
  proxy_http_version 1.1;
  proxy_set_header Connection "";
  proxy_request_buffering off;
  proxy_read_timeout 300s;
  proxy_temp_path        /run/registry-proxy/proxy_temp;
  client_body_temp_path  /run/registry-proxy/client_temp;
  fastcgi_temp_path      /run/registry-proxy/fastcgi_temp;
  uwsgi_temp_path        /run/registry-proxy/uwsgi_temp;
  scgi_temp_path         /run/registry-proxy/scgi_temp;

  upstream reg { server 127.0.0.1:5000; }

  # 443: unauthenticated pulls only
  server {
    listen <SERVER_IPV4>:443 ssl http2;
    # If IPv6, also:
    # listen [<SERVER_IPV6>]:443 ssl http2;
    ssl_certificate     /etc/registry/certs/registry.crt;
    ssl_certificate_key /etc/registry/certs/private/registry.key;
    ssl_protocols TLSv1.2 TLSv1.3;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

    # Hide catalog & tag listings
    location = /v2/_catalog      { return 403; }
    location ~ ^/v2/.+/tags/list { return 403; }

    location /v2/ {
      limit_req zone=reg_read burst=20 nodelay;
      limit_conn perip 20;
      proxy_pass http://reg;
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-Proto https;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      limit_except GET HEAD { return 403; }
      add_header Docker-Distribution-Api-Version "registry/2.0" always;
    }
  }

  # 4443: authenticated pushes only (mTLS)
  server {
    listen <SERVER_IPV4>:4443 ssl http2;
    # If IPv6, also:
    # listen [<SERVER_IPV6>]:4443 ssl http2;
    ssl_certificate     /etc/registry/certs/registry.crt;
    ssl_certificate_key /etc/registry/certs/private/registry.key;
    ssl_protocols TLSv1.2 TLSv1.3;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

    # mTLS client auth via private Client CA
    ssl_client_certificate /etc/registry/certs/clients/ca.crt;
    ssl_verify_client on;

    # Optional: enable client-cert revocation (publish a CRL and uncomment)
    # ssl_crl /etc/registry/certs/clients/ca.crl;

    location /v2/ {
      limit_req zone=reg_write burst=10;
      limit_conn perip 20;
      proxy_pass http://reg;
      proxy_set_header Host $host:$server_port;
      proxy_set_header X-Forwarded-Proto https;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      add_header Docker-Distribution-Api-Version "registry/2.0" always;
    }
  }
}
EOF

# Set proper permissions for nginx config (root-owned)
sudo chown root:root /etc/registry/nginx.conf
sudo chmod 644 /etc/registry/nginx.conf

**Note (Private CA):** OCSP stapling is disabled because our CA is private (no public OCSP responder). Clients trust the registry via the installed CA certificate. If migrating to a public CA later, enable stapling and add `resolver` + `ssl_trusted_certificate`.

2.6 Install Container Policy

# Install container policy file (root-owned)
sudo cp /opt/APP_NAME/registry/containers-policy.json /etc/containers/policy.json
sudo chown root:root /etc/containers/policy.json
sudo chmod 644 /etc/containers/policy.json

Step 3: Generate TLS Certificates

3.1 Generate Server and Client CA Certificates

# 1. Generate server CA and certificates with proper FHS-compliant structure
cd /etc/registry/certs

# Generate server CA private key in private subdirectory
sudo openssl genrsa -out private/ca.key 4096

# Generate server CA certificate in ca subdirectory
sudo openssl req -new -x509 -key private/ca.key \
    -out ca/ca.crt \
    -days 365 \
    -subj "/O=YOUR_ORGANIZATION/CN=APP_NAME-Registry-CA"

# Generate server private key in private subdirectory
sudo openssl genrsa -out private/registry.key 4096

# Copy and use the project's OpenSSL configuration file
sudo cp /opt/APP_NAME/registry/openssl.conf /etc/registry/certs/requests/
sudo chown root:root /etc/registry/certs/requests/openssl.conf

# Ensure OpenSSL config includes subjectAltName for REGISTRY_HOST and SERVER_IP
# The config should contain:
# [ req_ext ]
# subjectAltName = DNS:REGISTRY_HOST,IP:<SERVER_IPV4>
# If IPv6:
# subjectAltName = DNS:REGISTRY_HOST,IP:<SERVER_IPV4>,IP:<SERVER_IPV6>

# Generate server certificate signing request in requests subdirectory
sudo openssl req -new -key private/registry.key \
    -out requests/registry.csr \
    -config requests/openssl.conf

# Sign server certificate with CA
sudo openssl x509 -req -in requests/registry.csr \
    -CA ca/ca.crt -CAkey private/ca.key -CAcreateserial \
    -out registry.crt \
    -days 365 \
    -extensions req_ext \
    -extfile requests/openssl.conf

# 2. Generate client CA for mTLS authentication
# Generate client CA private key
sudo openssl genrsa -out private/client-ca.key 4096

# Generate client CA certificate
sudo openssl req -new -x509 -key private/client-ca.key \
    -out clients/ca.crt \
    -days 365 \
    -subj "/O=YOUR_ORGANIZATION/CN=APP_NAME-Client-CA"

# Generate client certificate for CI operations
sudo openssl genrsa -out private/client.key 4096

# Generate client certificate signing request
sudo openssl req -new -key private/client.key \
    -out requests/client.csr \
    -subj "/O=YOUR_ORGANIZATION/CN=APP_NAME-CI-Client"

# Sign client certificate with client CA
sudo openssl x509 -req -in requests/client.csr \
    -CA clients/ca.crt -CAkey private/client-ca.key -CAcreateserial \
    -out clients/client.crt \
    -days 365

# Set proper FHS-compliant permissions
sudo chmod 600 private/ca.key private/client-ca.key private/client.key  # Private keys - owner read/write only
sudo chmod 644 ca/ca.crt registry.crt clients/client.crt  # Certificates - world readable
sudo chmod 644 requests/registry.csr requests/client.csr requests/openssl.conf  # Requests - world readable

# Set registry key and client CA permissions for nginx proxy access
sudo chgrp registry-proxy /etc/registry/certs/private/registry.key /etc/registry/certs/clients/ca.crt
sudo chmod 640 /etc/registry/certs/private/registry.key /etc/registry/certs/clients/ca.crt

# Allow traversal into the private key directory for the proxy
sudo chgrp registry-proxy /etc/registry/certs/private
sudo chmod 750 /etc/registry/certs/private

# Ensure parent directory is traversable
sudo chmod 755 /etc/registry/certs

# Verify certificate creation
sudo openssl x509 -in /etc/registry/certs/registry.crt -text -noout | grep -E "(Subject:|DNS:|IP Address:)"

# Reminder: clients must use exactly REGISTRY_HOST (or those IPs) in pulls/pushes

# 3. Install server CA certificate in system trust store (for curl, wget, etc.)
sudo cp /etc/registry/certs/ca/ca.crt /usr/local/share/ca-certificates/registry-ca.crt

# This step should complete with "1 added, 0 removed". If it does not, there could be a problem with the certificate you generated, or you might already have a certificate in the trust store
sudo update-ca-certificates

# 4. Generate Cosign key pair for image signing
# Install Cosign (pinned + verified)
COSIGN_VERSION=v2.2.4
COSIGN_URL="https://github.com/sigstore/cosign/releases/download/${COSIGN_VERSION}/cosign-linux-amd64"
COSIGN_SHA256="<REPLACE_WITH_OFFICIAL_SHA256>"

**⚠️ IMPORTANT:** Replace `<REPLACE_WITH_OFFICIAL_SHA256>` with the official release SHA256 before running.

curl -fsSL -o /usr/local/bin/cosign "$COSIGN_URL"
echo "${COSIGN_SHA256}  /usr/local/bin/cosign" | sha256sum -c -
chmod +x /usr/local/bin/cosign

# Verify installation
cosign version --json

# Generate Cosign key pair (or use keyless OIDC in CI)
cosign generate-key-pair

# Create directory for Cosign public key distribution
sudo mkdir -p /etc/containers/keys
sudo cp cosign.pub /etc/containers/keys/org-cosign.pub
sudo chmod 644 /etc/containers/keys/org-cosign.pub

# Secure the private key (keep this safe and distribute to CI systems)
sudo chmod 600 cosign.key
echo "IMPORTANT: Secure cosign.key and distribute to CI systems for image signing"

Step 4: Configure Firewall and Start Services

4.1 Configure Firewall

# Open port 443 for unauthenticated pulls (public access)
sudo ufw allow 443/tcp

# Open port 4443 for authenticated pushes (restrict to known CIDRs)
# Example: restrict to specific networks (adjust CIDRs as needed)
sudo ufw delete allow 4443/tcp || true
sudo ufw allow from 10.0.0.0/8 to any port 4443 proto tcp
sudo ufw allow from 172.16.0.0/12 to any port 4443 proto tcp
sudo ufw allow from 192.168.0.0/16 to any port 4443 proto tcp

# For public access to 4443 (less secure), use:
# sudo ufw allow 4443/tcp

# Optional: Consider SystemCallFilter for additional hardening
# (validate exact syscall set for your distro's nginx build)
# SystemCallFilter=@system-service

# Note: Port 5000 is NOT opened - registry runs loopback-only

## Client Trust Configuration

For clients to trust your registry certificates and enforce image signatures, they should install the server CA certificate and Cosign public key:

**For pulls (port 443):**
```bash
# On client systems - use the actual FQDN/IP from your certificates
sudo mkdir -p /etc/containers/certs.d/REGISTRY_HOST
sudo cp /path/to/registry-ca.crt /etc/containers/certs.d/REGISTRY_HOST/ca.crt

# Install Cosign public key for signature verification
sudo mkdir -p /etc/containers/keys
sudo cp /path/to/org-cosign.pub /etc/containers/keys/org-cosign.pub

For pushes (port 4443, mTLS):

# On client systems - use the actual FQDN/IP from your certificates
sudo mkdir -p /etc/containers/certs.d/REGISTRY_HOST:4443
sudo cp /path/to/registry-ca.crt /etc/containers/certs.d/REGISTRY_HOST:4443/ca.crt
sudo cp /path/to/client.crt /etc/containers/certs.d/REGISTRY_HOST:4443/client.cert
sudo cp /path/to/client.key /etc/containers/certs.d/REGISTRY_HOST:4443/client.key

# Install Cosign public key for signature verification
sudo mkdir -p /etc/containers/keys
sudo cp /path/to/org-cosign.pub /etc/containers/keys/org-cosign.pub

Note: Replace REGISTRY_HOST with the actual FQDN or IP address that matches your certificate's Subject Alternative Name (SAN). For pushes, both the server CA certificate and client certificate/key are required for mTLS authentication. The Cosign public key is required for signature verification on both ports.

Best practice: pull/deploy by digest, not tag. Example: podman pull REGISTRY_HOST/namespace/image@sha256:<digest>

Security hardening notes:

• :U flag: ID-maps the host directory into the container's user namespace to avoid permission drift and tighten isolation.
• ssl_session_tickets off: Avoids long-lived TLS ticket key reuse unless you manage ticket key rotation.
• Client policy enforcement: The registry doesn't enforce signatures—clients do. Ensure containers-policy.json (with sigstoreSigned for REGISTRY_HOST and reject for docker.io) is deployed on every pulling host.

Security Model

This setup implements a multi-layered security approach:

1. mTLS on port 4443: Controls who can push to the registry
2. Cosign signatures: Controls what content is trusted - all images must be signed
3. Policy enforcement: The containers-policy.json rejects unsigned images from the registry
4. Port separation: Unauthenticated pulls (443) vs authenticated pushes (4443)
5. Rootless registry: Registry runs as non-root user with minimal privileges

### 4.2 Enable and Start Services

```bash
# Reload systemd and start services
sudo systemctl daemon-reload

# Start as the service user
sudo -u CI_SERVICE_USER sh -lc 'systemctl --user daemon-reload && systemctl --user enable --now registry.service'
sudo systemctl enable --now registry-proxy.service

# One-time: ensure host dir ownership matches rootless ID map
sudo -u CI_SERVICE_USER podman unshare chown -R 100000:100000 /var/lib/registry

### 4.1 Self-test

```bash
# 1) Listening sockets (should show nginx on the chosen IPs/ports):
sudo ss -ltnp '( sport = :443 or sport = :4443 )'

# 2) From another host on an allowed network:
curl -vk https://REGISTRY_HOST/v2/ | grep -i Docker-Distribution-Api-Version   # expect 200 + header
curl -vk https://REGISTRY_HOST/v2/_catalog                                      # expect 403 (blocked)
curl -vk --cert client.crt --key client.key https://REGISTRY_HOST:4443/v2/ | \
  grep -i Docker-Distribution-Api-Version                                       # expect 200 + header

# 3) Signature enforcement (on a client with CA + org-cosign.pub + policy.json):
#   - Pull unsigned image from your registry -> should FAIL
#   - Sign image with your org key, push via 4443, pull via 443 -> should SUCCEED

## Step 5: Verify Installation

### 5.1 Check Service Status

```bash
# Check that the services are running properly
sudo -u CI_SERVICE_USER -H sh -lc 'systemctl --user status registry.service'
sudo systemctl status registry-proxy.service

# Check that registry container is running
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman ps"

# Check registry logs
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman logs registry"

# Check nginx proxy logs
sudo journalctl -u registry-proxy.service -f --no-pager -n 50

# Quick validation of hardening
echo "=== Proxy Runtime & Logs ==="
systemctl status registry-proxy.service
ls -ld /run/registry-proxy /var/log/registry-proxy

echo "=== Nginx Logs ==="
tail -n1 /var/log/registry-proxy/error.log /var/log/registry-proxy/access.log

echo "=== Port Bindings ==="
ss -ltnp | egrep ':(443|4443)'; ss -ltnp | grep '127.0.0.1:5000'

echo "=== Security Tests ==="
# 443 forbids writes
curl -k -X PUT https://YOUR_ACTUAL_IP_ADDRESS/v2/ -I | grep 403 || echo "WARNING: 443 allows writes"

# 4443 requires client cert
curl -kI https://YOUR_ACTUAL_IP_ADDRESS:4443/v2/ | egrep '400|401|403' || echo "WARNING: 4443 may not require auth"

# 443 should return 200 with registry API header; catalog is forbidden
curl -vk https://REGISTRY_HOST/v2/ | grep -i Docker-Distribution-Api-Version
curl -vk https://REGISTRY_HOST/v2/_catalog   # expect 403

# 4443 requires mTLS, should return 200 with client certs
curl -vk --cert client.crt --key client.key https://REGISTRY_HOST:4443/v2/ | grep -i Docker-Distribution-Api-Version

# OCSP stapling is intentionally disabled; you should not expect stapling-related headers in responses.

# Verify Podman is using non-home paths
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman info --format '{{.Store.GraphRoot}} {{.Store.RunRoot}}'"

5.2 Test Registry Functionality

5.2.1 Test mTLS Authentication

# Switch to CI_SERVICE_USER for testing (CI_SERVICE_USER runs CI pipeline and Podman operations)
sudo su - CI_SERVICE_USER

# Navigate to the application directory
cd /opt/APP_NAME

# Configure Podman to use client certificates for mTLS authentication
mkdir -p ~/.config/containers
cat > ~/.config/containers/registries.conf << EOF
[[registry]]
location = "YOUR_ACTUAL_IP_ADDRESS:4443"
client_cert = "/etc/registry/certs/clients/client.crt"
client_key = "/etc/registry/certs/private/client.key"
EOF

# Test authenticated push using mTLS (port 4443)
echo "FROM alpine:latest" > /tmp/test.Dockerfile
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman build -f /tmp/test.Dockerfile -t YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest /tmp
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman push YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest

# Test unauthenticated pull from standard HTTPS endpoint (port 443)
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman pull YOUR_ACTUAL_IP_ADDRESS/APP_NAME/test:latest

# Test that unauthorized push to port 443 is blocked
echo "FROM alpine:latest" > /tmp/unauthorized.Dockerfile
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman build -f /tmp/unauthorized.Dockerfile -t YOUR_ACTUAL_IP_ADDRESS/APP_NAME/unauthorized:latest /tmp
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman push YOUR_ACTUAL_IP_ADDRESS/APP_NAME/unauthorized:latest
# Expected: This should fail with 403 Forbidden

# Clean up
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman rmi YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest 2>/dev/null || true
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman rmi YOUR_ACTUAL_IP_ADDRESS/APP_NAME/test:latest 2>/dev/null || true
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman rmi YOUR_ACTUAL_IP_ADDRESS/APP_NAME/unauthorized:latest 2>/dev/null || true
exit

Expected behavior:

• ✅ Push requires mTLS client certificate authentication on port 4443
• ✅ Pull works without authentication (public read access) on port 443
• ✅ Unauthorized push is blocked on port 443 (403 Forbidden)
• ✅ Registry accessible at https://YOUR_ACTUAL_IP_ADDRESS:4443 for authenticated operations (mTLS)
• ✅ Registry accessible at https://YOUR_ACTUAL_IP_ADDRESS for unauthenticated pulls
• ✅ Proper FHS-compliant certificate structure with secure permissions
• ✅ Port 443 allows unauthenticated Docker Registry v2 pulls; Port 4443 is for authenticated pushes (mTLS)

5.3 Troubleshooting TLS Errors

If you get a TLS error like remote error: tls: internal error when using self-signed certificates, verify the certificate installation and configuration:

# Verify the certificate was installed correctly in system trust store
ls -la /usr/local/share/ca-certificates/registry-ca.crt

# Verify certificate chain is valid
openssl verify -CAfile /etc/registry/certs/ca/ca.crt /etc/registry/certs/registry.crt

# Test the certificate connection
openssl s_client -connect YOUR_ACTUAL_IP_ADDRESS:4443 -servername YOUR_ACTUAL_IP_ADDRESS < /dev/null

# Test mTLS connection with client certificate
openssl s_client -connect YOUR_ACTUAL_IP_ADDRESS:4443 -servername YOUR_ACTUAL_IP_ADDRESS \
  -cert /etc/registry/certs/clients/client.crt \
  -key /etc/registry/certs/private/client.key < /dev/null

# Test Cosign signature verification
echo "Testing Cosign signature verification..."
cosign verify --key /etc/containers/keys/org-cosign.pub YOUR_ACTUAL_IP_ADDRESS/APP_NAME/test:latest

# Verify nginx is using the correct certificates
sudo ls -la /etc/registry/certs/registry.crt /etc/registry/certs/private/registry.key /etc/registry/certs/clients/ca.crt

# If issues persist, restart the services to reload certificates
sudo -u CI_SERVICE_USER -H sh -lc 'systemctl --user restart registry.service'
sudo systemctl restart registry-proxy.service

# Wait for services to restart, then test again
sleep 10

# Test mTLS authentication
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman pull YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest

Certificate Structure Summary

The project uses a two-port configuration:

• Port 443: Unauthenticated pulls (public read access)
• Port 4443: Authenticated pushes (mTLS client certificate required)

FHS-Compliant Certificate Locations:

• Private Keys: /etc/registry/certs/private/ (mode 600)
• CA Certificates: /etc/registry/certs/ca/ (mode 644)
• Client CA: /etc/registry/certs/clients/ (mode 640, readable by registry-proxy)
• Certificate Requests: /etc/registry/certs/requests/ (mode 644)
• Server Certificates: /etc/registry/certs/ (mode 644)
• System Trust Store: /usr/local/share/ca-certificates/registry-ca.crt

Security Notes

Certificate Security: The certificate files in /etc/registry/certs/ contain sensitive authentication data and should be:

• Backed up securely if needed for disaster recovery
• Never committed to version control
• Protected with proper permissions (600 for private keys, 640 for client CA)
• Rotated regularly by regenerating certificates and updating client configurations

Files to Preserve: DO NOT remove these files as they are needed for operation:

• /opt/APP_NAME/registry/containers-policy.json
• /etc/registry/certs/ (contains all certificates and keys)
• /etc/systemd/user/registry.service
• /etc/systemd/system/registry-proxy.service
• /etc/registry/nginx.conf

Architecture Benefits

1. 🔒 Enhanced Security: Keys stay on host, minimal capabilities, strict isolation
2. 🛡️ mTLS Authentication: Stronger than basic auth for push operations
3. 🚫 No Resource Contention: Registry isolated from other operations
4. 📁 No $HOME I/O: All state outside user home directory
5. 🔧 Robust Boot: User manager guaranteed to exist at boot time
6. 🎯 Clear Separation: Unauthenticated pulls vs authenticated pushes
7. ⚡ Low-Port Binding: Via minimal capability (CAP_NET_BIND_SERVICE)
8. 🛡️ Comprehensive Hardening: Multiple security layers and restrictions

🎉 Congratulations!

You have successfully set up a rootless Docker Registry v2 with host TLS reverse proxy featuring:

• ✅ Rootless registry via systemd user manager
• ✅ Host TLS reverse proxy with minimal capabilities
• ✅ mTLS authentication for secure push operations
• ✅ Unauthenticated pulls for public read access
• ✅ FHS-compliant directory structure for better organization and security
• ✅ No $HOME I/O policy with all state outside user home directory
• ✅ Port 443 allows unauthenticated Docker Registry v2 pulls; Port 4443 is for authenticated pushes (mTLS)
• ✅ Comprehensive security hardening with multiple protection layers

Your Docker Registry is now ready for secure image storage with proper authentication and isolation!