sharenet/Docker_Registry_Install_Guide.md

# Docker Registry Install Guide

This guide covers setting up a rootless Docker Registry v2 with host TLS reverse proxy for secure image storage. The registry runs rootless via Podman with loopback-only access, while a host nginx reverse proxy provides TLS termination and mTLS authentication.

## Architecture Overview

```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   External      │    │   Host TLS      │    │   Rootless      │
│   Clients       │    │   Reverse Proxy │    │   Registry      │
│                 │    │   (Nginx)       │    │   (Podman)      │
│                 │    │                 │    │                 │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         │                       │                       │
         └─── HTTPS :443 ────────┼───────────────────────┘
         (Unauthenticated pulls) │
                                 └─── HTTPS :4443 ───────┘
                                 (mTLS Authentication)
```

## Security Model

- **Rootless Registry**: Runs as `CI_SERVICE_USER` via systemd user manager
- **Host TLS Proxy**: Runs as `registry-proxy` with minimal capabilities
- **Loopback Isolation**: Registry only accessible via `127.0.0.1:5000`
- **mTLS Authentication**: Client certificates required for push operations
- **No $HOME I/O**: All Podman state outside user home directory
- **Port 443**: Unauthenticated pulls only (GET requests)
- **Port 4443**: Authenticated pushes only (mTLS required)

## Prerequisites

- Ubuntu 24.04 LTS with root access
- Podman installed and configured for rootless operation
- `CI_SERVICE_USER` created and configured
- Basic familiarity with Linux commands and SSH

## Step 1: Install Podman (if not already installed)

### 1.1 Install Podman

```bash
# Install Podman and related tools
sudo apt install -y podman slirp4netns fuse-overlayfs

# Verify installation
podman --version

# Configure Podman for rootless operation (optional but recommended)
echo 'kernel.unprivileged_userns_clone=1' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# Configure subuid/subgid for CI_SERVICE_USER
sudo usermod --add-subuids 100000-165535 CI_SERVICE_USER
sudo usermod --add-subgids 100000-165535 CI_SERVICE_USER
```

## Step 2: Set Up Rootless Docker Registry v2

### 2.1 Create System-wide Podman Configuration

```bash
# Create system-wide Podman configuration
sudo mkdir -p /etc/containers
sudo tee /etc/containers/registries.conf > /dev/null << 'EOF'
unqualified-search-registries = ["docker.io"]
EOF

# Set proper permissions for system-wide Podman config (root-owned)
sudo chown root:root /etc/containers/registries.conf
sudo chmod 644 /etc/containers/registries.conf
```

### 2.2 Enable User Manager and Create Directories

```bash
# Enable lingering for CI_SERVICE_USER to allow systemd user manager
sudo loginctl enable-linger CI_SERVICE_USER

# Create Podman rootless directories outside home
sudo mkdir -p /var/tmp/podman-$(id -u CI_SERVICE_USER)/{root,run,tmp,xdg-data,xdg-config}
sudo chown -R CI_SERVICE_USER:CI_SERVICE_USER /var/tmp/podman-$(id -u CI_SERVICE_USER)
sudo chmod 750 /var/tmp/podman-$(id -u CI_SERVICE_USER)

# Initialize Podman with rootless configuration (no home directory access)
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman system migrate"
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman info"
```

### 2.3 Create Registry User and Configuration Directories

```bash
# Create registry-proxy user for TLS reverse proxy
sudo useradd -r -s /bin/false registry-proxy

# Create registry configuration directories
sudo mkdir -p /etc/registry/certs/{private,clients,ca,requests}
sudo chown -R root:root /etc/registry/certs
sudo chmod 750 /etc/registry/certs/private
sudo chmod 755 /etc/registry/certs/{clients,ca,requests}

# Create registry data directory (systemd-managed)
sudo mkdir -p /var/lib/registry
sudo chown CI_SERVICE_USER:CI_SERVICE_USER /var/lib/registry
sudo chmod 750 /var/lib/registry

# Create log directory for nginx proxy
sudo install -d -o registry-proxy -g registry-proxy /var/log/registry-proxy
```

### 2.4 Install Systemd Services

#### 2.4.1 Rootless Registry Service

```bash
# Install systemd user service for rootless registry
sudo tee /etc/systemd/user/registry.service > /dev/null << 'EOF'
[Unit]
Description=Rootless Docker Registry v2 (loopback only)
After=network-online.target
Wants=network-online.target

[Service]
Environment=PODMAN_ROOT=/var/tmp/podman-%U/root
Environment=PODMAN_RUNROOT=/run/user/%U/podman-run
Environment=PODMAN_TMPDIR=/var/tmp/podman-%U/tmp
Environment=XDG_DATA_HOME=/var/tmp/podman-%U/xdg-data
Environment=XDG_CONFIG_HOME=/var/tmp/podman-%U/xdg-config
ExecStart=/usr/bin/podman --root=${PODMAN_ROOT} --runroot=${PODMAN_RUNROOT} --tmpdir=${PODMAN_TMPDIR} --events-backend=file \
  run --rm --name registry \
  -p 127.0.0.1:5000:5000 \
  --read-only --tmpfs /tmp:size=64m --cap-drop=ALL --security-opt=no-new-privileges \
  -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \
  -e REGISTRY_STORAGE_DELETE_ENABLED=false \
  -v /var/lib/registry:/var/lib/registry:z \
  docker.io/library/registry@sha256:8be26f81ffea54106bae012c6f349df70f4d5e7e2ec01b143c46e2c03b9e551d
ExecStop=/usr/bin/podman --root=${PODMAN_ROOT} --runroot=${PODMAN_RUNROOT} --tmpdir=${PODMAN_TMPDIR} stop -t 10 registry
Restart=on-failure
MemoryMax=1G
CPUQuota=100%

[Install]
WantedBy=default.target
EOF
```

#### 2.4.2 TLS Reverse Proxy Service

```bash
# Install systemd system service for TLS reverse proxy
sudo tee /etc/systemd/system/registry-proxy.service > /dev/null << 'EOF'
[Unit]
Description=TLS proxy for Docker Registry (443 pulls / 4443 pushes)
After=network-online.target
Wants=network-online.target

[Service]
User=registry-proxy
Group=registry-proxy
UMask=0077
RuntimeDirectory=registry-proxy
LogsDirectory=registry-proxy
ReadWritePaths=/run/registry-proxy /var/log/registry-proxy
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ProtectClock=yes
ProtectHostname=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
LimitNOFILE=65536
ExecStartPre=/usr/sbin/nginx -t -c /etc/registry/nginx.conf
ExecStart=/usr/sbin/nginx -g 'daemon off;' -c /etc/registry/nginx.conf
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
```

### 2.5 Create Nginx Configuration

```bash
# Create nginx configuration for TLS reverse proxy
sudo tee /etc/registry/nginx.conf > /dev/null << 'EOF'
worker_processes auto;
events { worker_connections 1024; }

pid /run/registry-proxy/nginx.pid;
access_log /var/log/registry-proxy/access.log;
error_log  /var/log/registry-proxy/error.log;

http {
  limit_req_zone $binary_remote_addr zone=reg_read:10m rate=10r/s;
  limit_req_zone $binary_remote_addr zone=reg_write:10m rate=5r/s;
  client_max_body_size 2g;
  ssl_ciphers HIGH:!aNULL:!MD5;
  ssl_prefer_server_ciphers on;
  ssl_verify_depth 2;
  ssl_session_cache shared:SSL:10m;
  ssl_session_timeout 10m;
  proxy_http_version 1.1;
  proxy_set_header Connection "";
  proxy_request_buffering off;
  proxy_read_timeout 300s;
  proxy_temp_path        /run/registry-proxy/proxy_temp;
  client_body_temp_path  /run/registry-proxy/client_temp;
  fastcgi_temp_path      /run/registry-proxy/fastcgi_temp;
  uwsgi_temp_path        /run/registry-proxy/uwsgi_temp;
  scgi_temp_path         /run/registry-proxy/scgi_temp;
  upstream reg { server 127.0.0.1:5000; }

  # 443: unauthenticated pulls only
  server {
    listen 443 ssl http2;
    ssl_certificate     /etc/registry/certs/registry.crt;
    ssl_certificate_key /etc/registry/certs/private/registry.key;
    ssl_protocols TLSv1.2 TLSv1.3;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    if ($request_method ~* ^(PUT|PATCH|POST|DELETE)$) { return 403; }
    location /v2/ {
      limit_req zone=reg_read burst=20 nodelay;
      proxy_pass http://reg;
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-Proto https;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
  }

  # 4443: authenticated pushes only (mTLS)
  server {
    listen 4443 ssl http2;
    ssl_certificate     /etc/registry/certs/registry.crt;
    ssl_certificate_key /etc/registry/certs/private/registry.key;
    ssl_protocols TLSv1.2 TLSv1.3;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    ssl_client_certificate /etc/registry/certs/clients/ca.crt;
    ssl_verify_client on;
    location /v2/ {
      limit_req zone=reg_write burst=10;
      proxy_pass http://reg;
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-Proto https;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
  }
}
EOF

# Set proper permissions for nginx config (root-owned)
sudo chown root:root /etc/registry/nginx.conf
sudo chmod 644 /etc/registry/nginx.conf
```

### 2.6 Install Container Policy

```bash
# Install container policy file (root-owned)
sudo cp /opt/APP_NAME/registry/containers-policy.json /etc/containers/policy.json
sudo chown root:root /etc/containers/policy.json
sudo chmod 644 /etc/containers/policy.json
```

## Step 3: Generate TLS Certificates

### 3.1 Generate Server and Client CA Certificates

```bash
# 1. Generate server CA and certificates with proper FHS-compliant structure
cd /etc/registry/certs

# Generate server CA private key in private subdirectory
sudo openssl genrsa -out private/ca.key 4096

# Generate server CA certificate in ca subdirectory
sudo openssl req -new -x509 -key private/ca.key \
    -out ca/ca.crt \
    -days 365 \
    -subj "/O=YOUR_ORGANIZATION/CN=APP_NAME-Registry-CA"

# Generate server private key in private subdirectory
sudo openssl genrsa -out private/registry.key 4096

# Copy and use the project's OpenSSL configuration file
sudo cp /opt/APP_NAME/registry/openssl.conf /etc/registry/certs/requests/
sudo chown root:root /etc/registry/certs/requests/openssl.conf

# Generate server certificate signing request in requests subdirectory
sudo openssl req -new -key private/registry.key \
    -out requests/registry.csr \
    -config requests/openssl.conf

# Sign server certificate with CA
sudo openssl x509 -req -in requests/registry.csr \
    -CA ca/ca.crt -CAkey private/ca.key -CAcreateserial \
    -out registry.crt \
    -days 365 \
    -extensions req_ext \
    -extfile requests/openssl.conf

# 2. Generate client CA for mTLS authentication
# Generate client CA private key
sudo openssl genrsa -out private/client-ca.key 4096

# Generate client CA certificate
sudo openssl req -new -x509 -key private/client-ca.key \
    -out clients/ca.crt \
    -days 365 \
    -subj "/O=YOUR_ORGANIZATION/CN=APP_NAME-Client-CA"

# Generate client certificate for CI operations
sudo openssl genrsa -out private/client.key 4096

# Generate client certificate signing request
sudo openssl req -new -key private/client.key \
    -out requests/client.csr \
    -subj "/O=YOUR_ORGANIZATION/CN=APP_NAME-CI-Client"

# Sign client certificate with client CA
sudo openssl x509 -req -in requests/client.csr \
    -CA clients/ca.crt -CAkey private/client-ca.key -CAcreateserial \
    -out clients/client.crt \
    -days 365

# Set proper FHS-compliant permissions
sudo chmod 600 private/ca.key private/client-ca.key private/client.key  # Private keys - owner read/write only
sudo chmod 644 ca/ca.crt registry.crt clients/client.crt  # Certificates - world readable
sudo chmod 644 requests/registry.csr requests/client.csr requests/openssl.conf  # Requests - world readable

# Set registry key and client CA permissions for nginx proxy access
sudo chgrp registry-proxy /etc/registry/certs/private/registry.key /etc/registry/certs/clients/ca.crt
sudo chmod 640 /etc/registry/certs/private/registry.key /etc/registry/certs/clients/ca.crt

# Verify certificate creation
sudo openssl x509 -in /etc/registry/certs/registry.crt -text -noout | grep -E "(Subject:|DNS:|IP Address:)"

# 3. Install server CA certificate in system trust store (for curl, wget, etc.)
sudo cp /etc/registry/certs/ca/ca.crt /usr/local/share/ca-certificates/registry-ca.crt

# This step should complete with "1 added, 0 removed". If it does not, there could be a problem with the certificate you generated, or you might already have a certificate in the trust store
sudo update-ca-certificates
```

## Step 4: Configure Firewall and Start Services

### 4.1 Configure Firewall

```bash
# Open port 443 for unauthenticated pulls (public access)
sudo ufw allow 443/tcp

# Open port 4443 for authenticated pushes (restrict to known CIDRs)
# Example: restrict to specific networks (adjust CIDRs as needed)
sudo ufw delete allow 4443/tcp || true
sudo ufw allow from 10.0.0.0/8 to any port 4443 proto tcp
sudo ufw allow from 172.16.0.0/12 to any port 4443 proto tcp
sudo ufw allow from 192.168.0.0/16 to any port 4443 proto tcp

# For public access to 4443 (less secure), use:
# sudo ufw allow 4443/tcp

# Optional: Consider SystemCallFilter for additional hardening
# (validate exact syscall set for your distro's nginx build)
# SystemCallFilter=@system-service

# Note: Port 5000 is NOT opened - registry runs loopback-only

## Client Trust Configuration

For clients to trust your registry certificates, they should install the server CA certificate:

**For pulls (port 443):**
```bash
# On client systems - use the actual FQDN/IP from your certificates
sudo mkdir -p /etc/containers/certs.d/REGISTRY_HOST
sudo cp /path/to/registry-ca.crt /etc/containers/certs.d/REGISTRY_HOST/ca.crt
```

**For pushes (port 4443, mTLS):**
```bash
# On client systems - use the actual FQDN/IP from your certificates
sudo mkdir -p /etc/containers/certs.d/REGISTRY_HOST:4443
sudo cp /path/to/registry-ca.crt /etc/containers/certs.d/REGISTRY_HOST:4443/ca.crt
sudo cp /path/to/client.crt /etc/containers/certs.d/REGISTRY_HOST:4443/client.cert
sudo cp /path/to/client.key /etc/containers/certs.d/REGISTRY_HOST:4443/client.key
```

**Note:** Replace `REGISTRY_HOST` with the actual FQDN or IP address that matches your certificate's Subject Alternative Name (SAN). For pushes, both the server CA certificate and client certificate/key are required for mTLS authentication.
```

### 4.2 Enable and Start Services

```bash
# Start as the service user
sudo -u CI_SERVICE_USER sh -lc 'systemctl --user daemon-reload && systemctl --user enable --now registry.service'
sudo systemctl enable --now registry-proxy.service
```

## Step 5: Verify Installation

### 5.1 Check Service Status

```bash
# Check that the services are running properly
sudo -u CI_SERVICE_USER -H sh -lc 'systemctl --user status registry.service'
sudo systemctl status registry-proxy.service

# Check that registry container is running
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman ps"

# Check registry logs
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman logs registry"

# Check nginx proxy logs
sudo journalctl -u registry-proxy.service -f --no-pager -n 50

# Quick validation of hardening
echo "=== Proxy Runtime & Logs ==="
systemctl status registry-proxy.service
ls -ld /run/registry-proxy /var/log/registry-proxy

echo "=== Nginx Logs ==="
tail -n1 /var/log/registry-proxy/error.log /var/log/registry-proxy/access.log

echo "=== Port Bindings ==="
ss -ltnp | egrep ':(443|4443)'; ss -ltnp | grep '127.0.0.1:5000'

echo "=== Security Tests ==="
# 443 forbids writes
curl -k -X PUT https://YOUR_ACTUAL_IP_ADDRESS/v2/ -I | grep 403 || echo "WARNING: 443 allows writes"

# 4443 requires client cert
curl -kI https://YOUR_ACTUAL_IP_ADDRESS:4443/v2/ | egrep '400|401|403' || echo "WARNING: 4443 may not require auth"

# Verify Podman is using non-home paths
sudo su - CI_SERVICE_USER -c "env PODMAN_ROOT=/var/tmp/podman-\$(id -u)/root PODMAN_RUNROOT=/run/user/\$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-\$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-\$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-\$(id -u)/xdg-config podman info --format '{{.Store.GraphRoot}} {{.Store.RunRoot}}'"
```

### 5.2 Test Registry Functionality

#### 5.2.1 Test mTLS Authentication

```bash
# Switch to CI_SERVICE_USER for testing (CI_SERVICE_USER runs CI pipeline and Podman operations)
sudo su - CI_SERVICE_USER

# Navigate to the application directory
cd /opt/APP_NAME

# Configure Podman to use client certificates for mTLS authentication
mkdir -p ~/.config/containers
cat > ~/.config/containers/registries.conf << EOF
[[registry]]
location = "YOUR_ACTUAL_IP_ADDRESS:4443"
client_cert = "/etc/registry/certs/clients/client.crt"
client_key = "/etc/registry/certs/private/client.key"
EOF

# Test authenticated push using mTLS (port 4443)
echo "FROM alpine:latest" > /tmp/test.Dockerfile
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman build -f /tmp/test.Dockerfile -t YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest /tmp
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman push YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest

# Test unauthenticated pull from standard HTTPS endpoint (port 443)
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman pull YOUR_ACTUAL_IP_ADDRESS/APP_NAME/test:latest

# Test that unauthorized push to port 443 is blocked
echo "FROM alpine:latest" > /tmp/unauthorized.Dockerfile
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman build -f /tmp/unauthorized.Dockerfile -t YOUR_ACTUAL_IP_ADDRESS/APP_NAME/unauthorized:latest /tmp
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman push YOUR_ACTUAL_IP_ADDRESS/APP_NAME/unauthorized:latest
# Expected: This should fail with 403 Forbidden

# Clean up
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman rmi YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest 2>/dev/null || true
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman rmi YOUR_ACTUAL_IP_ADDRESS/APP_NAME/test:latest 2>/dev/null || true
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman rmi YOUR_ACTUAL_IP_ADDRESS/APP_NAME/unauthorized:latest 2>/dev/null || true
exit
```

**Expected behavior**:
- ✅ Push requires mTLS client certificate authentication on port 4443
- ✅ Pull works without authentication (public read access) on port 443
- ✅ Unauthorized push is blocked on port 443 (403 Forbidden)
- ✅ Registry accessible at `https://YOUR_ACTUAL_IP_ADDRESS:4443` for authenticated operations (mTLS)
- ✅ Registry accessible at `https://YOUR_ACTUAL_IP_ADDRESS` for unauthenticated pulls
- ✅ Proper FHS-compliant certificate structure with secure permissions
- ✅ Port 443 allows unauthenticated Docker Registry v2 pulls; Port 4443 is for authenticated pushes (mTLS)

### 5.3 Troubleshooting TLS Errors

If you get a TLS error like `remote error: tls: internal error` when using self-signed certificates, verify the certificate installation and configuration:

```bash
# Verify the certificate was installed correctly in system trust store
ls -la /usr/local/share/ca-certificates/registry-ca.crt

# Verify certificate chain is valid
openssl verify -CAfile /etc/registry/certs/ca/ca.crt /etc/registry/certs/registry.crt

# Test the certificate connection
openssl s_client -connect YOUR_ACTUAL_IP_ADDRESS:4443 -servername YOUR_ACTUAL_IP_ADDRESS < /dev/null

# Test mTLS connection with client certificate
openssl s_client -connect YOUR_ACTUAL_IP_ADDRESS:4443 -servername YOUR_ACTUAL_IP_ADDRESS \
  -cert /etc/registry/certs/clients/client.crt \
  -key /etc/registry/certs/private/client.key < /dev/null

# Verify nginx is using the correct certificates
sudo ls -la /etc/registry/certs/registry.crt /etc/registry/certs/private/registry.key /etc/registry/certs/clients/ca.crt

# If issues persist, restart the services to reload certificates
sudo -u CI_SERVICE_USER -H sh -lc 'systemctl --user restart registry.service'
sudo systemctl restart registry-proxy.service

# Wait for services to restart, then test again
sleep 10

# Test mTLS authentication
env PODMAN_ROOT=/var/tmp/podman-$(id -u)/root PODMAN_RUNROOT=/run/user/$(id -u)/podman-run PODMAN_TMPDIR=/var/tmp/podman-$(id -u)/tmp XDG_DATA_HOME=/var/tmp/podman-$(id -u)/xdg-data XDG_CONFIG_HOME=/var/tmp/podman-$(id -u)/xdg-config podman pull YOUR_ACTUAL_IP_ADDRESS:4443/APP_NAME/test:latest
```

## Certificate Structure Summary

The project uses a two-port configuration:
- **Port 443**: Unauthenticated pulls (public read access)  
- **Port 4443**: Authenticated pushes (mTLS client certificate required)

**FHS-Compliant Certificate Locations:**
- **Private Keys**: `/etc/registry/certs/private/` (mode 600)
- **CA Certificates**: `/etc/registry/certs/ca/` (mode 644)
- **Client CA**: `/etc/registry/certs/clients/` (mode 640, readable by registry-proxy)
- **Certificate Requests**: `/etc/registry/certs/requests/` (mode 644)
- **Server Certificates**: `/etc/registry/certs/` (mode 644)
- **System Trust Store**: `/usr/local/share/ca-certificates/registry-ca.crt`

## Security Notes

**Certificate Security**: The certificate files in `/etc/registry/certs/` contain sensitive authentication data and should be:
- **Backed up securely** if needed for disaster recovery
- **Never committed to version control**
- **Protected with proper permissions** (600 for private keys, 640 for client CA)
- **Rotated regularly** by regenerating certificates and updating client configurations

**Files to Preserve**: DO NOT remove these files as they are needed for operation:
- `/opt/APP_NAME/registry/containers-policy.json`  
- `/etc/registry/certs/` (contains all certificates and keys)
- `/etc/systemd/user/registry.service`
- `/etc/systemd/system/registry-proxy.service`
- `/etc/registry/nginx.conf`

## Architecture Benefits

1. **🔒 Enhanced Security**: Keys stay on host, minimal capabilities, strict isolation
2. **🛡️ mTLS Authentication**: Stronger than basic auth for push operations
3. **🚫 No Resource Contention**: Registry isolated from other operations
4. **📁 No $HOME I/O**: All state outside user home directory
5. **🔧 Robust Boot**: User manager guaranteed to exist at boot time
6. **🎯 Clear Separation**: Unauthenticated pulls vs authenticated pushes
7. **⚡ Low-Port Binding**: Via minimal capability (`CAP_NET_BIND_SERVICE`)
8. **🛡️ Comprehensive Hardening**: Multiple security layers and restrictions

## 🎉 Congratulations!

You have successfully set up a rootless Docker Registry v2 with host TLS reverse proxy featuring:

- ✅ **Rootless registry** via systemd user manager
- ✅ **Host TLS reverse proxy** with minimal capabilities
- ✅ **mTLS authentication** for secure push operations
- ✅ **Unauthenticated pulls** for public read access
- ✅ **FHS-compliant directory structure** for better organization and security
- ✅ **No $HOME I/O** policy with all state outside user home directory
- ✅ **Port 443 allows unauthenticated Docker Registry v2 pulls; Port 4443 is for authenticated pushes (mTLS)**
- ✅ **Comprehensive security hardening** with multiple protection layers

Your Docker Registry is now ready for secure image storage with proper authentication and isolation!