From 416f8b8714f2eac19f8cee69fc733f0437673cdb Mon Sep 17 00:00:00 2001 From: continuist Date: Tue, 2 Sep 2025 20:59:27 -0400 Subject: [PATCH] Make podman setup more secure --- CI_CD_PIPELINE_SETUP_GUIDE.md | 222 ++++++++++++++++++++++++++-------- 1 file changed, 171 insertions(+), 51 deletions(-) diff --git a/CI_CD_PIPELINE_SETUP_GUIDE.md b/CI_CD_PIPELINE_SETUP_GUIDE.md index 52eabef..7e5da94 100644 --- a/CI_CD_PIPELINE_SETUP_GUIDE.md +++ b/CI_CD_PIPELINE_SETUP_GUIDE.md @@ -575,9 +575,43 @@ sudo apt install -y \ apache2-utils ``` -#### 1.5 Install Podman +#### 1.5 Configure Firewall FIRST - Before Any Container Operations -**Note**: Podman is required for container operations and will be installed in the following steps. +**SECURITY FIRST**: Configure firewall BEFORE installing Podman to prevent any accidental exposure. + +```bash +# Configure secure firewall defaults +sudo ufw --force reset +sudo ufw default deny incoming +sudo ufw default allow outgoing + +# ONLY allow SSH (port 22) +sudo ufw allow ssh + +# Enable firewall immediately +sudo ufw --force enable + +# Verify firewall configuration +sudo ufw status verbose + +# Expected output: +# Status: active +# Logging: on (low) +# Default: deny (incoming), allow (outgoing), disabled (routed) +# New profiles: skip +# +# To Action From +# -- ------ ---- +# 22/tcp ALLOW IN Anywhere + +# Test that essential services still work +curl -I https://docker.io +curl -I https://quay.io +``` + +#### 1.6 Install Podman + +**Note**: Podman is required for container operations and will be installed AFTER firewall is configured. ```bash # Install Podman and related tools @@ -829,10 +863,10 @@ sudo chmod 600 /etc/forgejo-runner/.runner - Registers the runner with your Forgejo instance - Sets up the runner with appropriate labels for Ubuntu and Docker environments -**Step 5: Create and Enable Systemd User Service** +**Step 5: Create and Enable Systemd Service** ```bash -sudo tee /etc/systemd/user/forgejo-runner.service > /dev/null << 'EOF' +sudo tee /etc/systemd/system/forgejo-runner.service > /dev/null << 'EOF' [Unit] Description=Forgejo Actions Runner After=network.target @@ -845,11 +879,11 @@ Restart=always RestartSec=10 [Install] -WantedBy=default.target +WantedBy=multi-user.target EOF # Enable the service via user manager -sudo systemctl --global enable forgejo-runner.service +sudo systemctl enable forgejo-runner.service ``` **What this does**: @@ -895,11 +929,63 @@ sudo journalctl -u forgejo-runner.service -f --no-pager - Check network: Ensure the runner can reach your Forgejo instance - Restart service: `sudo systemctl restart forgejo-runner.service` -### Step 6: Set Up Podman-in-Podman (PiP) for CI Operations +### Step 7: Set Up Podman-in-Podman (PiP) for CI Operations -**Important**: This step sets up a Podman-in-Podman container that provides an isolated environment for CI/CD operations, eliminating resource contention with Forgejo Container Registry and simplifying cleanup. +#### 7.0 Configure Container Policy (Required for PiP) -#### 6.1 Create Containerized CI/CD Environment +**Important**: Before setting up the PiP container, you need to configure a simplified container policy that allows pulling images from quay.io and other necessary registries while maintaining security. + +```bash +# Switch to root to configure system-wide container policy +sudo su - + +# Create a simplified container policy that allows necessary registries +cat > /etc/containers/policy.json << 'EOF' +{ + "default": [ + { + "type": "insecureAcceptAnything" + } + ], + "transports": { + "docker": { + "quay.io": [ + { + "type": "insecureAcceptAnything" + } + ], + "docker.io": [ + { + "type": "insecureAcceptAnything" + } + ], + "registry-1.docker.io": [ + { + "type": "insecureAcceptAnything" + } + ] + } + } +} +EOF + +# Verify the policy was created correctly +cat /etc/containers/policy.json + +# Exit root shell +exit +``` + +**What this does**: +- **Simplified security**: Allows pulling from quay.io, docker.io, and other common registries +- **Maintains security**: Still provides a policy framework for future restrictions +- **Required for PiP**: The PiP container needs to pull the podman image from quay.io + +**Security Note**: This policy allows pulling from common registries. For production environments, you may want to implement stricter policies with signature verification for specific registries. + +#### 7.1 SECURE Containerized CI/CD Environment Setup + +**CRITICAL SECURITY NOTE**: This setup uses UNIX socket communication only - NO network ports exposed. ```bash # Switch to CI_SERVICE_USER (who has Podman access) @@ -908,20 +994,41 @@ sudo su - CI_SERVICE_USER # Navigate to the application directory cd /opt/APP_NAME -# Start PiP container for isolated Podman operations +# Create secure PiP container with NO network exposure podman run -d \ --name ci-pip \ - --privileged \ - -p 2375:2375 \ - -e DOCKER_TLS_CERTDIR="" \ - quay.io/podman/stable:latest + --security-opt=no-new-privileges \ + --cap-drop=ALL \ + --cap-add=SETGID \ + --cap-add=SETUID \ + --cap-add=CHOWN \ + -v /var/run/podman.sock:/var/run/podman.sock \ + quay.io/podman/stable:latest \ + podman system service --time=0 unix:///var/run/podman.sock -# Wait for a minute or two for PiP to be ready (wait for Podman daemon inside PiP) +# Wait for PiP to be ready +sleep 10 -# Test PiP connectivity +# Check container status to ensure it's running +podman ps + +# Test PiP connectivity through secure socket podman exec ci-pip podman version + +# Verify NO network ports are exposed +podman inspect ci-pip | grep -A 10 "Ports" +# Should show empty or only internal ports + +# Test image pulling capability (uses host's network stack securely) +podman exec ci-pip podman pull alpine:latest ``` +**How This Works Securely**: +- **NO exposed ports**: Management API only accessible through UNIX socket +- **Image pulling**: PiP container uses host's network stack to pull images from docker.io +- **No privileges**: Minimal capabilities, no root access +- **Host firewall protection**: UFW blocks all unnecessary ports + **What this does**: - **Creates isolated PiP environment**: Provides isolated Podman environment for all CI/CD operations - **Health checks**: Ensures PiP is fully ready before proceeding @@ -929,37 +1036,33 @@ podman exec ci-pip podman version **Why CI_SERVICE_USER**: The CI_SERVICE_USER has Podman access and runs the CI pipeline, so it needs direct access to the PiP container for seamless CI/CD operations. -#### 6.2 Configure PiP for Forgejo Container Registry +#### 7.2 Configure PiP for Forgejo Container Registry ```bash # Navigate to the application directory cd /opt/APP_NAME -# Configure PiP to use client certificates for mTLS authentication -podman exec ci-pip mkdir -p /etc/containers -podman exec ci-pip tee /etc/containers/registries.conf > /dev/null << 'EOF' -[[registry]] -location = "YOUR_CI_CD_IP:4443" -# Forgejo Container Registry uses Personal Access Tokens for authentication -EOF - # Copy client certificates to PiP container # Forgejo Container Registry authentication is handled through environment variables # Test Forgejo Container Registry connectivity from PiP podman exec ci-pip podman pull alpine:latest -podman exec ci-pip podman tag alpine:latest YOUR_CI_CD_IP:4443/APP_NAME/test:latest -podman exec ci-pip podman push YOUR_CI_CD_IP:4443/APP_NAME/test:latest +podman exec ci-pip podman tag alpine:latest your-forgejo-domain.com/your-username/APP_NAME/test:latest + +# Log in to Forgejo Container Registry +podman exec ci-pip podman login git.gcdo.org -u YOUR_USERNAME -p YOUR_PERSONAL_ACCESS_TOKEN + +# Push the image +podman exec ci-pip podman push your-forgejo-domain.com/your-username/APP_NAME/test:latest # Test unauthenticated pull from standard port 443 -podman exec ci-pip podman pull YOUR_CI_CD_IP/APP_NAME/test:latest +podman exec ci-pip podman pull your-forgejo-domain.com/your-username/APP_NAME/test:latest # Clean up test images -podman exec ci-pip podman rmi YOUR_CI_CD_IP:4443/APP_NAME/test:latest -podman exec ci-pip podman rmi YOUR_CI_CD_IP/APP_NAME/test:latest +podman exec ci-pip podman rmi your-forgejo-domain.com/your-username/APP_NAME/test:latest ``` -#### 6.3 Set Up Workspace Directory +#### 7.3 Set Up Workspace Directory **Important**: The CI workflow needs a workspace directory for code checkout. This directory will be used by the Forgejo Actions runner. @@ -1130,43 +1233,60 @@ chmod +x scripts/monitor.sh **Note**: The repository script is more comprehensive and includes proper error handling, colored output, and support for both CI/CD and production environments. It automatically detects the environment and provides appropriate monitoring information. -### Step 7: Configure Firewall - -#### 7.1 Configure UFW Firewall - -```bash -sudo ufw --force enable -sudo ufw default deny incoming -sudo ufw default allow outgoing -sudo ufw allow ssh -# Note: Forgejo Container Registry access is configured through Forgejo itself -``` - **Security Model**: - **Forgejo Registry**: Integrated authentication and authorization - **SSH**: Restricted to your IP addresses - **All other ports**: Blocked -### Step 8: Test CI/CD Setup +### Step 8: Security Verification and Testing -#### 8.1 Test Podman Installation +#### 8.1 Security Audit - Verify NO Exposure + +```bash +# Verify NO containers expose ports to external network +podman ps --format "table {{.Names}}\t{{.Ports}}" +# Should show empty or only internal ports + +# Verify firewall is active and blocking all unnecessary ports +sudo ufw status verbose + +# Check listening ports - should only show SSH (22) and system services +sudo ss -tulpn | grep -E "(2375|2376|4443|5000)" +# Should return empty - no container management ports exposed + +# Verify PiP container security settings +podman inspect ci-pip | grep -E "(Privileged|SecurityOpt|Capabilities)" +# Should show: "Privileged": false, security options present, limited capabilities +``` + +#### 8.2 Test Podman Installation ```bash podman --version ``` -#### 8.2 Check Forgejo Container Registry Status +#### 8.3 Test Secure PiP Functionality ```bash -cd /opt/APP_NAME -podman pod ps +# Test PiP operations through secure socket +podman exec ci-pip podman version + +# Test image pulling through secure environment +podman exec ci-pip podman pull alpine:latest + +# Test container operations within PiP +podman exec ci-pip podman run --rm alpine:latest echo "Secure PiP working" ``` -#### 8.3 Test Forgejo Container Registry Access +#### 8.4 Network Security Test ```bash -# Test Forgejo Container Registry access -# Registry access is handled through your Forgejo instance's web interface +# Attempt to access container ports from external perspective (should fail) +curl -v http://localhost:2375/_ping # Should fail with connection refused +curl -v http://0.0.0.0:2375/_ping # Should fail with connection refused + +# Test that essential outgoing connections work +podman exec ci-pip podman search alpine ``` ---