From 080bb126308efb8823136b4ccc1dad82831d7849 Mon Sep 17 00:00:00 2001 From: continuist Date: Sun, 29 Jun 2025 23:46:24 -0400 Subject: [PATCH] Add section for verifying the test docker compose file works correctly --- CI_CD_PIPELINE_SETUP_GUIDE.md | 117 ++++++++++++++++++++++++++++------ 1 file changed, 98 insertions(+), 19 deletions(-) diff --git a/CI_CD_PIPELINE_SETUP_GUIDE.md b/CI_CD_PIPELINE_SETUP_GUIDE.md index b812d3d..bf988c1 100644 --- a/CI_CD_PIPELINE_SETUP_GUIDE.md +++ b/CI_CD_PIPELINE_SETUP_GUIDE.md @@ -1135,13 +1135,17 @@ sudo journalctl -u forgejo-runner.service -f --no-pager #### 8.1 Create Containerized CI/CD Environment ```bash -# Start DinD container using the repository's CI/CD Docker Compose file -cd /opt/APP_NAME -docker compose -f ci-cd-compose.yml up -d +# Start DinD container for isolated Docker operations +docker run -d \ + --name ci-cd-dind \ + --privileged \ + -p 2375:2375 \ + -e DOCKER_TLS_CERTDIR="" \ + docker:dind # Wait for DinD to be ready echo "Waiting for DinD container to be ready..." -timeout 60 bash -c 'until docker compose -f ci-cd-compose.yml ps | grep -q "healthy"; do sleep 2; done' +timeout 60 bash -c 'until docker exec ci-cd-dind docker version; do sleep 2; done' # Test DinD connectivity docker exec ci-cd-dind docker version @@ -1149,9 +1153,8 @@ docker exec ci-cd-dind docker version **What this does**: - **Creates isolated DinD environment**: Provides isolated Docker environment for all CI/CD operations -- **Persistent storage**: `ci-cd-data` volume preserves data between restarts - **Health checks**: Ensures DinD is fully ready before proceeding -- **Configuration management**: Uses the repository's `ci-cd-compose.yml` file +- **Simple setup**: Direct Docker commands for maximum flexibility #### 8.2 Configure DinD for Harbor Registry @@ -1179,21 +1182,44 @@ docker exec ci-cd-dind docker rmi localhost:5000/test/alpine:latest - **Tests connectivity**: Verifies DinD can pull, tag, and push images to Harbor - **Validates setup**: Ensures the complete CI/CD pipeline will work -#### 8.3 DinD Environment Management +#### 8.3 CI/CD Workflow Architecture -The DinD container is managed as an isolated environment where all CI/CD operations run inside the DinD container, providing complete isolation from the host system. +The CI/CD pipeline uses a three-stage approach with dedicated environments for each stage: + +**Job 1 (Testing) - `docker-compose.test.yml`:** +- **Purpose**: Comprehensive testing with multiple containers +- **Environment**: DinD with PostgreSQL, Rust, and Node.js containers +- **Services**: + - PostgreSQL database for backend tests + - Rust toolchain for backend testing and migrations + - Node.js toolchain for frontend testing +- **Network**: All containers communicate through `ci-cd-test-network` +- **Cleanup**: `docker compose -f docker-compose.test.yml down` + +**Job 2 (Building) - Direct Docker Commands:** +- **Purpose**: Isolated image building and pushing to Harbor +- **Environment**: Single DinD container for Docker operations +- **Process**: + - Uses Docker Buildx for efficient building + - Builds backend and frontend images separately + - Pushes images to Harbor registry +- **Cleanup**: Simple container stop/remove + +**Job 3 (Deployment) - `docker-compose.prod.yml`:** +- **Purpose**: Production deployment with pre-built images +- **Environment**: Production runner on Production Linode +- **Process**: + - Pulls images from Harbor registry + - Deploys complete application stack + - Verifies all services are healthy +- **Services**: PostgreSQL, backend, frontend, Nginx **Key Benefits:** -- **๐Ÿงน Complete Isolation**: All testing and building runs inside DinD -- **๐Ÿšซ No Host Contamination**: No containers run directly on host Docker -- **โšก Consistent Environment**: Same isolation level for all operations -- **๐ŸŽฏ Resource Isolation**: CI/CD operations can't interfere with host services -- **๐Ÿ”„ Parallel Safety**: Multiple operations can run safely - -**How it works:** -- **Job 1 (Testing)**: Creates PostgreSQL, Rust, and Node.js containers inside DinD -- **Job 2 (Building)**: Uses DinD directly for building and pushing Docker images to Harbor -- **Job 3 (Deployment)**: Production runner pulls images from Harbor and deploys using `docker-compose.prod.yml` +- **๐Ÿงน Complete Isolation**: Each job has its own dedicated environment +- **๐Ÿšซ No Resource Contention**: Testing and building don't interfere with Harbor +- **โšก Consistent Environment**: Same setup every time +- **๐ŸŽฏ Purpose-Specific**: Each Docker Compose file serves a specific purpose +- **๐Ÿ”„ Parallel Safety**: Jobs can run safely in parallel **Testing DinD Setup:** @@ -2104,4 +2130,57 @@ You have successfully set up a complete CI/CD pipeline with: - โœ… **SSL/TLS support** for production (optional) - โœ… **Zero resource contention** between CI/CD and Harbor -Your application is now ready for continuous deployment with proper security, monitoring, and maintenance procedures in place! \ No newline at end of file +Your application is now ready for continuous deployment with proper security, monitoring, and maintenance procedures in place! + +### Step 8.6 CI/CD Workflow Summary Table + +| Stage | What Runs | How/Where | +|---------|--------------------------|--------------------------| +| Test | All integration/unit tests| `docker-compose.test.yml`| +| Build | Build & push images | Direct Docker commands | +| Deploy | Deploy to production | `docker-compose.prod.yml`| + +**How it works:** +- **Test:** The workflow spins up a full test environment using `docker-compose.test.yml` (Postgres, backend, frontend, etc.) and runs all tests inside containers. +- **Build:** If tests pass, the workflow uses direct Docker commands (no compose file) to build backend and frontend images and push them to Harbor. +- **Deploy:** The production runner pulls images from Harbor and deploys the stack using `docker-compose.prod.yml`. + +**Expected Output:** +- Each stage runs in its own isolated environment. +- Test failures stop the pipeline before any images are built or deployed. +- Only tested images are deployed to production. + + +### Manual Testing with docker-compose.test.yml + +You can use the same test environment locally that the CI pipeline uses for integration testing. This is useful for debugging, development, or verifying your setup before pushing changes. + +#### Start the Test Environment +```bash +docker compose -f docker-compose.test.yml up -d +``` +This will start all services needed for integration tests (PostgreSQL, backend, frontend, etc.) in the background. + +#### Check Service Health +```bash +docker compose -f docker-compose.test.yml ps +``` +Look for the `healthy` status in the output to ensure all services are ready. + +#### Run Tests Manually +You can now exec into the containers to run tests or commands as needed. For example: +```bash +# Run backend tests +docker exec ci-cd-test-rust cargo test --all + +# Run frontend tests +docker exec ci-cd-test-node npm run test +``` + +#### Cleanup +When you're done, stop and remove all test containers: +```bash +docker compose -f docker-compose.test.yml down +``` + +**Tip:** This is the same environment and process used by the CI pipeline, so passing tests here means they should also pass in CI. \ No newline at end of file