Add section for verifying the test docker compose file works correctly

CI_CD_PIPELINE_SETUP_GUIDE.md

@@ -1135,13 +1135,17 @@ sudo journalctl -u forgejo-runner.service -f --no-pager
 #### 8.1 Create Containerized CI/CD Environment
 
 ```bash
-# Start DinD container using the repository's CI/CD Docker Compose file
+# Start DinD container for isolated Docker operations
-cd /opt/APP_NAME
+docker run -d \
-docker compose -f ci-cd-compose.yml up -d
+  --name ci-cd-dind \
+  --privileged \
+  -p 2375:2375 \
+  -e DOCKER_TLS_CERTDIR="" \
+  docker:dind
 
 # Wait for DinD to be ready
 echo "Waiting for DinD container to be ready..."
-timeout 60 bash -c 'until docker compose -f ci-cd-compose.yml ps | grep -q "healthy"; do sleep 2; done'
+timeout 60 bash -c 'until docker exec ci-cd-dind docker version; do sleep 2; done'
 
 # Test DinD connectivity
 docker exec ci-cd-dind docker version
@@ -1149,9 +1153,8 @@ docker exec ci-cd-dind docker version
 
 **What this does**:
 - **Creates isolated DinD environment**: Provides isolated Docker environment for all CI/CD operations
-- **Persistent storage**: `ci-cd-data` volume preserves data between restarts
 - **Health checks**: Ensures DinD is fully ready before proceeding
-- **Configuration management**: Uses the repository's `ci-cd-compose.yml` file
+- **Simple setup**: Direct Docker commands for maximum flexibility
 
 #### 8.2 Configure DinD for Harbor Registry
 
@@ -1179,21 +1182,44 @@ docker exec ci-cd-dind docker rmi localhost:5000/test/alpine:latest
 - **Tests connectivity**: Verifies DinD can pull, tag, and push images to Harbor
 - **Validates setup**: Ensures the complete CI/CD pipeline will work
 
-#### 8.3 DinD Environment Management
+#### 8.3 CI/CD Workflow Architecture
 
-The DinD container is managed as an isolated environment where all CI/CD operations run inside the DinD container, providing complete isolation from the host system.
+The CI/CD pipeline uses a three-stage approach with dedicated environments for each stage:
+
+**Job 1 (Testing) - `docker-compose.test.yml`:**
+- **Purpose**: Comprehensive testing with multiple containers
+- **Environment**: DinD with PostgreSQL, Rust, and Node.js containers
+- **Services**: 
+  - PostgreSQL database for backend tests
+  - Rust toolchain for backend testing and migrations
+  - Node.js toolchain for frontend testing
+- **Network**: All containers communicate through `ci-cd-test-network`
+- **Cleanup**: `docker compose -f docker-compose.test.yml down`
+
+**Job 2 (Building) - Direct Docker Commands:**
+- **Purpose**: Isolated image building and pushing to Harbor
+- **Environment**: Single DinD container for Docker operations
+- **Process**: 
+  - Uses Docker Buildx for efficient building
+  - Builds backend and frontend images separately
+  - Pushes images to Harbor registry
+- **Cleanup**: Simple container stop/remove
+
+**Job 3 (Deployment) - `docker-compose.prod.yml`:**
+- **Purpose**: Production deployment with pre-built images
+- **Environment**: Production runner on Production Linode
+- **Process**:
+  - Pulls images from Harbor registry
+  - Deploys complete application stack
+  - Verifies all services are healthy
+- **Services**: PostgreSQL, backend, frontend, Nginx
 
 **Key Benefits:**
-- **🧹 Complete Isolation**: All testing and building runs inside DinD
+- **🧹 Complete Isolation**: Each job has its own dedicated environment
-- **🚫 No Host Contamination**: No containers run directly on host Docker
+- **🚫 No Resource Contention**: Testing and building don't interfere with Harbor
-- **⚡ Consistent Environment**: Same isolation level for all operations
+- **⚡ Consistent Environment**: Same setup every time
-- **🎯 Resource Isolation**: CI/CD operations can't interfere with host services
+- **🎯 Purpose-Specific**: Each Docker Compose file serves a specific purpose
-- **🔄 Parallel Safety**: Multiple operations can run safely
+- **🔄 Parallel Safety**: Jobs can run safely in parallel
-
-**How it works:**
-- **Job 1 (Testing)**: Creates PostgreSQL, Rust, and Node.js containers inside DinD
-- **Job 2 (Building)**: Uses DinD directly for building and pushing Docker images to Harbor
-- **Job 3 (Deployment)**: Production runner pulls images from Harbor and deploys using `docker-compose.prod.yml`
 
 **Testing DinD Setup:**
 
@@ -2104,4 +2130,57 @@ You have successfully set up a complete CI/CD pipeline with:
 - ✅ **SSL/TLS support** for production (optional)
 - ✅ **Zero resource contention** between CI/CD and Harbor
 
-Your application is now ready for continuous deployment with proper security, monitoring, and maintenance procedures in place!
+Your application is now ready for continuous deployment with proper security, monitoring, and maintenance procedures in place!
+
+### Step 8.6 CI/CD Workflow Summary Table
+
+| Stage   | What Runs                | How/Where                |
+|---------|--------------------------|--------------------------|
+| Test    | All integration/unit tests| `docker-compose.test.yml`|
+| Build   | Build & push images      | Direct Docker commands   |
+| Deploy  | Deploy to production     | `docker-compose.prod.yml`|
+
+**How it works:**
+- **Test:** The workflow spins up a full test environment using `docker-compose.test.yml` (Postgres, backend, frontend, etc.) and runs all tests inside containers.
+- **Build:** If tests pass, the workflow uses direct Docker commands (no compose file) to build backend and frontend images and push them to Harbor.
+- **Deploy:** The production runner pulls images from Harbor and deploys the stack using `docker-compose.prod.yml`.
+
+**Expected Output:**
+- Each stage runs in its own isolated environment.
+- Test failures stop the pipeline before any images are built or deployed.
+- Only tested images are deployed to production.
+
+
+### Manual Testing with docker-compose.test.yml
+
+You can use the same test environment locally that the CI pipeline uses for integration testing. This is useful for debugging, development, or verifying your setup before pushing changes.
+
+#### Start the Test Environment
+```bash
+docker compose -f docker-compose.test.yml up -d
+```
+This will start all services needed for integration tests (PostgreSQL, backend, frontend, etc.) in the background.
+
+#### Check Service Health
+```bash
+docker compose -f docker-compose.test.yml ps
+```
+Look for the `healthy` status in the output to ensure all services are ready.
+
+#### Run Tests Manually
+You can now exec into the containers to run tests or commands as needed. For example:
+```bash
+# Run backend tests
+docker exec ci-cd-test-rust cargo test --all
+
+# Run frontend tests
+docker exec ci-cd-test-node npm run test
+```
+
+#### Cleanup
+When you're done, stop and remove all test containers:
+```bash
+docker compose -f docker-compose.test.yml down
+```
+
+**Tip:** This is the same environment and process used by the CI pipeline, so passing tests here means they should also pass in CI.