Update CI guide to use official Harbor installer and remove registry folder from repo
Some checks are pending
CI/CD Pipeline / Test Backend (push) Waiting to run
CI/CD Pipeline / Test Frontend (push) Waiting to run
CI/CD Pipeline / Build and Push Docker Images (push) Blocked by required conditions
CI/CD Pipeline / Deploy to Production (push) Blocked by required conditions

This commit is contained in:
continuist 2025-06-29 12:28:18 -04:00
parent 54e6b9a35c
commit 3bb04786b2
5 changed files with 92 additions and 613 deletions

View file

@ -479,29 +479,7 @@ sudo chmod 644 /etc/ssl/registry/registry.crt
- `registry.key`: `600` (owner read/write only) - private key must be secure
- `registry.crt`: `644` (owner read/write, group/others read) - certificate can be read by services
#### 5.2 Update Harbor Configuration with Actual IP Address
```bash
# Switch to SERVICE_USER (registry directory owner)
sudo su - SERVICE_USER
cd /opt/APP_NAME/registry
# Get your actual IP address
YOUR_ACTUAL_IP=$(curl -4 -s ifconfig.me)
echo "Your IP address is: $YOUR_ACTUAL_IP"
# Replace placeholder IP addresses in Harbor configuration files
sed -i "s/YOUR_CI_CD_IP/$YOUR_ACTUAL_IP/g" harbor.yml
sed -i "s/YOUR_CI_CD_IP/$YOUR_ACTUAL_IP/g" docker-compose.yml
# Exit SERVICE_USER shell
exit
```
**Important**: This step replaces all instances of `YOUR_CI_CD_IP` with your actual CI/CD Linode IP address in the Harbor configuration files.
#### 5.3 Set Harbor Environment Variables
#### 5.2 Generate Secure Passwords and Secrets
```bash
# Set environment variables for Harbor
@ -520,16 +498,6 @@ echo "DB_PASSWORD: $DB_PASSWORD"
echo "CORE_SECRET: $CORE_SECRET"
echo "JOBSERVICE_SECRET: $JOBSERVICE_SECRET"
# Update Harbor configuration with secure passwords and secrets
cd /opt/APP_NAME/registry
sed -i "s/Harbor12345/$HARBOR_ADMIN_PASSWORD/g" harbor.yml
sed -i "s/your-db-password/$DB_PASSWORD/g" harbor.yml
sed -i "s/your-db-password/$DB_PASSWORD/g" docker-compose.yml
# Update Harbor secrets in docker-compose.yml
sed -i "s/your-core-secret/$CORE_SECRET/g" docker-compose.yml
sed -i "s/your-jobservice-secret/$JOBSERVICE_SECRET/g" docker-compose.yml
# Save secrets securely for future reference
cat > /opt/APP_NAME/harbor-secrets.txt << EOF
# Harbor Secrets - KEEP THESE SECURE!
@ -562,13 +530,88 @@ echo "IMPORTANT: Keep this file secure and backed up!"
- Store these secrets securely as they will be needed for Harbor upgrades or troubleshooting.
- **CRITICAL**: The secrets file contains sensitive information. Keep it secure and backed up!
#### 5.4 Start Harbor
#### 5.3 Install Harbor Using Official Installer
```bash
# Switch to SERVICE_USER (registry directory owner)
# Switch to DEPLOY_USER (who has sudo access)
sudo su - DEPLOY_USER
cd /opt/APP_NAME
# Download Harbor 2.10.0 offline installer
sudo wget https://github.com/goharbor/harbor/releases/download/v2.10.0/harbor-offline-installer-v2.10.0.tgz
sudo tar -xzf harbor-offline-installer-v2.10.0.tgz
cd harbor
sudo cp harbor.yml.tmpl harbor.yml
# Edit harbor.yml configuration
sudo nano harbor.yml
```
**Important**: In the `harbor.yml` file, update the following variables:
- `hostname: YOUR_CI_CD_IP` (replace with your actual IP)
- `certificate: /etc/ssl/registry/registry.crt`
- `private_key: /etc/ssl/registry/registry.key`
- `password: <the DB_PASSWORD generated in Step 5.2>`
**Note**: Leave `harbor_admin_password` as `Harbor12345` for now. This will be changed at first login through the UI after launching Harbor.
#### 5.4 Prepare and Install Harbor
```bash
# Prepare Harbor configuration
sudo ./prepare
# Install Harbor with Trivy vulnerability scanner
sudo ./install.sh --with-trivy
cd ..
# Change harbor folder permissions recursively to SERVICE_USER
sudo chown -R SERVICE_USER:SERVICE_USER harbor
# Switch to SERVICE_USER to run installation again as non-root
sudo su - SERVICE_USER
cd /opt/APP_NAME/registry
cd /opt/APP_NAME/harbor
# Install Harbor as SERVICE_USER (permissions are partially adjusted correctly)
./install.sh --with-trivy
# Exit SERVICE_USER shell
exit
```
#### 5.5 Fix Permission Issues
```bash
# Switch back to DEPLOY_USER to adjust the permissions for various env files
cd /opt/APP_NAME/harbor
sudo chown SERVICE_USER:SERVICE_USER common/config/jobservice/env
sudo chown SERVICE_USER:SERVICE_USER common/config/db/env
sudo chown SERVICE_USER:SERVICE_USER common/config/registryctl/env
sudo chown SERVICE_USER:SERVICE_USER common/config/trivy-adapter/env
sudo chown SERVICE_USER:SERVICE_USER common/config/core/env
# Exit DEPLOY_USER shell
exit
```
#### 5.6 Test Harbor Installation
```bash
# Switch to SERVICE_USER
sudo su - SERVICE_USER
cd /opt/APP_NAME/harbor
# Verify you can stop Harbor. All Harbor containers should stop.
docker compose down
# Verify you can bring Harbor back up. All Harbor containers should start back up.
docker compose up -d
# Exit SERVICE_USER shell
@ -577,21 +620,21 @@ exit
**Important**: Harbor startup can take 2-3 minutes as it initializes the database and downloads vulnerability databases. The health check will ensure all services are running properly.
#### 5.5 Wait for Harbor Startup
#### 5.7 Wait for Harbor Startup
```bash
# Monitor Harbor startup progress
cd /opt/APP_NAME/registry
cd /opt/APP_NAME/harbor
docker compose logs -f
```
**Expected output**: You should see logs from all Harbor services (core, database, redis, registry, portal, nginx, jobservice, trivy) starting up. Wait until you see "Harbor has been installed and started successfully" or similar success messages.
#### 5.6 Test Harbor Setup
#### 5.8 Test Harbor Setup
```bash
# Check if all Harbor containers are running
cd /opt/APP_NAME/registry
cd /opt/APP_NAME/harbor
docker compose ps
# Test Harbor API (HTTPS)
@ -605,40 +648,7 @@ curl -k -I https://localhost:8080
**Important**: All Harbor services should show as "Up" in the `docker compose ps` output. The health check should return a JSON response indicating all services are healthy.
#### 5.7 Test Harbor Access
```bash
# Test Docker login to Harbor
docker login YOUR_CI_CD_IP:8080
# Enter: ci-user and your-secure-password
# Create a test image
echo "FROM alpine:latest" > /tmp/test.Dockerfile
echo "RUN echo 'Hello from Harbor test image'" >> /tmp/test.Dockerfile
# Build and tag test image for public project
docker build -f /tmp/test.Dockerfile -t YOUR_CI_CD_IP:8080/public/test:latest /tmp
# Push to Harbor (requires authentication)
docker push YOUR_CI_CD_IP:8080/public/test:latest
# Verify image is in Harbor
curl -k https://localhost:8080/v2/_catalog
# Test public pull (no authentication required)
docker logout YOUR_CI_CD_IP:8080
docker pull YOUR_CI_CD_IP:8080/public/test:latest
# Clean up test image
docker rmi YOUR_CI_CD_IP:8080/public/test:latest
```
**Expected behavior**:
- ✅ **Push requires authentication**: `docker push` only works when logged in
- ✅ **Pull works without authentication**: `docker pull` works without login for public projects
- ✅ **Web UI accessible**: Harbor UI is available at `https://YOUR_CI_CD_IP:8080`
#### 5.8 Access Harbor Web UI
#### 5.9 Access Harbor Web UI
1. **Open your browser** and navigate to: `https://YOUR_CI_CD_IP:8080`
2. **Login with default credentials**:
@ -646,7 +656,7 @@ docker rmi YOUR_CI_CD_IP:8080/public/test:latest
- Password: `Harbor12345` (or your configured password)
3. **Change the admin password** when prompted (required on first login)
#### 5.9 Configure Harbor for Public Read, Authenticated Write
#### 5.10 Configure Harbor for Public Read, Authenticated Write
1. **Create a Public Project**:
- Go to **Projects** → **New Project**
@ -668,7 +678,7 @@ docker rmi YOUR_CI_CD_IP:8080/public/test:latest
- Set **Role**: `Developer`
- Click **OK**
#### 5.10 Test Harbor Authentication and Access Model
#### 5.11 Test Harbor Authentication and Access Model
```bash
# Test Docker login to Harbor
@ -701,7 +711,7 @@ docker rmi YOUR_CI_CD_IP:8080/public/test:latest
- ✅ **Pull works without authentication**: `docker pull` works without login for public projects
- ✅ **Web UI accessible**: Harbor UI is available at `https://YOUR_CI_CD_IP:8080`
#### 5.11 Harbor Access Model Summary
#### 5.12 Harbor Access Model Summary
Your Harbor registry is now configured with the following access model:

View file

@ -1,219 +0,0 @@
# Harbor Registry Configuration
This folder contains the configuration files for the Harbor Registry setup used in the CI/CD pipeline.
## Files
- `docker-compose.yml` - Docker Compose configuration for Harbor services
- `harbor.yml` - Harbor configuration file
- `README.md` - This file
## Architecture
This setup uses a hybrid approach for optimal maintainability and security:
### Repository Files (Version Controlled)
- Configuration files in `/opt/APP_NAME/registry/`
- Easy to update via git pull
- Version controlled and tracked
### System Files (Not Version Controlled)
- SSL certificates in `/etc/ssl/registry/`
- Authentication files in `/etc/registry/auth/`
- Registry data in Docker volume `/var/lib/registry`
## Usage
The setup process will:
1. Clone the repository to `/opt/APP_NAME/`
2. Create system directories for certificates and auth
3. Generate SSL certificates in `/etc/ssl/registry/`
4. Create authentication files in `/etc/registry/auth/`
5. Start the registry services using the hybrid configuration
## Configuration Notes
- **Registry**: Runs on port 5000 with HTTPS
- **Nginx**: Provides SSL termination and reverse proxy on port 8080
- **Registry UI**: Web interface accessible via nginx on port 8080
- **Authentication**: Uses htpasswd for push authentication
- **Storage**: Uses Docker volume for persistent data
- **Configuration**: Version controlled in repository
- **Certificates**: Stored in system SSL directory
## Security
- SSL certificates are self-signed and stored in system SSL directory
- Authentication files are stored in system auth directory
- Configuration is version controlled and easily auditable
- All communication uses HTTPS
- Clear separation between config, auth, and data
# Harbor Registry Setup
This directory contains the configuration for Harbor, an enterprise-grade container registry that provides:
- **Public read access** - Anyone can pull images without authentication
- **Authenticated write access** - Only authenticated users can push images
- **Web UI** - Modern web interface for managing images
- **Vulnerability scanning** - Built-in security scanning with Trivy
- **Role-based access control** - Fine-grained permissions
- **Multi-tenancy** - Project-based organization
## Prerequisites
1. Docker and Docker Compose installed
2. SSL certificates for HTTPS (recommended for production)
3. At least 4GB RAM and 10GB disk space
## Configuration
### 1. Update Configuration Files
Before starting Harbor, update the following files:
- `harbor.yml`: Update `hostname` and `harbor_admin_password`
- `docker-compose.yml`: Update secrets and passwords
### 2. SSL Certificates
Place your SSL certificates in `/etc/ssl/registry/`:
- `registry.crt` - SSL certificate
- `registry.key` - SSL private key
### 3. Environment Variables
Set the following environment variables:
```bash
export HARBOR_HOSTNAME=YOUR_CI_CD_IP
export HARBOR_ADMIN_PASSWORD=your-secure-password
export DB_PASSWORD=your-db-password
```
## Installation
1. **Stop existing registry** (if running):
```bash
docker compose down
```
2. **Start Harbor**:
```bash
docker compose up -d
```
3. **Wait for startup** (can take 2-3 minutes):
```bash
docker compose logs -f
```
## Initial Setup
1. **Access Harbor UI**: https://YOUR_CI_CD_IP:8080
2. **Login with default credentials**:
- Username: `admin`
- Password: `Harbor12345` (or your configured password)
3. **Change admin password** on first login
## Configuration for Public Read, Authenticated Write
### 1. Create a Public Project
1. Go to **Projects** → **New Project**
2. Set **Project Name**: `public`
3. Set **Access Level**: `Public`
4. Click **OK**
### 2. Create a Private Project (for authenticated writes)
1. Go to **Projects** → **New Project**
2. Set **Project Name**: `private`
3. Set **Access Level**: `Private`
4. Click **OK**
### 3. Create Users
1. Go to **Administration****Users** → **New User**
2. Create users with appropriate roles:
- **Developer**: Can push/pull to private projects
- **Guest**: Can only pull from public projects
## Usage
### Docker Login
```bash
docker login YOUR_CI_CD_IP:8080
```
### Push Images
```bash
# Tag your image
docker tag myimage:latest YOUR_CI_CD_IP:8080/public/myimage:latest
# Push to public project (requires authentication)
docker push YOUR_CI_CD_IP:8080/public/myimage:latest
```
### Pull Images
```bash
# Pull from public project (no authentication required)
docker pull YOUR_CI_CD_IP:8080/public/myimage:latest
```
## Security Features
- **Vulnerability Scanning**: Automatic CVE scanning with Trivy
- **Image Signing**: Content trust and image signing
- **RBAC**: Role-based access control
- **Audit Logs**: Complete audit trail of all operations
## Maintenance
### Backup
```bash
# Backup Harbor data
docker compose exec harbor-db pg_dump -U postgres registry > backup.sql
```
### Update
```bash
# Pull latest images
docker compose pull
# Restart services
docker compose up -d
```
### Logs
```bash
# View all logs
docker compose logs
# View specific service logs
docker compose logs harbor-core
```
## Troubleshooting
### Common Issues
1. **Startup takes too long**: Harbor needs time to initialize database and download vulnerability databases
2. **SSL certificate errors**: Ensure certificates are properly placed and have correct permissions
3. **Authentication issues**: Check user permissions and project access levels
### Health Check
```bash
# Check service status
docker compose ps
# Check Harbor health
curl -k https://YOUR_CI_CD_IP:8080/api/v2.0/health
```
## Resources
- [Harbor Documentation](https://goharbor.io/docs/)
- [Harbor GitHub](https://github.com/goharbor/harbor)
- [CNCF Harbor](https://landscape.cncf.io/card-mode?category=container-registry&grouping=category&selected=harbor)

View file

@ -1,135 +0,0 @@
services:
harbor-core:
image: goharbor/harbor-core:v2.10.0
container_name: harbor-core
restart: unless-stopped
depends_on:
- harbor-db
- harbor-redis
environment:
- CONFIG_PATH=/etc/harbor/app.conf
- CORE_SECRET=your-core-secret
- JOBSERVICE_SECRET=your-jobservice-secret
- TRIVY_ADAPTER_URL=http://harbor-trivy:8080
- TRIVY_ADAPTER_URL_INSECURE=true
volumes:
- ./harbor.yml:/etc/harbor/app.conf:ro
- harbor_core:/var/log/harbor
- harbor_data:/data
networks:
- harbor
harbor-db:
image: goharbor/harbor-db:v2.10.0
container_name: harbor-db
restart: unless-stopped
environment:
- POSTGRES_DB=registry
- POSTGRES_PASSWORD=your-db-password
- POSTGRES_USER=postgres
volumes:
- harbor_db:/var/lib/postgresql/data
networks:
- harbor
harbor-redis:
image: goharbor/redis-photon:v2.10.0
container_name: harbor-redis
restart: unless-stopped
command: redis-server --appendonly yes
volumes:
- harbor_redis:/data
networks:
- harbor
harbor-registry:
image: goharbor/registry-photon:v2.10.0
container_name: harbor-registry
restart: unless-stopped
depends_on:
- harbor-core
environment:
- REGISTRY_STORAGE_DELETE_ENABLED=true
- REGISTRY_STORAGE_CACHE_BLOBDESCRIPTOR=inmemory
- REGISTRY_STORAGE_FILESYSTEM_MAXTHREADS=100
volumes:
- harbor_registry:/storage
networks:
- harbor
harbor-portal:
image: goharbor/harbor-portal:v2.10.0
container_name: harbor-portal
restart: unless-stopped
depends_on:
- harbor-core
environment:
- CONFIG_PATH=/etc/harbor/app.conf
volumes:
- ./harbor.yml:/etc/harbor/app.conf:ro
networks:
- harbor
harbor-nginx:
image: goharbor/nginx-photon:v2.10.0
container_name: harbor-nginx
restart: unless-stopped
ports:
- "8080:8080"
depends_on:
- harbor-core
- harbor-portal
- harbor-registry
volumes:
- ./harbor.yml:/etc/harbor/app.conf:ro
- /etc/ssl/registry:/etc/ssl/certs:ro
- harbor_nginx:/var/log/harbor
networks:
- harbor
harbor-jobservice:
image: goharbor/harbor-jobservice:v2.10.0
container_name: harbor-jobservice
restart: unless-stopped
depends_on:
- harbor-core
- harbor-redis
environment:
- CONFIG_PATH=/etc/harbor/app.conf
- CORE_SECRET=your-core-secret
- JOBSERVICE_SECRET=your-jobservice-secret
volumes:
- ./harbor.yml:/etc/harbor/app.conf:ro
- harbor_jobservice:/var/log/harbor
networks:
- harbor
harbor-trivy:
image: goharbor/trivy-adapter-photon:v2.10.0
container_name: harbor-trivy
restart: unless-stopped
depends_on:
- harbor-core
environment:
- CONFIG_PATH=/etc/harbor/app.conf
- CORE_SECRET=your-core-secret
volumes:
- ./harbor.yml:/etc/harbor/app.conf:ro
- harbor_trivy:/var/log/harbor
networks:
- harbor
volumes:
harbor_core:
harbor_data:
harbor_db:
harbor_redis:
harbor_registry:
harbor_portal:
harbor_nginx:
harbor_jobservice:
harbor_trivy:
networks:
harbor:
driver: bridge

View file

@ -1,177 +0,0 @@
# Configuration file of Harbor
# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: YOUR_CI_CD_IP
# http related config
http:
# port for http, default is 80. If https enabled, this port will redirect to https port
port: 8080
# https related config
https:
# https port for harbor, default is 443
port: 8080
# The path of cert and key files for nginx
certificate: /etc/ssl/certs/registry.crt
private_key: /etc/ssl/certs/registry.key
# Uncomment external_url if you want to enable external proxy
# And when it enabled the hostname will no longer used
# external_url: https://reg.mydomain.com:8433
# The initial password of Harbor admin
# Change it from default after updating for the first time
harbor_admin_password: Harbor12345
# Harbor DB configuration
database:
# The password for the root user of Harbor DB. Change this before any production use.
password: your-db-password
# The maximum number of connections in the idle connection pool. If it <=0, no idle connections are retained.
max_idle_conns: 50
# The maximum number of open connections to the database. If it <= 0, then there is no limit on the number of open connections.
# Note: the default number of connections is 100 for postgres.
max_open_conns: 100
# The default data volume
data_volume: /data
# Harbor Storage settings by default is using /data as default volume.
# Uncomment storage_service setting If you want to using external storage.
# storage_service:
# # ca_bundle is the path to the custom root ca certificate, which will be injected into the truststore
# # of registry's and chart repository's containers. This is usually needed when the user hosts a internal storage with self signed certificate.
# ca_bundle:
# # storage backend, default is filesystem, options include filesystem, azure, gcs, s3, swift and oss
# # for more info about this configuration please refer https://docs.docker.com/registry/configuration/
# filesystem:
# maxthreads: 100
# # set disable_cache to true want to disable cache of redis in registry
# # set disable_cache to false want to enable cache of redis in registry
# disable_cache: true
# Trivy configuration
trivy:
# enabled the flag to enable Trivy scanner
enabled: true
# ignore update if the CVEs are already in the whitelist. It only works when scanner is V1
ignore_unfixed: false
# skip update if the CVEs are already in the whitelist, It only works when scanner is V2
skip_update: false
# generate a scan report in the JSON format
json_output: false
# in online mode, Trivy will download the latest database from GitHub and scan it offline
# in offline mode, if the database does not exist locally, Trivy will exit with an error
offline_scan: false
# insecure Skip tls certificate verification
insecure: false
# github_token the github access token to download Trivy DB (see https://github.com/settings/tokens)
# This is required only when the GitHub rate limiting is exceeded
github_token: ""
# Jobservice configuration
jobservice:
# Maximum number of job workers in job service
max_job_workers: 10
# Notification configuration
notification:
# Maximum retry count for webhook job
webhook_job_max_retry: 10
# Log configurations
log:
# options are debug, info, warning, error, fatal
level: info
# configs for logs in local storage
local:
# Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.
rotate_count: 50
# Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
# If the M is followed by M, the size is assumed to be in megabytes. If the G is followed by G, the size is assumed to be in gigabytes. So size 100, size 100k, size 100M and size 100G are all valid.
rotate_size: 200M
# The directory on your host that store log
location: /var/log/harbor
# Uncomment following lines to enable external syslog endpoint.
# external_endpoint:
# # protocol used to transmit log to external endpoint, options is tcp or udp
# protocol: tcp
# # The host of external endpoint
# host: localhost
# # Port of external endpoint
# port: 5140
#This attribute is for migrator to detect the version of the .cfg file, DO NOT MODIFY!
_version: 2.10.0
# Uncomment external_database if using external database.
# external_database:
# harbor:
# host: harbor_db_host
# port: harbor_db_port
# db_name: harbor
# username: user
# password: password
# ssl_mode: disable
# max_idle_conns: 2
# max_open_conns: 0
# notary_signer:
# host: notary_signer_db_host
# port: notary_signer_db_port
# db_name: notary_signer
# username: user
# password: password
# ssl_mode: disable
# notary_server:
# host: notary_server_db_host
# port: notary_server_db_port
# db_name: notary_server
# username: user
# password: password
# ssl_mode: disable
# Uncomment redis if using external Redis server
# redis:
# host: redis_host
# port: redis_port
# password: redis_password
# # db_index 0 is for core, it's unchangeable
# registry_db_index: 1
# jobservice_db_index: 2
# chartmuseum_db_index: 3
# trivy_db_index: 5
# idle_timeout_seconds: 30
# Uncomment uaa for trusting the certificate of uaa instance that is hosted via self-signed cert.
# uaa:
# ca_file: /path/to/ca
# Global proxy settings
# http_proxy:
# https_proxy:
# no_proxy:
# - 127.0.0.1
# - localhost
# - core
# - redis
# - postgresql
# - notary-db
# - notary-signer
# - clair
# - trivy-adapter
# - trivy
# - chartmuseum
# - jobservice
# - registry
# - portal
# - log
# - nginx
# metric:
# enabled: false
# port: 9090
# path: /metrics

View file

@ -94,28 +94,28 @@ cleanup_registry() {
log_info "Cleaning up Harbor registry..."
if [ ! -d "$REGISTRY_DIR" ]; then
log_warning "Registry directory not found: $REGISTRY_DIR"
log_warning "Harbor directory not found: $REGISTRY_DIR"
return
fi
if [ "$DRY_RUN" = "true" ]; then
log_warning "DRY RUN MODE - No changes will be made"
echo "Would run: cd $REGISTRY_DIR && docker-compose exec harbor-registry registry garbage-collect"
echo "Would run: cd $REGISTRY_DIR && docker-compose exec registry registry garbage-collect"
return
fi
# Change to registry directory
# Change to Harbor directory
cd "$REGISTRY_DIR"
# Check if Harbor is running
if ! docker-compose ps | grep -q "harbor-registry.*Up"; then
if ! docker-compose ps | grep -q "registry.*Up"; then
log_warning "Harbor registry is not running, skipping registry cleanup"
return
fi
# Run Harbor registry garbage collection
log_info "Running Harbor registry garbage collection..."
docker-compose exec -T harbor-registry registry garbage-collect
docker-compose exec -T registry registry garbage-collect
log_success "Harbor registry cleanup completed"
}