diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..58e14bf --- /dev/null +++ b/.env.example @@ -0,0 +1,24 @@ +# Keycloak Environment Configuration +# Copy this file to .env and fill in the actual values + +# Database Configuration (Required) +KC_DB_USERNAME=keycloak +KC_DB_PASSWORD=your_db_password_here +POSTGRES_PASSWORD=your_db_password_here + +# Keycloak Admin (Only needed for initial setup) +# KEYCLOAK_ADMIN=admin +# KEYCLOAK_ADMIN_PASSWORD=admin_password_here + +# Backup Configuration (Optional) +# RUN_BACKUP=true +# BACKUP_DATA_ENV=sandbox +# INIT_DATA_ENV=sandbox + +# Distributed High Availability Configuration +# For multi-VM setup with PostgreSQL replication +# PRIMARY_DB_HOST=primary-vm-ip-address + +# Replication Configuration +# POSTGRES_REPLICATION_USER=replicator +# POSTGRES_REPLICATION_PASSWORD=replication_password_here \ No newline at end of file diff --git a/DISTRIBUTED_HA_GUIDE.md b/DISTRIBUTED_HA_GUIDE.md new file mode 100644 index 0000000..f24bbf0 --- /dev/null +++ b/DISTRIBUTED_HA_GUIDE.md @@ -0,0 +1,268 @@ +# Distributed High Availability Setup Guide + +This guide provides detailed instructions for setting up Keycloak in a distributed high availability configuration across two VMs with PostgreSQL streaming replication. + +## Architecture Overview + +``` +┌─────────────────┐ ┌─────────────────┐ +│ Primary VM │ │ Replica VM │ +│ │ │ │ +│ ┌─────────────┐ │ │ ┌─────────────┐ │ +│ │ Keycloak │ │ │ │ Keycloak │ │ +│ │ (R/W) │ │ │ │ (R/O) │ │ +│ │ Port: 3123 │ │ │ │ Port: 3124 │ │ +│ └─────────────┘ │ │ └─────────────┘ │ +│ │ │ │ │ │ +│ ┌─────────────┐ │ Repl │ ┌─────────────┐ │ +│ │ PostgreSQL │◄├─────────┤►│ PostgreSQL │ │ +│ │ (Primary) │ │ │ │ (Replica) │ │ +│ │ Port: 5432 │ │ │ │ Port: 5433 │ │ +│ └─────────────┘ │ │ └─────────────┘ │ +└─────────────────┘ └─────────────────┘ +``` + +## Prerequisites + +### Infrastructure +- Two VMs with Docker and Docker Compose installed +- Network connectivity between VMs on ports 5432 and 5433 +- Sufficient disk space for database storage and WAL files + +### Software Requirements +- Docker 20.10+ +- Docker Compose 2.0+ +- Network tools: `nc` (netcat) for connectivity testing + +## Step-by-Step Setup + +### Step 1: Prepare Primary VM + +1. **Clone repository and navigate to directory:** + ```bash + git clone + cd keycloak-deployment + ``` + +2. **Run primary setup:** + ```bash + ./cmds/setup-distributed.sh primary + ``` + + This will prompt for: + - Database username (default: keycloak) + - Database password + - Replication user password + +3. **Deploy primary instance:** + ```bash + ./cmds/deploy-primary.sh up + ``` + +4. **Verify primary deployment:** + ```bash + ./cmds/deploy-primary.sh status + ``` + +5. **Note the primary VM's IP address** for replica configuration: + ```bash + ip addr show | grep inet + ``` + +### Step 2: Prepare Replica VM + +1. **Clone repository and navigate to directory:** + ```bash + git clone + cd keycloak-deployment + ``` + +2. **Run replica setup:** + ```bash + ./cmds/setup-distributed.sh replica + ``` + + This will prompt for: + - Primary VM IP address + - Database username (must match primary) + - Database password (must match primary) + - Replication user password (must match primary) + +3. **Deploy replica instance:** + ```bash + ./cmds/deploy-replica.sh up + ``` + +4. **Verify replica deployment:** + ```bash + ./cmds/deploy-replica.sh status + ``` + +### Step 3: Verify Replication + +1. **On replica VM, check replication status:** + ```bash + ./cmds/health-check-distributed.sh replica + ``` + +2. **On primary VM, check connected replicas:** + ```bash + ./cmds/health-check-distributed.sh primary + ``` + +3. **Test data replication:** + - Create test data in primary Keycloak + - Verify it appears in replica Keycloak (read-only) + +## Daily Operations + +### Health Monitoring + +```bash +# Check all instances +./cmds/health-check-distributed.sh all + +# Continuous monitoring +./cmds/health-check-distributed.sh monitor +``` + +### Viewing Logs + +```bash +# Primary VM +./cmds/deploy-primary.sh logs + +# Replica VM +./cmds/deploy-replica.sh logs +``` + +### Stopping Services + +```bash +# Primary VM +./cmds/deploy-primary.sh down + +# Replica VM +./cmds/deploy-replica.sh down +``` + +## Troubleshooting + +### Common Issues + +1. **Replica cannot connect to primary:** + - Check network connectivity: `nc -z 5432` + - Verify firewall settings + - Check primary VM is running + +2. **Replication lag is high:** + - Check network bandwidth between VMs + - Monitor disk I/O on both VMs + - Review PostgreSQL logs + +3. **Keycloak replica shows as unhealthy:** + - Check if database replica is ready + - Verify database connectivity + - Review Keycloak logs for specific errors + +### Log Locations + +- **Docker Compose logs:** `docker compose logs ` +- **PostgreSQL logs:** Inside container at `/var/log/postgresql/` +- **Keycloak logs:** Shown in Docker Compose logs + +## Disaster Recovery + +### Primary VM Failure + +1. **Immediate response:** + - Update DNS/load balancer to point to replica VM + - Monitor replica for increased load + +2. **Promote replica to primary (manual process):** + ```bash + # On replica VM + docker compose -f docker-compose.replica.yaml exec db-replica \ + psql -U keycloak -d postgres -c "SELECT pg_promote();" + ``` + +3. **Restart Keycloak without read-only restrictions:** + - Stop replica Keycloak + - Update configuration to remove read-only settings + - Restart Keycloak + +### Replica VM Failure + +1. **Primary continues serving traffic** +2. **Fix replica VM issues** +3. **Rebuild replica from primary:** + ```bash + ./cmds/deploy-replica.sh down + # Remove old data + docker volume rm keycloak-deployment_db-replica-data + ./cmds/deploy-replica.sh up + ``` + +## Maintenance + +### Updating Keycloak Version + +1. **Update config.sh with new version** +2. **Update primary first:** + ```bash + ./cmds/deploy-primary.sh down + docker compose -f docker-compose.primary.yaml pull + ./cmds/deploy-primary.sh up + ``` +3. **Update replica after primary is verified:** + ```bash + ./cmds/deploy-replica.sh down + docker compose -f docker-compose.replica.yaml pull + ./cmds/deploy-replica.sh up + ``` + +### Database Maintenance + +- **Backups:** Run on primary VM only +- **Vacuum/Analyze:** Performed automatically +- **Monitoring:** Use provided health check scripts + +## Security Considerations + +1. **Network Security:** + - Use VPN or private networks between VMs + - Configure firewalls to allow only necessary ports + - Use strong passwords for replication user + +2. **Database Security:** + - Encrypt replication traffic (configure SSL) + - Regularly rotate passwords + - Monitor access logs + +3. **Application Security:** + - Keep Keycloak updated + - Monitor for security advisories + - Use HTTPS for all Keycloak traffic + +## Performance Tuning + +### PostgreSQL Settings + +Edit `config/postgresql.conf` for your environment: +- `shared_buffers`: 25% of RAM +- `wal_keep_size`: Based on network reliability +- `max_connections`: Based on expected load + +### Keycloak Settings + +- Monitor JVM heap usage +- Adjust database connection pool sizes +- Configure session storage appropriately + +## Support + +For issues with this deployment: +1. Check this documentation +2. Review logs using provided scripts +3. Test connectivity between VMs +4. Verify configuration files match between VMs \ No newline at end of file diff --git a/README.md b/README.md index a806ff3..15b9ae8 100644 --- a/README.md +++ b/README.md @@ -1,26 +1,175 @@ -# Keyloak Deployment +# Keycloak Deployment [Keycloak](https://www.keycloak.org/) is the Identity and Access Management system used by the UC Davis Library. It primarily acts as an identity broker for the UC Davis Central Authentication System (CAS). -After making any changes, you can deploy by following these steps: -1. Check `config.sh` to ensure that everything looks good. -2. Run `./cmds/generate-deployment-files.sh`. -3. Check changes into github and tag the release. -4. ```ssh auth.library.ucdavis.edu``` -5. `cd /opt` and then into the version you are deploying - currently `prod` or `sandbox`. -6. git pull either the tag or branch you need. -7. `docker compose pull` -8. If you made changes to the apache config, move it. `mv apache/keycloak.conf /etc/httpd/conf.d/prod.conf` -9. Verify that your env file is good. -10. `docker compose up -d` +## High Availability Deployment Options -## Env +This deployment supports two high availability configurations: +### 1. Single-VM Hot Spare (Legacy) +A simpler setup where both Keycloak instances run on the same VM sharing one database. + +### 2. Distributed High Availability (Recommended) +A robust multi-VM setup with PostgreSQL streaming replication and read-only replica Keycloak instances. + +## Distributed High Availability Setup + +The distributed setup provides true high availability across separate VMs with PostgreSQL streaming replication. + +### Architecture +- **Primary VM**: Keycloak + PostgreSQL primary database +- **Replica VM**: Keycloak (read-only) + PostgreSQL replica database +- **Automatic Replication**: Real-time data synchronization between databases +- **Failover**: Manual or automated failover to replica VM + +### Initial Setup + +#### On Primary VM: +```bash +# Setup primary configuration +./cmds/setup-distributed.sh primary + +# Deploy primary instance +./cmds/deploy-primary.sh up +``` + +#### On Replica VM: +```bash +# Setup replica configuration (requires primary VM IP) +./cmds/setup-distributed.sh replica + +# Deploy replica instance +./cmds/deploy-replica.sh up +``` + +### Management Commands + +#### Primary VM Operations: +```bash +# Deploy/start primary +./cmds/deploy-primary.sh up + +# Stop primary +./cmds/deploy-primary.sh down + +# Check primary status +./cmds/deploy-primary.sh status + +# View primary logs +./cmds/deploy-primary.sh logs +``` + +#### Replica VM Operations: +```bash +# Deploy/start replica +./cmds/deploy-replica.sh up + +# Stop replica +./cmds/deploy-replica.sh down + +# Check replica status +./cmds/deploy-replica.sh status + +# View replica logs +./cmds/deploy-replica.sh logs +``` + +### Health Monitoring + +```bash +# Check all instances +./cmds/health-check-distributed.sh all + +# Check only primary +./cmds/health-check-distributed.sh primary + +# Check only replica +./cmds/health-check-distributed.sh replica + +# Continuous monitoring +./cmds/health-check-distributed.sh monitor +``` + +### Single-VM Hot Spare (Legacy) + +For backward compatibility, the original single-VM hot spare is still available: + +```bash +# Enable hot spare +./cmds/manage-hot-spare.sh enable + +# Deploy with hot spare +docker compose --profile hot-spare up -d + +# Check status +./cmds/health-check.sh +``` + +## Environment Variables + +### Basic Configuration | Variable | Description | Required? | | -------- | ----------- | --------- | -| KC_DB_USERNAME | PG user | Y | -| KC_DB_PASSWORD | PG password | Y | -| POSTGRES_PASSWORD | Same as above | Y | -| KEYCLOAK_ADMIN | Creates KC admin user on start | Only use if setting up KC for first time | -| KEYCLOAK_ADMIN_PASSWORD | KC admin user password created on start | Only use if setting up KC for first time | +| KC_DB_USERNAME | PostgreSQL username | Y | +| KC_DB_PASSWORD | PostgreSQL password | Y | +| POSTGRES_PASSWORD | Same as KC_DB_PASSWORD | Y | +| KEYCLOAK_ADMIN | Creates KC admin user on start | Only for initial setup | +| KEYCLOAK_ADMIN_PASSWORD | KC admin user password | Only for initial setup | + +### Distributed HA Configuration +| Variable | Description | Required? | +| -------- | ----------- | --------- | +| PRIMARY_DB_HOST | IP address of primary VM | Y (replica only) | +| POSTGRES_REPLICATION_USER | Replication username | Y (both VMs) | +| POSTGRES_REPLICATION_PASSWORD | Replication password | Y (both VMs) | + +### Network Requirements + +For distributed deployment: +- **Port 5432**: PostgreSQL primary (primary VM) +- **Port 5433**: PostgreSQL replica (replica VM) +- **Port 3123**: Keycloak primary (primary VM) +- **Port 3124**: Keycloak replica (replica VM) +- **Network connectivity** between VMs for database replication + +## Deployment Process + +### Distributed Deployment (Recommended) + +1. **Setup Primary VM:** + ```bash + ./cmds/setup-distributed.sh primary + ./cmds/deploy-primary.sh up + ``` + +2. **Setup Replica VM:** + ```bash + ./cmds/setup-distributed.sh replica + ./cmds/deploy-replica.sh up + ``` + +3. **Verify deployment:** + ```bash + ./cmds/health-check-distributed.sh all + ``` + +### Legacy Single-VM Deployment + +1. Check `config.sh` configuration +2. Run `./cmds/generate-deployment-files.sh` +3. Deploy: `docker compose up -d` or `docker compose --profile hot-spare up -d` + +## Failover Procedures + +### Manual Failover to Replica +1. Stop primary services: `./cmds/deploy-primary.sh down` +2. Promote replica to primary (requires manual PostgreSQL promotion) +3. Update DNS/load balancer to point to replica VM +4. Restart replica Keycloak without read-only restrictions + +### Recovery Procedures +1. Fix primary VM issues +2. Restore primary from replica backup +3. Re-establish replication +4. Failback to primary when ready diff --git a/apache/keycloak.conf b/apache/keycloak.conf index 8865dcf..cbd1eb7 100644 --- a/apache/keycloak.conf +++ b/apache/keycloak.conf @@ -6,6 +6,14 @@ ServerName sandbox.auth.library.ucdavis.edu + + RewriteEngine On + # remove all GA and Hotjar cookies + RewriteCond %{HTTP:Cookie} ^(.*)(ga_[^;]+|_hj[^;]+;?)(.*)$ [NC] + RewriteRule .* - [E=MODIFIED_COOKIE:%1%3] + Header set Cookie "%{MODIFIED_COOKIE}e" env=MODIFIED_COOKIE + + Require ip 172.19.19.0/24 169.237.102.0/26 169.237.102.128/25 @@ -27,8 +35,27 @@ RequestHeader set x-forwarded-proto "https" RequestHeader set x-ssl-client-cert "%{SSL_CLIENT_CERT}s" - ProxyPass / http://localhost:3123/ - ProxyPassReverse / http://localhost:3123/ + # Hot spare configuration with health checks + # Define upstream servers + ProxyPass /health/balancer-manager ! + + SetHandler balancer-manager + Require ip 172.19.19.0/24 169.237.102.0/26 169.237.102.128/25 + + + # Primary and spare Keycloak instances with health checks + ProxyPass / balancer://keycloak-cluster/ + ProxyPassReverse / balancer://keycloak-cluster/ + + + # Primary instance + BalancerMember http://localhost:3123 status=+H + # Hot spare instance + BalancerMember http://localhost:3124 status=+H + # Health check configuration + ProxySet retry=300 + ProxySet ping=5 + SSLEngine on SSLCertificateFile /etc/ssl/certs/auth_library_ucdavis_edu_cert.cer diff --git a/cmds/deploy-primary.sh b/cmds/deploy-primary.sh new file mode 100755 index 0000000..10bf50f --- /dev/null +++ b/cmds/deploy-primary.sh @@ -0,0 +1,103 @@ +#!/bin/bash + +### +# Deploy primary Keycloak instance with PostgreSQL primary database +### + +set -e +CMDS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" +cd $CMDS_DIR/.. + +source ./config.sh + +function usage() { + echo "Usage: $0 [up|down|status|logs]" + echo "" + echo "Commands:" + echo " up - Start primary instance" + echo " down - Stop primary instance" + echo " status - Show status of primary services" + echo " logs - Show logs of primary services" + exit 1 +} + +function deploy_primary() { + echo "Deploying primary Keycloak instance..." + + # Check if replication user environment variables are set + if [ -z "$POSTGRES_REPLICATION_USER" ] || [ -z "$POSTGRES_REPLICATION_PASSWORD" ]; then + echo "Warning: POSTGRES_REPLICATION_USER and POSTGRES_REPLICATION_PASSWORD should be set in .env for replication" + fi + + # Create archive directory for WAL files + mkdir -p ./archive + + # Start primary services + docker compose -f docker-compose.primary.yaml up -d + + echo "" + echo "Primary instance deployed successfully!" + echo "Keycloak: http://localhost:$KC_HOST_PORT" + echo "Database: localhost:5432" + echo "" + echo "To deploy replica on another VM, use:" + echo " ./cmds/deploy-replica.sh up" +} + +function stop_primary() { + echo "Stopping primary Keycloak instance..." + docker compose -f docker-compose.primary.yaml down + echo "Primary instance stopped." +} + +function show_status() { + echo "=== Primary Instance Status ===" + docker compose -f docker-compose.primary.yaml ps + + echo "" + echo "=== Health Checks ===" + + # Check Keycloak health + echo -n "Keycloak health: " + if curl -sf "http://localhost:$KC_HOST_PORT/health/ready" > /dev/null 2>&1; then + echo "✓ HEALTHY" + else + echo "✗ UNHEALTHY" + fi + + # Check database health + echo -n "Database health: " + if docker compose -f docker-compose.primary.yaml exec -T db pg_isready > /dev/null 2>&1; then + echo "✓ HEALTHY" + else + echo "✗ UNHEALTHY" + fi + + # Check replication status + echo "" + echo "=== Replication Status ===" + docker compose -f docker-compose.primary.yaml exec -T db psql -U $KC_DB_USERNAME -d postgres -c "SELECT client_addr, state, sync_state FROM pg_stat_replication;" 2>/dev/null || echo "No replicas connected" +} + +function show_logs() { + docker compose -f docker-compose.primary.yaml logs -f +} + +# Parse command line arguments +case "${1:-}" in + up) + deploy_primary + ;; + down) + stop_primary + ;; + status) + show_status + ;; + logs) + show_logs + ;; + *) + usage + ;; +esac \ No newline at end of file diff --git a/cmds/deploy-replica.sh b/cmds/deploy-replica.sh new file mode 100755 index 0000000..3c411ff --- /dev/null +++ b/cmds/deploy-replica.sh @@ -0,0 +1,143 @@ +#!/bin/bash + +### +# Deploy replica Keycloak instance with PostgreSQL replica database +### + +set -e +CMDS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" +cd $CMDS_DIR/.. + +source ./config.sh + +function usage() { + echo "Usage: $0 [up|down|status|logs]" + echo "" + echo "Commands:" + echo " up - Start replica instance" + echo " down - Stop replica instance" + echo " status - Show status of replica services" + echo " logs - Show logs of replica services" + echo "" + echo "Prerequisites:" + echo " - PRIMARY_DB_HOST must be set in .env file" + echo " - Primary instance must be running and accessible" + echo " - Replication credentials must match primary" + exit 1 +} + +function deploy_replica() { + echo "Deploying replica Keycloak instance..." + + # Check required environment variables + if [ -z "$PRIMARY_DB_HOST" ]; then + echo "ERROR: PRIMARY_DB_HOST must be set in .env file" + echo "Example: PRIMARY_DB_HOST=192.168.1.100" + exit 1 + fi + + if [ -z "$POSTGRES_REPLICATION_USER" ] || [ -z "$POSTGRES_REPLICATION_PASSWORD" ]; then + echo "ERROR: POSTGRES_REPLICATION_USER and POSTGRES_REPLICATION_PASSWORD must be set in .env" + exit 1 + fi + + # Test connectivity to primary database + echo "Testing connectivity to primary database at $PRIMARY_DB_HOST:5432..." + if ! nc -z "$PRIMARY_DB_HOST" 5432; then + echo "ERROR: Cannot connect to primary database at $PRIMARY_DB_HOST:5432" + echo "Please ensure:" + echo " 1. Primary instance is running" + echo " 2. PRIMARY_DB_HOST is correct" + echo " 3. Network connectivity exists" + exit 1 + fi + + echo "Primary database is accessible." + + # Start replica services + docker compose -f docker-compose.replica.yaml up -d + + echo "" + echo "Replica instance deployment started!" + echo "Keycloak Replica: http://localhost:3124 (read-only)" + echo "Database Replica: localhost:5433" + echo "" + echo "Note: Initial replication setup may take a few minutes." + echo "Use './cmds/deploy-replica.sh status' to monitor progress." +} + +function stop_replica() { + echo "Stopping replica Keycloak instance..." + docker compose -f docker-compose.replica.yaml down + echo "Replica instance stopped." +} + +function show_status() { + echo "=== Replica Instance Status ===" + docker compose -f docker-compose.replica.yaml ps + + echo "" + echo "=== Health Checks ===" + + # Check Keycloak replica health + echo -n "Keycloak replica health: " + if curl -sf "http://localhost:3124/health/ready" > /dev/null 2>&1; then + echo "✓ HEALTHY (read-only)" + else + echo "✗ UNHEALTHY" + fi + + # Check database replica health + echo -n "Database replica health: " + if docker compose -f docker-compose.replica.yaml exec -T db-replica pg_isready > /dev/null 2>&1; then + echo "✓ HEALTHY" + else + echo "✗ UNHEALTHY" + fi + + # Check replication status + echo "" + echo "=== Replication Status ===" + echo -n "Primary connection: " + if [ -n "$PRIMARY_DB_HOST" ] && nc -z "$PRIMARY_DB_HOST" 5432; then + echo "✓ CONNECTED" + + # Check replication lag + echo "Checking replication lag..." + docker compose -f docker-compose.replica.yaml exec -T db-replica psql -U $KC_DB_USERNAME -d postgres -c " + SELECT + CASE + WHEN pg_is_in_recovery() THEN 'Replica (standby)' + ELSE 'Primary' + END as server_type, + pg_last_wal_receive_lsn() as last_received, + pg_last_wal_replay_lsn() as last_replayed, + EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp())) as lag_seconds; + " 2>/dev/null || echo "Unable to check replication status" + else + echo "✗ DISCONNECTED" + fi +} + +function show_logs() { + docker compose -f docker-compose.replica.yaml logs -f +} + +# Parse command line arguments +case "${1:-}" in + up) + deploy_replica + ;; + down) + stop_replica + ;; + status) + show_status + ;; + logs) + show_logs + ;; + *) + usage + ;; +esac \ No newline at end of file diff --git a/cmds/health-check-distributed.sh b/cmds/health-check-distributed.sh new file mode 100755 index 0000000..f201837 --- /dev/null +++ b/cmds/health-check-distributed.sh @@ -0,0 +1,231 @@ +#!/bin/bash + +### +# Comprehensive health check for distributed Keycloak deployment +### + +set -e +CMDS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" +cd $CMDS_DIR/.. + +source ./config.sh + +function usage() { + echo "Usage: $0 [primary|replica|all|monitor]" + echo "" + echo "Commands:" + echo " primary - Check primary instance health" + echo " replica - Check replica instance health" + echo " all - Check both primary and replica" + echo " monitor - Continuous monitoring (Ctrl+C to stop)" + exit 1 +} + +function check_primary() { + echo "=== Primary Instance Health Check ===" + echo "Timestamp: $(date)" + echo "" + + # Check if primary services are running + local primary_running=false + if docker compose -f docker-compose.primary.yaml ps --services --filter "status=running" | grep -q keycloak; then + primary_running=true + fi + + if [ "$primary_running" = true ]; then + echo "✓ Primary services are running" + + # Check Keycloak health + echo -n "Keycloak health: " + if curl -sf "http://localhost:$KC_HOST_PORT/health/ready" > /dev/null 2>&1; then + echo "✓ HEALTHY" + primary_kc_healthy=true + else + echo "✗ UNHEALTHY" + primary_kc_healthy=false + fi + + # Check database health + echo -n "Database health: " + if docker compose -f docker-compose.primary.yaml exec -T db pg_isready > /dev/null 2>&1; then + echo "✓ HEALTHY" + primary_db_healthy=true + else + echo "✗ UNHEALTHY" + primary_db_healthy=false + fi + + # Check for connected replicas + echo "" + echo "Connected replicas:" + docker compose -f docker-compose.primary.yaml exec -T db psql -U $KC_DB_USERNAME -d postgres -c " + SELECT + client_addr as replica_ip, + state, + sync_state, + EXTRACT(EPOCH FROM (now() - backend_start)) as connection_duration_seconds + FROM pg_stat_replication; + " 2>/dev/null || echo "No replicas connected" + + else + echo "✗ Primary services are not running" + primary_kc_healthy=false + primary_db_healthy=false + fi + + return 0 +} + +function check_replica() { + echo "=== Replica Instance Health Check ===" + echo "Timestamp: $(date)" + echo "" + + # Check if replica services are running + local replica_running=false + if docker compose -f docker-compose.replica.yaml ps --services --filter "status=running" | grep -q keycloak-replica; then + replica_running=true + fi + + if [ "$replica_running" = true ]; then + echo "✓ Replica services are running" + + # Check Keycloak replica health + echo -n "Keycloak replica health: " + if curl -sf "http://localhost:3124/health/ready" > /dev/null 2>&1; then + echo "✓ HEALTHY (read-only)" + replica_kc_healthy=true + else + echo "✗ UNHEALTHY" + replica_kc_healthy=false + fi + + # Check database replica health + echo -n "Database replica health: " + if docker compose -f docker-compose.replica.yaml exec -T db-replica pg_isready > /dev/null 2>&1; then + echo "✓ HEALTHY" + replica_db_healthy=true + else + echo "✗ UNHEALTHY" + replica_db_healthy=false + fi + + # Check replication status and lag + echo "" + echo "Replication status:" + if [ -n "$PRIMARY_DB_HOST" ] && nc -z "$PRIMARY_DB_HOST" 5432 2>/dev/null; then + echo "✓ Connected to primary at $PRIMARY_DB_HOST:5432" + + docker compose -f docker-compose.replica.yaml exec -T db-replica psql -U $KC_DB_USERNAME -d postgres -c " + SELECT + CASE + WHEN pg_is_in_recovery() THEN 'Replica (standby)' + ELSE 'Primary (ERROR: should be replica!)' + END as server_type, + pg_last_wal_receive_lsn() as last_received, + pg_last_wal_replay_lsn() as last_replayed, + ROUND(EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp())), 2) as lag_seconds; + " 2>/dev/null || echo "Unable to check replication status" + else + echo "✗ Cannot connect to primary database" + replica_db_healthy=false + fi + + else + echo "✗ Replica services are not running" + replica_kc_healthy=false + replica_db_healthy=false + fi + + return 0 +} + +function check_all() { + check_primary + echo "" + echo "================================" + echo "" + check_replica + + echo "" + echo "=== Overall System Status ===" + + local overall_status="HEALTHY" + local status_details="" + + if [ "$primary_kc_healthy" = true ] && [ "$primary_db_healthy" = true ]; then + status_details="✓ Primary: HEALTHY" + else + status_details="✗ Primary: UNHEALTHY" + overall_status="DEGRADED" + fi + + if [ "$replica_kc_healthy" = true ] && [ "$replica_db_healthy" = true ]; then + status_details="$status_details, ✓ Replica: HEALTHY" + else + status_details="$status_details, ✗ Replica: UNHEALTHY" + if [ "$overall_status" = "DEGRADED" ]; then + overall_status="CRITICAL" + else + overall_status="DEGRADED" + fi + fi + + echo "Status: $overall_status" + echo "Details: $status_details" + + case "$overall_status" in + "HEALTHY") + echo "✓ Both primary and replica are functioning normally" + return 0 + ;; + "DEGRADED") + echo "⚠ One instance is down but service can continue" + return 1 + ;; + "CRITICAL") + echo "✗ Both instances have issues - immediate attention required" + return 2 + ;; + esac +} + +function monitor_continuous() { + echo "Starting continuous monitoring (Ctrl+C to stop)..." + echo "Checking every 30 seconds..." + echo "" + + while true; do + clear + echo "=== Continuous Monitoring ===" + check_all + echo "" + echo "Next check in 30 seconds... (Ctrl+C to stop)" + sleep 30 + done +} + +# Initialize health variables +primary_kc_healthy=false +primary_db_healthy=false +replica_kc_healthy=false +replica_db_healthy=false + +# Parse command line arguments +case "${1:-all}" in + primary) + check_primary + ;; + replica) + check_replica + ;; + all) + check_all + ;; + monitor) + monitor_continuous + ;; + *) + usage + ;; +esac \ No newline at end of file diff --git a/cmds/health-check.sh b/cmds/health-check.sh new file mode 100755 index 0000000..32e4824 --- /dev/null +++ b/cmds/health-check.sh @@ -0,0 +1,98 @@ +#!/bin/bash + +### +# Health check script for Keycloak hot spare deployment +### + +set -e +CMDS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" +cd $CMDS_DIR/.. + +source ./config.sh + +function check_instance() { + local port=$1 + local name=$2 + + echo -n "Checking $name (port $port)... " + + if curl -sf "http://localhost:$port/health/ready" > /dev/null 2>&1; then + echo "✓ HEALTHY" + return 0 + else + echo "✗ UNHEALTHY" + return 1 + fi +} + +function check_database() { + echo -n "Checking database... " + + if docker compose exec -T db pg_isready > /dev/null 2>&1; then + echo "✓ HEALTHY" + return 0 + else + echo "✗ UNHEALTHY" + return 1 + fi +} + +echo "=== Keycloak Hot Spare Health Check ===" +echo "Timestamp: $(date)" +echo "" + +# Check if hot spare is enabled +if grep -q "ENABLE_HOT_SPARE=true" config.sh; then + echo "Hot spare is ENABLED" + echo "" + + # Check database first + check_database + echo "" + + # Check primary instance + primary_healthy=0 + check_instance $KC_HOST_PORT "Primary Keycloak" || primary_healthy=1 + + # Check spare instance if it should be running + spare_healthy=0 + if docker compose ps | grep -q "keycloak-spare"; then + check_instance $KC_SPARE_HOST_PORT "Spare Keycloak" || spare_healthy=1 + else + echo "Spare Keycloak (port $KC_SPARE_HOST_PORT)... NOT RUNNING" + spare_healthy=1 + fi + + echo "" + echo "=== Summary ===" + if [ $primary_healthy -eq 0 ] && [ $spare_healthy -eq 0 ]; then + echo "Status: HEALTHY - Both instances running" + exit 0 + elif [ $primary_healthy -eq 0 ] || [ $spare_healthy -eq 0 ]; then + echo "Status: DEGRADED - One instance running" + exit 1 + else + echo "Status: UNHEALTHY - No instances running" + exit 2 + fi +else + echo "Hot spare is DISABLED" + echo "" + + # Check database + check_database + echo "" + + # Check primary instance only + if check_instance $KC_HOST_PORT "Primary Keycloak"; then + echo "" + echo "=== Summary ===" + echo "Status: HEALTHY - Single instance running" + exit 0 + else + echo "" + echo "=== Summary ===" + echo "Status: UNHEALTHY - Primary instance down" + exit 2 + fi +fi \ No newline at end of file diff --git a/cmds/manage-hot-spare.sh b/cmds/manage-hot-spare.sh new file mode 100755 index 0000000..a12b069 --- /dev/null +++ b/cmds/manage-hot-spare.sh @@ -0,0 +1,81 @@ +#!/bin/bash + +### +# Manage hot spare configuration for Keycloak deployment +### + +set -e +CMDS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" +cd $CMDS_DIR/.. + +source ./config.sh + +function usage() { + echo "Usage: $0 [enable|disable|status]" + echo "" + echo "Commands:" + echo " enable - Enable hot spare instance" + echo " disable - Disable hot spare instance" + echo " status - Show current hot spare status" + exit 1 +} + +function enable_hot_spare() { + echo "Enabling hot spare configuration..." + + # Update config.sh to enable hot spare + sed -i 's/ENABLE_HOT_SPARE=false/ENABLE_HOT_SPARE=true/' config.sh + + # Regenerate deployment files + ./cmds/generate-deployment-files.sh + + echo "Hot spare enabled. To start with hot spare, run:" + echo "docker compose --profile hot-spare up -d" +} + +function disable_hot_spare() { + echo "Disabling hot spare configuration..." + + # Update config.sh to disable hot spare + sed -i 's/ENABLE_HOT_SPARE=true/ENABLE_HOT_SPARE=false/' config.sh + + # Regenerate deployment files + ./cmds/generate-deployment-files.sh + + echo "Hot spare disabled. Normal single-instance deployment will be used." +} + +function show_status() { + echo "Hot spare status:" + if grep -q "ENABLE_HOT_SPARE=true" config.sh; then + echo " Status: ENABLED" + echo " Primary port: $KC_HOST_PORT" + echo " Spare port: $KC_SPARE_HOST_PORT" + echo "" + echo "To start with hot spare: docker compose --profile hot-spare up -d" + echo "To start without hot spare: docker compose up -d" + else + echo " Status: DISABLED" + echo " Only primary instance will run on port: $KC_HOST_PORT" + fi + + echo "" + echo "Running containers:" + docker compose ps 2>/dev/null || echo " No containers running" +} + +# Parse command line arguments +case "${1:-}" in + enable) + enable_hot_spare + ;; + disable) + disable_hot_spare + ;; + status) + show_status + ;; + *) + usage + ;; +esac \ No newline at end of file diff --git a/cmds/setup-distributed.sh b/cmds/setup-distributed.sh new file mode 100755 index 0000000..b91c300 --- /dev/null +++ b/cmds/setup-distributed.sh @@ -0,0 +1,183 @@ +#!/bin/bash + +### +# Setup script for distributed Keycloak deployment +### + +set -e +CMDS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" +cd $CMDS_DIR/.. + +function usage() { + echo "Usage: $0 [primary|replica]" + echo "" + echo "Commands:" + echo " primary - Setup primary VM configuration" + echo " replica - Setup replica VM configuration" + echo "" + echo "This script helps configure the .env file for distributed deployment." + exit 1 +} + +function setup_primary() { + echo "=== Setting up Primary VM Configuration ===" + echo "" + + # Check if .env exists + if [ ! -f .env ]; then + echo "Creating .env file from .env.example..." + cp .env.example .env + fi + + echo "Configuring .env for primary VM..." + + # Prompt for basic database credentials + read -p "Enter database username (default: keycloak): " db_user + db_user=${db_user:-keycloak} + + read -s -p "Enter database password: " db_pass + echo "" + + read -s -p "Enter replication user password: " repl_pass + echo "" + + # Update .env file + sed -i "s/KC_DB_USERNAME=.*/KC_DB_USERNAME=$db_user/" .env + sed -i "s/KC_DB_PASSWORD=.*/KC_DB_PASSWORD=$db_pass/" .env + sed -i "s/POSTGRES_PASSWORD=.*/POSTGRES_PASSWORD=$db_pass/" .env + + # Add replication settings + if ! grep -q "POSTGRES_REPLICATION_USER" .env; then + echo "" >> .env + echo "# Replication Configuration" >> .env + echo "POSTGRES_REPLICATION_USER=replicator" >> .env + echo "POSTGRES_REPLICATION_PASSWORD=$repl_pass" >> .env + else + sed -i "s/POSTGRES_REPLICATION_PASSWORD=.*/POSTGRES_REPLICATION_PASSWORD=$repl_pass/" .env + fi + + echo "" + echo "✓ Primary VM configuration complete!" + echo "" + echo "Next steps:" + echo "1. Deploy primary instance: ./cmds/deploy-primary.sh up" + echo "2. Get the IP address of this VM for replica configuration" + echo "3. Configure replica VM using this IP address" +} + +function setup_replica() { + echo "=== Setting up Replica VM Configuration ===" + echo "" + + # Check if .env exists + if [ ! -f .env ]; then + echo "Creating .env file from .env.example..." + cp .env.example .env + fi + + echo "Configuring .env for replica VM..." + + # Prompt for primary database details + read -p "Enter primary VM IP address: " primary_ip + if [ -z "$primary_ip" ]; then + echo "ERROR: Primary IP address is required" + exit 1 + fi + + read -p "Enter database username (default: keycloak): " db_user + db_user=${db_user:-keycloak} + + read -s -p "Enter database password: " db_pass + echo "" + + read -s -p "Enter replication user password: " repl_pass + echo "" + + # Update .env file + sed -i "s/KC_DB_USERNAME=.*/KC_DB_USERNAME=$db_user/" .env + sed -i "s/KC_DB_PASSWORD=.*/KC_DB_PASSWORD=$db_pass/" .env + sed -i "s/POSTGRES_PASSWORD=.*/POSTGRES_PASSWORD=$db_pass/" .env + + # Add primary and replication settings + if ! grep -q "PRIMARY_DB_HOST" .env; then + echo "" >> .env + echo "# Primary Database Connection" >> .env + echo "PRIMARY_DB_HOST=$primary_ip" >> .env + else + sed -i "s/PRIMARY_DB_HOST=.*/PRIMARY_DB_HOST=$primary_ip/" .env + fi + + if ! grep -q "POSTGRES_REPLICATION_USER" .env; then + echo "" >> .env + echo "# Replication Configuration" >> .env + echo "POSTGRES_REPLICATION_USER=replicator" >> .env + echo "POSTGRES_REPLICATION_PASSWORD=$repl_pass" >> .env + else + sed -i "s/POSTGRES_REPLICATION_PASSWORD=.*/POSTGRES_REPLICATION_PASSWORD=$repl_pass/" .env + fi + + # Test connectivity to primary + echo "" + echo "Testing connectivity to primary database at $primary_ip:5432..." + if nc -z "$primary_ip" 5432; then + echo "✓ Primary database is accessible" + else + echo "⚠ Warning: Cannot connect to primary database" + echo " Make sure the primary instance is running and accessible" + fi + + echo "" + echo "✓ Replica VM configuration complete!" + echo "" + echo "Next steps:" + echo "1. Ensure primary instance is running" + echo "2. Deploy replica instance: ./cmds/deploy-replica.sh up" + echo "3. Monitor health: ./cmds/health-check-distributed.sh all" +} + +function show_status() { + echo "=== Current Configuration Status ===" + echo "" + + if [ -f .env ]; then + echo "Configuration file (.env) exists" + + if grep -q "PRIMARY_DB_HOST=" .env && [ -n "$(grep "PRIMARY_DB_HOST=" .env | cut -d'=' -f2)" ]; then + echo "✓ Configured as: REPLICA VM" + echo " Primary DB Host: $(grep "PRIMARY_DB_HOST=" .env | cut -d'=' -f2)" + else + echo "✓ Configured as: PRIMARY VM" + fi + + echo "" + echo "Database user: $(grep "KC_DB_USERNAME=" .env | cut -d'=' -f2)" + echo "Replication user: $(grep "POSTGRES_REPLICATION_USER=" .env 2>/dev/null | cut -d'=' -f2 || echo 'Not configured')" + else + echo "✗ Configuration file (.env) not found" + echo " Run setup to create configuration" + fi + + echo "" + echo "Available deployment files:" + [ -f docker-compose.primary.yaml ] && echo " ✓ docker-compose.primary.yaml" + [ -f docker-compose.replica.yaml ] && echo " ✓ docker-compose.replica.yaml" + [ -f docker-compose.yaml ] && echo " ✓ docker-compose.yaml (original single-node)" +} + +# Parse command line arguments +case "${1:-}" in + primary) + setup_primary + ;; + replica) + setup_replica + ;; + status) + show_status + ;; + *) + show_status + echo "" + usage + ;; +esac \ No newline at end of file diff --git a/config.sh b/config.sh index 42bb76e..8881c97 100755 --- a/config.sh +++ b/config.sh @@ -29,6 +29,15 @@ KC_TAG=23.0.4 KC_HOSTNAME=sandbox.auth.library.ucdavis.edu KC_HOST_PORT=3123 +# Hot Spare Configuration (Legacy single-VM) +ENABLE_HOT_SPARE=false +KC_SPARE_HOST_PORT=3124 + +# Distributed HA Configuration +ENABLE_DISTRIBUTED_HA=true +REPLICA_KC_PORT=3124 +REPLICA_DB_PORT=5433 + # Postgres POSTGRES_TAG=15.3 diff --git a/config/init-primary.sh b/config/init-primary.sh new file mode 100755 index 0000000..24aa2a9 --- /dev/null +++ b/config/init-primary.sh @@ -0,0 +1,29 @@ +#!/bin/bash + +# PostgreSQL initialization script for primary server +# Sets up replication user and permissions + +set -e + +echo "Setting up PostgreSQL replication..." + +# Wait for PostgreSQL to be ready +until pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB"; do + echo "Waiting for PostgreSQL to be ready..." + sleep 2 +done + +# Create replication user if it doesn't exist +psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL + SELECT 'CREATE USER replicator' WHERE NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = 'replicator')\gexec + ALTER USER replicator WITH REPLICATION ENCRYPTED PASSWORD '${POSTGRES_REPLICATION_PASSWORD}'; + GRANT CONNECT ON DATABASE postgres TO replicator; +EOSQL + +echo "Replication user setup complete." + +# Create WAL archive directory +mkdir -p /var/lib/postgresql/archive +chown postgres:postgres /var/lib/postgresql/archive + +echo "PostgreSQL primary setup complete." \ No newline at end of file diff --git a/config/pg_hba.conf b/config/pg_hba.conf new file mode 100644 index 0000000..0463a49 --- /dev/null +++ b/config/pg_hba.conf @@ -0,0 +1,22 @@ +# PostgreSQL Client Authentication Configuration File for Primary Server + +# TYPE DATABASE USER ADDRESS METHOD + +# "local" is for Unix domain socket connections only +local all all trust + +# IPv4 local connections: +host all all 127.0.0.1/32 trust +host all all 0.0.0.0/0 md5 + +# IPv6 local connections: +host all all ::1/128 trust + +# Allow replication connections from localhost, by a user with the +# replication privilege. +local replication all trust +host replication all 127.0.0.1/32 trust +host replication all ::1/128 trust + +# Allow replication connections from any IP address for the replicator user +host replication replicator 0.0.0.0/0 md5 \ No newline at end of file diff --git a/config/postgresql-replica.conf b/config/postgresql-replica.conf new file mode 100644 index 0000000..29f0c55 --- /dev/null +++ b/config/postgresql-replica.conf @@ -0,0 +1,28 @@ +# PostgreSQL configuration for replica server + +# Basic settings +listen_addresses = '*' +port = 5432 +max_connections = 100 +shared_buffers = 128MB +dynamic_shared_memory_type = posix + +# Hot standby configuration +hot_standby = on +hot_standby_feedback = on + +# Recovery configuration (for replica) +# This is handled by the standby.signal file created by pg_basebackup + +# Logging +log_destination = 'stderr' +logging_collector = on +log_directory = 'log' +log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' +log_rotation_age = 1d +log_rotation_size = 10MB +log_min_messages = warning +log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,app=%a,client=%h ' + +# Read-only settings for replica +default_transaction_read_only = on \ No newline at end of file diff --git a/config/postgresql.conf b/config/postgresql.conf new file mode 100644 index 0000000..65bb196 --- /dev/null +++ b/config/postgresql.conf @@ -0,0 +1,35 @@ +# PostgreSQL configuration for primary server with streaming replication + +# Basic settings +listen_addresses = '*' +port = 5432 +max_connections = 100 +shared_buffers = 128MB +dynamic_shared_memory_type = posix + +# Write Ahead Logging (WAL) configuration for replication +wal_level = replica +max_wal_senders = 3 +wal_keep_size = 1GB +synchronous_commit = off + +# Archive settings +archive_mode = on +archive_command = 'test ! -f /var/lib/postgresql/archive/%f && cp %p /var/lib/postgresql/archive/%f' + +# Hot standby configuration +hot_standby = on +hot_standby_feedback = on + +# Logging +log_destination = 'stderr' +logging_collector = on +log_directory = 'log' +log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' +log_rotation_age = 1d +log_rotation_size = 10MB +log_min_messages = warning +log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,app=%a,client=%h ' + +# Connection settings +max_connections = 100 \ No newline at end of file diff --git a/docker-compose.primary.yaml b/docker-compose.primary.yaml new file mode 100644 index 0000000..cf023fd --- /dev/null +++ b/docker-compose.primary.yaml @@ -0,0 +1,79 @@ +services: + keycloak: + image: quay.io/keycloak/keycloak:23.0.4 + environment: + KC_HOSTNAME: sandbox.auth.library.ucdavis.edu + KC_HEALTH_ENABLED: true + KC_METRICS_ENABLED: true + KC_PROXY: edge + KC_DB: postgres + KC_DB_URL: jdbc:postgresql://db/postgres + depends_on: + db: + condition: service_started + init: + condition: service_completed_successfully + env_file: + - .env + ports: + - 3123:8080 + command: ["start"] + healthcheck: + test: ["CMD-SHELL", "curl -f http://localhost:8080/health/ready || exit 1"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s + db: + image: postgres:15.3 + env_file: + - .env + environment: + # Standard PostgreSQL environment + POSTGRES_USER: ${KC_DB_USERNAME} + POSTGRES_DB: postgres + volumes: + - db-data:/var/lib/postgresql/data + - ./config/postgresql.conf:/etc/postgresql/postgresql.conf + - ./config/pg_hba.conf:/etc/postgresql/pg_hba.conf + - ./config/init-primary.sh:/docker-entrypoint-initdb.d/init-primary.sh + command: > + postgres + -c config_file=/etc/postgresql/postgresql.conf + -c hba_file=/etc/postgresql/pg_hba.conf + ports: + - "5432:5432" + backup: + image: gcr.io/ucdlib-pubreg/auth-kc-utils:v1.0.1 + env_file: + - .env + depends_on: + - db + environment: + PGHOST: 'db' + PGDATABASE: 'postgres' + PGPORT: 5432 + GOOGLE_APPLICATION_CREDENTIALS: /etc/service-account.json + GC_BACKUP_BUCKET: itis-iam/keycloak + BACKUP_FILE_NAME: db.sql.gz + volumes: + - ./gc-writer-key.json:/etc/service-account.json + command: ./backup/entrypoint.sh + init: + image: gcr.io/ucdlib-pubreg/auth-kc-utils:v1.0.1 + env_file: + - .env + depends_on: + - db + environment: + PGHOST: 'db' + PGDATABASE: 'postgres' + PGPORT: 5432 + GOOGLE_APPLICATION_CREDENTIALS: /etc/service-account.json + GC_BACKUP_BUCKET: itis-iam/keycloak + BACKUP_FILE_NAME: db.sql.gz + volumes: + - ./gc-reader-key.json:/etc/service-account.json + command: ./init/init.sh +volumes: + db-data: \ No newline at end of file diff --git a/docker-compose.replica.yaml b/docker-compose.replica.yaml new file mode 100644 index 0000000..c15fde4 --- /dev/null +++ b/docker-compose.replica.yaml @@ -0,0 +1,57 @@ +services: + keycloak-replica: + image: quay.io/keycloak/keycloak:23.0.4 + environment: + KC_HOSTNAME: sandbox.auth.library.ucdavis.edu + KC_HEALTH_ENABLED: true + KC_METRICS_ENABLED: true + KC_PROXY: edge + KC_DB: postgres + KC_DB_URL: jdbc:postgresql://db-replica/postgres + # Read-only mode configuration + KC_TRANSACTION_XA_ENABLED: false + depends_on: + db-replica: + condition: service_started + env_file: + - .env + ports: + - 3124:8080 + command: ["start"] + healthcheck: + test: ["CMD-SHELL", "curl -f http://localhost:8080/health/ready || exit 1"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s + db-replica: + image: postgres:15.3 + env_file: + - .env + environment: + POSTGRES_USER: ${KC_DB_USERNAME} + POSTGRES_DB: postgres + PGPASSWORD: ${POSTGRES_REPLICATION_PASSWORD} + volumes: + - db-replica-data:/var/lib/postgresql/data + - ./config/postgresql-replica.conf:/etc/postgresql/postgresql.conf + command: > + bash -c " + echo 'Waiting for primary database...' && + until pg_isready -h ${PRIMARY_DB_HOST} -p 5432; do + echo 'Waiting for primary database to be ready...' + sleep 5 + done && + echo 'Primary database is ready. Checking replica status...' && + if [ ! -f /var/lib/postgresql/data/PG_VERSION ]; then + echo 'Initializing replica from primary...' + PGPASSWORD=${POSTGRES_REPLICATION_PASSWORD} pg_basebackup -h ${PRIMARY_DB_HOST} -p 5432 -U replicator -D /var/lib/postgresql/data -v -P -W -R + echo 'primary_conninfo = ''host=${PRIMARY_DB_HOST} port=5432 user=replicator password=${POSTGRES_REPLICATION_PASSWORD}''' >> /var/lib/postgresql/data/postgresql.auto.conf + fi && + echo 'Starting PostgreSQL replica...' && + postgres -c config_file=/etc/postgresql/postgresql.conf + " + ports: + - "5433:5432" +volumes: + db-replica-data: \ No newline at end of file diff --git a/docker-compose.yaml b/docker-compose.yaml index f9f28fc..8880241 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -19,6 +19,39 @@ services: ports: - 3123:8080 command: ["start"] + healthcheck: + test: ["CMD-SHELL", "curl -f http://localhost:8080/health/ready || exit 1"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s + keycloak-spare: + image: quay.io/keycloak/keycloak:23.0.4 + environment: + KC_HOSTNAME: sandbox.auth.library.ucdavis.edu + KC_HEALTH_ENABLED: true + KC_METRICS_ENABLED: true + KC_PROXY: edge + KC_DB: postgres + KC_DB_URL: jdbc:postgresql://db/postgres + depends_on: + db: + condition: service_started + init: + condition: service_completed_successfully + env_file: + - .env + ports: + - 3124:8080 + command: ["start"] + healthcheck: + test: ["CMD-SHELL", "curl -f http://localhost:8080/health/ready || exit 1"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s + profiles: + - hot-spare db: image: postgres:15.3 env_file: diff --git a/templates/apache.conf b/templates/apache.conf index 080916c..48ec207 100644 --- a/templates/apache.conf +++ b/templates/apache.conf @@ -35,8 +35,27 @@ RequestHeader set x-forwarded-proto "https" RequestHeader set x-ssl-client-cert "%{SSL_CLIENT_CERT}s" - ProxyPass / http://localhost:{{KC_HOST_PORT}}/ - ProxyPassReverse / http://localhost:{{KC_HOST_PORT}}/ + # Hot spare configuration with health checks + # Define upstream servers + ProxyPass /health/balancer-manager ! + + SetHandler balancer-manager + Require ip 172.19.19.0/24 169.237.102.0/26 169.237.102.128/25 + + + # Primary and spare Keycloak instances with health checks + ProxyPass / balancer://keycloak-cluster/ + ProxyPassReverse / balancer://keycloak-cluster/ + + + # Primary instance + BalancerMember http://localhost:{{KC_HOST_PORT}} status=+H + # Hot spare instance + BalancerMember http://localhost:{{KC_SPARE_HOST_PORT}} status=+H + # Health check configuration + ProxySet retry=300 + ProxySet ping=5 + SSLEngine on SSLCertificateFile /etc/ssl/certs/auth_library_ucdavis_edu_cert.cer diff --git a/templates/deployment.yaml b/templates/deployment.yaml index 79d0959..516396f 100644 --- a/templates/deployment.yaml +++ b/templates/deployment.yaml @@ -19,6 +19,39 @@ services: ports: - {{KC_HOST_PORT}}:8080 command: ["start"] + healthcheck: + test: ["CMD-SHELL", "curl -f http://localhost:8080/health/ready || exit 1"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s + keycloak-spare: + image: quay.io/keycloak/keycloak:{{KC_TAG}} + environment: + KC_HOSTNAME: {{KC_HOSTNAME}} + KC_HEALTH_ENABLED: true + KC_METRICS_ENABLED: true + KC_PROXY: edge + KC_DB: postgres + KC_DB_URL: jdbc:postgresql://db/postgres + depends_on: + db: + condition: service_started + init: + condition: service_completed_successfully + env_file: + - .env + ports: + - {{KC_SPARE_HOST_PORT}}:8080 + command: ["start"] + healthcheck: + test: ["CMD-SHELL", "curl -f http://localhost:8080/health/ready || exit 1"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s + profiles: + - hot-spare db: image: postgres:{{POSTGRES_TAG}} env_file: