ai-tax-agent/docs/DEPLOYMENT_PROGRESS.md

Deployment Progress Report

Date: 2025-10-04
Status: Ready for Deployment
Next Step: Build Docker Images

---

✅ Completed Tasks

1. Production Compose Files Created

Created three production-ready Docker Compose files in infra/base/:

infrastructure.yaml

• Vault (secrets management)
• MinIO (object storage)
• Qdrant (vector database)
• Neo4j (knowledge graph)
• Postgres (relational database)
• Redis (cache)
• NATS (event bus with JetStream)

Key Features:

• Uses shared frontend and backend networks
• All services exposed via Traefik with SSL (GoDaddy cert resolver)
• Protected by Authentik ForwardAuth middleware
• Production-ready health checks
• Persistent volumes for data

services.yaml

• All microservices (svc-ingestion, svc-extract, svc-kg, svc-rag-retriever, svc-forms, svc-hmrc, svc-ocr)
• Review UI (ui-review)

Key Features:

• Images pulled from Gitea registry: gitea.harkon.co.uk/ai-tax-agent/*
• All services routed through api.harkon.co.uk with path prefixes
• UI exposed at app.harkon.co.uk
• Rate limiting and authentication middleware
• Environment variables from .env.production

monitoring.yaml

• Prometheus (metrics collection)
• Grafana (visualization with Authentik OAuth)
• Loki (log aggregation)
• Promtail (log shipper)

Key Features:

• 30-day metrics retention
• Grafana integrated with Authentik SSO
• Loki for centralized logging
• All services exposed via Traefik with SSL

2. Deployment Scripts Created

scripts/generate-production-secrets.sh

• Generates strong passwords for all services
• Uses openssl rand for cryptographically secure secrets
• Creates backup of .env.production before modification
• Displays important credentials (admin password, Vault token, etc.)

Usage:

chmod +x scripts/generate-production-secrets.sh
./scripts/generate-production-secrets.sh

scripts/build-and-push-images.sh

• Builds all Docker images for production
• Tags with version numbers
• Pushes to Gitea registry
• Supports custom registry and version

Usage:

chmod +x scripts/build-and-push-images.sh
./scripts/build-and-push-images.sh gitea.harkon.co.uk v1.0.0

scripts/deploy-to-production.sh

• Automated deployment to remote server
• Step-by-step or full deployment
• Backup, prepare, deploy, verify
• View logs and service status

Usage:

chmod +x scripts/deploy-to-production.sh

# Full deployment
./scripts/deploy-to-production.sh all

# Step-by-step
./scripts/deploy-to-production.sh backup
./scripts/deploy-to-production.sh prepare
./scripts/deploy-to-production.sh infrastructure
./scripts/deploy-to-production.sh services
./scripts/deploy-to-production.sh monitoring
./scripts/deploy-to-production.sh verify

# View logs
./scripts/deploy-to-production.sh logs svc-ingestion

3. Documentation Created

infra/base manifests

Comprehensive production deployment guide including:

• Prerequisites checklist
• Three deployment options (automated, step-by-step, manual)
• Post-deployment initialization steps
• Service URLs (public and admin)
• Monitoring and troubleshooting
• Rollback procedures
• Maintenance tasks
• Security notes

4. Environment Configuration

.env.production

• Created from env.example
• Ready for secret generation
• Configured for production:
  • DOMAIN=harkon.co.uk
  • DEBUG=false
  • DEVELOPMENT_MODE=false
  • GoDaddy API credentials
  • All service passwords (to be generated)

.gitignore

• Updated to exclude .env.production
• Prevents accidental commit of secrets
• Also excludes .env.*.backup files

---

📋 Current Status

What's Ready

✅ Production compose files (infrastructure, services, monitoring)
✅ Deployment automation scripts
✅ Secret generation script
✅ Image build and push script
✅ Comprehensive documentation
✅ Environment file template
✅ Git ignore rules for secrets

What's Pending

⏳ Generate production secrets
⏳ Build Docker images
⏳ Push images to Gitea registry
⏳ Create backup of remote server
⏳ Deploy to production
⏳ Initialize infrastructure (Vault, MinIO, NATS)
⏳ Configure Authentik OAuth providers
⏳ Verify deployment

---

🚀 Next Steps

Step 1: Generate Production Secrets (5 minutes)

cd /Users/harris/Projects/ai-tax-agent
chmod +x scripts/generate-production-secrets.sh
./scripts/generate-production-secrets.sh

Important: Save the output credentials in your password manager!

Step 2: Build and Push Docker Images (30-60 minutes)

# Login to Gitea registry
docker login gitea.harkon.co.uk

# Build and push all images
chmod +x scripts/build-and-push-images.sh
./scripts/build-and-push-images.sh gitea.harkon.co.uk v1.0.0

This will build and push:

• svc-ingestion
• svc-extract
• svc-kg
• svc-rag-retriever
• svc-rag-indexer
• svc-forms
• svc-hmrc
• svc-ocr
• svc-rpa
• svc-normalize-map
• svc-reason
• svc-firm-connectors
• svc-coverage
• ui-review

Step 3: Deploy to Production (15-30 minutes)

# Full automated deployment
chmod +x scripts/deploy-to-production.sh
./scripts/deploy-to-production.sh all

Or step-by-step:

./scripts/deploy-to-production.sh backup
./scripts/deploy-to-production.sh prepare
./scripts/deploy-to-production.sh infrastructure
# Verify infrastructure is healthy
./scripts/deploy-to-production.sh verify
./scripts/deploy-to-production.sh services
./scripts/deploy-to-production.sh monitoring
./scripts/deploy-to-production.sh verify

Step 4: Post-Deployment Configuration (20-30 minutes)

1. Initialize Vault

   ssh deploy@141.136.35.199
   cd /opt/ai-tax-agent
   docker exec -it vault vault operator init
   # Save unseal keys!
   docker exec -it vault vault operator unseal
   

2. Create MinIO Buckets

   docker exec -it minio mc alias set local http://localhost:9092 admin <PASSWORD>
   docker exec -it minio mc mb local/documents
   docker exec -it minio mc mb local/models
   

3. Create NATS Streams

   docker exec -it nats nats stream add TAX_AGENT_EVENTS \
     --subjects="tax.>" \
     --storage=file \
     --retention=limits \
     --max-age=7d
   

4. Configure Authentik

   • Login to https://authentik.harkon.co.uk
   • Create groups: app-admin, app-user, app-reviewer
   • Create OAuth providers for Review UI and Grafana
   • Configure ForwardAuth outpost

Step 5: Verify Deployment (10 minutes)

# Check all services
./scripts/deploy-to-production.sh verify

# Test endpoints
curl -I https://app.harkon.co.uk
curl -I https://api.harkon.co.uk/healthz
curl -I https://grafana.harkon.co.uk

# View logs
./scripts/deploy-to-production.sh logs svc-ingestion

---

📊 Architecture Overview

Network Topology

Internet
    ↓
Traefik (Port 80/443)
    ↓
┌─────────────────────────────────────────┐
│         Frontend Network                │
│  - Traefik                              │
│  - Authentik (Server + Outpost)         │
│  - All exposed services                 │
└─────────────────────────────────────────┘
    ↓
┌─────────────────────────────────────────┐
│         Backend Network                 │
│  - Postgres, Redis, Neo4j               │
│  - MinIO, Qdrant, Vault                 │
│  - NATS, Prometheus, Loki               │
│  - All microservices                    │
└─────────────────────────────────────────┘

Service Domains

Public Services:

• app.harkon.co.uk - Review UI
• api.harkon.co.uk - API Gateway (all microservices)
• grafana.harkon.co.uk - Monitoring Dashboard

Admin Services (Auth Required):

• vault.harkon.co.uk - Secrets Management
• minio.harkon.co.uk - Object Storage Console
• neo4j.harkon.co.uk - Knowledge Graph Browser
• qdrant.harkon.co.uk - Vector Database UI
• prometheus.harkon.co.uk - Metrics
• loki.harkon.co.uk - Logs
• nats.harkon.co.uk - Event Bus Monitor

Company Services (Existing):

• authentik.harkon.co.uk - SSO
• traefik.harkon.co.uk - Reverse Proxy Dashboard
• gitea.harkon.co.uk - Git Repository
• cloud.harkon.co.uk - Nextcloud
• portainer.harkon.co.uk - Docker Management

---

🔒 Security Considerations

1. Secrets Management

   • All secrets generated with openssl rand
   • .env.production excluded from git
   • Vault for runtime secret storage
   • Authentik for authentication

2. Network Security

   • Services isolated in backend network
   • Only necessary services on frontend network
   • All traffic encrypted with SSL (Let's Encrypt via GoDaddy DNS)
   • ForwardAuth middleware on all admin services

3. Access Control

   • Authentik SSO for all services
   • Role-based access (admin, user, reviewer)
   • OAuth2 for service-to-service auth

---

📝 Important Notes

1. Backup Before Deployment

   • Always create backup before making changes
   • Script includes automatic backup step
   • Backups stored in ~/backups/ on remote server

2. Incremental Deployment

   • Deploy infrastructure first
   • Verify health before deploying services
   • Monitor logs during deployment

3. Rollback Plan

   • Backups available in ~/backups/
   • Can restore previous state
   • Company services unaffected

4. Monitoring

   • Grafana dashboards for all services
   • Loki for centralized logging
   • Prometheus for metrics
   • Alerts configured in Grafana

---

🎯 Success Criteria

Deployment is successful when:

• All infrastructure services are running and healthy
• All application services are running and healthy
• All monitoring services are running and healthy
• UI accessible at https://app.harkon.co.uk
• API accessible at https://api.harkon.co.uk
• Grafana accessible at https://grafana.harkon.co.uk
• All services protected by Authentik
• SSL certificates valid
• No errors in logs
• Company services still operational

---

📞 Support

If you encounter issues:

1. Check logs: ./scripts/deploy-to-production.sh logs <service>
2. Verify status: ./scripts/deploy-to-production.sh verify
3. Review manifests: infra/base/*.yaml
4. Check deployment plan: docs/DEPLOYMENT_PLAN.md
5. Follow checklist: docs/DEPLOYMENT_CHECKLIST.md