ai-tax-agent/docs/POST_BUILD_DEPLOYMENT.md

Post-Build Deployment Guide

This guide covers the deployment steps after Docker images have been built and pushed to the Gitea registry.

Prerequisites

✅ Docker images built and pushed to gitea.harkon.co.uk/ai-tax-agent/*:v1.0.0
✅ Production environment file generated (.env.production)
✅ SSH access to production server (deploy@141.136.35.199)
✅ Gitea access token created with write:package scope

---

Deployment Steps

Step 1: Prepare Remote Server Directory Structure

# Create directory structure on remote server
ssh deploy@141.136.35.199 << 'EOF'
mkdir -p /opt/ai-tax-agent/{compose/production,data,logs,backups}
mkdir -p /opt/ai-tax-agent/compose/{prometheus,loki,grafana}
EOF

Step 2: Copy Configuration Files

# Copy production compose files
scp infra/base/infrastructure.yaml deploy@141.136.35.199:/opt/ai-tax-agent/compose/production/
scp infra/base/services.yaml deploy@141.136.35.199:/opt/ai-tax-agent/compose/production/
scp infra/base/monitoring.yaml deploy@141.136.35.199:/opt/ai-tax-agent/compose/production/

# Copy environment file
scp infra/environments/production/.env deploy@141.136.35.199:/opt/ai-tax-agent/compose/.env

# Copy monitoring configs
scp infra/compose/prometheus/prometheus.yml deploy@141.136.35.199:/opt/ai-tax-agent/compose/prometheus/
scp infra/compose/loki/loki-config.yml deploy@141.136.35.199:/opt/ai-tax-agent/compose/loki/loki.yml
scp infra/compose/promtail/promtail-config.yml deploy@141.136.35.199:/opt/ai-tax-agent/compose/loki/promtail-config.yml

Step 3: Update Traefik Configuration

Add the AI Tax Agent middleware to Traefik's dynamic configuration:

# Create Traefik dynamic config for AI Tax Agent
ssh deploy@141.136.35.199 << 'EOF'
cat > /opt/compose/traefik/config/ai-tax-agent.yaml << 'TRAEFIK'
http:
  middlewares:
    # Rate limiting for API
    api-ratelimit:
      rateLimit:
        average: 100
        burst: 50
        period: 1s
    
    # CORS headers
    api-cors:
      headers:
        accessControlAllowMethods:
          - GET
          - POST
          - PUT
          - DELETE
          - OPTIONS
        accessControlAllowOriginList:
          - "https://app.harkon.co.uk"
        accessControlAllowHeaders:
          - "Content-Type"
          - "Authorization"
        accessControlMaxAge: 100
        addVaryHeader: true
    
    # Security headers
    security-headers:
      headers:
        frameDeny: true
        browserXssFilter: true
        contentTypeNosniff: true
        stsSeconds: 31536000
        stsIncludeSubdomains: true
        stsPreload: true
TRAEFIK
EOF

Step 4: Deploy Infrastructure Services

# Use the deployment script
./scripts/deploy-to-production.sh infrastructure

# Or manually:
ssh deploy@141.136.35.199 << 'EOF'
cd /opt/ai-tax-agent
docker compose -f compose/production/infrastructure.yaml up -d
EOF

Wait 2-3 minutes for infrastructure services to initialize.

Step 5: Initialize Vault

# Initialize Vault (first time only)
ssh deploy@141.136.35.199 << 'EOF'
# Vault will auto-unseal if configured, otherwise:
docker exec vault vault operator init -key-shares=5 -key-threshold=3 > ~/vault-keys.txt
docker exec vault vault operator unseal <unseal-key-1>
docker exec vault vault operator unseal <unseal-key-2>
docker exec vault vault operator unseal <unseal-key-3>
EOF

# IMPORTANT: Save vault-keys.txt securely and delete from server!
scp deploy@141.136.35.199:~/vault-keys.txt ./vault-keys-SECURE.txt
ssh deploy@141.136.35.199 "rm ~/vault-keys.txt"

Step 6: Initialize MinIO

# MinIO is ready immediately, access at:
# https://minio.harkon.co.uk
# Username: admin (from .env.production MINIO_ROOT_USER)
# Password: <from .env.production MINIO_ROOT_PASSWORD>

# Create required buckets
ssh deploy@141.136.35.199 << 'EOF'
docker exec apa-minio mc alias set local http://localhost:9000 admin <MINIO_ROOT_PASSWORD>
docker exec apa-minio mc mb local/documents
docker exec apa-minio mc mb local/processed
docker exec apa-minio mc mb local/models
docker exec apa-minio mc mb local/temp
EOF

Step 7: Initialize Neo4j

# Access Neo4j Browser at:
# https://neo4j.harkon.co.uk
# Username: neo4j
# Password: <from .env.production NEO4J_PASSWORD>

# Verify connection
ssh deploy@141.136.35.199 << 'EOF'
docker exec apa-neo4j cypher-shell -u neo4j -p <NEO4J_PASSWORD> "RETURN 'Connected' as status;"
EOF

Step 8: Deploy Application Services

# Deploy all application services
./scripts/deploy-to-production.sh services

# Or manually:
ssh deploy@141.136.35.199 << 'EOF'
cd /opt/ai-tax-agent
docker compose -f compose/production/services.yaml up -d
EOF

Wait 1-2 minutes for services to start.

Step 9: Deploy Monitoring Stack

# Deploy monitoring
./scripts/deploy-to-production.sh monitoring

# Or manually:
ssh deploy@141.136.35.199 << 'EOF'
cd /opt/ai-tax-agent
docker compose -f compose/production/monitoring.yaml up -d
EOF

Step 10: Configure Authentik OAuth for Grafana

1. Login to Authentik: https://auth.harkon.co.uk

2. Create OAuth Provider:

   • Applications → Providers → Create
   • Type: OAuth2/OpenID Provider
   • Name: Grafana
   • Client ID: grafana (copy this)
   • Client Secret: Generate and copy
   • Redirect URIs: https://grafana.harkon.co.uk/login/generic_oauth
   • Scopes: openid, profile, email, groups

3. Create Application:

   • Applications → Create
   • Name: Grafana
   • Slug: grafana
   • Provider: Select the provider created above
   • Launch URL: https://grafana.harkon.co.uk

4. Update Environment Variables:

   # On remote server, update .env.production
   ssh deploy@141.136.35.199
   nano /opt/ai-tax-agent/compose/.env.production
   
   # Update these values:
   GRAFANA_OAUTH_CLIENT_ID=grafana
   GRAFANA_OAUTH_CLIENT_SECRET=<secret-from-authentik>
   
   # Restart Grafana
   cd /opt/ai-tax-agent
   docker compose -f compose/production/monitoring.yaml restart apa-grafana
   

Step 11: Verify Deployment

# Run verification script
./scripts/verify-deployment.sh

# Or check manually:
./scripts/health-check.sh

Step 12: Access Services

Service	URL	Authentication
Application UI	https://app.harkon.co.uk	Authentik SSO
API Gateway	https://api.harkon.co.uk	Authentik SSO
Grafana	https://grafana.harkon.co.uk	Authentik OAuth
Prometheus	https://prometheus.harkon.co.uk	Authentik SSO
Vault	https://vault.harkon.co.uk	Vault Token
MinIO Console	https://minio-console.harkon.co.uk	MinIO Credentials
Neo4j Browser	https://neo4j.harkon.co.uk	Neo4j Credentials
Qdrant	https://qdrant.harkon.co.uk	Authentik SSO

---

Post-Deployment Tasks

1. Configure Grafana Dashboards

1. Login to Grafana: https://grafana.harkon.co.uk

2. Add Prometheus data source:

   • Configuration → Data Sources → Add data source
   • Type: Prometheus
   • URL: http://prometheus:9090
   • Save & Test

3. Add Loki data source:

   • Configuration → Data Sources → Add data source
   • Type: Loki
   • URL: http://loki:3100
   • Save & Test

4. Import dashboards (optional):

   • Create → Import
   • Dashboard ID: 1860 (Node Exporter Full)
   • Dashboard ID: 7362 (Docker Monitoring)

2. Set Up Alerts (Optional)

Create alert rules in Prometheus or Grafana for:

• Service health checks
• High memory usage
• High CPU usage
• Disk space warnings
• Failed authentication attempts

3. Configure Backups

# Set up automated backups (cron job on server)
ssh deploy@141.136.35.199
crontab -e

# Add daily backup at 2 AM
0 2 * * * /opt/ai-tax-agent/scripts/backup.sh

4. Test Application Workflows

1. Upload a document via UI
2. Check ingestion service logs
3. Verify extraction in Neo4j
4. Test RAG retrieval via API
5. Review results in UI

---

Troubleshooting

Services Not Starting

# Check logs
ssh deploy@141.136.35.199 "docker logs <container-name>"

# Check resource usage
ssh deploy@141.136.35.199 "docker stats"

# Restart specific service
ssh deploy@141.136.35.199 "cd /opt/ai-tax-agent && docker compose -f compose/production/services.yaml restart <service-name>"

SSL Certificate Issues

# Check Traefik logs
ssh deploy@141.136.35.199 "docker logs traefik --tail 100"

# Force certificate renewal
ssh deploy@141.136.35.199 "docker exec traefik rm /var/traefik/certs/godaddy-acme.json && docker restart traefik"

Database Connection Issues

# Check PostgreSQL
ssh deploy@141.136.35.199 "docker exec postgres pg_isready"

# Check Neo4j
ssh deploy@141.136.35.199 "docker exec neo4j cypher-shell -u neo4j -p <password> 'RETURN 1;'"

# Check Redis
ssh deploy@141.136.35.199 "docker exec redis redis-cli ping"

---

Rollback Procedure

If deployment fails:

# Use rollback script
./scripts/rollback-deployment.sh

# Or manually restore from backup
ssh deploy@141.136.35.199 << 'EOF'
cd /opt/ai-tax-agent
docker compose -f compose/production/services.yaml down
docker compose -f compose/production/infrastructure.yaml down
docker compose -f compose/production/monitoring.yaml down

# Restore from backup
tar -xzf backups/backup-<timestamp>.tar.gz -C /opt/ai-tax-agent/

# Restart services
docker compose -f compose/production/infrastructure.yaml up -d
sleep 30
docker compose -f compose/production/services.yaml up -d
docker compose -f compose/production/monitoring.yaml up -d
EOF

---

Next Steps

1. ✅ Monitor application logs for errors
2. ✅ Set up automated backups
3. ✅ Configure alerting rules
4. ✅ Document any custom configurations
5. ✅ Train users on the application
6. ✅ Plan for scaling (if needed)

---

Support

For issues or questions:

• Check logs: ./scripts/verify-deployment.sh
• Review documentation: docs/DEPLOYMENT_CHECKLIST.md
• Contact: [Your support contact]