Initial commit

docs/POST_BUILD_DEPLOYMENT.md

@@ -0,0 +1,378 @@
+# Post-Build Deployment Guide
+
+This guide covers the deployment steps **after** Docker images have been built and pushed to the Gitea registry.
+
+## Prerequisites
+
+✅ Docker images built and pushed to `gitea.harkon.co.uk/ai-tax-agent/*:v1.0.0`  
+✅ Production environment file generated (`.env.production`)  
+✅ SSH access to production server (`deploy@141.136.35.199`)  
+✅ Gitea access token created with `write:package` scope  
+
+---
+
+## Deployment Steps
+
+### Step 1: Prepare Remote Server Directory Structure
+
+```bash
+# Create directory structure on remote server
+ssh deploy@141.136.35.199 << 'EOF'
+mkdir -p /opt/ai-tax-agent/{compose/production,data,logs,backups}
+mkdir -p /opt/ai-tax-agent/compose/{prometheus,loki,grafana}
+EOF
+```
+
+### Step 2: Copy Configuration Files
+
+```bash
+# Copy production compose files
+scp infra/compose/production/infrastructure.yaml deploy@141.136.35.199:/opt/ai-tax-agent/compose/production/
+scp infra/compose/production/services.yaml deploy@141.136.35.199:/opt/ai-tax-agent/compose/production/
+scp infra/compose/production/monitoring.yaml deploy@141.136.35.199:/opt/ai-tax-agent/compose/production/
+
+# Copy environment file
+scp infra/compose/.env.production deploy@141.136.35.199:/opt/ai-tax-agent/compose/.env.production
+
+# Copy monitoring configs
+scp infra/compose/prometheus/prometheus.yml deploy@141.136.35.199:/opt/ai-tax-agent/compose/prometheus/
+scp infra/compose/loki/loki-config.yml deploy@141.136.35.199:/opt/ai-tax-agent/compose/loki/loki.yml
+scp infra/compose/promtail/promtail-config.yml deploy@141.136.35.199:/opt/ai-tax-agent/compose/loki/promtail-config.yml
+```
+
+### Step 3: Update Traefik Configuration
+
+Add the AI Tax Agent middleware to Traefik's dynamic configuration:
+
+```bash
+# Create Traefik dynamic config for AI Tax Agent
+ssh deploy@141.136.35.199 << 'EOF'
+cat > /opt/compose/traefik/config/ai-tax-agent.yaml << 'TRAEFIK'
+http:
+  middlewares:
+    # Rate limiting for API
+    api-ratelimit:
+      rateLimit:
+        average: 100
+        burst: 50
+        period: 1s
+    
+    # CORS headers
+    api-cors:
+      headers:
+        accessControlAllowMethods:
+          - GET
+          - POST
+          - PUT
+          - DELETE
+          - OPTIONS
+        accessControlAllowOriginList:
+          - "https://app.harkon.co.uk"
+        accessControlAllowHeaders:
+          - "Content-Type"
+          - "Authorization"
+        accessControlMaxAge: 100
+        addVaryHeader: true
+    
+    # Security headers
+    security-headers:
+      headers:
+        frameDeny: true
+        browserXssFilter: true
+        contentTypeNosniff: true
+        stsSeconds: 31536000
+        stsIncludeSubdomains: true
+        stsPreload: true
+TRAEFIK
+EOF
+```
+
+### Step 4: Deploy Infrastructure Services
+
+```bash
+# Use the deployment script
+./scripts/deploy-to-production.sh infrastructure
+
+# Or manually:
+ssh deploy@141.136.35.199 << 'EOF'
+cd /opt/ai-tax-agent
+docker compose -f compose/production/infrastructure.yaml up -d
+EOF
+```
+
+**Wait 2-3 minutes** for infrastructure services to initialize.
+
+### Step 5: Initialize Vault
+
+```bash
+# Initialize Vault (first time only)
+ssh deploy@141.136.35.199 << 'EOF'
+# Vault will auto-unseal if configured, otherwise:
+docker exec vault vault operator init -key-shares=5 -key-threshold=3 > ~/vault-keys.txt
+docker exec vault vault operator unseal <unseal-key-1>
+docker exec vault vault operator unseal <unseal-key-2>
+docker exec vault vault operator unseal <unseal-key-3>
+EOF
+
+# IMPORTANT: Save vault-keys.txt securely and delete from server!
+scp deploy@141.136.35.199:~/vault-keys.txt ./vault-keys-SECURE.txt
+ssh deploy@141.136.35.199 "rm ~/vault-keys.txt"
+```
+
+### Step 6: Initialize MinIO
+
+```bash
+# MinIO is ready immediately, access at:
+# https://minio-console.harkon.co.uk
+# Username: admin (from .env.production MINIO_ROOT_USER)
+# Password: <from .env.production MINIO_ROOT_PASSWORD>
+
+# Create required buckets
+ssh deploy@141.136.35.199 << 'EOF'
+docker exec minio mc alias set local http://localhost:9000 admin <MINIO_ROOT_PASSWORD>
+docker exec minio mc mb local/documents
+docker exec minio mc mb local/processed
+docker exec minio mc mb local/models
+docker exec minio mc mb local/temp
+EOF
+```
+
+### Step 7: Initialize Neo4j
+
+```bash
+# Access Neo4j Browser at:
+# https://neo4j.harkon.co.uk
+# Username: neo4j
+# Password: <from .env.production NEO4J_PASSWORD>
+
+# Verify connection
+ssh deploy@141.136.35.199 << 'EOF'
+docker exec neo4j cypher-shell -u neo4j -p <NEO4J_PASSWORD> "RETURN 'Connected' as status;"
+EOF
+```
+
+### Step 8: Deploy Application Services
+
+```bash
+# Deploy all application services
+./scripts/deploy-to-production.sh services
+
+# Or manually:
+ssh deploy@141.136.35.199 << 'EOF'
+cd /opt/ai-tax-agent
+docker compose -f compose/production/services.yaml up -d
+EOF
+```
+
+**Wait 1-2 minutes** for services to start.
+
+### Step 9: Deploy Monitoring Stack
+
+```bash
+# Deploy monitoring
+./scripts/deploy-to-production.sh monitoring
+
+# Or manually:
+ssh deploy@141.136.35.199 << 'EOF'
+cd /opt/ai-tax-agent
+docker compose -f compose/production/monitoring.yaml up -d
+EOF
+```
+
+### Step 10: Configure Authentik OAuth for Grafana
+
+1. **Login to Authentik**: https://authentik.harkon.co.uk
+2. **Create OAuth Provider**:
+   - Applications → Providers → Create
+   - Type: OAuth2/OpenID Provider
+   - Name: `Grafana`
+   - Client ID: `grafana` (copy this)
+   - Client Secret: Generate and copy
+   - Redirect URIs: `https://grafana.harkon.co.uk/login/generic_oauth`
+   - Scopes: `openid`, `profile`, `email`, `groups`
+
+3. **Create Application**:
+   - Applications → Create
+   - Name: `Grafana`
+   - Slug: `grafana`
+   - Provider: Select the provider created above
+   - Launch URL: `https://grafana.harkon.co.uk`
+
+4. **Update Environment Variables**:
+   ```bash
+   # On remote server, update .env.production
+   ssh deploy@141.136.35.199
+   nano /opt/ai-tax-agent/compose/.env.production
+   
+   # Update these values:
+   GRAFANA_OAUTH_CLIENT_ID=grafana
+   GRAFANA_OAUTH_CLIENT_SECRET=<secret-from-authentik>
+   
+   # Restart Grafana
+   cd /opt/ai-tax-agent
+   docker compose -f compose/production/monitoring.yaml restart grafana
+   ```
+
+### Step 11: Verify Deployment
+
+```bash
+# Run verification script
+./scripts/verify-deployment.sh
+
+# Or check manually:
+./scripts/health-check.sh
+```
+
+### Step 12: Access Services
+
+| Service | URL | Authentication |
+|---------|-----|----------------|
+| **Application UI** | https://app.harkon.co.uk | Authentik SSO |
+| **API Gateway** | https://api.harkon.co.uk | Authentik SSO |
+| **Grafana** | https://grafana.harkon.co.uk | Authentik OAuth |
+| **Prometheus** | https://prometheus.harkon.co.uk | Authentik SSO |
+| **Vault** | https://vault.harkon.co.uk | Vault Token |
+| **MinIO Console** | https://minio-console.harkon.co.uk | MinIO Credentials |
+| **Neo4j Browser** | https://neo4j.harkon.co.uk | Neo4j Credentials |
+| **Qdrant** | https://qdrant.harkon.co.uk | Authentik SSO |
+
+---
+
+## Post-Deployment Tasks
+
+### 1. Configure Grafana Dashboards
+
+1. Login to Grafana: https://grafana.harkon.co.uk
+2. Add Prometheus data source:
+   - Configuration → Data Sources → Add data source
+   - Type: Prometheus
+   - URL: `http://prometheus:9090`
+   - Save & Test
+
+3. Add Loki data source:
+   - Configuration → Data Sources → Add data source
+   - Type: Loki
+   - URL: `http://loki:3100`
+   - Save & Test
+
+4. Import dashboards (optional):
+   - Create → Import
+   - Dashboard ID: 1860 (Node Exporter Full)
+   - Dashboard ID: 7362 (Docker Monitoring)
+
+### 2. Set Up Alerts (Optional)
+
+Create alert rules in Prometheus or Grafana for:
+- Service health checks
+- High memory usage
+- High CPU usage
+- Disk space warnings
+- Failed authentication attempts
+
+### 3. Configure Backups
+
+```bash
+# Set up automated backups (cron job on server)
+ssh deploy@141.136.35.199
+crontab -e
+
+# Add daily backup at 2 AM
+0 2 * * * /opt/ai-tax-agent/scripts/backup.sh
+```
+
+### 4. Test Application Workflows
+
+1. **Upload a document** via UI
+2. **Check ingestion** service logs
+3. **Verify extraction** in Neo4j
+4. **Test RAG retrieval** via API
+5. **Review results** in UI
+
+---
+
+## Troubleshooting
+
+### Services Not Starting
+
+```bash
+# Check logs
+ssh deploy@141.136.35.199 "docker logs <container-name>"
+
+# Check resource usage
+ssh deploy@141.136.35.199 "docker stats"
+
+# Restart specific service
+ssh deploy@141.136.35.199 "cd /opt/ai-tax-agent && docker compose -f compose/production/services.yaml restart <service-name>"
+```
+
+### SSL Certificate Issues
+
+```bash
+# Check Traefik logs
+ssh deploy@141.136.35.199 "docker logs traefik --tail 100"
+
+# Force certificate renewal
+ssh deploy@141.136.35.199 "docker exec traefik rm /var/traefik/certs/godaddy-acme.json && docker restart traefik"
+```
+
+### Database Connection Issues
+
+```bash
+# Check PostgreSQL
+ssh deploy@141.136.35.199 "docker exec postgres pg_isready"
+
+# Check Neo4j
+ssh deploy@141.136.35.199 "docker exec neo4j cypher-shell -u neo4j -p <password> 'RETURN 1;'"
+
+# Check Redis
+ssh deploy@141.136.35.199 "docker exec redis redis-cli ping"
+```
+
+---
+
+## Rollback Procedure
+
+If deployment fails:
+
+```bash
+# Use rollback script
+./scripts/rollback-deployment.sh
+
+# Or manually restore from backup
+ssh deploy@141.136.35.199 << 'EOF'
+cd /opt/ai-tax-agent
+docker compose -f compose/production/services.yaml down
+docker compose -f compose/production/infrastructure.yaml down
+docker compose -f compose/production/monitoring.yaml down
+
+# Restore from backup
+tar -xzf backups/backup-<timestamp>.tar.gz -C /opt/ai-tax-agent/
+
+# Restart services
+docker compose -f compose/production/infrastructure.yaml up -d
+sleep 30
+docker compose -f compose/production/services.yaml up -d
+docker compose -f compose/production/monitoring.yaml up -d
+EOF
+```
+
+---
+
+## Next Steps
+
+1. ✅ Monitor application logs for errors
+2. ✅ Set up automated backups
+3. ✅ Configure alerting rules
+4. ✅ Document any custom configurations
+5. ✅ Train users on the application
+6. ✅ Plan for scaling (if needed)
+
+---
+
+## Support
+
+For issues or questions:
+- Check logs: `./scripts/verify-deployment.sh`
+- Review documentation: `docs/DEPLOYMENT_CHECKLIST.md`
+- Contact: [Your support contact]
+