ai-tax-agent/docs/POST_BUILD_DEPLOYMENT.md

# Post-Build Deployment Guide

This guide covers the deployment steps **after** Docker images have been built and pushed to the Gitea registry.

## Prerequisites

✅ Docker images built and pushed to `gitea.harkon.co.uk/ai-tax-agent/*:v1.0.0`  
✅ Production environment file generated (`.env.production`)  
✅ SSH access to production server (`deploy@141.136.35.199`)  
✅ Gitea access token created with `write:package` scope  

---

## Deployment Steps

### Step 1: Prepare Remote Server Directory Structure

```bash
# Create directory structure on remote server
ssh deploy@141.136.35.199 << 'EOF'
mkdir -p /opt/ai-tax-agent/{compose/production,data,logs,backups}
mkdir -p /opt/ai-tax-agent/compose/{prometheus,loki,grafana}
EOF
```

### Step 2: Copy Configuration Files

```bash
# Copy production compose files
scp infra/base/infrastructure.yaml deploy@141.136.35.199:/opt/ai-tax-agent/compose/production/
scp infra/base/services.yaml deploy@141.136.35.199:/opt/ai-tax-agent/compose/production/
scp infra/base/monitoring.yaml deploy@141.136.35.199:/opt/ai-tax-agent/compose/production/

# Copy environment file
scp infra/environments/production/.env deploy@141.136.35.199:/opt/ai-tax-agent/compose/.env

# Copy monitoring configs
scp infra/compose/prometheus/prometheus.yml deploy@141.136.35.199:/opt/ai-tax-agent/compose/prometheus/
scp infra/compose/loki/loki-config.yml deploy@141.136.35.199:/opt/ai-tax-agent/compose/loki/loki.yml
scp infra/compose/promtail/promtail-config.yml deploy@141.136.35.199:/opt/ai-tax-agent/compose/loki/promtail-config.yml
```

### Step 3: Update Traefik Configuration

Add the AI Tax Agent middleware to Traefik's dynamic configuration:

```bash
# Create Traefik dynamic config for AI Tax Agent
ssh deploy@141.136.35.199 << 'EOF'
cat > /opt/compose/traefik/config/ai-tax-agent.yaml << 'TRAEFIK'
http:
  middlewares:
    # Rate limiting for API
    api-ratelimit:
      rateLimit:
        average: 100
        burst: 50
        period: 1s
    
    # CORS headers
    api-cors:
      headers:
        accessControlAllowMethods:
          - GET
          - POST
          - PUT
          - DELETE
          - OPTIONS
        accessControlAllowOriginList:
          - "https://app.harkon.co.uk"
        accessControlAllowHeaders:
          - "Content-Type"
          - "Authorization"
        accessControlMaxAge: 100
        addVaryHeader: true
    
    # Security headers
    security-headers:
      headers:
        frameDeny: true
        browserXssFilter: true
        contentTypeNosniff: true
        stsSeconds: 31536000
        stsIncludeSubdomains: true
        stsPreload: true
TRAEFIK
EOF
```

### Step 4: Deploy Infrastructure Services

```bash
# Use the deployment script
./scripts/deploy-to-production.sh infrastructure

# Or manually:
ssh deploy@141.136.35.199 << 'EOF'
cd /opt/ai-tax-agent
docker compose -f compose/production/infrastructure.yaml up -d
EOF
```

**Wait 2-3 minutes** for infrastructure services to initialize.

### Step 5: Initialize Vault

```bash
# Initialize Vault (first time only)
ssh deploy@141.136.35.199 << 'EOF'
# Vault will auto-unseal if configured, otherwise:
docker exec vault vault operator init -key-shares=5 -key-threshold=3 > ~/vault-keys.txt
docker exec vault vault operator unseal <unseal-key-1>
docker exec vault vault operator unseal <unseal-key-2>
docker exec vault vault operator unseal <unseal-key-3>
EOF

# IMPORTANT: Save vault-keys.txt securely and delete from server!
scp deploy@141.136.35.199:~/vault-keys.txt ./vault-keys-SECURE.txt
ssh deploy@141.136.35.199 "rm ~/vault-keys.txt"
```

### Step 6: Initialize MinIO

```bash
# MinIO is ready immediately, access at:
# https://minio.harkon.co.uk
# Username: admin (from .env.production MINIO_ROOT_USER)
# Password: <from .env.production MINIO_ROOT_PASSWORD>

# Create required buckets
ssh deploy@141.136.35.199 << 'EOF'
docker exec apa-minio mc alias set local http://localhost:9000 admin <MINIO_ROOT_PASSWORD>
docker exec apa-minio mc mb local/documents
docker exec apa-minio mc mb local/processed
docker exec apa-minio mc mb local/models
docker exec apa-minio mc mb local/temp
EOF
```

### Step 7: Initialize Neo4j

```bash
# Access Neo4j Browser at:
# https://neo4j.harkon.co.uk
# Username: neo4j
# Password: <from .env.production NEO4J_PASSWORD>

# Verify connection
ssh deploy@141.136.35.199 << 'EOF'
docker exec apa-neo4j cypher-shell -u neo4j -p <NEO4J_PASSWORD> "RETURN 'Connected' as status;"
EOF
```

### Step 8: Deploy Application Services

```bash
# Deploy all application services
./scripts/deploy-to-production.sh services

# Or manually:
ssh deploy@141.136.35.199 << 'EOF'
cd /opt/ai-tax-agent
docker compose -f compose/production/services.yaml up -d
EOF
```

**Wait 1-2 minutes** for services to start.

### Step 9: Deploy Monitoring Stack

```bash
# Deploy monitoring
./scripts/deploy-to-production.sh monitoring

# Or manually:
ssh deploy@141.136.35.199 << 'EOF'
cd /opt/ai-tax-agent
docker compose -f compose/production/monitoring.yaml up -d
EOF
```

### Step 10: Configure Authentik OAuth for Grafana

1. **Login to Authentik**: https://auth.harkon.co.uk
2. **Create OAuth Provider**:
   - Applications → Providers → Create
   - Type: OAuth2/OpenID Provider
   - Name: `Grafana`
   - Client ID: `grafana` (copy this)
   - Client Secret: Generate and copy
   - Redirect URIs: `https://grafana.harkon.co.uk/login/generic_oauth`
   - Scopes: `openid`, `profile`, `email`, `groups`

3. **Create Application**:
   - Applications → Create
   - Name: `Grafana`
   - Slug: `grafana`
   - Provider: Select the provider created above
   - Launch URL: `https://grafana.harkon.co.uk`

4. **Update Environment Variables**:
   ```bash
   # On remote server, update .env.production
   ssh deploy@141.136.35.199
   nano /opt/ai-tax-agent/compose/.env.production
   
   # Update these values:
   GRAFANA_OAUTH_CLIENT_ID=grafana
   GRAFANA_OAUTH_CLIENT_SECRET=<secret-from-authentik>
   
   # Restart Grafana
   cd /opt/ai-tax-agent
   docker compose -f compose/production/monitoring.yaml restart apa-grafana
   ```

### Step 11: Verify Deployment

```bash
# Run verification script
./scripts/verify-deployment.sh

# Or check manually:
./scripts/health-check.sh
```

### Step 12: Access Services

| Service | URL | Authentication |
|---------|-----|----------------|
| **Application UI** | https://app.harkon.co.uk | Authentik SSO |
| **API Gateway** | https://api.harkon.co.uk | Authentik SSO |
| **Grafana** | https://grafana.harkon.co.uk | Authentik OAuth |
| **Prometheus** | https://prometheus.harkon.co.uk | Authentik SSO |
| **Vault** | https://vault.harkon.co.uk | Vault Token |
| **MinIO Console** | https://minio-console.harkon.co.uk | MinIO Credentials |
| **Neo4j Browser** | https://neo4j.harkon.co.uk | Neo4j Credentials |
| **Qdrant** | https://qdrant.harkon.co.uk | Authentik SSO |

---

## Post-Deployment Tasks

### 1. Configure Grafana Dashboards

1. Login to Grafana: https://grafana.harkon.co.uk
2. Add Prometheus data source:
   - Configuration → Data Sources → Add data source
   - Type: Prometheus
   - URL: `http://prometheus:9090`
   - Save & Test

3. Add Loki data source:
   - Configuration → Data Sources → Add data source
   - Type: Loki
   - URL: `http://loki:3100`
   - Save & Test

4. Import dashboards (optional):
   - Create → Import
   - Dashboard ID: 1860 (Node Exporter Full)
   - Dashboard ID: 7362 (Docker Monitoring)

### 2. Set Up Alerts (Optional)

Create alert rules in Prometheus or Grafana for:
- Service health checks
- High memory usage
- High CPU usage
- Disk space warnings
- Failed authentication attempts

### 3. Configure Backups

```bash
# Set up automated backups (cron job on server)
ssh deploy@141.136.35.199
crontab -e

# Add daily backup at 2 AM
0 2 * * * /opt/ai-tax-agent/scripts/backup.sh
```

### 4. Test Application Workflows

1. **Upload a document** via UI
2. **Check ingestion** service logs
3. **Verify extraction** in Neo4j
4. **Test RAG retrieval** via API
5. **Review results** in UI

---

## Troubleshooting

### Services Not Starting

```bash
# Check logs
ssh deploy@141.136.35.199 "docker logs <container-name>"

# Check resource usage
ssh deploy@141.136.35.199 "docker stats"

# Restart specific service
ssh deploy@141.136.35.199 "cd /opt/ai-tax-agent && docker compose -f compose/production/services.yaml restart <service-name>"
```

### SSL Certificate Issues

```bash
# Check Traefik logs
ssh deploy@141.136.35.199 "docker logs traefik --tail 100"

# Force certificate renewal
ssh deploy@141.136.35.199 "docker exec traefik rm /var/traefik/certs/godaddy-acme.json && docker restart traefik"
```

### Database Connection Issues

```bash
# Check PostgreSQL
ssh deploy@141.136.35.199 "docker exec postgres pg_isready"

# Check Neo4j
ssh deploy@141.136.35.199 "docker exec neo4j cypher-shell -u neo4j -p <password> 'RETURN 1;'"

# Check Redis
ssh deploy@141.136.35.199 "docker exec redis redis-cli ping"
```

---

## Rollback Procedure

If deployment fails:

```bash
# Use rollback script
./scripts/rollback-deployment.sh

# Or manually restore from backup
ssh deploy@141.136.35.199 << 'EOF'
cd /opt/ai-tax-agent
docker compose -f compose/production/services.yaml down
docker compose -f compose/production/infrastructure.yaml down
docker compose -f compose/production/monitoring.yaml down

# Restore from backup
tar -xzf backups/backup-<timestamp>.tar.gz -C /opt/ai-tax-agent/

# Restart services
docker compose -f compose/production/infrastructure.yaml up -d
sleep 30
docker compose -f compose/production/services.yaml up -d
docker compose -f compose/production/monitoring.yaml up -d
EOF
```

---

## Next Steps

1. ✅ Monitor application logs for errors
2. ✅ Set up automated backups
3. ✅ Configure alerting rules
4. ✅ Document any custom configurations
5. ✅ Train users on the application
6. ✅ Plan for scaling (if needed)

---

## Support

For issues or questions:
- Check logs: `./scripts/verify-deployment.sh`
- Review documentation: `docs/DEPLOYMENT_CHECKLIST.md`
- Contact: [Your support contact]