ai-tax-agent/docs/DEPLOYMENT_PROGRESS.md

# Deployment Progress Report

**Date**: 2025-10-04  
**Status**: Ready for Deployment  
**Next Step**: Build Docker Images

---

## ✅ Completed Tasks

### 1. Production Compose Files Created

Created three production-ready Docker Compose files in `infra/base/`:

#### **infrastructure.yaml**
- Vault (secrets management)
- MinIO (object storage)
- Qdrant (vector database)
- Neo4j (knowledge graph)
- Postgres (relational database)
- Redis (cache)
- NATS (event bus with JetStream)

**Key Features:**
- Uses shared `frontend` and `backend` networks
- All services exposed via Traefik with SSL (GoDaddy cert resolver)
- Protected by Authentik ForwardAuth middleware
- Production-ready health checks
- Persistent volumes for data

#### **services.yaml**
- All microservices (svc-ingestion, svc-extract, svc-kg, svc-rag-retriever, svc-forms, svc-hmrc, svc-ocr)
- Review UI (ui-review)

**Key Features:**
- Images pulled from Gitea registry: `gitea.harkon.co.uk/ai-tax-agent/*`
- All services routed through `api.harkon.co.uk` with path prefixes
- UI exposed at `app.harkon.co.uk`
- Rate limiting and authentication middleware
- Environment variables from `.env.production`

#### **monitoring.yaml**
- Prometheus (metrics collection)
- Grafana (visualization with Authentik OAuth)
- Loki (log aggregation)
- Promtail (log shipper)

**Key Features:**
- 30-day metrics retention
- Grafana integrated with Authentik SSO
- Loki for centralized logging
- All services exposed via Traefik with SSL

### 2. Deployment Scripts Created

#### **scripts/generate-production-secrets.sh**
- Generates strong passwords for all services
- Uses `openssl rand` for cryptographically secure secrets
- Creates backup of `.env.production` before modification
- Displays important credentials (admin password, Vault token, etc.)

**Usage:**
```bash
chmod +x scripts/generate-production-secrets.sh
./scripts/generate-production-secrets.sh
```

#### **scripts/build-and-push-images.sh**
- Builds all Docker images for production
- Tags with version numbers
- Pushes to Gitea registry
- Supports custom registry and version

**Usage:**
```bash
chmod +x scripts/build-and-push-images.sh
./scripts/build-and-push-images.sh gitea.harkon.co.uk v1.0.0
```

#### **scripts/deploy-to-production.sh**
- Automated deployment to remote server
- Step-by-step or full deployment
- Backup, prepare, deploy, verify
- View logs and service status

**Usage:**
```bash
chmod +x scripts/deploy-to-production.sh

# Full deployment
./scripts/deploy-to-production.sh all

# Step-by-step
./scripts/deploy-to-production.sh backup
./scripts/deploy-to-production.sh prepare
./scripts/deploy-to-production.sh infrastructure
./scripts/deploy-to-production.sh services
./scripts/deploy-to-production.sh monitoring
./scripts/deploy-to-production.sh verify

# View logs
./scripts/deploy-to-production.sh logs svc-ingestion
```

### 3. Documentation Created

#### **infra/base manifests**
Comprehensive production deployment guide including:
- Prerequisites checklist
- Three deployment options (automated, step-by-step, manual)
- Post-deployment initialization steps
- Service URLs (public and admin)
- Monitoring and troubleshooting
- Rollback procedures
- Maintenance tasks
- Security notes

### 4. Environment Configuration

#### **.env.production**
- Created from `env.example`
- Ready for secret generation
- Configured for production:
  - `DOMAIN=harkon.co.uk`
  - `DEBUG=false`
  - `DEVELOPMENT_MODE=false`
  - GoDaddy API credentials
  - All service passwords (to be generated)

#### **.gitignore**
- Updated to exclude `.env.production`
- Prevents accidental commit of secrets
- Also excludes `.env.*.backup` files

---

## 📋 Current Status

### What's Ready
✅ Production compose files (infrastructure, services, monitoring)  
✅ Deployment automation scripts  
✅ Secret generation script  
✅ Image build and push script  
✅ Comprehensive documentation  
✅ Environment file template  
✅ Git ignore rules for secrets  

### What's Pending
⏳ Generate production secrets  
⏳ Build Docker images  
⏳ Push images to Gitea registry  
⏳ Create backup of remote server  
⏳ Deploy to production  
⏳ Initialize infrastructure (Vault, MinIO, NATS)  
⏳ Configure Authentik OAuth providers  
⏳ Verify deployment  

---

## 🚀 Next Steps

### Step 1: Generate Production Secrets (5 minutes)

```bash
cd /Users/harris/Projects/ai-tax-agent
chmod +x scripts/generate-production-secrets.sh
./scripts/generate-production-secrets.sh
```

**Important:** Save the output credentials in your password manager!

### Step 2: Build and Push Docker Images (30-60 minutes)

```bash
# Login to Gitea registry
docker login gitea.harkon.co.uk

# Build and push all images
chmod +x scripts/build-and-push-images.sh
./scripts/build-and-push-images.sh gitea.harkon.co.uk v1.0.0
```

This will build and push:
- svc-ingestion
- svc-extract
- svc-kg
- svc-rag-retriever
- svc-rag-indexer
- svc-forms
- svc-hmrc
- svc-ocr
- svc-rpa
- svc-normalize-map
- svc-reason
- svc-firm-connectors
- svc-coverage
- ui-review

### Step 3: Deploy to Production (15-30 minutes)

```bash
# Full automated deployment
chmod +x scripts/deploy-to-production.sh
./scripts/deploy-to-production.sh all
```

Or step-by-step:
```bash
./scripts/deploy-to-production.sh backup
./scripts/deploy-to-production.sh prepare
./scripts/deploy-to-production.sh infrastructure
# Verify infrastructure is healthy
./scripts/deploy-to-production.sh verify
./scripts/deploy-to-production.sh services
./scripts/deploy-to-production.sh monitoring
./scripts/deploy-to-production.sh verify
```

### Step 4: Post-Deployment Configuration (20-30 minutes)

1. **Initialize Vault**
   ```bash
   ssh deploy@141.136.35.199
   cd /opt/ai-tax-agent
   docker exec -it vault vault operator init
   # Save unseal keys!
   docker exec -it vault vault operator unseal
   ```

2. **Create MinIO Buckets**
   ```bash
   docker exec -it minio mc alias set local http://localhost:9092 admin <PASSWORD>
   docker exec -it minio mc mb local/documents
   docker exec -it minio mc mb local/models
   ```

3. **Create NATS Streams**
   ```bash
   docker exec -it nats nats stream add TAX_AGENT_EVENTS \
     --subjects="tax.>" \
     --storage=file \
     --retention=limits \
     --max-age=7d
   ```

4. **Configure Authentik**
   - Login to https://authentik.harkon.co.uk
   - Create groups: `app-admin`, `app-user`, `app-reviewer`
   - Create OAuth providers for Review UI and Grafana
   - Configure ForwardAuth outpost

### Step 5: Verify Deployment (10 minutes)

```bash
# Check all services
./scripts/deploy-to-production.sh verify

# Test endpoints
curl -I https://app.harkon.co.uk
curl -I https://api.harkon.co.uk/healthz
curl -I https://grafana.harkon.co.uk

# View logs
./scripts/deploy-to-production.sh logs svc-ingestion
```

---

## 📊 Architecture Overview

### Network Topology
```
Internet
    ↓
Traefik (Port 80/443)
    ↓
┌─────────────────────────────────────────┐
│         Frontend Network                │
│  - Traefik                              │
│  - Authentik (Server + Outpost)         │
│  - All exposed services                 │
└─────────────────────────────────────────┘
    ↓
┌─────────────────────────────────────────┐
│         Backend Network                 │
│  - Postgres, Redis, Neo4j               │
│  - MinIO, Qdrant, Vault                 │
│  - NATS, Prometheus, Loki               │
│  - All microservices                    │
└─────────────────────────────────────────┘
```

### Service Domains

**Public Services:**
- `app.harkon.co.uk` - Review UI
- `api.harkon.co.uk` - API Gateway (all microservices)
- `grafana.harkon.co.uk` - Monitoring Dashboard

**Admin Services (Auth Required):**
- `vault.harkon.co.uk` - Secrets Management
- `minio.harkon.co.uk` - Object Storage Console
- `neo4j.harkon.co.uk` - Knowledge Graph Browser
- `qdrant.harkon.co.uk` - Vector Database UI
- `prometheus.harkon.co.uk` - Metrics
- `loki.harkon.co.uk` - Logs
- `nats.harkon.co.uk` - Event Bus Monitor

**Company Services (Existing):**
- `authentik.harkon.co.uk` - SSO
- `traefik.harkon.co.uk` - Reverse Proxy Dashboard
- `gitea.harkon.co.uk` - Git Repository
- `cloud.harkon.co.uk` - Nextcloud
- `portainer.harkon.co.uk` - Docker Management

---

## 🔒 Security Considerations

1. **Secrets Management**
   - All secrets generated with `openssl rand`
   - `.env.production` excluded from git
   - Vault for runtime secret storage
   - Authentik for authentication

2. **Network Security**
   - Services isolated in backend network
   - Only necessary services on frontend network
   - All traffic encrypted with SSL (Let's Encrypt via GoDaddy DNS)
   - ForwardAuth middleware on all admin services

3. **Access Control**
   - Authentik SSO for all services
   - Role-based access (admin, user, reviewer)
   - OAuth2 for service-to-service auth

---

## 📝 Important Notes

1. **Backup Before Deployment**
   - Always create backup before making changes
   - Script includes automatic backup step
   - Backups stored in `~/backups/` on remote server

2. **Incremental Deployment**
   - Deploy infrastructure first
   - Verify health before deploying services
   - Monitor logs during deployment

3. **Rollback Plan**
   - Backups available in `~/backups/`
   - Can restore previous state
   - Company services unaffected

4. **Monitoring**
   - Grafana dashboards for all services
   - Loki for centralized logging
   - Prometheus for metrics
   - Alerts configured in Grafana

---

## 🎯 Success Criteria

Deployment is successful when:
- [ ] All infrastructure services are running and healthy
- [ ] All application services are running and healthy
- [ ] All monitoring services are running and healthy
- [ ] UI accessible at https://app.harkon.co.uk
- [ ] API accessible at https://api.harkon.co.uk
- [ ] Grafana accessible at https://grafana.harkon.co.uk
- [ ] All services protected by Authentik
- [ ] SSL certificates valid
- [ ] No errors in logs
- [ ] Company services still operational

---

## 📞 Support

If you encounter issues:
1. Check logs: `./scripts/deploy-to-production.sh logs <service>`
2. Verify status: `./scripts/deploy-to-production.sh verify`
3. Review manifests: `infra/base/*.yaml`
4. Check deployment plan: `docs/DEPLOYMENT_PLAN.md`
5. Follow checklist: `docs/DEPLOYMENT_CHECKLIST.md`