ai-tax-agent/infra/MIGRATION_GUIDE.md

# Infrastructure Migration Guide

This guide helps you migrate from the old infrastructure structure to the new organized multi-environment setup.

## Old Structure vs New Structure

### Old Structure
```
infra/
├── compose/
│   ├── docker-compose.local.yml (1013 lines - everything)
│   ├── docker-compose.backend.yml (1014 lines - everything)
│   ├── authentik/compose.yaml
│   ├── gitea/compose.yaml
│   ├── nextcloud/compose.yaml
│   ├── portainer/docker-compose.yaml
│   └── traefik/compose.yaml
├── production/
│   ├── infrastructure.yaml
│   ├── services.yaml
│   └── monitoring.yaml
├── .env.production
└── various config folders
```

### New Structure
```
infra/
├── base/                      # Shared compose files
│   ├── infrastructure.yaml
│   ├── services.yaml
│   ├── monitoring.yaml
│   └── external.yaml
├── environments/              # Environment-specific configs
│   ├── local/.env
│   ├── development/.env
│   └── production/.env
├── configs/                   # Service configurations
│   ├── traefik/
│   ├── grafana/
│   ├── prometheus/
│   └── ...
└── scripts/
    └── deploy.sh              # Unified deployment script
```

## Migration Steps

### Step 1: Backup Current Setup

```bash
# Backup current environment files
cp infra/.env.production infra/.env.production.backup
cp infra/compose/.env infra/compose/.env.backup

# Backup compose files
tar -czf infra-backup-$(date +%Y%m%d).tar.gz infra/
```

### Step 2: Stop Current Services (if migrating live)

```bash
# Stop services (if running)
cd infra/compose
docker compose -f docker-compose.local.yml down

# Or for production
cd infra/production
docker compose -f infrastructure.yaml down
docker compose -f services.yaml down
docker compose -f monitoring.yaml down
```

### Step 3: Create Environment Files

```bash
# For local development
cp infra/environments/local/.env.example infra/environments/local/.env
vim infra/environments/local/.env

# For development server
cp infra/environments/development/.env.example infra/environments/development/.env
vim infra/environments/development/.env

# For production (copy from existing)
cp infra/.env.production infra/environments/production/.env
```

### Step 4: Move Configuration Files

```bash
# Move Traefik configs
cp -r infra/traefik/* infra/configs/traefik/

# Move Grafana configs
cp -r infra/grafana/* infra/configs/grafana/

# Move Prometheus configs
cp -r infra/prometheus/* infra/configs/prometheus/

# Move Loki configs
cp -r infra/loki/* infra/configs/loki/

# Move Vault configs
cp -r infra/vault/* infra/configs/vault/

# Move Authentik configs
cp -r infra/authentik/* infra/configs/authentik/
```

### Step 5: Update Volume Names (if needed)

If you want to preserve existing data, you have two options:

#### Option A: Keep Existing Volumes (Recommended)

The new compose files use the same volume names, so your data will be preserved automatically.

#### Option B: Rename Volumes

If you want environment-specific volume names:

```bash
# List current volumes
docker volume ls

# Rename volumes (example for production)
docker volume create prod_postgres_data
docker run --rm -v postgres_data:/from -v prod_postgres_data:/to alpine sh -c "cd /from && cp -av . /to"

# Repeat for each volume
```

### Step 6: Setup Networks

```bash
# Create Docker networks
./infra/scripts/setup-networks.sh
```

### Step 7: Deploy New Structure

```bash
# For local
./infra/scripts/deploy.sh local all

# For development
./infra/scripts/deploy.sh development all

# For production
./infra/scripts/deploy.sh production all
```

### Step 8: Verify Services

```bash
# Check running services
docker ps

# Check logs
docker compose -f infra/base/infrastructure.yaml --env-file infra/environments/production/.env logs -f

# Test endpoints
curl https://vault.harkon.co.uk
curl https://minio.harkon.co.uk
curl https://grafana.harkon.co.uk
```

## Handling External Services

If you have existing Traefik, Authentik, Gitea, Nextcloud, or Portainer:

### Option 1: Keep Existing (Recommended for Production)

Don't deploy `external.yaml`. Just ensure:

1. Networks are shared:
```yaml
networks:
  frontend:
    external: true
  backend:
    external: true
```

2. Services can discover each other via network

### Option 2: Migrate to New Structure

1. Stop existing services
2. Update their compose files to use new structure
3. Deploy via `external.yaml`

## Environment-Specific Differences

### Local Development

- Uses `localhost` or `*.local.harkon.co.uk`
- Self-signed SSL certificates
- Simple passwords
- Optional Authentik
- Traefik dashboard exposed on port 8080

### Development Server

- Uses `*.dev.harkon.co.uk`
- Let's Encrypt SSL via DNS-01 challenge
- Strong passwords (generated)
- Authentik SSO enabled
- Gitea container registry

### Production Server

- Uses `*.harkon.co.uk`
- Let's Encrypt SSL via DNS-01 challenge
- Strong passwords (generated)
- Authentik SSO enabled
- Gitea container registry
- No debug ports exposed

## Troubleshooting

### Issue: Services can't find each other

**Solution**: Ensure networks are created and services are on the correct networks

```bash
docker network ls
docker network inspect frontend
docker network inspect backend
```

### Issue: Volumes not found

**Solution**: Check volume names match

```bash
docker volume ls
docker compose -f infra/base/infrastructure.yaml --env-file infra/environments/production/.env config
```

### Issue: Environment variables not loaded

**Solution**: Check .env file exists and is in correct location

```bash
ls -la infra/environments/production/.env
cat infra/environments/production/.env | grep DOMAIN
```

### Issue: Traefik routing not working

**Solution**: Check labels and ensure Traefik can see containers

```bash
docker logs traefik | grep -i error
docker inspect <container> | grep -A 20 Labels
```

## Rollback Plan

If migration fails:

```bash
# Stop new services
./infra/scripts/deploy.sh production down

# Restore old structure
cd infra/compose
docker compose -f docker-compose.backend.yml up -d

# Or for production
cd infra/production
docker compose -f infrastructure.yaml up -d
docker compose -f services.yaml up -d
docker compose -f monitoring.yaml up -d
```

## Post-Migration Cleanup

After successful migration and verification:

```bash
# Remove old compose files (optional)
rm -rf infra/compose/docker-compose.*.yml

# Remove old production folder (optional)
rm -rf infra/production.old

# Remove backup files
rm infra/.env.production.backup
rm infra-backup-*.tar.gz
```

## Benefits of New Structure

✅ **Multi-environment support** - Easy to deploy to local, dev, prod
✅ **Cleaner organization** - Configs separated by purpose
✅ **Unified deployment** - Single script for all environments
✅ **Better security** - Environment-specific secrets
✅ **Easier maintenance** - Clear separation of concerns
✅ **Scalable** - Easy to add new environments or services

## Next Steps

1. Test in local environment first
2. Deploy to development server
3. Verify all services work
4. Deploy to production
5. Update documentation
6. Train team on new structure