ai-tax-agent/infra/FINAL_STRUCTURE.md

# AI Tax Agent Infrastructure - Final Structure

## Overview

The infrastructure is organized into two main categories:

1. **External Services** - Production-only services deployed individually
2. **Application Infrastructure** - Multi-environment services for the application

---

## Directory Structure

```
ai-tax-agent/
├── infra/
│   ├── compose/                          # External services (production)
│   │   ├── traefik/                     # Reverse proxy
│   │   │   ├── compose.yaml
│   │   │   ├── config/                  # Traefik configuration (source of truth)
│   │   │   ├── certs/
│   │   │   └── .provider.env
│   │   ├── authentik/                   # SSO provider
│   │   │   ├── compose.yaml
│   │   │   ├── .env
│   │   │   ├── media/
│   │   │   └── custom-templates/
│   │   ├── gitea/                       # Git + Container Registry
│   │   │   ├── compose.yaml
│   │   │   └── .env
│   │   ├── nextcloud/                   # File storage
│   │   │   └── compose.yaml
│   │   ├── portainer/                   # Docker management
│   │   │   └── docker-compose.yaml
│   │   ├── docker-compose.local.yml     # Local dev (all-in-one)
│   │   ├── docker-compose.backend.yml   # Backend services
│   │   └── README.md
│   │
│   ├── base/                            # Application infrastructure (multi-env)
│   │   ├── infrastructure.yaml          # Core services (Vault, MinIO, DBs, etc.)
│   │   ├── services.yaml                # Application microservices (14 services)
│   │   └── monitoring.yaml              # Monitoring stack (Prometheus, Grafana, Loki)
│   │
│   ├── environments/                    # Environment-specific configs
│   │   ├── local/
│   │   │   ├── .env.example
│   │   │   └── .env                     # Local development config
│   │   ├── development/
│   │   │   ├── .env.example
│   │   │   └── .env                     # Development server config
│   │   └── production/
│   │       ├── .env.example
│   │       └── .env                     # Production server config
│   │
│   ├── configs/                         # Application service configs
│   │   ├── traefik/
│   │   │   └── app-middlewares.yml      # App-specific Traefik middlewares
│   │   ├── authentik/
│   │   │   └── bootstrap.yaml           # App-specific Authentik bootstrap
│   │   ├── grafana/
│   │   │   ├── dashboards/
│   │   │   └── provisioning/
│   │   ├── prometheus/
│   │   │   └── prometheus.yml
│   │   ├── loki/
│   │   │   └── loki-config.yml
│   │   └── vault/
│   │       └── config/
│   │
│   ├── docker/                          # Dockerfile templates
│   │   ├── base-runtime.Dockerfile
│   │   ├── base-ml.Dockerfile
│   │   └── Dockerfile.ml-service.template
│   │
│   ├── certs/                           # SSL certificates
│   │   ├── local/
│   │   ├── development/
│   │   └── production/
│   │
│   ├── scripts/                         # Infrastructure deployment scripts
│   │   ├── deploy.sh                    # Deploy application infrastructure
│   │   ├── setup-networks.sh            # Create Docker networks
│   │   └── reorganize-structure.sh
│   │
│   ├── README.md                        # Main infrastructure docs
│   ├── QUICK_START.md                   # Quick start guide
│   ├── DEPLOYMENT_GUIDE.md              # Complete deployment guide
│   ├── MIGRATION_GUIDE.md               # Migration from old structure
│   ├── STRUCTURE_OVERVIEW.md            # Architecture overview
│   ├── STRUCTURE_CLEANUP.md             # Cleanup plan
│   └── FINAL_STRUCTURE.md               # This file
│
├── scripts/                             # Project-wide scripts
│   ├── deploy-external.sh               # Deploy external services
│   ├── cleanup-infra-structure.sh       # Cleanup and align structure
│   ├── build-and-push-images.sh         # Build and push Docker images
│   ├── generate-secrets.sh              # Generate secrets
│   └── ...
│
└── Makefile                             # Project commands
```

---

## Deployment Workflows

### 1. Local Development

```bash
# Option A: Use Makefile (recommended)
make bootstrap
make run

# Option B: Use compose directly
cd infra/compose
docker compose -f docker-compose.local.yml up -d

# Option C: Use new multi-env structure
cp infra/environments/local/.env.example infra/environments/local/.env
./infra/scripts/setup-networks.sh
./infra/scripts/deploy.sh local all
```

### 2. Production - External Services

Deploy individually on remote server:

```bash
# SSH to server
ssh deploy@141.136.35.199

# Deploy all external services
cd /opt/ai-tax-agent
./scripts/deploy-external.sh all

# Or deploy individually
cd /opt/ai-tax-agent/infra/compose/traefik
docker compose up -d

cd /opt/ai-tax-agent/infra/compose/authentik
docker compose up -d

cd /opt/ai-tax-agent/infra/compose/gitea
docker compose up -d
```

### 3. Production - Application Infrastructure

```bash
# SSH to server
ssh deploy@141.136.35.199
cd /opt/ai-tax-agent

# Deploy infrastructure
./infra/scripts/deploy.sh production infrastructure

# Deploy monitoring
./infra/scripts/deploy.sh production monitoring

# Deploy services
./infra/scripts/deploy.sh production services

# Or use Makefile
make deploy-infra-prod
make deploy-monitoring-prod
make deploy-services-prod
```

---

## Makefile Commands

### Local Development

```bash
make bootstrap              # Setup development environment
make run                    # Start all services (local)
make stop                   # Stop all services
make restart                # Restart all services
make logs                   # Show logs from all services
make status                 # Show status of all services
make health                 # Check health of all services
```

### External Services (Production)

```bash
make deploy-external        # Deploy all external services
make deploy-traefik         # Deploy Traefik only
make deploy-authentik       # Deploy Authentik only
make deploy-gitea           # Deploy Gitea only
make deploy-nextcloud       # Deploy Nextcloud only
make deploy-portainer       # Deploy Portainer only
```

### Application Infrastructure (Multi-Environment)

```bash
# Local
make deploy-infra-local
make deploy-services-local
make deploy-monitoring-local

# Development
make deploy-infra-dev
make deploy-services-dev
make deploy-monitoring-dev

# Production
make deploy-infra-prod
make deploy-services-prod
make deploy-monitoring-prod
```

### Development Tools

```bash
make test                   # Run all tests
make lint                   # Run linting
make format                 # Format code
make build                  # Build Docker images
make clean                  # Clean up containers and volumes
```

---

## Configuration Management

### External Services

Each external service has its own configuration:

- **Traefik**: `infra/compose/traefik/config/` (source of truth)
- **Authentik**: `infra/compose/authentik/.env`
- **Gitea**: `infra/compose/gitea/.env`

### Application Infrastructure

Application-specific configurations:

- **Environment Variables**: `infra/environments/<env>/.env`
- **Traefik Middlewares**: `infra/configs/traefik/app-middlewares.yml`
- **Authentik Bootstrap**: `infra/configs/authentik/bootstrap.yaml`
- **Grafana Dashboards**: `infra/configs/grafana/dashboards/`
- **Prometheus Config**: `infra/configs/prometheus/prometheus.yml`

---

## Key Differences

### External Services vs Application Infrastructure

| Aspect | External Services | Application Infrastructure |
|--------|------------------|---------------------------|
| **Location** | `infra/compose/` | `infra/base/` + `infra/environments/` |
| **Deployment** | Individual compose files | Unified deployment script |
| **Environment** | Production only | Local, Dev, Prod |
| **Purpose** | Shared company services | AI Tax Agent application |
| **Examples** | Traefik, Authentik, Gitea | Vault, MinIO, Microservices |

---

## Networks

All services use two shared Docker networks:

- **frontend**: Public-facing services (connected to Traefik)
- **backend**: Internal services (databases, message queues)

Create networks:

```bash
docker network create frontend
docker network create backend

# Or use script
./infra/scripts/setup-networks.sh

# Or use Makefile
make networks
```

---

## Service Access

### Local Development

- **Grafana**: http://localhost:3000
- **MinIO**: http://localhost:9093
- **Vault**: http://localhost:8200
- **Traefik Dashboard**: http://localhost:8080

### Production

- **Traefik**: https://traefik.harkon.co.uk
- **Authentik**: https://authentik.harkon.co.uk
- **Gitea**: https://gitea.harkon.co.uk
- **Grafana**: https://grafana.harkon.co.uk
- **MinIO**: https://minio.harkon.co.uk
- **Vault**: https://vault.harkon.co.uk
- **UI Review**: https://ui-review.harkon.co.uk

---

## Best Practices

### 1. Configuration Management

- ✅ External service configs live with their compose files
- ✅ Application configs live in `infra/configs/`
- ✅ Environment-specific settings in `.env` files
- ✅ Never commit `.env` files (use `.env.example`)

### 2. Deployment

- ✅ Test in local first
- ✅ Deploy to development before production
- ✅ Deploy external services before application infrastructure
- ✅ Deploy infrastructure before services

### 3. Secrets Management

- ✅ Use `./scripts/generate-secrets.sh` for production
- ✅ Store secrets in `.env` files (gitignored)
- ✅ Use Vault for runtime secrets
- ✅ Rotate secrets regularly

### 4. Monitoring

- ✅ Check logs after deployment
- ✅ Verify health endpoints
- ✅ Monitor Grafana dashboards
- ✅ Set up alerts for production

---

## Troubleshooting

### Services Not Starting

```bash
# Check logs
docker compose logs -f <service>

# Check status
docker ps -a

# Check networks
docker network ls
docker network inspect frontend
```

### Configuration Issues

```bash
# Verify environment file
cat infra/environments/production/.env | grep DOMAIN

# Check compose file syntax
docker compose -f infra/base/infrastructure.yaml config

# Validate Traefik config
docker exec traefik traefik version
```

### Network Issues

```bash
# Recreate networks
docker network rm frontend backend
./infra/scripts/setup-networks.sh

# Check network connectivity
docker exec <service> ping <other-service>
```

---

## Migration from Old Structure

If you have the old structure, run:

```bash
./scripts/cleanup-infra-structure.sh
```

This will:
- Remove duplicate configurations
- Align Traefik configs
- Create app-specific middlewares
- Update .gitignore
- Create documentation

---

## Next Steps

1. ✅ Structure cleaned up and aligned
2. 📖 Read [QUICK_START.md](QUICK_START.md) for quick deployment
3. 📚 Read [DEPLOYMENT_GUIDE.md](DEPLOYMENT_GUIDE.md) for detailed instructions
4. 🧪 Test local deployment: `make run`
5. 🚀 Deploy to production: `make deploy-infra-prod`

---

## Support

For issues or questions:

1. Check logs: `make logs`
2. Check health: `make health`
3. Review documentation in `infra/`
4. Check Traefik dashboard for routing issues