ai-tax-agent/infra/FINAL_STRUCTURE.md

AI Tax Agent Infrastructure - Final Structure

Overview

The infrastructure is organized into two main categories:

1. External Services - Production-only services deployed individually
2. Application Infrastructure - Multi-environment services for the application

---

Directory Structure

ai-tax-agent/
├── infra/
│   ├── compose/                          # External services (production)
│   │   ├── traefik/                     # Reverse proxy
│   │   │   ├── compose.yaml
│   │   │   ├── config/                  # Traefik configuration (source of truth)
│   │   │   ├── certs/
│   │   │   └── .provider.env
│   │   ├── authentik/                   # SSO provider
│   │   │   ├── compose.yaml
│   │   │   ├── .env
│   │   │   ├── media/
│   │   │   └── custom-templates/
│   │   ├── gitea/                       # Git + Container Registry
│   │   │   ├── compose.yaml
│   │   │   └── .env
│   │   ├── nextcloud/                   # File storage
│   │   │   └── compose.yaml
│   │   ├── portainer/                   # Docker management
│   │   │   └── docker-compose.yaml
│   │   ├── docker-compose.local.yml     # Local dev (all-in-one)
│   │   ├── docker-compose.backend.yml   # Backend services
│   │   └── README.md
│   │
│   ├── base/                            # Application infrastructure (multi-env)
│   │   ├── infrastructure.yaml          # Core services (Vault, MinIO, DBs, etc.)
│   │   ├── services.yaml                # Application microservices (14 services)
│   │   └── monitoring.yaml              # Monitoring stack (Prometheus, Grafana, Loki)
│   │
│   ├── environments/                    # Environment-specific configs
│   │   ├── local/
│   │   │   ├── .env.example
│   │   │   └── .env                     # Local development config
│   │   ├── development/
│   │   │   ├── .env.example
│   │   │   └── .env                     # Development server config
│   │   └── production/
│   │       ├── .env.example
│   │       └── .env                     # Production server config
│   │
│   ├── configs/                         # Application service configs
│   │   ├── traefik/
│   │   │   └── app-middlewares.yml      # App-specific Traefik middlewares
│   │   ├── authentik/
│   │   │   └── bootstrap.yaml           # App-specific Authentik bootstrap
│   │   ├── grafana/
│   │   │   ├── dashboards/
│   │   │   └── provisioning/
│   │   ├── prometheus/
│   │   │   └── prometheus.yml
│   │   ├── loki/
│   │   │   └── loki-config.yml
│   │   └── vault/
│   │       └── config/
│   │
│   ├── docker/                          # Dockerfile templates
│   │   ├── base-runtime.Dockerfile
│   │   ├── base-ml.Dockerfile
│   │   └── Dockerfile.ml-service.template
│   │
│   ├── certs/                           # SSL certificates
│   │   ├── local/
│   │   ├── development/
│   │   └── production/
│   │
│   ├── scripts/                         # Infrastructure deployment scripts
│   │   ├── deploy.sh                    # Deploy application infrastructure
│   │   ├── setup-networks.sh            # Create Docker networks
│   │   └── reorganize-structure.sh
│   │
│   ├── README.md                        # Main infrastructure docs
│   ├── QUICK_START.md                   # Quick start guide
│   ├── DEPLOYMENT_GUIDE.md              # Complete deployment guide
│   ├── MIGRATION_GUIDE.md               # Migration from old structure
│   ├── STRUCTURE_OVERVIEW.md            # Architecture overview
│   ├── STRUCTURE_CLEANUP.md             # Cleanup plan
│   └── FINAL_STRUCTURE.md               # This file
│
├── scripts/                             # Project-wide scripts
│   ├── deploy-external.sh               # Deploy external services
│   ├── cleanup-infra-structure.sh       # Cleanup and align structure
│   ├── build-and-push-images.sh         # Build and push Docker images
│   ├── generate-secrets.sh              # Generate secrets
│   └── ...
│
└── Makefile                             # Project commands

---

Deployment Workflows

1. Local Development

# Option A: Use Makefile (recommended)
make bootstrap
make run

# Option B: Use compose directly
cd infra/compose
docker compose -f docker-compose.local.yml up -d

# Option C: Use new multi-env structure
cp infra/environments/local/.env.example infra/environments/local/.env
./infra/scripts/setup-networks.sh
./infra/scripts/deploy.sh local all

2. Production - External Services

Deploy individually on remote server:

# SSH to server
ssh deploy@141.136.35.199

# Deploy all external services
cd /opt/ai-tax-agent
./scripts/deploy-external.sh all

# Or deploy individually
cd /opt/ai-tax-agent/infra/compose/traefik
docker compose up -d

cd /opt/ai-tax-agent/infra/compose/authentik
docker compose up -d

cd /opt/ai-tax-agent/infra/compose/gitea
docker compose up -d

3. Production - Application Infrastructure

# SSH to server
ssh deploy@141.136.35.199
cd /opt/ai-tax-agent

# Deploy infrastructure
./infra/scripts/deploy.sh production infrastructure

# Deploy monitoring
./infra/scripts/deploy.sh production monitoring

# Deploy services
./infra/scripts/deploy.sh production services

# Or use Makefile
make deploy-infra-prod
make deploy-monitoring-prod
make deploy-services-prod

---

Makefile Commands

Local Development

make bootstrap              # Setup development environment
make run                    # Start all services (local)
make stop                   # Stop all services
make restart                # Restart all services
make logs                   # Show logs from all services
make status                 # Show status of all services
make health                 # Check health of all services

External Services (Production)

make deploy-external        # Deploy all external services
make deploy-traefik         # Deploy Traefik only
make deploy-authentik       # Deploy Authentik only
make deploy-gitea           # Deploy Gitea only
make deploy-nextcloud       # Deploy Nextcloud only
make deploy-portainer       # Deploy Portainer only

Application Infrastructure (Multi-Environment)

# Local
make deploy-infra-local
make deploy-services-local
make deploy-monitoring-local

# Development
make deploy-infra-dev
make deploy-services-dev
make deploy-monitoring-dev

# Production
make deploy-infra-prod
make deploy-services-prod
make deploy-monitoring-prod

Development Tools

make test                   # Run all tests
make lint                   # Run linting
make format                 # Format code
make build                  # Build Docker images
make clean                  # Clean up containers and volumes

---

Configuration Management

External Services

Each external service has its own configuration:

• Traefik: infra/compose/traefik/config/ (source of truth)
• Authentik: infra/compose/authentik/.env
• Gitea: infra/compose/gitea/.env

Application Infrastructure

Application-specific configurations:

• Environment Variables: infra/environments/<env>/.env
• Traefik Middlewares: infra/configs/traefik/app-middlewares.yml
• Authentik Bootstrap: infra/configs/authentik/bootstrap.yaml
• Grafana Dashboards: infra/configs/grafana/dashboards/
• Prometheus Config: infra/configs/prometheus/prometheus.yml

---

Key Differences

External Services vs Application Infrastructure

Aspect	External Services	Application Infrastructure
Location	infra/compose/	infra/base/ + infra/environments/
Deployment	Individual compose files	Unified deployment script
Environment	Production only	Local, Dev, Prod
Purpose	Shared company services	AI Tax Agent application
Examples	Traefik, Authentik, Gitea	Vault, MinIO, Microservices

---

Networks

All services use two shared Docker networks:

• frontend: Public-facing services (connected to Traefik)
• backend: Internal services (databases, message queues)

Create networks:

docker network create frontend
docker network create backend

# Or use script
./infra/scripts/setup-networks.sh

# Or use Makefile
make networks

---

Service Access

Local Development

• Grafana: http://localhost:3000
• MinIO: http://localhost:9093
• Vault: http://localhost:8200
• Traefik Dashboard: http://localhost:8080

Production

• Traefik: https://traefik.harkon.co.uk
• Authentik: https://authentik.harkon.co.uk
• Gitea: https://gitea.harkon.co.uk
• Grafana: https://grafana.harkon.co.uk
• MinIO: https://minio.harkon.co.uk
• Vault: https://vault.harkon.co.uk
• UI Review: https://ui-review.harkon.co.uk

---

Best Practices

1. Configuration Management

• ✅ External service configs live with their compose files
• ✅ Application configs live in infra/configs/
• ✅ Environment-specific settings in .env files
• ✅ Never commit .env files (use .env.example)

2. Deployment

• ✅ Test in local first
• ✅ Deploy to development before production
• ✅ Deploy external services before application infrastructure
• ✅ Deploy infrastructure before services

3. Secrets Management

• ✅ Use ./scripts/generate-secrets.sh for production
• ✅ Store secrets in .env files (gitignored)
• ✅ Use Vault for runtime secrets
• ✅ Rotate secrets regularly

4. Monitoring

• ✅ Check logs after deployment
• ✅ Verify health endpoints
• ✅ Monitor Grafana dashboards
• ✅ Set up alerts for production

---

Troubleshooting

Services Not Starting

# Check logs
docker compose logs -f <service>

# Check status
docker ps -a

# Check networks
docker network ls
docker network inspect frontend

Configuration Issues

# Verify environment file
cat infra/environments/production/.env | grep DOMAIN

# Check compose file syntax
docker compose -f infra/base/infrastructure.yaml config

# Validate Traefik config
docker exec traefik traefik version

Network Issues

# Recreate networks
docker network rm frontend backend
./infra/scripts/setup-networks.sh

# Check network connectivity
docker exec <service> ping <other-service>

---

Migration from Old Structure

If you have the old structure, run:

./scripts/cleanup-infra-structure.sh

This will:

• Remove duplicate configurations
• Align Traefik configs
• Create app-specific middlewares
• Update .gitignore
• Create documentation

---

Next Steps

1. ✅ Structure cleaned up and aligned
2. 📖 Read QUICK_START.md for quick deployment
3. 📚 Read DEPLOYMENT_GUIDE.md for detailed instructions
4. 🧪 Test local deployment: make run
5. 🚀 Deploy to production: make deploy-infra-prod

---

Support

For issues or questions:

1. Check logs: make logs
2. Check health: make health
3. Review documentation in infra/
4. Check Traefik dashboard for routing issues