# AI Tax Agent Infrastructure - Final Structure ## Overview The infrastructure is organized into two main categories: 1. **External Services** - Production-only services deployed individually 2. **Application Infrastructure** - Multi-environment services for the application --- ## Directory Structure ``` ai-tax-agent/ ├── infra/ │ ├── compose/ # External services (production) │ │ ├── traefik/ # Reverse proxy │ │ │ ├── compose.yaml │ │ │ ├── config/ # Traefik configuration (source of truth) │ │ │ ├── certs/ │ │ │ └── .provider.env │ │ ├── authentik/ # SSO provider │ │ │ ├── compose.yaml │ │ │ ├── .env │ │ │ ├── media/ │ │ │ └── custom-templates/ │ │ ├── gitea/ # Git + Container Registry │ │ │ ├── compose.yaml │ │ │ └── .env │ │ ├── nextcloud/ # File storage │ │ │ └── compose.yaml │ │ ├── portainer/ # Docker management │ │ │ └── docker-compose.yaml │ │ ├── docker-compose.local.yml # Local dev (all-in-one) │ │ ├── docker-compose.backend.yml # Backend services │ │ └── README.md │ │ │ ├── base/ # Application infrastructure (multi-env) │ │ ├── infrastructure.yaml # Core services (Vault, MinIO, DBs, etc.) │ │ ├── services.yaml # Application microservices (14 services) │ │ └── monitoring.yaml # Monitoring stack (Prometheus, Grafana, Loki) │ │ │ ├── environments/ # Environment-specific configs │ │ ├── local/ │ │ │ ├── .env.example │ │ │ └── .env # Local development config │ │ ├── development/ │ │ │ ├── .env.example │ │ │ └── .env # Development server config │ │ └── production/ │ │ ├── .env.example │ │ └── .env # Production server config │ │ │ ├── configs/ # Application service configs │ │ ├── traefik/ │ │ │ └── app-middlewares.yml # App-specific Traefik middlewares │ │ ├── authentik/ │ │ │ └── bootstrap.yaml # App-specific Authentik bootstrap │ │ ├── grafana/ │ │ │ ├── dashboards/ │ │ │ └── provisioning/ │ │ ├── prometheus/ │ │ │ └── prometheus.yml │ │ ├── loki/ │ │ │ └── loki-config.yml │ │ └── vault/ │ │ └── config/ │ │ │ ├── docker/ # Dockerfile templates │ │ ├── base-runtime.Dockerfile │ │ ├── base-ml.Dockerfile │ │ └── Dockerfile.ml-service.template │ │ │ ├── certs/ # SSL certificates │ │ ├── local/ │ │ ├── development/ │ │ └── production/ │ │ │ ├── scripts/ # Infrastructure deployment scripts │ │ ├── deploy.sh # Deploy application infrastructure │ │ ├── setup-networks.sh # Create Docker networks │ │ └── reorganize-structure.sh │ │ │ ├── README.md # Main infrastructure docs │ ├── QUICK_START.md # Quick start guide │ ├── DEPLOYMENT_GUIDE.md # Complete deployment guide │ ├── MIGRATION_GUIDE.md # Migration from old structure │ ├── STRUCTURE_OVERVIEW.md # Architecture overview │ ├── STRUCTURE_CLEANUP.md # Cleanup plan │ └── FINAL_STRUCTURE.md # This file │ ├── scripts/ # Project-wide scripts │ ├── deploy-external.sh # Deploy external services │ ├── cleanup-infra-structure.sh # Cleanup and align structure │ ├── build-and-push-images.sh # Build and push Docker images │ ├── generate-secrets.sh # Generate secrets │ └── ... │ └── Makefile # Project commands ``` --- ## Deployment Workflows ### 1. Local Development ```bash # Option A: Use Makefile (recommended) make bootstrap make run # Option B: Use compose directly cd infra/compose docker compose -f docker-compose.local.yml up -d # Option C: Use new multi-env structure cp infra/environments/local/.env.example infra/environments/local/.env ./infra/scripts/setup-networks.sh ./infra/scripts/deploy.sh local all ``` ### 2. Production - External Services Deploy individually on remote server: ```bash # SSH to server ssh deploy@141.136.35.199 # Deploy all external services cd /opt/ai-tax-agent ./scripts/deploy-external.sh all # Or deploy individually cd /opt/ai-tax-agent/infra/compose/traefik docker compose up -d cd /opt/ai-tax-agent/infra/compose/authentik docker compose up -d cd /opt/ai-tax-agent/infra/compose/gitea docker compose up -d ``` ### 3. Production - Application Infrastructure ```bash # SSH to server ssh deploy@141.136.35.199 cd /opt/ai-tax-agent # Deploy infrastructure ./infra/scripts/deploy.sh production infrastructure # Deploy monitoring ./infra/scripts/deploy.sh production monitoring # Deploy services ./infra/scripts/deploy.sh production services # Or use Makefile make deploy-infra-prod make deploy-monitoring-prod make deploy-services-prod ``` --- ## Makefile Commands ### Local Development ```bash make bootstrap # Setup development environment make run # Start all services (local) make stop # Stop all services make restart # Restart all services make logs # Show logs from all services make status # Show status of all services make health # Check health of all services ``` ### External Services (Production) ```bash make deploy-external # Deploy all external services make deploy-traefik # Deploy Traefik only make deploy-authentik # Deploy Authentik only make deploy-gitea # Deploy Gitea only make deploy-nextcloud # Deploy Nextcloud only make deploy-portainer # Deploy Portainer only ``` ### Application Infrastructure (Multi-Environment) ```bash # Local make deploy-infra-local make deploy-services-local make deploy-monitoring-local # Development make deploy-infra-dev make deploy-services-dev make deploy-monitoring-dev # Production make deploy-infra-prod make deploy-services-prod make deploy-monitoring-prod ``` ### Development Tools ```bash make test # Run all tests make lint # Run linting make format # Format code make build # Build Docker images make clean # Clean up containers and volumes ``` --- ## Configuration Management ### External Services Each external service has its own configuration: - **Traefik**: `infra/compose/traefik/config/` (source of truth) - **Authentik**: `infra/compose/authentik/.env` - **Gitea**: `infra/compose/gitea/.env` ### Application Infrastructure Application-specific configurations: - **Environment Variables**: `infra/environments//.env` - **Traefik Middlewares**: `infra/configs/traefik/app-middlewares.yml` - **Authentik Bootstrap**: `infra/configs/authentik/bootstrap.yaml` - **Grafana Dashboards**: `infra/configs/grafana/dashboards/` - **Prometheus Config**: `infra/configs/prometheus/prometheus.yml` --- ## Key Differences ### External Services vs Application Infrastructure | Aspect | External Services | Application Infrastructure | |--------|------------------|---------------------------| | **Location** | `infra/compose/` | `infra/base/` + `infra/environments/` | | **Deployment** | Individual compose files | Unified deployment script | | **Environment** | Production only | Local, Dev, Prod | | **Purpose** | Shared company services | AI Tax Agent application | | **Examples** | Traefik, Authentik, Gitea | Vault, MinIO, Microservices | --- ## Networks All services use two shared Docker networks: - **frontend**: Public-facing services (connected to Traefik) - **backend**: Internal services (databases, message queues) Create networks: ```bash docker network create frontend docker network create backend # Or use script ./infra/scripts/setup-networks.sh # Or use Makefile make networks ``` --- ## Service Access ### Local Development - **Grafana**: http://localhost:3000 - **MinIO**: http://localhost:9093 - **Vault**: http://localhost:8200 - **Traefik Dashboard**: http://localhost:8080 ### Production - **Traefik**: https://traefik.harkon.co.uk - **Authentik**: https://authentik.harkon.co.uk - **Gitea**: https://gitea.harkon.co.uk - **Grafana**: https://grafana.harkon.co.uk - **MinIO**: https://minio.harkon.co.uk - **Vault**: https://vault.harkon.co.uk - **UI Review**: https://ui-review.harkon.co.uk --- ## Best Practices ### 1. Configuration Management - ✅ External service configs live with their compose files - ✅ Application configs live in `infra/configs/` - ✅ Environment-specific settings in `.env` files - ✅ Never commit `.env` files (use `.env.example`) ### 2. Deployment - ✅ Test in local first - ✅ Deploy to development before production - ✅ Deploy external services before application infrastructure - ✅ Deploy infrastructure before services ### 3. Secrets Management - ✅ Use `./scripts/generate-secrets.sh` for production - ✅ Store secrets in `.env` files (gitignored) - ✅ Use Vault for runtime secrets - ✅ Rotate secrets regularly ### 4. Monitoring - ✅ Check logs after deployment - ✅ Verify health endpoints - ✅ Monitor Grafana dashboards - ✅ Set up alerts for production --- ## Troubleshooting ### Services Not Starting ```bash # Check logs docker compose logs -f # Check status docker ps -a # Check networks docker network ls docker network inspect frontend ``` ### Configuration Issues ```bash # Verify environment file cat infra/environments/production/.env | grep DOMAIN # Check compose file syntax docker compose -f infra/base/infrastructure.yaml config # Validate Traefik config docker exec traefik traefik version ``` ### Network Issues ```bash # Recreate networks docker network rm frontend backend ./infra/scripts/setup-networks.sh # Check network connectivity docker exec ping ``` --- ## Migration from Old Structure If you have the old structure, run: ```bash ./scripts/cleanup-infra-structure.sh ``` This will: - Remove duplicate configurations - Align Traefik configs - Create app-specific middlewares - Update .gitignore - Create documentation --- ## Next Steps 1. ✅ Structure cleaned up and aligned 2. 📖 Read [QUICK_START.md](QUICK_START.md) for quick deployment 3. 📚 Read [DEPLOYMENT_GUIDE.md](DEPLOYMENT_GUIDE.md) for detailed instructions 4. 🧪 Test local deployment: `make run` 5. 🚀 Deploy to production: `make deploy-infra-prod` --- ## Support For issues or questions: 1. Check logs: `make logs` 2. Check health: `make health` 3. Review documentation in `infra/` 4. Check Traefik dashboard for routing issues