ai-tax-agent/infra/STRUCTURE_CLEANUP.md

Infrastructure Structure Cleanup Plan

Current Situation

We have two parallel structures that need to be aligned:

1. External Services (Production/Remote)

Located in infra/compose/ - These are deployed individually on the remote server:

• Traefik - infra/compose/traefik/
• Authentik - infra/compose/authentik/
• Gitea - infra/compose/gitea/
• Nextcloud - infra/compose/nextcloud/
• Portainer - infra/compose/portainer/

2. Application Infrastructure (Multi-Environment)

Located in infra/base/ and infra/environments/:

• Infrastructure services (Vault, MinIO, DBs, etc.)
• Application services (14 microservices)
• Monitoring stack (Prometheus, Grafana, Loki)

3. Configuration Duplication

• infra/compose/traefik/config/ - Production Traefik config
• infra/configs/traefik/ - Application Traefik config (copied)
• Similar duplication for other services

---

Cleanup Strategy

Phase 1: Consolidate Configurations

Traefik

• Keep: infra/compose/traefik/ as the source of truth for production
• Symlink: infra/configs/traefik/ → ../compose/traefik/
• Reason: External service configs should live with their compose files

Authentik

• Keep: infra/compose/authentik/ for production
• Keep: infra/configs/authentik/ for application-specific bootstrap
• Reason: Different purposes - one for service, one for app integration

Grafana/Prometheus/Loki

• Keep: infra/configs/grafana/, infra/configs/prometheus/, infra/configs/loki/
• Reason: These are application-specific, not external services

Phase 2: Update References

Makefile

• Update paths to reference correct locations
• Add targets for external service deployment
• Separate local dev from production deployment

Scripts

• Update scripts/deploy.sh to handle external services
• Create scripts/deploy-external.sh for production external services
• Update infra/scripts/deploy.sh for application infrastructure

Phase 3: Documentation

• Clear separation between:
  • External services (production only)
  • Application infrastructure (multi-environment)
  • Development environment (local only)

---

Proposed Final Structure

ai-tax-agent/
├── infra/
│   ├── compose/                    # External services (production)
│   │   ├── traefik/
│   │   │   ├── compose.yaml       # Traefik service definition
│   │   │   ├── config/            # Traefik configuration
│   │   │   ├── certs/             # SSL certificates
│   │   │   └── .provider.env      # GoDaddy API credentials
│   │   ├── authentik/
│   │   │   ├── compose.yaml
│   │   │   ├── .env
│   │   │   ├── media/
│   │   │   └── custom-templates/
│   │   ├── gitea/
│   │   │   ├── compose.yaml
│   │   │   └── .env
│   │   ├── nextcloud/
│   │   │   └── compose.yaml
│   │   ├── portainer/
│   │   │   └── docker-compose.yaml
│   │   ├── docker-compose.local.yml    # Local dev (all-in-one)
│   │   └── docker-compose.backend.yml  # Backend services
│   │
│   ├── base/                       # Application infrastructure (multi-env)
│   │   ├── infrastructure.yaml    # Core infra services
│   │   ├── services.yaml          # Application microservices
│   │   └── monitoring.yaml        # Monitoring stack
│   │
│   ├── environments/               # Environment-specific configs
│   │   ├── local/
│   │   │   ├── .env.example
│   │   │   └── .env
│   │   ├── development/
│   │   │   ├── .env.example
│   │   │   └── .env
│   │   └── production/
│   │       ├── .env.example
│   │       └── .env
│   │
│   ├── configs/                    # Application service configs
│   │   ├── authentik/             # App-specific Authentik bootstrap
│   │   ├── grafana/               # Grafana dashboards
│   │   ├── prometheus/            # Prometheus scrape configs
│   │   ├── loki/                  # Loki config
│   │   └── vault/                 # Vault config
│   │
│   └── scripts/                    # Infrastructure deployment scripts
│       ├── deploy.sh              # Deploy application infrastructure
│       ├── setup-networks.sh      # Create Docker networks
│       └── reorganize-structure.sh
│
├── scripts/                        # Project-wide scripts
│   ├── deploy-external.sh         # Deploy external services (production)
│   ├── build-and-push-images.sh   # Build and push Docker images
│   ├── generate-secrets.sh        # Generate secrets
│   └── ...
│
└── Makefile                        # Project commands

---

Deployment Workflows

Local Development

# Use all-in-one compose file
make bootstrap
make run
# OR
cd infra/compose
docker compose -f docker-compose.local.yml up -d

Production - External Services

# Deploy individually on remote server
cd /opt/ai-tax-agent/infra/compose/traefik
docker compose up -d

cd /opt/ai-tax-agent/infra/compose/authentik
docker compose up -d

cd /opt/ai-tax-agent/infra/compose/gitea
docker compose up -d

Production - Application Infrastructure

# Deploy application infrastructure
./infra/scripts/deploy.sh production infrastructure
./infra/scripts/deploy.sh production monitoring
./infra/scripts/deploy.sh production services

---

Migration Steps

Step 1: Align Traefik Configs

# Remove duplicate configs
rm -rf infra/configs/traefik/config/traefik-dynamic.yml

# Keep only app-specific middleware
# Move production configs to compose/traefik/config/

Step 2: Update Makefile

• Add targets for external service deployment
• Update paths to reference correct locations
• Separate local dev from production

Step 3: Update Scripts

• Create scripts/deploy-external.sh for production
• Update infra/scripts/deploy.sh for application infra
• Update all path references

Step 4: Documentation

• Update README files
• Create deployment guides for each environment
• Document external vs application services

---

Key Decisions

1. External Services Location

Decision: Keep in infra/compose/ with individual folders Reason: These are production-only, deployed separately, have their own configs

2. Application Infrastructure Location

Decision: Keep in infra/base/ with environment-specific .env files Reason: Multi-environment support, shared compose files

3. Configuration Management

Decision:

• External service configs live with their compose files
• Application configs live in infra/configs/ Reason: Clear separation of concerns

4. Makefile Targets

Decision:

• make run - Local development (all-in-one)
• make deploy-external - Production external services
• make deploy-infra - Application infrastructure Reason: Clear separation of deployment targets

---

Benefits

✅ Clear Separation - External vs application services ✅ No Duplication - Single source of truth for configs ✅ Multi-Environment - Easy to deploy to local/dev/prod ✅ Maintainable - Logical organization ✅ Scalable - Easy to add new services ✅ Production-Ready - Matches actual deployment

---

Next Steps

1. Run cleanup script to align configurations
2. Update Makefile with new targets
3. Update deployment scripts
4. Test local deployment
5. Test production deployment
6. Update documentation