Initial commit

infra/STRUCTURE_CLEANUP.md

@@ -0,0 +1,243 @@
+# Infrastructure Structure Cleanup Plan
+
+## Current Situation
+
+We have two parallel structures that need to be aligned:
+
+### 1. External Services (Production/Remote)
+Located in `infra/compose/` - These are deployed individually on the remote server:
+- **Traefik** - `infra/compose/traefik/`
+- **Authentik** - `infra/compose/authentik/`
+- **Gitea** - `infra/compose/gitea/`
+- **Nextcloud** - `infra/compose/nextcloud/`
+- **Portainer** - `infra/compose/portainer/`
+
+### 2. Application Infrastructure (Multi-Environment)
+Located in `infra/base/` and `infra/environments/`:
+- Infrastructure services (Vault, MinIO, DBs, etc.)
+- Application services (14 microservices)
+- Monitoring stack (Prometheus, Grafana, Loki)
+
+### 3. Configuration Duplication
+- `infra/compose/traefik/config/` - Production Traefik config
+- `infra/configs/traefik/` - Application Traefik config (copied)
+- Similar duplication for other services
+
+---
+
+## Cleanup Strategy
+
+### Phase 1: Consolidate Configurations
+
+#### Traefik
+- **Keep**: `infra/compose/traefik/` as the source of truth for production
+- **Symlink**: `infra/configs/traefik/` → `../compose/traefik/`
+- **Reason**: External service configs should live with their compose files
+
+#### Authentik
+- **Keep**: `infra/compose/authentik/` for production
+- **Keep**: `infra/configs/authentik/` for application-specific bootstrap
+- **Reason**: Different purposes - one for service, one for app integration
+
+#### Grafana/Prometheus/Loki
+- **Keep**: `infra/configs/grafana/`, `infra/configs/prometheus/`, `infra/configs/loki/`
+- **Reason**: These are application-specific, not external services
+
+### Phase 2: Update References
+
+#### Makefile
+- Update paths to reference correct locations
+- Add targets for external service deployment
+- Separate local dev from production deployment
+
+#### Scripts
+- Update `scripts/deploy.sh` to handle external services
+- Create `scripts/deploy-external.sh` for production external services
+- Update `infra/scripts/deploy.sh` for application infrastructure
+
+### Phase 3: Documentation
+
+- Clear separation between:
+  - External services (production only)
+  - Application infrastructure (multi-environment)
+  - Development environment (local only)
+
+---
+
+## Proposed Final Structure
+
+```
+ai-tax-agent/
+├── infra/
+│   ├── compose/                    # External services (production)
+│   │   ├── traefik/
+│   │   │   ├── compose.yaml       # Traefik service definition
+│   │   │   ├── config/            # Traefik configuration
+│   │   │   ├── certs/             # SSL certificates
+│   │   │   └── .provider.env      # GoDaddy API credentials
+│   │   ├── authentik/
+│   │   │   ├── compose.yaml
+│   │   │   ├── .env
+│   │   │   ├── media/
+│   │   │   └── custom-templates/
+│   │   ├── gitea/
+│   │   │   ├── compose.yaml
+│   │   │   └── .env
+│   │   ├── nextcloud/
+│   │   │   └── compose.yaml
+│   │   ├── portainer/
+│   │   │   └── docker-compose.yaml
+│   │   ├── docker-compose.local.yml    # Local dev (all-in-one)
+│   │   └── docker-compose.backend.yml  # Backend services
+│   │
+│   ├── base/                       # Application infrastructure (multi-env)
+│   │   ├── infrastructure.yaml    # Core infra services
+│   │   ├── services.yaml          # Application microservices
+│   │   └── monitoring.yaml        # Monitoring stack
+│   │
+│   ├── environments/               # Environment-specific configs
+│   │   ├── local/
+│   │   │   ├── .env.example
+│   │   │   └── .env
+│   │   ├── development/
+│   │   │   ├── .env.example
+│   │   │   └── .env
+│   │   └── production/
+│   │       ├── .env.example
+│   │       └── .env
+│   │
+│   ├── configs/                    # Application service configs
+│   │   ├── authentik/             # App-specific Authentik bootstrap
+│   │   ├── grafana/               # Grafana dashboards
+│   │   ├── prometheus/            # Prometheus scrape configs
+│   │   ├── loki/                  # Loki config
+│   │   └── vault/                 # Vault config
+│   │
+│   └── scripts/                    # Infrastructure deployment scripts
+│       ├── deploy.sh              # Deploy application infrastructure
+│       ├── setup-networks.sh      # Create Docker networks
+│       └── reorganize-structure.sh
+│
+├── scripts/                        # Project-wide scripts
+│   ├── deploy-external.sh         # Deploy external services (production)
+│   ├── build-and-push-images.sh   # Build and push Docker images
+│   ├── generate-secrets.sh        # Generate secrets
+│   └── ...
+│
+└── Makefile                        # Project commands
+```
+
+---
+
+## Deployment Workflows
+
+### Local Development
+```bash
+# Use all-in-one compose file
+make bootstrap
+make run
+# OR
+cd infra/compose
+docker compose -f docker-compose.local.yml up -d
+```
+
+### Production - External Services
+```bash
+# Deploy individually on remote server
+cd /opt/ai-tax-agent/infra/compose/traefik
+docker compose up -d
+
+cd /opt/ai-tax-agent/infra/compose/authentik
+docker compose up -d
+
+cd /opt/ai-tax-agent/infra/compose/gitea
+docker compose up -d
+```
+
+### Production - Application Infrastructure
+```bash
+# Deploy application infrastructure
+./infra/scripts/deploy.sh production infrastructure
+./infra/scripts/deploy.sh production monitoring
+./infra/scripts/deploy.sh production services
+```
+
+---
+
+## Migration Steps
+
+### Step 1: Align Traefik Configs
+
+```bash
+# Remove duplicate configs
+rm -rf infra/configs/traefik/config/traefik-dynamic.yml
+
+# Keep only app-specific middleware
+# Move production configs to compose/traefik/config/
+```
+
+### Step 2: Update Makefile
+
+- Add targets for external service deployment
+- Update paths to reference correct locations
+- Separate local dev from production
+
+### Step 3: Update Scripts
+
+- Create `scripts/deploy-external.sh` for production
+- Update `infra/scripts/deploy.sh` for application infra
+- Update all path references
+
+### Step 4: Documentation
+
+- Update README files
+- Create deployment guides for each environment
+- Document external vs application services
+
+---
+
+## Key Decisions
+
+### 1. External Services Location
+**Decision**: Keep in `infra/compose/` with individual folders
+**Reason**: These are production-only, deployed separately, have their own configs
+
+### 2. Application Infrastructure Location
+**Decision**: Keep in `infra/base/` with environment-specific `.env` files
+**Reason**: Multi-environment support, shared compose files
+
+### 3. Configuration Management
+**Decision**: 
+- External service configs live with their compose files
+- Application configs live in `infra/configs/`
+**Reason**: Clear separation of concerns
+
+### 4. Makefile Targets
+**Decision**: 
+- `make run` - Local development (all-in-one)
+- `make deploy-external` - Production external services
+- `make deploy-infra` - Application infrastructure
+**Reason**: Clear separation of deployment targets
+
+---
+
+## Benefits
+
+✅ **Clear Separation** - External vs application services
+✅ **No Duplication** - Single source of truth for configs
+✅ **Multi-Environment** - Easy to deploy to local/dev/prod
+✅ **Maintainable** - Logical organization
+✅ **Scalable** - Easy to add new services
+✅ **Production-Ready** - Matches actual deployment
+
+---
+
+## Next Steps
+
+1. Run cleanup script to align configurations
+2. Update Makefile with new targets
+3. Update deployment scripts
+4. Test local deployment
+5. Test production deployment
+6. Update documentation
+