ai-tax-agent/infra/STRUCTURE_CLEANUP.md

# Infrastructure Structure Cleanup Plan

## Current Situation

We have two parallel structures that need to be aligned:

### 1. External Services (Production/Remote)
Located in `infra/compose/` - These are deployed individually on the remote server:
- **Traefik** - `infra/compose/traefik/`
- **Authentik** - `infra/compose/authentik/`
- **Gitea** - `infra/compose/gitea/`
- **Nextcloud** - `infra/compose/nextcloud/`
- **Portainer** - `infra/compose/portainer/`

### 2. Application Infrastructure (Multi-Environment)
Located in `infra/base/` and `infra/environments/`:
- Infrastructure services (Vault, MinIO, DBs, etc.)
- Application services (14 microservices)
- Monitoring stack (Prometheus, Grafana, Loki)

### 3. Configuration Duplication
- `infra/compose/traefik/config/` - Production Traefik config
- `infra/configs/traefik/` - Application Traefik config (copied)
- Similar duplication for other services

---

## Cleanup Strategy

### Phase 1: Consolidate Configurations

#### Traefik
- **Keep**: `infra/compose/traefik/` as the source of truth for production
- **Symlink**: `infra/configs/traefik/` → `../compose/traefik/`
- **Reason**: External service configs should live with their compose files

#### Authentik
- **Keep**: `infra/compose/authentik/` for production
- **Keep**: `infra/configs/authentik/` for application-specific bootstrap
- **Reason**: Different purposes - one for service, one for app integration

#### Grafana/Prometheus/Loki
- **Keep**: `infra/configs/grafana/`, `infra/configs/prometheus/`, `infra/configs/loki/`
- **Reason**: These are application-specific, not external services

### Phase 2: Update References

#### Makefile
- Update paths to reference correct locations
- Add targets for external service deployment
- Separate local dev from production deployment

#### Scripts
- Update `scripts/deploy.sh` to handle external services
- Create `scripts/deploy-external.sh` for production external services
- Update `infra/scripts/deploy.sh` for application infrastructure

### Phase 3: Documentation

- Clear separation between:
  - External services (production only)
  - Application infrastructure (multi-environment)
  - Development environment (local only)

---

## Proposed Final Structure

```
ai-tax-agent/
├── infra/
│   ├── compose/                    # External services (production)
│   │   ├── traefik/
│   │   │   ├── compose.yaml       # Traefik service definition
│   │   │   ├── config/            # Traefik configuration
│   │   │   ├── certs/             # SSL certificates
│   │   │   └── .provider.env      # GoDaddy API credentials
│   │   ├── authentik/
│   │   │   ├── compose.yaml
│   │   │   ├── .env
│   │   │   ├── media/
│   │   │   └── custom-templates/
│   │   ├── gitea/
│   │   │   ├── compose.yaml
│   │   │   └── .env
│   │   ├── nextcloud/
│   │   │   └── compose.yaml
│   │   ├── portainer/
│   │   │   └── docker-compose.yaml
│   │   ├── docker-compose.local.yml    # Local dev (all-in-one)
│   │   └── docker-compose.backend.yml  # Backend services
│   │
│   ├── base/                       # Application infrastructure (multi-env)
│   │   ├── infrastructure.yaml    # Core infra services
│   │   ├── services.yaml          # Application microservices
│   │   └── monitoring.yaml        # Monitoring stack
│   │
│   ├── environments/               # Environment-specific configs
│   │   ├── local/
│   │   │   ├── .env.example
│   │   │   └── .env
│   │   ├── development/
│   │   │   ├── .env.example
│   │   │   └── .env
│   │   └── production/
│   │       ├── .env.example
│   │       └── .env
│   │
│   ├── configs/                    # Application service configs
│   │   ├── authentik/             # App-specific Authentik bootstrap
│   │   ├── grafana/               # Grafana dashboards
│   │   ├── prometheus/            # Prometheus scrape configs
│   │   ├── loki/                  # Loki config
│   │   └── vault/                 # Vault config
│   │
│   └── scripts/                    # Infrastructure deployment scripts
│       ├── deploy.sh              # Deploy application infrastructure
│       ├── setup-networks.sh      # Create Docker networks
│       └── reorganize-structure.sh
│
├── scripts/                        # Project-wide scripts
│   ├── deploy-external.sh         # Deploy external services (production)
│   ├── build-and-push-images.sh   # Build and push Docker images
│   ├── generate-secrets.sh        # Generate secrets
│   └── ...
│
└── Makefile                        # Project commands
```

---

## Deployment Workflows

### Local Development
```bash
# Use all-in-one compose file
make bootstrap
make run
# OR
cd infra/compose
docker compose -f docker-compose.local.yml up -d
```

### Production - External Services
```bash
# Deploy individually on remote server
cd /opt/ai-tax-agent/infra/compose/traefik
docker compose up -d

cd /opt/ai-tax-agent/infra/compose/authentik
docker compose up -d

cd /opt/ai-tax-agent/infra/compose/gitea
docker compose up -d
```

### Production - Application Infrastructure
```bash
# Deploy application infrastructure
./infra/scripts/deploy.sh production infrastructure
./infra/scripts/deploy.sh production monitoring
./infra/scripts/deploy.sh production services
```

---

## Migration Steps

### Step 1: Align Traefik Configs

```bash
# Remove duplicate configs
rm -rf infra/configs/traefik/config/traefik-dynamic.yml

# Keep only app-specific middleware
# Move production configs to compose/traefik/config/
```

### Step 2: Update Makefile

- Add targets for external service deployment
- Update paths to reference correct locations
- Separate local dev from production

### Step 3: Update Scripts

- Create `scripts/deploy-external.sh` for production
- Update `infra/scripts/deploy.sh` for application infra
- Update all path references

### Step 4: Documentation

- Update README files
- Create deployment guides for each environment
- Document external vs application services

---

## Key Decisions

### 1. External Services Location
**Decision**: Keep in `infra/compose/` with individual folders
**Reason**: These are production-only, deployed separately, have their own configs

### 2. Application Infrastructure Location
**Decision**: Keep in `infra/base/` with environment-specific `.env` files
**Reason**: Multi-environment support, shared compose files

### 3. Configuration Management
**Decision**: 
- External service configs live with their compose files
- Application configs live in `infra/configs/`
**Reason**: Clear separation of concerns

### 4. Makefile Targets
**Decision**: 
- `make run` - Local development (all-in-one)
- `make deploy-external` - Production external services
- `make deploy-infra` - Application infrastructure
**Reason**: Clear separation of deployment targets

---

## Benefits

✅ **Clear Separation** - External vs application services
✅ **No Duplication** - Single source of truth for configs
✅ **Multi-Environment** - Easy to deploy to local/dev/prod
✅ **Maintainable** - Logical organization
✅ **Scalable** - Easy to add new services
✅ **Production-Ready** - Matches actual deployment

---

## Next Steps

1. Run cleanup script to align configurations
2. Update Makefile with new targets
3. Update deployment scripts
4. Test local deployment
5. Test production deployment
6. Update documentation