completed local setup with compose

infra/compose/README.md

@@ -1,133 +1,23 @@
-# External Services
+# Compose Stacks
 
-This directory contains Docker Compose configurations for external services that run on the production server.
+This folder is for the self-contained local stack (self-signed TLS) and Traefik assets. Remote environments use the shared compose files in `infra/base` together with `infra/scripts/deploy.sh`.
 
-## Services
+## Local development (self-signed TLS)
+- Copy envs: `cp infra/compose/env.example infra/compose/.env` then set passwords/secrets and the dev domain (defaults to `local.lan`).
+- Host aliases: add the domain to `/etc/hosts` (e.g. `127.0.0.1 auth.local.lan api.local.lan grafana.local.lan vault.local.lan minio.local.lan`).
+- Networks: `./infra/scripts/setup-networks.sh` (creates `apa-frontend` and `apa-backend` used everywhere).
+- Run: `cd infra/compose && docker compose --env-file .env -f docker-compose.local.yml up -d`.
+- Stop: `docker compose --env-file .env -f docker-compose.local.yml down`.
+- TLS: Traefik mounts `infra/compose/traefik/certs/local.{crt,key}`. Regenerate if needed with `openssl req -x509 -newkey rsa:2048 -nodes -keyout infra/compose/traefik/certs/local.key -out infra/compose/traefik/certs/local.crt -days 365 -subj "/CN=*.local.lan"`.
 
-### Traefik
-- **Location**: `traefik/`
-- **Purpose**: Reverse proxy and load balancer for all services
-- **Deploy**: `cd traefik && docker compose up -d`
-- **Access**: https://traefik.harkon.co.uk
+## Cloud / remote (Let’s Encrypt)
+- Config lives in `infra/base` with env files in `infra/environments/{development,production}/.env`.
+- Create the same docker networks on the host (`./infra/scripts/setup-networks.sh`) so Traefik and services share `apa-frontend` / `apa-backend`.
+- Deploy on the server: `./infra/scripts/deploy.sh <environment> all` (or `infrastructure`, `monitoring`, `services`).
+- Certificates: Traefik uses DNS-01 via GoDaddy from the provider env in `infra/base/traefik/config` (make sure `DOMAIN`, ACME email, and provider creds are set in the env file).
 
-### Authentik
-- **Location**: `authentik/`
-- **Purpose**: SSO and authentication provider
-- **Deploy**: `cd authentik && docker compose up -d`
-- **Access**: https://authentik.harkon.co.uk
-
-### Gitea
-- **Location**: `gitea/`
-- **Purpose**: Git repository hosting and container registry
-- **Deploy**: `cd gitea && docker compose up -d`
-- **Access**: https://gitea.harkon.co.uk
-
-### Nextcloud
-- **Location**: `nextcloud/`
-- **Purpose**: File storage and collaboration
-- **Deploy**: `cd nextcloud && docker compose up -d`
-- **Access**: https://nextcloud.harkon.co.uk
-
-### Portainer
-- **Location**: `portainer/`
-- **Purpose**: Docker management UI
-- **Deploy**: `cd portainer && docker compose up -d`
-- **Access**: https://portainer.harkon.co.uk
-
-## Deployment
-
-### Production (Remote Server)
-
-```bash
-# SSH to server
-ssh deploy@141.136.35.199
-
-# Navigate to service directory
-cd /opt/ai-tax-agent/infra/compose/<service>
-
-# Deploy service
-docker compose up -d
-
-# Check logs
-docker compose logs -f
-
-# Check status
-docker compose ps
-```
-
-### Local Development
-
-For local development, use the all-in-one compose file:
-
-```bash
-cd infra/compose
-docker compose -f docker-compose.local.yml up -d
-```
-
-## Configuration
-
-Each service has its own `.env` file for environment-specific configuration:
-
-- `traefik/.provider.env` - GoDaddy API credentials
-- `authentik/.env` - Authentik secrets
-- `gitea/.env` - Gitea database credentials
-
-## Networks
-
-All services use shared Docker networks:
-
-- `frontend` - Public-facing services
-- `backend` - Internal services
-
-Create networks before deploying:
-
-```bash
-docker network create frontend
-docker network create backend
-```
-
-## Maintenance
-
-### Update Service
-
-```bash
-cd /opt/ai-tax-agent/infra/compose/<service>
-docker compose pull
-docker compose up -d
-```
-
-### Restart Service
-
-```bash
-cd /opt/ai-tax-agent/infra/compose/<service>
-docker compose restart
-```
-
-### View Logs
-
-```bash
-cd /opt/ai-tax-agent/infra/compose/<service>
-docker compose logs -f
-```
-
-### Backup Data
-
-```bash
-# Backup volumes
-docker run --rm -v <service>_data:/data -v $(pwd):/backup alpine tar czf /backup/<service>-backup.tar.gz /data
-```
-
-## Integration with Application
-
-These external services are used by the application infrastructure:
-
-- **Traefik** - Routes traffic to application services
-- **Authentik** - Provides SSO for application UIs
-- **Gitea** - Hosts Docker images for application services
-
-The application infrastructure is deployed separately using:
-
-```bash
-./infra/scripts/deploy.sh production infrastructure
-./infra/scripts/deploy.sh production services
-```
+## Files of note
+- `docker-compose.local.yml` – full local stack.
+- `traefik/traefik.local.yml` and `traefik/traefik-dynamic.local.yml` – static/dynamic Traefik config for local.
+- `traefik/certs/` – self-signed certs used by the local proxy.
+- `env.example` – defaults for local `.env`.

infra/compose/compose.override.yaml

@@ -0,0 +1,156 @@
+# FILE: infra/compose/compose.override.yaml
+# Local development overrides
+# Automatically loaded by docker compose when compose.yaml is present
+
+services:
+  # --- Infrastructure Overrides ---
+
+  apa-traefik:
+    volumes:
+      - ./traefik/traefik.local.yml:/etc/traefik/traefik.yml:ro
+      - ./traefik/traefik-dynamic.local.yml:/etc/traefik/traefik-dynamic.yml:ro
+      - ./traefik/certs/:/var/traefik/certs/:ro
+    ports:
+      - "8080:8080" # Dashboard (admin entrypoint, insecure mode only for local)
+
+  apa-authentik-server:
+    environment:
+      AUTHENTIK_ERROR_REPORTING__ENABLED: "false"
+      DOMAIN: ${DOMAIN:-local.lan}
+      GRAFANA_OAUTH_CLIENT_ID: ${GRAFANA_OAUTH_CLIENT_ID}
+      GRAFANA_OAUTH_CLIENT_SECRET: ${GRAFANA_OAUTH_CLIENT_SECRET}
+      AUTHENTIK_MINIO_CLIENT_SECRET: ${AUTHENTIK_MINIO_CLIENT_SECRET}
+      AUTHENTIK_VAULT_CLIENT_SECRET: ${AUTHENTIK_VAULT_CLIENT_SECRET}
+      AUTHENTIK_OUTPOST_TOKEN: ${AUTHENTIK_OUTPOST_TOKEN}
+    volumes:
+      - ../authentik/bootstrap.yaml:/blueprints/ai-tax-agent-bootstrap.yaml:ro
+
+  apa-authentik-worker:
+    environment:
+      DOMAIN: ${DOMAIN:-local.lan}
+      GRAFANA_OAUTH_CLIENT_ID: ${GRAFANA_OAUTH_CLIENT_ID}
+      GRAFANA_OAUTH_CLIENT_SECRET: ${GRAFANA_OAUTH_CLIENT_SECRET}
+      AUTHENTIK_MINIO_CLIENT_SECRET: ${AUTHENTIK_MINIO_CLIENT_SECRET}
+      AUTHENTIK_VAULT_CLIENT_SECRET: ${AUTHENTIK_VAULT_CLIENT_SECRET}
+      AUTHENTIK_OUTPOST_TOKEN: ${AUTHENTIK_OUTPOST_TOKEN}
+    volumes:
+      - ../authentik/bootstrap.yaml:/blueprints/ai-tax-agent-bootstrap.yaml:ro
+
+  apa-vault:
+    volumes:
+      - ./traefik/certs/:/certs:ro
+
+  # --- Service Build Overrides ---
+  # Pointing to local source code for building
+
+  apa-svc-ingestion:
+    build:
+      context: ../../
+      dockerfile: apps/svc_ingestion/Dockerfile
+    image: ai-tax-agent/svc-ingestion:local
+    pull_policy: never
+
+  apa-svc-extract:
+    build:
+      context: ../../
+      dockerfile: apps/svc_extract/Dockerfile
+    image: ai-tax-agent/svc-extract:local
+    pull_policy: never
+
+  apa-svc-kg:
+    build:
+      context: ../../
+      dockerfile: apps/svc_kg/Dockerfile
+    image: ai-tax-agent/svc-kg:local
+    pull_policy: never
+
+  apa-svc-rag-retriever:
+    build:
+      context: ../../
+      dockerfile: apps/svc_rag_retriever/Dockerfile
+    image: ai-tax-agent/svc-rag-retriever:local
+    pull_policy: never
+
+  apa-svc-forms:
+    build:
+      context: ../../
+      dockerfile: apps/svc_forms/Dockerfile
+    image: ai-tax-agent/svc-forms:local
+    pull_policy: never
+
+  apa-svc-hmrc:
+    build:
+      context: ../../
+      dockerfile: apps/svc_hmrc/Dockerfile
+    image: ai-tax-agent/svc-hmrc:local
+    pull_policy: never
+
+  apa-svc-ocr:
+    build:
+      context: ../../
+      dockerfile: apps/svc_ocr/Dockerfile
+    image: ai-tax-agent/svc-ocr:local
+    pull_policy: never
+    restart: on-failure
+
+  apa-svc-rag-indexer:
+    build:
+      context: ../../
+      dockerfile: apps/svc_rag_indexer/Dockerfile
+    image: ai-tax-agent/svc-rag-indexer:local
+    pull_policy: never
+
+  apa-svc-reason:
+    build:
+      context: ../../
+      dockerfile: apps/svc_reason/Dockerfile
+    image: ai-tax-agent/svc-reason:local
+    pull_policy: never
+
+  apa-svc-rpa:
+    build:
+      context: ../../
+      dockerfile: apps/svc_rpa/Dockerfile
+    image: ai-tax-agent/svc-rpa:local
+    pull_policy: never
+
+  apa-svc-normalize-map:
+    build:
+      context: ../../
+      dockerfile: apps/svc_normalize_map/Dockerfile
+    image: ai-tax-agent/svc-normalize-map:local
+    pull_policy: never
+
+  apa-svc-coverage:
+    build:
+      context: ../../
+      dockerfile: apps/svc_coverage/Dockerfile
+    image: ai-tax-agent/svc-coverage:local
+    pull_policy: never
+
+  apa-svc-firm-connectors:
+    build:
+      context: ../../
+      dockerfile: apps/svc_firm_connectors/Dockerfile
+    image: ai-tax-agent/svc-firm-connectors:local
+    pull_policy: never
+
+  apa-ui-review:
+    # UI might not have a Dockerfile in root/ui-review/Dockerfile based on previous file view
+    # Assuming standard build context if it exists, otherwise comment out build
+    # build:
+    #   context: ../../ui-review
+    #   dockerfile: Dockerfile
+    image: alpine:latest
+    profiles: ["disabled"]
+    environment:
+      - NEXTAUTH_URL=https://app.local.lan
+      - API_BASE_URL=https://api.local.lan
+
+  apa-minio:
+    volumes:
+      - ./traefik/certs/local.crt:/root/.minio/certs/CAs/local.crt:ro
+
+  # --- Local Development Specific Services ---
+  # Services that only exist in local dev (e.g. mailhog if used, or specific tools)
+  # None identified from docker-compose.local.yml that aren't in base

infra/compose/compose.yaml

@@ -0,0 +1,14 @@
+# FILE: infra/compose/compose.yaml
+# Main entry point for Docker Compose
+# Includes base configurations from infra/base/
+
+include:
+  - ../base/infrastructure.yaml
+  - ../base/services.yaml
+  # Monitoring stack is optional for local dev but included for completeness
+  # Can be disabled via profiles if needed, but keeping simple for now
+  - ../base/monitoring.yaml
+
+# Define project name to match existing convention if needed,
+# though 'compose' directory name usually defaults to 'compose'
+name: ai-tax-agent

infra/compose/env.example

@@ -1,7 +1,7 @@
 # FILE: infra/compose/env.example
 
 # Domain Configuration
-DOMAIN=local
+DOMAIN=local.lan
 EMAIL=admin@local.lan
 
 # Database Passwords
@@ -26,6 +26,7 @@ AUTHENTIK_SECRET_KEY=changeme
 AUTHENTIK_OUTPOST_TOKEN=changeme
 AUTHENTIK_BOOTSTRAP_EMAIL=admin@local.lan
 AUTHENTIK_BOOTSTRAP_PASSWORD=admin123
+# AUTHENTIK_BOOTSTRAP_TOKEN: This value will be automatically updated after the initial setup.
 AUTHENTIK_BOOTSTRAP_TOKEN=
 
 # Monitoring
@@ -80,7 +81,7 @@ PII_LOG_RETENTION_DAYS=30
 
 # Backup & DR
 BACKUP_ENABLED=true
-BACKUP_SCHEDULE=0 2 * * *
+BACKUP_SCHEDULE="0 2 * * *"
 BACKUP_RETENTION_DAYS=30
 
 # Performance Tuning

infra/compose/traefik/traefik-dynamic.local.yml

@@ -0,0 +1,89 @@
+http:
+  middlewares:
+    authentik-forwardauth:
+      forwardAuth:
+        address: "http://apa-authentik-outpost:9000/outpost.goauthentik.io/auth/traefik"
+        trustForwardHeader: true
+        authResponseHeaders:
+          - X-authentik-username
+          - X-authentik-groups
+          - X-authentik-email
+          - X-authentik-name
+          - X-authentik-uid
+          - X-authentik-jwt
+          - X-authentik-meta-jwks
+          - X-authentik-meta-outpost
+          - X-authentik-meta-provider
+          - X-authentik-meta-app
+          - X-authentik-meta-version
+
+    # Large upload middleware for Gitea registry
+    gitea-large-upload:
+      buffering:
+        maxRequestBodyBytes: 5368709120 # 5GB
+        memRequestBodyBytes: 104857600 # 100MB
+        maxResponseBodyBytes: 5368709120 # 5GB
+        memResponseBodyBytes: 104857600 # 100MB
+        retryExpression: "IsNetworkError() && Attempts() < 3"
+
+    # Rate limiting for public APIs
+    rate-limit:
+      rateLimit:
+        average: 100
+        burst: 50
+        period: 1s
+
+    # Security headers
+    security-headers:
+      headers:
+        frameDeny: true
+        sslRedirect: true
+        browserXssFilter: true
+        contentTypeNosniff: true
+        stsIncludeSubdomains: true
+        stsPreload: true
+        stsSeconds: 31536000
+
+    # CORS headers
+    api-cors:
+      headers:
+        accessControlAllowMethods:
+          - GET
+          - POST
+          - PUT
+          - DELETE
+          - OPTIONS
+        accessControlAllowOriginList:
+          - "https://app.harkon.co.uk"
+        accessControlAllowHeaders:
+          - "Content-Type"
+          - "Authorization"
+        accessControlMaxAge: 100
+        addVaryHeader: true
+
+    # Strip API prefixes
+    strip-api-prefixes:
+      stripPrefix:
+        prefixes:
+          - "/rag-indexer"
+          - "/firm-connectors"
+          - "/normalize-map"
+          - "/ingestion"
+          - "/extract"
+          - "/forms"
+          - "/hmrc"
+          - "/ocr"
+          - "/reason"
+          - "/rpa"
+          - "/coverage"
+          - "/kg"
+          - "/rag"
+
+tls:
+  certificates:
+    - certFile: /var/traefik/certs/local.crt
+      keyFile: /var/traefik/certs/local.key
+  options:
+    default:
+      minVersion: VersionTLS12
+      sniStrict: false

infra/compose/traefik/traefik.local.yml

@@ -0,0 +1,35 @@
+# Traefik static configuration for local development (self-signed TLS)
+entryPoints:
+  web:
+    address: ":80"
+    http:
+      redirections:
+        entryPoint:
+          to: websecure
+          scheme: https
+  websecure:
+    address: ":443"
+    http:
+      tls:
+        options: default
+
+providers:
+  docker:
+    endpoint: "unix:///var/run/docker.sock"
+    exposedByDefault: false
+    network: "apa-frontend"
+  file:
+    filename: "/etc/traefik/traefik-dynamic.yml"
+    watch: true
+
+api:
+  dashboard: true
+  insecure: true
+
+serversTransport:
+  insecureSkipVerify: true
+
+log:
+  level: INFO
+
+accessLog: {}