harkon/ai-tax-agent
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

completed local setup with compose
Some checks failed
CI/CD Pipeline / Generate SBOM (push) Has been cancelled
Details
CI/CD Pipeline / Deploy to Staging (push) Has been cancelled
Details
CI/CD Pipeline / Deploy to Production (push) Has been cancelled
Details
CI/CD Pipeline / Code Quality & Linting (push) Has been cancelled
Details
CI/CD Pipeline / Policy Validation (push) Has been cancelled
Details
CI/CD Pipeline / Test Suite (push) Has been cancelled
Details
CI/CD Pipeline / Build Docker Images (svc-coverage) (push) Has been cancelled
Details
CI/CD Pipeline / Build Docker Images (svc-extract) (push) Has been cancelled
Details
CI/CD Pipeline / Build Docker Images (svc-firm-connectors) (push) Has been cancelled
Details
CI/CD Pipeline / Build Docker Images (svc-forms) (push) Has been cancelled
Details
CI/CD Pipeline / Build Docker Images (svc-hmrc) (push) Has been cancelled
Details
CI/CD Pipeline / Build Docker Images (svc-ingestion) (push) Has been cancelled
Details
CI/CD Pipeline / Build Docker Images (svc-kg) (push) Has been cancelled
Details
CI/CD Pipeline / Build Docker Images (svc-normalize-map) (push) Has been cancelled
Details
CI/CD Pipeline / Build Docker Images (svc-ocr) (push) Has been cancelled
Details
CI/CD Pipeline / Build Docker Images (svc-rag-indexer) (push) Has been cancelled
Details
CI/CD Pipeline / Build Docker Images (svc-rag-retriever) (push) Has been cancelled
Details
CI/CD Pipeline / Build Docker Images (svc-reason) (push) Has been cancelled
Details
CI/CD Pipeline / Build Docker Images (svc-rpa) (push) Has been cancelled
Details
CI/CD Pipeline / Build Docker Images (ui-review) (push) Has been cancelled
Details
CI/CD Pipeline / Security Scanning (svc-coverage) (push) Has been cancelled
Details
CI/CD Pipeline / Security Scanning (svc-extract) (push) Has been cancelled
Details
CI/CD Pipeline / Security Scanning (svc-kg) (push) Has been cancelled
Details
CI/CD Pipeline / Security Scanning (svc-rag-retriever) (push) Has been cancelled
Details
CI/CD Pipeline / Security Scanning (ui-review) (push) Has been cancelled
Details
CI/CD Pipeline / Notifications (push) Has been cancelled
Details

Browse Source
This commit is contained in:
harkon
2025-11-26 13:17:17 +00:00
parent 8fe5e62fee
commit fdba81809f
87 changed files with 5610 additions and 3376 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
docs/ARCHITECT.md
View File

@@ -42,8 +42,8 @@ Deliver a complete, implementable solution—ontology, extraction pipeline, RAG+
2. **svc-rpa** — Playwright RPA for firm/client portals; Prefect-scheduled; emits `doc.ingested`.
3. **svc-ocr** — Tesseract (local) or Textract (scale); de-skew/rotation/layout; emits `doc.ocr_ready`.
4. **svc-extract** — LLM + rules + table detectors → **schema-constrained JSON** (kv + tables + bbox/page); emits `doc.extracted`.
5. **svc-normalize-map** — normalize currency/dates; entity resolution; assign tax year; map to KG nodes/edges with **Evidence** anchors; emits `kg.upserted`.
6. **svc-kg** — Neo4j DDL + **SHACL** validation; **bitemporal** writes `{valid_from, valid_to, asserted_at}`; RDF export.
5. **svc-normalize-map** — Consumes `doc.extracted` events; normalizes extracted data (currencies, dates); performs entity resolution; assigns tax year; maps to KG nodes/edges with **Evidence** anchors; emits `kg.upsert.ready` events.
6. **svc-kg** — Consumes `kg.upsert.ready` events; performs Neo4j DDL operations + **SHACL** validation; **bitemporal** writes `{valid_from, valid_to, asserted_at}`; RDF export; emits `kg.upserted` events.
7. **svc-rag-indexer** — chunk/de-identify/embed; upsert **Qdrant** collections (firm knowledge, legislation, best practices, glossary).
8. **svc-rag-retriever** — **hybrid retrieval** (dense + sparse) + rerank + **KG-fusion**; returns chunks + citations + KG join hints.
9. **svc-reason** — deterministic calculators (employment, self-employment, property, dividends/interest, allowances, NIC, HICBC, student loans); Cypher materializers; explanations.
@@ -51,11 +51,12 @@ Deliver a complete, implementable solution—ontology, extraction pipeline, RAG+
11. **svc-hmrc** — submit stub|sandbox|live; rate-limit & retries; submission audit.
12. **svc-firm-connectors** — read-only connectors to Firm Databases; sync to **Secure Client Data Store** with lineage.
13. **ui-review** — Next.js reviewer portal (SSO via Traefik+Authentik); reviewers accept/override extractions.
14. **svc-coverage** — Evaluates document coverage against policies, identifies gaps, and generates clarifying questions.
## Orchestration & Messaging
- **Prefect 2.x** for local orchestration; **Temporal** for production scale (sagas, retries, idempotency).
- Events: Kafka (or SQS/SNS) — `doc.ingested`, `doc.ocr_ready`, `doc.extracted`, `kg.upserted`, `rag.indexed`, `calc.schedule_ready`, `form.filled`, `hmrc.submitted`, `review.requested`, `review.completed`, `firm.sync.completed`.
- Events: Kafka (or SQS/SNS) — `doc.ingested`, `doc.ocr_ready`, `doc.extracted`, `kg.upsert.ready`, `kg.upserted`, `rag.indexed`, `calc.schedule_ready`, `form.filled`, `hmrc.submitted`, `review.requested`, `review.completed`, `firm.sync.completed`.
## Concrete Stack (pin/assume unless replaced)
@@ -103,7 +104,7 @@ repo/
svc-ingestion/ svc-rpa/ svc-ocr/ svc-extract/
svc-normalize-map/ svc-kg/ svc-rag-indexer/ svc-rag-retriever/
svc-reason/ svc-forms/ svc-hmrc/ svc-firm-connectors/
ui-review/
svc-coverage/ ui-review/
kg/
ONTOLOGY.md
schemas/{nodes_and_edges.schema.json, context.jsonld, shapes.ttl}

48
docs/DEVELOPMENT.md
View File

@@ -7,6 +7,7 @@ This guide explains how to run services locally for development.
### Prerequisites
1. **Infrastructure Services Running**: Ensure Docker Compose infrastructure is running:
```bash
make deploy-infra
```
@@ -39,17 +40,17 @@ DISABLE_AUTH=true cd apps/svc_ingestion && uvicorn main:app --reload --host 0.0.
### Environment Variables for Development
| Variable | Description | Default | Dev Value |
|----------|-------------|---------|-----------|
| `DISABLE_AUTH` | Disable authentication middleware | `false` | `true` |
| `DEV_MODE` | Enable development mode | `false` | `true` |
| `VAULT_ADDR` | Vault server address | `http://vault:8200` | - |
| `VAULT_TOKEN` | Vault token (dev only) | - | `root` |
| `MINIO_ENDPOINT` | MinIO endpoint | `minio:9000` | `minio:9092` |
| `POSTGRES_URL` | PostgreSQL connection URL | - | `postgresql://postgres:postgres@localhost:5432/tax_system` |
| `REDIS_URL` | Redis connection URL | `redis://redis:6379` | `redis://localhost:6379` |
| `NEO4J_URI` | Neo4j connection URI | `bolt://neo4j:7687` | `bolt://localhost:7687` |
| `NATS_SERVERS` | NATS server URLs | `nats://nats:4222` | `nats://localhost:4222` |
| Variable | Description | Default | Dev Value |
| ---------------- | --------------------------------- | -------------------- | ---------------------------------------------------------- |
| `DISABLE_AUTH` | Disable authentication middleware | `false` | `true` |
| `DEV_MODE` | Enable development mode | `false` | `true` |
| `VAULT_ADDR` | Vault server address | `http://vault:8200` | - |
| `VAULT_TOKEN` | Vault token (dev only) | - | `root` |
| `MINIO_ENDPOINT` | MinIO endpoint | `minio:9000` | `minio:9092` |
| `POSTGRES_URL` | PostgreSQL connection URL | - | `postgresql://postgres:postgres@localhost:5432/tax_system` |
| `REDIS_URL` | Redis connection URL | `redis://redis:6379` | `redis://localhost:6379` |
| `NEO4J_URI` | Neo4j connection URI | `bolt://neo4j:7687` | `bolt://localhost:7687` |
| `NATS_SERVERS` | NATS server URLs | `nats://nats:4222` | `nats://localhost:4222` |
### Testing with Postman
@@ -68,6 +69,7 @@ Authorization: Bearer dev-token-12345
#### With Development Mode (DISABLE_AUTH=true)
No authentication headers required! The middleware automatically sets:
- User: `dev-user`
- Email: `dev@example.com`
- Roles: `["developers"]`
@@ -123,17 +125,20 @@ Create a Postman environment called "AI Tax Agent - Dev":
### Example Requests
#### Health Check
```bash
curl http://localhost:8000/healthz
```
#### Upload Document (Development Mode)
```bash
curl -X POST http://localhost:8000/upload \
-F "file=@/path/to/document.pdf"
```
#### Upload Document (Production Mode)
```bash
curl -X POST http://localhost:8000/upload \
-H "X-Authenticated-User: dev-user" \
@@ -145,41 +150,47 @@ curl -X POST http://localhost:8000/upload \
### Debugging
#### Check Service Logs
```bash
# Local development
# Logs appear in terminal where service is running
# Docker Compose
docker-compose -f infra/compose/docker-compose.local.yml logs -f svc-ingestion
docker compose logs -f svc-ingestion
```
#### Verify Infrastructure Services
```bash
# Check all services status
docker-compose -f infra/compose/docker-compose.local.yml ps
docker compose ps
# Check specific service health
docker-compose -f infra/compose/docker-compose.local.yml exec postgres pg_isready
docker-compose -f infra/compose/docker-compose.local.yml exec redis redis-cli ping
docker-compose -f infra/compose/docker-compose.local.yml exec minio mc --version
docker compose exec postgres pg_isready
docker compose exec redis redis-cli ping
docker compose exec minio mc --version
```
#### Common Issues
**Issue**: `401 Unauthorized` errors
- **Solution**: Set `DISABLE_AUTH=true` when running locally, or add authentication headers
**Issue**: `Connection refused` to database/redis/etc
- **Solution**: Ensure infrastructure services are running with `make deploy-infra`
- **Solution**: Use `localhost` instead of service names when running locally
**Issue**: `Module not found` errors
- **Solution**: Ensure you're running from project root and virtual environment is activated
- **Solution**: Install dependencies: `pip install -r apps/SERVICE_NAME/requirements.txt -r libs/requirements.txt`
### Hot Reload
When running with `uvicorn --reload`, the service automatically reloads when you save changes to:
- Python files in `apps/SERVICE_NAME/`
- Python files in `libs/`
@@ -191,7 +202,7 @@ To run multiple services simultaneously for integration testing:
# Terminal 1: Run ingestion service
DISABLE_AUTH=true make dev-service SERVICE=svc_ingestion
# Terminal 2: Run extraction service
# Terminal 2: Run extraction service
DISABLE_AUTH=true make dev-service SERVICE=svc_extract
# Terminal 3: Run knowledge graph service
@@ -210,7 +221,7 @@ DISABLE_AUTH=true cd apps/svc_extract && uvicorn main:app --reload --host 0.0.0.
All Docker Compose services are configured with health checks and should show as `healthy`:
```bash
$ docker-compose -f infra/compose/docker-compose.local.yml ps
$ docker compose ps
NAME STATUS
authentik-db Up 35 hours (healthy)
authentik-outpost Up 35 hours (healthy)
@@ -237,4 +248,3 @@ vault Up 35 hours
- See [README.md](README.md) for architecture overview
- See [TESTING.md](TESTING.md) for testing guidelines (if available)
- See service-specific README files in `apps/SERVICE_NAME/` directories

72
docs/ENVIRONMENT_COMPARISON.md
View File

@@ -6,22 +6,23 @@ This document compares the local development environment with the production env
## Quick Reference
| Aspect | Local Development | Production |
|--------|------------------|------------|
| **Domain** | `*.local.lan` | `*.harkon.co.uk` |
| **SSL** | Self-signed certificates | Let's Encrypt (GoDaddy DNS) |
| **Networks** | `ai-tax-agent-frontend`<br/>`ai-tax-agent-backend` | `frontend`<br/>`backend` |
| **Compose File** | `docker-compose.local.yml` | `infrastructure.yaml`<br/>`services.yaml`<br/>`monitoring.yaml` |
| **Location** | Local machine | `deploy@141.136.35.199:/opt/ai-tax-agent/` |
| **Traefik** | Isolated instance | Shared with company services |
| **Authentik** | Isolated instance | Shared with company services |
| **Data Persistence** | Local Docker volumes | Remote Docker volumes + backups |
| Aspect | Local Development | Production |
| -------------------- | -------------------------------------------------- | --------------------------------------------------------------- |
| **Domain** | `*.local.lan` | `*.harkon.co.uk` |
| **SSL** | Self-signed certificates | Let's Encrypt (GoDaddy DNS) |
| **Networks** | `ai-tax-agent-frontend`<br/>`ai-tax-agent-backend` | `frontend`<br/>`backend` |
| **Compose File** | `compose.yaml` | `infrastructure.yaml`<br/>`services.yaml`<br/>`monitoring.yaml` |
| **Location** | Local machine | `deploy@141.136.35.199:/opt/ai-tax-agent/` |
| **Traefik** | Isolated instance | Shared with company services |
| **Authentik** | Isolated instance | Shared with company services |
| **Data Persistence** | Local Docker volumes | Remote Docker volumes + backups |
## Detailed Comparison
### 1. Domain & URLs
#### Local Development
```
Frontend:
- Review UI: https://review.local.lan
@@ -42,6 +43,7 @@ Admin Interfaces:
```
#### Production
```
Frontend:
- Review UI: https://app.harkon.co.uk
@@ -69,6 +71,7 @@ Company Services (shared):
### 2. SSL/TLS Configuration
#### Local Development
- **Certificate Type**: Self-signed
- **Generation**: `scripts/generate-dev-certs.sh`
- **Location**: `infra/compose/certs/local.crt`, `infra/compose/certs/local.key`
@@ -76,6 +79,7 @@ Company Services (shared):
- **Renewal**: Manual (when expired)
#### Production
- **Certificate Type**: Let's Encrypt
- **Challenge**: DNS-01 (GoDaddy)
- **Location**: `/opt/compose/traefik/certs/godaddy-acme.json`
@@ -85,6 +89,7 @@ Company Services (shared):
### 3. Network Configuration
#### Local Development
```yaml
networks:
frontend:
@@ -96,12 +101,14 @@ networks:
```
**Creation**:
```bash
docker network create ai-tax-agent-frontend
docker network create ai-tax-agent-backend
```
#### Production
```yaml
networks:
frontend:
@@ -117,12 +124,14 @@ networks:
### 4. Service Isolation
#### Local Development
- **Traefik**: Dedicated instance for AI Tax Agent
- **Authentik**: Dedicated instance for AI Tax Agent
- **Isolation**: Complete - no shared services
- **Impact**: Changes don't affect other services
#### Production
- **Traefik**: Shared with company services
- **Authentik**: Shared with company services
- **Isolation**: Partial - infrastructure shared, application isolated
@@ -131,14 +140,16 @@ networks:
### 5. Authentication & Authorization
#### Local Development
- **Bootstrap Admin**: `admin@local.lan` / `admin123`
- **Groups**: Auto-created via bootstrap
- **OAuth Clients**: Auto-configured
- **Users**: Test users only
#### Production
- **Bootstrap Admin**: Real admin credentials
- **Groups**:
- **Groups**:
- `company` - Company services access
- `app-admin` - Full app access
- `app-user` - App user access
@@ -149,6 +160,7 @@ networks:
### 6. Data Persistence
#### Local Development
```bash
# Volume location
/var/lib/docker/volumes/
@@ -168,6 +180,7 @@ networks:
**Retention**: Until `make clean`
#### Production
```bash
# Volume location
/var/lib/docker/volumes/
@@ -188,6 +201,7 @@ networks:
### 7. Environment Variables
#### Local Development (`.env`)
```bash
DOMAIN=local.lan
EMAIL=admin@local.lan
@@ -200,6 +214,7 @@ DEVELOPMENT_MODE=true
```
#### Production (`.env.production`)
```bash
DOMAIN=harkon.co.uk
EMAIL=admin@harkon.co.uk
@@ -214,11 +229,13 @@ DEVELOPMENT_MODE=false
### 8. Resource Limits
#### Local Development
- **No limits**: Uses available resources
- **Suitable for**: Development and testing
- **Scaling**: Not configured
#### Production
```yaml
# Example resource limits
services:
@@ -226,22 +243,24 @@ services:
deploy:
resources:
limits:
cpus: '1.0'
cpus: "1.0"
memory: 1G
reservations:
cpus: '0.5'
cpus: "0.5"
memory: 512M
```
### 9. Logging & Monitoring
#### Local Development
- **Logs**: Docker logs (`docker compose logs`)
- **Retention**: Until container restart
- **Monitoring**: Optional (Grafana available but not required)
- **Alerts**: Disabled
#### Production
- **Logs**: Centralized in Loki
- **Retention**: 30 days
- **Monitoring**: Required (Prometheus + Grafana)
@@ -250,6 +269,7 @@ services:
### 10. Deployment Process
#### Local Development
```bash
# Start everything
make bootstrap
@@ -259,7 +279,7 @@ make up
./scripts/create-networks.sh
./scripts/generate-dev-certs.sh
cd infra/compose
docker compose -f docker-compose.local.yml up -d
docker compose up -d
# Stop everything
make down
@@ -269,6 +289,7 @@ make clean
```
#### Production
```bash
# Deploy infrastructure
cd /opt/ai-tax-agent
@@ -287,11 +308,13 @@ docker compose -f services.yaml up -d --no-deps svc-ingestion
### 11. Database Migrations
#### Local Development
- **Automatic**: Migrations run on startup
- **Rollback**: `make clean` and restart
- **Data Loss**: Acceptable
#### Production
- **Manual**: Migrations run explicitly
- **Rollback**: Requires backup restoration
- **Data Loss**: NOT acceptable
@@ -299,11 +322,13 @@ docker compose -f services.yaml up -d --no-deps svc-ingestion
### 12. Secrets Management
#### Local Development
- **Storage**: `.env` file (committed to git as example)
- **Vault**: Dev mode (unsealed automatically)
- **Security**: Low (development only)
#### Production
- **Storage**: `.env.production` (NOT committed to git)
- **Vault**: Production mode (manual unseal required)
- **Security**: High (encrypted, access controlled)
@@ -311,11 +336,13 @@ docker compose -f services.yaml up -d --no-deps svc-ingestion
### 13. CI/CD Integration
#### Local Development
- **CI/CD**: Not applicable
- **Testing**: Manual
- **Deployment**: Manual
#### Production
- **CI/CD**: Gitea Actions (planned)
- **Testing**: Automated (unit, integration, e2e)
- **Deployment**: Automated with approval gates
@@ -323,12 +350,14 @@ docker compose -f services.yaml up -d --no-deps svc-ingestion
### 14. Backup & Recovery
#### Local Development
- **Backup**: Not configured
- **Recovery**: Rebuild from scratch
- **RTO**: N/A
- **RPO**: N/A
#### Production
- **Backup**: Daily automated backups
- **Recovery**: Restore from backup
- **RTO**: 1 hour
@@ -337,11 +366,13 @@ docker compose -f services.yaml up -d --no-deps svc-ingestion
### 15. Cost Considerations
#### Local Development
- **Infrastructure**: Free (local machine)
- **Compute**: Uses local resources
- **Storage**: Uses local disk
#### Production
- **Infrastructure**: Server rental (~$50/month)
- **Compute**: Shared with company services
- **Storage**: Included in server
@@ -353,16 +384,19 @@ docker compose -f services.yaml up -d --no-deps svc-ingestion
### From Local to Production
1. **Build images locally**:
```bash
docker compose -f docker-compose.local.yml build
docker compose build
```
2. **Tag for production**:
```bash
docker tag svc-ingestion:latest gitea.harkon.co.uk/ai-tax-agent/svc-ingestion:v1.0.0
```
3. **Push to registry**:
```bash
docker push gitea.harkon.co.uk/ai-tax-agent/svc-ingestion:v1.0.0
```
@@ -378,23 +412,26 @@ docker compose -f services.yaml up -d --no-deps svc-ingestion
### From Production to Local (for debugging)
1. **Pull production image**:
```bash
docker pull gitea.harkon.co.uk/ai-tax-agent/svc-ingestion:v1.0.0
```
2. **Tag for local use**:
```bash
docker tag gitea.harkon.co.uk/ai-tax-agent/svc-ingestion:v1.0.0 svc-ingestion:latest
```
3. **Run locally**:
```bash
docker compose -f docker-compose.local.yml up -d svc-ingestion
docker compose up -d svc-ingestion
```
## Best Practices
### Local Development
1. ✅ Use `make` commands for consistency
2. ✅ Keep `.env` file updated from `env.example`
3. ✅ Run tests before committing
@@ -402,6 +439,7 @@ docker compose -f services.yaml up -d --no-deps svc-ingestion
5. ✅ Clean up regularly with `make clean`
### Production
1. ✅ Never commit `.env.production` to git
2. ✅ Always backup before making changes
3. ✅ Test in local environment first
@@ -413,12 +451,14 @@ docker compose -f services.yaml up -d --no-deps svc-ingestion
## Troubleshooting
### Local Development Issues
- **Port conflicts**: Check if ports 80, 443, 8080 are in use
- **Network errors**: Recreate networks with `make networks`
- **Certificate errors**: Regenerate with `./scripts/generate-dev-certs.sh`
- **Service won't start**: Check logs with `docker compose logs <service>`
### Production Issues
- **Service unreachable**: Check Traefik routing and DNS
- **Authentication fails**: Verify Authentik configuration
- **SSL errors**: Check certificate renewal in Traefik

66
docs/NATS_DOCKER_COMPOSE_SUMMARY.md
View File

@@ -8,9 +8,10 @@ Successfully integrated NATS.io message broker with JetStream support into the A
### 1. Added NATS Service to Docker Compose
**File**: `infra/compose/docker-compose.local.yml`
**File**: `infra/compose/compose.yaml`
#### NATS Service Configuration:
```yaml
nats:
image: nats:2.10-alpine
@@ -19,9 +20,9 @@ nats:
networks:
- backend
ports:
- "4222:4222" # NATS client connections
- "8222:8222" # HTTP monitoring
- "6222:6222" # Cluster routing (for future clustering)
- "4222:4222" # NATS client connections
- "8222:8222" # HTTP monitoring
- "6222:6222" # Cluster routing (for future clustering)
volumes:
- nats_data:/data
command: >
@@ -33,7 +34,15 @@ nats:
environment:
NATS_LOG_LEVEL: ${NATS_LOG_LEVEL:-info}
healthcheck:
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8222/healthz"]
test:
[
"CMD",
"wget",
"--no-verbose",
"--tries=1",
"--spider",
"http://localhost:8222/healthz",
]
interval: 30s
timeout: 10s
retries: 3
@@ -47,6 +56,7 @@ nats:
```
#### Key Features:
- **JetStream Enabled**: Persistent messaging with file-based storage
- **Monitoring**: HTTP monitoring interface on port 8222
- **Cluster Ready**: Port 6222 configured for future clustering
@@ -63,6 +73,7 @@ Added `nats_data:` volume to the volumes section for persistent storage.
Updated **13 application services** to include NATS configuration:
#### Services Updated:
1. `svc-ingestion`
2. `svc-extract`
3. `svc-kg`
@@ -78,6 +89,7 @@ Updated **13 application services** to include NATS configuration:
13. `svc-rpa`
#### Environment Variables Added to Each Service:
```yaml
environment:
# ... existing variables ...
@@ -95,6 +107,7 @@ depends_on:
**File**: `infra/compose/env.example`
Added NATS configuration variables:
```bash
# Event Bus Configuration
EVENT_BUS_TYPE=memory
@@ -119,18 +132,20 @@ cd infra/compose
cp env.example .env
# Start all services including NATS
docker-compose -f docker-compose.local.yml up -d
docker compose up -d
# Check NATS status
docker-compose -f docker-compose.local.yml logs nats
docker compose logs nats
```
### Using NATS in Applications
#### Option 1: Environment Variable Configuration
Set `EVENT_BUS_TYPE=nats` in your environment to use NATS instead of memory/kafka.
#### Option 2: Direct Configuration
```python
from libs.events import create_event_bus
@@ -177,17 +192,18 @@ nats --server=nats://localhost:4222 stream info TAX_AGENT_EVENTS
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `NATS_SERVERS` | `nats://nats:4222` | NATS server connection string |
| `NATS_STREAM_NAME` | `TAX_AGENT_EVENTS` | JetStream stream name |
| `NATS_CONSUMER_GROUP` | `tax-agent` | Consumer group name |
| `NATS_LOG_LEVEL` | `info` | NATS server log level |
| `EVENT_BUS_TYPE` | `memory` | Event bus type (memory/kafka/nats) |
| Variable | Default | Description |
| --------------------- | ------------------ | ---------------------------------- |
| `NATS_SERVERS` | `nats://nats:4222` | NATS server connection string |
| `NATS_STREAM_NAME` | `TAX_AGENT_EVENTS` | JetStream stream name |
| `NATS_CONSUMER_GROUP` | `tax-agent` | Consumer group name |
| `NATS_LOG_LEVEL` | `info` | NATS server log level |
| `EVENT_BUS_TYPE` | `memory` | Event bus type (memory/kafka/nats) |
### NATS Server Configuration
The NATS server is configured with:
- **JetStream**: Enabled for persistent messaging
- **File Storage**: 10GB maximum
- **Memory Storage**: 1GB maximum
@@ -219,26 +235,31 @@ The NATS server is configured with:
## Benefits
### 1. **High Performance**
- Very low latency messaging
- High throughput with minimal overhead
- Efficient binary protocol
### 2. **Operational Simplicity**
- Single binary deployment
- Minimal configuration required
- Built-in monitoring and health checks
### 3. **Reliability**
- JetStream provides persistence
- Automatic message acknowledgment
- Configurable retry policies
### 4. **Scalability**
- Ready for clustering (port 6222 configured)
- Horizontal scaling support
- Load balancing across consumers
### 5. **Integration**
- Seamless integration with existing services
- Traefik routing for web UI
- Authentik authentication for monitoring
@@ -246,27 +267,30 @@ The NATS server is configured with:
## Next Steps
1. **Test the Integration**:
```bash
# Start the stack
docker-compose -f docker-compose.local.yml up -d
docker compose up -d
# Check NATS is running
docker-compose -f docker-compose.local.yml ps nats
docker compose ps nats
# View NATS logs
docker-compose -f docker-compose.local.yml logs nats
docker compose logs nats
```
2. **Switch to NATS**:
```bash
# Update environment
echo "EVENT_BUS_TYPE=nats" >> .env
# Restart services
docker-compose -f docker-compose.local.yml restart
docker compose restart
```
3. **Monitor Usage**:
- Access monitoring at `https://nats.local`
- Use NATS CLI for detailed monitoring
- Check application logs for event processing

94
docs/QUICK_REFERENCE.md
View File

@@ -20,16 +20,16 @@ curl http://localhost:8000/healthz
```bash
# Start all services
cd infra/compose
docker-compose -f docker-compose.local.yml up -d
docker compose up -d
# Check status
docker-compose -f docker-compose.local.yml ps
docker compose ps
# View logs
docker-compose -f docker-compose.local.yml logs -f svc-ingestion
docker compose logs -f svc-ingestion
# Stop all services
docker-compose -f docker-compose.local.yml down
docker compose down
```
## 🔍 Checking Status
@@ -39,13 +39,13 @@ docker-compose -f docker-compose.local.yml down
```bash
# Check all services
cd infra/compose
docker-compose -f docker-compose.local.yml ps
docker compose ps
# Count healthy services
docker-compose -f docker-compose.local.yml ps | grep -c "healthy"
docker compose ps | grep -c "healthy"
# Check specific service
docker-compose -f docker-compose.local.yml ps svc-ingestion
docker compose ps svc-ingestion
```
### Logs
@@ -53,16 +53,16 @@ docker-compose -f docker-compose.local.yml ps svc-ingestion
```bash
# View service logs
cd infra/compose
docker-compose -f docker-compose.local.yml logs -f SERVICE_NAME
docker compose logs -f SERVICE_NAME
# View last 50 lines
docker-compose -f docker-compose.local.yml logs --tail=50 SERVICE_NAME
docker compose logs --tail=50 SERVICE_NAME
# View logs since 5 minutes ago
docker-compose -f docker-compose.local.yml logs --since 5m SERVICE_NAME
docker compose logs --since 5m SERVICE_NAME
# Search logs for errors
docker-compose -f docker-compose.local.yml logs SERVICE_NAME | grep -i error
docker compose logs SERVICE_NAME | grep -i error
```
### Health Checks
@@ -70,7 +70,7 @@ docker-compose -f docker-compose.local.yml logs SERVICE_NAME | grep -i error
```bash
# Check Traefik health check status
cd infra/compose
docker-compose -f docker-compose.local.yml logs traefik --since 5m | grep -i "health"
docker compose logs traefik --since 5m | grep -i "health"
# Should show no errors (only certificate warnings are OK)
```
@@ -119,13 +119,13 @@ curl -X POST http://localhost:8000/upload \
```bash
# Check logs for errors
cd infra/compose
docker-compose -f docker-compose.local.yml logs SERVICE_NAME --tail=100
docker compose logs SERVICE_NAME --tail=100
# Restart service
docker-compose -f docker-compose.local.yml restart SERVICE_NAME
docker compose restart SERVICE_NAME
# Rebuild and restart
docker-compose -f docker-compose.local.yml up -d --build SERVICE_NAME
docker compose up -d --build SERVICE_NAME
```
### Infrastructure Issues
@@ -133,13 +133,13 @@ docker-compose -f docker-compose.local.yml up -d --build SERVICE_NAME
```bash
# Check infrastructure services
cd infra/compose
docker-compose -f docker-compose.local.yml ps postgres redis minio neo4j
docker compose ps postgres redis minio neo4j
# Restart infrastructure
docker-compose -f docker-compose.local.yml restart postgres redis minio neo4j
docker compose restart postgres redis minio neo4j
# Check connectivity
docker-compose -f docker-compose.local.yml exec svc-ingestion ping -c 3 postgres
docker compose exec svc-ingestion ping -c 3 postgres
```
### Health Check Failures
@@ -147,13 +147,13 @@ docker-compose -f docker-compose.local.yml exec svc-ingestion ping -c 3 postgres
```bash
# Check Traefik logs
cd infra/compose
docker-compose -f docker-compose.local.yml logs traefik --tail=100 | grep -i "health\|error"
docker compose logs traefik --tail=100 | grep -i "health\|error"
# Test health endpoint directly
docker-compose -f docker-compose.local.yml exec SERVICE_NAME curl -f http://localhost:8000/healthz
docker compose exec SERVICE_NAME curl -f http://localhost:8000/healthz
# Restart Traefik
docker-compose -f docker-compose.local.yml restart traefik
docker compose restart traefik
```
### Authentication Issues
@@ -191,10 +191,10 @@ open http://localhost:8080
```bash
# PostgreSQL
docker-compose -f infra/compose/docker-compose.local.yml exec postgres psql -U postgres
docker compose exec postgres psql -U postgres
# Redis
docker-compose -f infra/compose/docker-compose.local.yml exec redis redis-cli
docker compose exec redis redis-cli
# Neo4j Browser
open http://localhost:7474
@@ -206,14 +206,14 @@ open http://localhost:7474
```bash
cd infra/compose
docker-compose -f docker-compose.local.yml restart
docker compose restart
```
### Restart Single Service
```bash
cd infra/compose
docker-compose -f docker-compose.local.yml restart svc-ingestion
docker compose restart svc-ingestion
```
### View Service Configuration
@@ -280,6 +280,7 @@ make dev-service SERVICE=svc_ingestion
1. **Create Environment**: "AI Tax Agent - Development"
2. **Add Variables**:
- `base_url`: `http://localhost:8000`
- `auth_user`: `dev-user`
- `auth_email`: `dev@example.com`
@@ -337,13 +338,13 @@ docker-compose -f docker-compose.local.yml ps | grep svc-ingestion
### Common Issues
| Issue | Solution |
|-------|----------|
| 401 Unauthorized | Use `DISABLE_AUTH=true` or add auth headers |
| Connection refused | Check service is running: `docker-compose ps` |
| 500 Internal Error | Check logs: `docker-compose logs SERVICE_NAME` |
| Issue | Solution |
| -------------------- | ------------------------------------------------- |
| 401 Unauthorized | Use `DISABLE_AUTH=true` or add auth headers |
| Connection refused | Check service is running: `docker-compose ps` |
| 500 Internal Error | Check logs: `docker-compose logs SERVICE_NAME` |
| Health check failing | Check Traefik logs: `docker-compose logs traefik` |
| Port already in use | Stop conflicting service or change port |
| Port already in use | Stop conflicting service or change port |
## 🎯 Quick Commands
@@ -366,22 +367,22 @@ cd infra/compose && docker-compose -f docker-compose.local.yml down
## 🔄 Service Ports
| Service | Port | Access |
|---------|------|--------|
| svc-ingestion | 8000 | http://localhost:8000 |
| PostgreSQL | 5432 | localhost:5432 |
| Redis | 6379 | localhost:6379 |
| MinIO Console | 9093 | http://localhost:9093 |
| MinIO API | 9092 | http://localhost:9092 |
| Neo4j Browser | 7474 | http://localhost:7474 |
| Neo4j Bolt | 7687 | bolt://localhost:7687 |
| Qdrant | 6333 | http://localhost:6333 |
| NATS | 4222 | nats://localhost:4222 |
| Prometheus | 9090 | http://localhost:9090 |
| Grafana | 3000 | http://localhost:3000 |
| Service | Port | Access |
| ----------------- | ---- | --------------------- |
| svc-ingestion | 8000 | http://localhost:8000 |
| PostgreSQL | 5432 | localhost:5432 |
| Redis | 6379 | localhost:6379 |
| MinIO Console | 9093 | http://localhost:9093 |
| MinIO API | 9092 | http://localhost:9092 |
| Neo4j Browser | 7474 | http://localhost:7474 |
| Neo4j Bolt | 7687 | bolt://localhost:7687 |
| Qdrant | 6333 | http://localhost:6333 |
| NATS | 4222 | nats://localhost:4222 |
| Prometheus | 9090 | http://localhost:9090 |
| Grafana | 3000 | http://localhost:3000 |
| Traefik Dashboard | 8080 | http://localhost:8080 |
| Vault | 8200 | http://localhost:8200 |
| Unleash | 4242 | http://localhost:4242 |
| Vault | 8200 | http://localhost:8200 |
| Unleash | 4242 | http://localhost:4242 |
## ✅ Health Check
@@ -413,4 +414,3 @@ fi
```
Save this as `check-health.sh` and run with `bash check-health.sh`

BIN
docs/SA150-Notes-2025.pdf Normal file
View File

Binary file not shown.