ai-tax-agent/docs/NATS_DOCKER_COMPOSE_SUMMARY.md

# NATS Docker Compose Integration Summary

## Overview

Successfully integrated NATS.io message broker with JetStream support into the AI Tax Agent's Docker Compose infrastructure. The NATS service is now available alongside other infrastructure services like Redis, PostgreSQL, and Neo4j.

## Changes Made

### 1. Added NATS Service to Docker Compose

**File**: `infra/compose/docker-compose.local.yml`

#### NATS Service Configuration:
```yaml
nats:
  image: nats:2.10-alpine
  container_name: nats
  restart: unless-stopped
  networks:
    - backend
  ports:
    - "4222:4222"  # NATS client connections
    - "8222:8222"  # HTTP monitoring
    - "6222:6222"  # Cluster routing (for future clustering)
  volumes:
    - nats_data:/data
  command: >
    --jetstream
    --store_dir=/data
    --http_port=8222
    --max_file_store=10GB
    --max_mem_store=1GB
  environment:
    NATS_LOG_LEVEL: ${NATS_LOG_LEVEL:-info}
  healthcheck:
    test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8222/healthz"]
    interval: 30s
    timeout: 10s
    retries: 3
  labels:
    - "traefik.enable=true"
    - "traefik.http.routers.nats-monitor.rule=Host(`nats.${DOMAIN:-local}`)"
    - "traefik.http.routers.nats-monitor.entrypoints=websecure"
    - "traefik.http.routers.nats-monitor.tls=true"
    - "traefik.http.routers.nats-monitor.middlewares=authentik-forwardauth@file"
    - "traefik.http.services.nats-monitor.loadbalancer.server.port=8222"
```

#### Key Features:
- **JetStream Enabled**: Persistent messaging with file-based storage
- **Monitoring**: HTTP monitoring interface on port 8222
- **Cluster Ready**: Port 6222 configured for future clustering
- **Health Checks**: Automated health monitoring
- **Traefik Integration**: Web UI accessible at `https://nats.local`
- **Storage Limits**: 10GB file storage, 1GB memory storage

### 2. Added NATS Volume

Added `nats_data:` volume to the volumes section for persistent storage.

### 3. Updated All Application Services

Updated **13 application services** to include NATS configuration:

#### Services Updated:
1. `svc-ingestion`
2. `svc-extract`
3. `svc-kg`
4. `svc-rag-retriever`
5. `svc-coverage`
6. `svc-firm-connectors`
7. `svc-forms`
8. `svc-hmrc`
9. `svc-normalize-map`
10. `svc-ocr`
11. `svc-rag-indexer`
12. `svc-reason`
13. `svc-rpa`

#### Environment Variables Added to Each Service:
```yaml
environment:
  # ... existing variables ...
  - NATS_SERVERS=${NATS_SERVERS:-nats://nats:4222}
  - NATS_STREAM_NAME=${NATS_STREAM_NAME:-TAX_AGENT_EVENTS}
  - NATS_CONSUMER_GROUP=${NATS_CONSUMER_GROUP:-tax-agent}

depends_on:
  # ... existing dependencies ...
  - nats
```

### 4. Updated Environment Configuration

**File**: `infra/compose/env.example`

Added NATS configuration variables:
```bash
# Event Bus Configuration
EVENT_BUS_TYPE=memory
KAFKA_BOOTSTRAP_SERVERS=

# NATS Configuration
NATS_SERVERS=nats://nats:4222
NATS_STREAM_NAME=TAX_AGENT_EVENTS
NATS_CONSUMER_GROUP=tax-agent
NATS_LOG_LEVEL=info
```

## Usage

### Starting the Stack

```bash
# Navigate to compose directory
cd infra/compose

# Copy environment file
cp env.example .env

# Start all services including NATS
docker-compose -f docker-compose.local.yml up -d

# Check NATS status
docker-compose -f docker-compose.local.yml logs nats
```

### Using NATS in Applications

#### Option 1: Environment Variable Configuration
Set `EVENT_BUS_TYPE=nats` in your environment to use NATS instead of memory/kafka.

#### Option 2: Direct Configuration
```python
from libs.events import create_event_bus

# Use environment variables (recommended)
bus = create_event_bus(
    "nats",
    servers=os.getenv("NATS_SERVERS", "nats://nats:4222"),
    stream_name=os.getenv("NATS_STREAM_NAME", "TAX_AGENT_EVENTS"),
    consumer_group=os.getenv("NATS_CONSUMER_GROUP", "tax-agent")
)

# Or direct configuration
bus = create_event_bus(
    "nats",
    servers="nats://nats:4222",
    stream_name="TAX_AGENT_EVENTS",
    consumer_group="tax-agent"
)
```

### Accessing NATS Monitoring

- **URL**: `https://nats.local` (requires Authentik authentication)
- **Direct Access**: `http://localhost:8222` (when running locally)
- **Health Check**: `http://localhost:8222/healthz`

### NATS CLI Access

```bash
# Install NATS CLI
go install github.com/nats-io/natscli/nats@latest

# Connect to NATS server
nats --server=nats://localhost:4222 server info

# List streams
nats --server=nats://localhost:4222 stream list

# Monitor stream
nats --server=nats://localhost:4222 stream info TAX_AGENT_EVENTS
```

## Configuration Options

### Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `NATS_SERVERS` | `nats://nats:4222` | NATS server connection string |
| `NATS_STREAM_NAME` | `TAX_AGENT_EVENTS` | JetStream stream name |
| `NATS_CONSUMER_GROUP` | `tax-agent` | Consumer group name |
| `NATS_LOG_LEVEL` | `info` | NATS server log level |
| `EVENT_BUS_TYPE` | `memory` | Event bus type (memory/kafka/nats) |

### NATS Server Configuration

The NATS server is configured with:
- **JetStream**: Enabled for persistent messaging
- **File Storage**: 10GB maximum
- **Memory Storage**: 1GB maximum
- **Data Directory**: `/data` (persistent volume)
- **Monitoring**: HTTP interface on port 8222

## Network Architecture

```
┌─────────────────┐    ┌──────────────┐    ┌─────────────────┐
│   Application   │───▶│     NATS     │◀───│   Application   │
│   Services      │    │   (4222)     │    │   Services      │
│                 │    │              │    │                 │
│ svc-ingestion   │    │  JetStream   │    │ svc-extract     │
│ svc-kg          │    │  Enabled     │    │ svc-rag-*       │
│ svc-forms       │    │              │    │ svc-reason      │
│ ...             │    │              │    │ ...             │
└─────────────────┘    └──────────────┘    └─────────────────┘
                              │
                              ▼
                    ┌──────────────────┐
                    │   Monitoring     │
                    │   (8222)         │
                    │                  │
                    │ https://nats.local│
                    └──────────────────┘
```

## Benefits

### 1. **High Performance**
- Very low latency messaging
- High throughput with minimal overhead
- Efficient binary protocol

### 2. **Operational Simplicity**
- Single binary deployment
- Minimal configuration required
- Built-in monitoring and health checks

### 3. **Reliability**
- JetStream provides persistence
- Automatic message acknowledgment
- Configurable retry policies

### 4. **Scalability**
- Ready for clustering (port 6222 configured)
- Horizontal scaling support
- Load balancing across consumers

### 5. **Integration**
- Seamless integration with existing services
- Traefik routing for web UI
- Authentik authentication for monitoring

## Next Steps

1. **Test the Integration**:
   ```bash
   # Start the stack
   docker-compose -f docker-compose.local.yml up -d
   
   # Check NATS is running
   docker-compose -f docker-compose.local.yml ps nats
   
   # View NATS logs
   docker-compose -f docker-compose.local.yml logs nats
   ```

2. **Switch to NATS**:
   ```bash
   # Update environment
   echo "EVENT_BUS_TYPE=nats" >> .env
   
   # Restart services
   docker-compose -f docker-compose.local.yml restart
   ```

3. **Monitor Usage**:
   - Access monitoring at `https://nats.local`
   - Use NATS CLI for detailed monitoring
   - Check application logs for event processing

4. **Production Deployment**:
   - Configure NATS clustering for high availability
   - Set up proper authentication and TLS
   - Configure monitoring and alerting
   - Tune storage and memory limits based on usage

The NATS integration is now complete and ready for use across all AI Tax Agent services!