ai-tax-agent/docs/NATS_DOCKER_COMPOSE_SUMMARY.md

NATS Docker Compose Integration Summary

Overview

Successfully integrated NATS.io message broker with JetStream support into the AI Tax Agent's Docker Compose infrastructure. The NATS service is now available alongside other infrastructure services like Redis, PostgreSQL, and Neo4j.

Changes Made

1. Added NATS Service to Docker Compose

File: infra/compose/docker-compose.local.yml

NATS Service Configuration:

nats:
  image: nats:2.10-alpine
  container_name: nats
  restart: unless-stopped
  networks:
    - backend
  ports:
    - "4222:4222"  # NATS client connections
    - "8222:8222"  # HTTP monitoring
    - "6222:6222"  # Cluster routing (for future clustering)
  volumes:
    - nats_data:/data
  command: >
    --jetstream
    --store_dir=/data
    --http_port=8222
    --max_file_store=10GB
    --max_mem_store=1GB
  environment:
    NATS_LOG_LEVEL: ${NATS_LOG_LEVEL:-info}
  healthcheck:
    test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8222/healthz"]
    interval: 30s
    timeout: 10s
    retries: 3
  labels:
    - "traefik.enable=true"
    - "traefik.http.routers.nats-monitor.rule=Host(`nats.${DOMAIN:-local}`)"
    - "traefik.http.routers.nats-monitor.entrypoints=websecure"
    - "traefik.http.routers.nats-monitor.tls=true"
    - "traefik.http.routers.nats-monitor.middlewares=authentik-forwardauth@file"
    - "traefik.http.services.nats-monitor.loadbalancer.server.port=8222"

Key Features:

• JetStream Enabled: Persistent messaging with file-based storage
• Monitoring: HTTP monitoring interface on port 8222
• Cluster Ready: Port 6222 configured for future clustering
• Health Checks: Automated health monitoring
• Traefik Integration: Web UI accessible at https://nats.local
• Storage Limits: 10GB file storage, 1GB memory storage

2. Added NATS Volume

Added nats_data: volume to the volumes section for persistent storage.

3. Updated All Application Services

Updated 13 application services to include NATS configuration:

Services Updated:

1. svc-ingestion
2. svc-extract
3. svc-kg
4. svc-rag-retriever
5. svc-coverage
6. svc-firm-connectors
7. svc-forms
8. svc-hmrc
9. svc-normalize-map
10. svc-ocr
11. svc-rag-indexer
12. svc-reason
13. svc-rpa

Environment Variables Added to Each Service:

environment:
  # ... existing variables ...
  - NATS_SERVERS=${NATS_SERVERS:-nats://nats:4222}
  - NATS_STREAM_NAME=${NATS_STREAM_NAME:-TAX_AGENT_EVENTS}
  - NATS_CONSUMER_GROUP=${NATS_CONSUMER_GROUP:-tax-agent}

depends_on:
  # ... existing dependencies ...
  - nats

4. Updated Environment Configuration

File: infra/compose/env.example

Added NATS configuration variables:

# Event Bus Configuration
EVENT_BUS_TYPE=memory
KAFKA_BOOTSTRAP_SERVERS=

# NATS Configuration
NATS_SERVERS=nats://nats:4222
NATS_STREAM_NAME=TAX_AGENT_EVENTS
NATS_CONSUMER_GROUP=tax-agent
NATS_LOG_LEVEL=info

Usage

Starting the Stack

# Navigate to compose directory
cd infra/compose

# Copy environment file
cp env.example .env

# Start all services including NATS
docker-compose -f docker-compose.local.yml up -d

# Check NATS status
docker-compose -f docker-compose.local.yml logs nats

Using NATS in Applications

Option 1: Environment Variable Configuration

Set EVENT_BUS_TYPE=nats in your environment to use NATS instead of memory/kafka.

Option 2: Direct Configuration

from libs.events import create_event_bus

# Use environment variables (recommended)
bus = create_event_bus(
    "nats",
    servers=os.getenv("NATS_SERVERS", "nats://nats:4222"),
    stream_name=os.getenv("NATS_STREAM_NAME", "TAX_AGENT_EVENTS"),
    consumer_group=os.getenv("NATS_CONSUMER_GROUP", "tax-agent")
)

# Or direct configuration
bus = create_event_bus(
    "nats",
    servers="nats://nats:4222",
    stream_name="TAX_AGENT_EVENTS",
    consumer_group="tax-agent"
)

Accessing NATS Monitoring

• URL: https://nats.local (requires Authentik authentication)
• Direct Access: http://localhost:8222 (when running locally)
• Health Check: http://localhost:8222/healthz

NATS CLI Access

# Install NATS CLI
go install github.com/nats-io/natscli/nats@latest

# Connect to NATS server
nats --server=nats://localhost:4222 server info

# List streams
nats --server=nats://localhost:4222 stream list

# Monitor stream
nats --server=nats://localhost:4222 stream info TAX_AGENT_EVENTS

Configuration Options

Environment Variables

Variable	Default	Description
NATS_SERVERS	nats://nats:4222	NATS server connection string
NATS_STREAM_NAME	TAX_AGENT_EVENTS	JetStream stream name
NATS_CONSUMER_GROUP	tax-agent	Consumer group name
NATS_LOG_LEVEL	info	NATS server log level
EVENT_BUS_TYPE	memory	Event bus type (memory/kafka/nats)

NATS Server Configuration

The NATS server is configured with:

• JetStream: Enabled for persistent messaging
• File Storage: 10GB maximum
• Memory Storage: 1GB maximum
• Data Directory: /data (persistent volume)
• Monitoring: HTTP interface on port 8222

Network Architecture

┌─────────────────┐    ┌──────────────┐    ┌─────────────────┐
│   Application   │───▶│     NATS     │◀───│   Application   │
│   Services      │    │   (4222)     │    │   Services      │
│                 │    │              │    │                 │
│ svc-ingestion   │    │  JetStream   │    │ svc-extract     │
│ svc-kg          │    │  Enabled     │    │ svc-rag-*       │
│ svc-forms       │    │              │    │ svc-reason      │
│ ...             │    │              │    │ ...             │
└─────────────────┘    └──────────────┘    └─────────────────┘
                              │
                              ▼
                    ┌──────────────────┐
                    │   Monitoring     │
                    │   (8222)         │
                    │                  │
                    │ https://nats.local│
                    └──────────────────┘

Benefits

1. High Performance

• Very low latency messaging
• High throughput with minimal overhead
• Efficient binary protocol

2. Operational Simplicity

• Single binary deployment
• Minimal configuration required
• Built-in monitoring and health checks

3. Reliability

• JetStream provides persistence
• Automatic message acknowledgment
• Configurable retry policies

4. Scalability

• Ready for clustering (port 6222 configured)
• Horizontal scaling support
• Load balancing across consumers

5. Integration

• Seamless integration with existing services
• Traefik routing for web UI
• Authentik authentication for monitoring

Next Steps

1. Test the Integration:

   # Start the stack
   docker-compose -f docker-compose.local.yml up -d
   
   # Check NATS is running
   docker-compose -f docker-compose.local.yml ps nats
   
   # View NATS logs
   docker-compose -f docker-compose.local.yml logs nats
   

2. Switch to NATS:

   # Update environment
   echo "EVENT_BUS_TYPE=nats" >> .env
   
   # Restart services
   docker-compose -f docker-compose.local.yml restart
   

3. Monitor Usage:

   • Access monitoring at https://nats.local
   • Use NATS CLI for detailed monitoring
   • Check application logs for event processing

4. Production Deployment:

   • Configure NATS clustering for high availability
   • Set up proper authentication and TLS
   • Configure monitoring and alerting
   • Tune storage and memory limits based on usage

The NATS integration is now complete and ready for use across all AI Tax Agent services!