ai-tax-agent/docs/NATS_README.md

# NATS.io Event Bus with JetStream

This document describes the NATS.io event bus implementation with JetStream support for the AI Tax Agent project.

## Overview

The `NATSEventBus` class provides a robust, scalable event streaming solution using NATS.io with JetStream for persistent messaging. It implements the same `EventBus` interface as other event bus implementations (Kafka, SQS, Memory) for consistency.

## Features

- **JetStream Integration**: Uses NATS JetStream for persistent, reliable message delivery
- **Automatic Stream Management**: Creates and manages JetStream streams automatically
- **Pull-based Consumers**: Uses pull-based consumers for better flow control
- **Cluster Support**: Supports NATS cluster configurations for high availability
- **Error Handling**: Comprehensive error handling with automatic retries
- **Message Acknowledgment**: Explicit message acknowledgment with configurable retry policies
- **Durable Consumers**: Creates durable consumers for guaranteed message processing

## Configuration

### Basic Configuration

```python
from libs.events import NATSEventBus

# Single server
bus = NATSEventBus(
    servers="nats://localhost:4222",
    stream_name="TAX_AGENT_EVENTS",
    consumer_group="tax-agent"
)

# Multiple servers (cluster)
bus = NATSEventBus(
    servers=[
        "nats://nats1.example.com:4222",
        "nats://nats2.example.com:4222",
        "nats://nats3.example.com:4222"
    ],
    stream_name="PRODUCTION_EVENTS",
    consumer_group="tax-agent-prod"
)
```

### Factory Configuration

```python
from libs.events import create_event_bus

bus = create_event_bus(
    "nats",
    servers="nats://localhost:4222",
    stream_name="TAX_AGENT_EVENTS",
    consumer_group="tax-agent"
)
```

## Usage

### Publishing Events

```python
from libs.events import EventPayload

# Create event payload
payload = EventPayload(
    data={"user_id": "123", "action": "login"},
    actor="user-service",
    tenant_id="tenant-456",
    trace_id="trace-789"
)

# Publish event
success = await bus.publish("user.login", payload)
if success:
    print("Event published successfully")
```

### Subscribing to Events

```python
async def handle_user_login(topic: str, payload: EventPayload) -> None:
    print(f"User {payload.data['user_id']} logged in")
    # Process the event...

# Subscribe to topic
await bus.subscribe("user.login", handle_user_login)
```

### Complete Example

```python
import asyncio
from libs.events import NATSEventBus, EventPayload

async def main():
    bus = NATSEventBus()
    
    try:
        # Start the bus
        await bus.start()
        
        # Subscribe to events
        await bus.subscribe("user.created", handle_user_created)
        
        # Publish an event
        payload = EventPayload(
            data={"user_id": "123", "email": "user@example.com"},
            actor="registration-service",
            tenant_id="tenant-456"
        )
        await bus.publish("user.created", payload)
        
        # Wait for processing
        await asyncio.sleep(1)
        
    finally:
        await bus.stop()

asyncio.run(main())
```

## JetStream Configuration

The NATS event bus automatically creates and configures JetStream streams with the following settings:

- **Retention Policy**: Work Queue (messages are removed after acknowledgment)
- **Max Age**: 7 days (messages older than 7 days are automatically deleted)
- **Storage**: File-based storage for persistence
- **Subject Pattern**: `{stream_name}.*` (e.g., `TAX_AGENT_EVENTS.*`)

### Consumer Configuration

- **Durable Consumers**: Each topic subscription creates a durable consumer
- **Ack Policy**: Explicit acknowledgment required
- **Deliver Policy**: New messages only (doesn't replay old messages)
- **Max Deliver**: 3 attempts before message is considered failed
- **Ack Wait**: 30 seconds timeout for acknowledgment

## Error Handling

The NATS event bus includes comprehensive error handling:

### Publishing Errors
- Network failures are logged and return `False`
- Automatic retry logic can be implemented at the application level

### Consumer Errors
- Handler exceptions are caught and logged
- Failed messages are negatively acknowledged (NAK) for retry
- Messages that fail multiple times are moved to a dead letter queue (if configured)

### Connection Errors
- Automatic reconnection is handled by the NATS client
- Consumer tasks are gracefully shut down on connection loss

## Monitoring and Observability

The implementation includes structured logging with the following information:

- Event publishing success/failure
- Consumer subscription status
- Message processing metrics
- Error details and stack traces

### Log Examples

```
INFO: Event published topic=user.created event_id=01HK... stream_seq=123
INFO: Subscribed to topic topic=user.login consumer=tax-agent-user.login
ERROR: Handler failed topic=user.created event_id=01HK... error=...
```

## Performance Considerations

### Throughput
- Pull-based consumers allow for controlled message processing
- Batch fetching (up to 10 messages per fetch) improves throughput
- Async processing enables high concurrency

### Memory Usage
- File-based storage keeps memory usage low
- Configurable message retention prevents unbounded growth

### Network Efficiency
- Binary protocol with minimal overhead
- Connection pooling and reuse
- Efficient subject-based routing

## Deployment

### Docker Compose Example

```yaml
services:
  nats:
    image: nats:2.10-alpine
    ports:
      - "4222:4222"
      - "8222:8222"
    command: 
      - "--jetstream"
      - "--store_dir=/data"
      - "--http_port=8222"
    volumes:
      - nats_data:/data

volumes:
  nats_data:
```

### Kubernetes Example

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nats
spec:
  serviceName: nats
  replicas: 3
  selector:
    matchLabels:
      app: nats
  template:
    metadata:
      labels:
        app: nats
    spec:
      containers:
      - name: nats
        image: nats:2.10-alpine
        args:
          - "--cluster_name=nats-cluster"
          - "--jetstream"
          - "--store_dir=/data"
        ports:
        - containerPort: 4222
        - containerPort: 6222
        - containerPort: 8222
        volumeMounts:
        - name: nats-storage
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: nats-storage
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi
```

## Dependencies

The NATS event bus requires the following Python package:

```
nats-py>=2.6.0
```

This is automatically included in `libs/requirements.txt`.

## Comparison with Other Event Buses

| Feature | NATS | Kafka | SQS |
|---------|------|-------|-----|
| Setup Complexity | Low | Medium | Low |
| Throughput | High | Very High | Medium |
| Latency | Very Low | Low | Medium |
| Persistence | Yes (JetStream) | Yes | Yes |
| Ordering | Per Subject | Per Partition | FIFO Queues |
| Clustering | Built-in | Built-in | Managed |
| Operational Overhead | Low | High | None |

## Best Practices

1. **Use meaningful subject names**: Follow a hierarchical naming convention (e.g., `service.entity.action`)
2. **Handle failures gracefully**: Implement proper error handling in event handlers
3. **Monitor consumer lag**: Track message processing delays
4. **Use appropriate retention**: Configure message retention based on business requirements
5. **Test failure scenarios**: Verify behavior during network partitions and service failures