completed local setup with compose
Some checks failed
CI/CD Pipeline / Generate SBOM (push) Has been cancelled
CI/CD Pipeline / Deploy to Staging (push) Has been cancelled
CI/CD Pipeline / Deploy to Production (push) Has been cancelled
CI/CD Pipeline / Code Quality & Linting (push) Has been cancelled
CI/CD Pipeline / Policy Validation (push) Has been cancelled
CI/CD Pipeline / Test Suite (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-coverage) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-extract) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-firm-connectors) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-forms) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-hmrc) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-ingestion) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-kg) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-normalize-map) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-ocr) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-rag-indexer) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-rag-retriever) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-reason) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-rpa) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (ui-review) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (svc-coverage) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (svc-extract) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (svc-kg) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (svc-rag-retriever) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (ui-review) (push) Has been cancelled
CI/CD Pipeline / Notifications (push) Has been cancelled

This commit is contained in:
harkon
2025-11-26 13:17:17 +00:00
parent 8fe5e62fee
commit fdba81809f
87 changed files with 5610 additions and 3376 deletions

View File

@@ -8,9 +8,10 @@ Successfully integrated NATS.io message broker with JetStream support into the A
### 1. Added NATS Service to Docker Compose
**File**: `infra/compose/docker-compose.local.yml`
**File**: `infra/compose/compose.yaml`
#### NATS Service Configuration:
```yaml
nats:
image: nats:2.10-alpine
@@ -19,9 +20,9 @@ nats:
networks:
- backend
ports:
- "4222:4222" # NATS client connections
- "8222:8222" # HTTP monitoring
- "6222:6222" # Cluster routing (for future clustering)
- "4222:4222" # NATS client connections
- "8222:8222" # HTTP monitoring
- "6222:6222" # Cluster routing (for future clustering)
volumes:
- nats_data:/data
command: >
@@ -33,7 +34,15 @@ nats:
environment:
NATS_LOG_LEVEL: ${NATS_LOG_LEVEL:-info}
healthcheck:
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8222/healthz"]
test:
[
"CMD",
"wget",
"--no-verbose",
"--tries=1",
"--spider",
"http://localhost:8222/healthz",
]
interval: 30s
timeout: 10s
retries: 3
@@ -47,6 +56,7 @@ nats:
```
#### Key Features:
- **JetStream Enabled**: Persistent messaging with file-based storage
- **Monitoring**: HTTP monitoring interface on port 8222
- **Cluster Ready**: Port 6222 configured for future clustering
@@ -63,6 +73,7 @@ Added `nats_data:` volume to the volumes section for persistent storage.
Updated **13 application services** to include NATS configuration:
#### Services Updated:
1. `svc-ingestion`
2. `svc-extract`
3. `svc-kg`
@@ -78,6 +89,7 @@ Updated **13 application services** to include NATS configuration:
13. `svc-rpa`
#### Environment Variables Added to Each Service:
```yaml
environment:
# ... existing variables ...
@@ -95,6 +107,7 @@ depends_on:
**File**: `infra/compose/env.example`
Added NATS configuration variables:
```bash
# Event Bus Configuration
EVENT_BUS_TYPE=memory
@@ -119,18 +132,20 @@ cd infra/compose
cp env.example .env
# Start all services including NATS
docker-compose -f docker-compose.local.yml up -d
docker compose up -d
# Check NATS status
docker-compose -f docker-compose.local.yml logs nats
docker compose logs nats
```
### Using NATS in Applications
#### Option 1: Environment Variable Configuration
Set `EVENT_BUS_TYPE=nats` in your environment to use NATS instead of memory/kafka.
#### Option 2: Direct Configuration
```python
from libs.events import create_event_bus
@@ -177,17 +192,18 @@ nats --server=nats://localhost:4222 stream info TAX_AGENT_EVENTS
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `NATS_SERVERS` | `nats://nats:4222` | NATS server connection string |
| `NATS_STREAM_NAME` | `TAX_AGENT_EVENTS` | JetStream stream name |
| `NATS_CONSUMER_GROUP` | `tax-agent` | Consumer group name |
| `NATS_LOG_LEVEL` | `info` | NATS server log level |
| `EVENT_BUS_TYPE` | `memory` | Event bus type (memory/kafka/nats) |
| Variable | Default | Description |
| --------------------- | ------------------ | ---------------------------------- |
| `NATS_SERVERS` | `nats://nats:4222` | NATS server connection string |
| `NATS_STREAM_NAME` | `TAX_AGENT_EVENTS` | JetStream stream name |
| `NATS_CONSUMER_GROUP` | `tax-agent` | Consumer group name |
| `NATS_LOG_LEVEL` | `info` | NATS server log level |
| `EVENT_BUS_TYPE` | `memory` | Event bus type (memory/kafka/nats) |
### NATS Server Configuration
The NATS server is configured with:
- **JetStream**: Enabled for persistent messaging
- **File Storage**: 10GB maximum
- **Memory Storage**: 1GB maximum
@@ -219,26 +235,31 @@ The NATS server is configured with:
## Benefits
### 1. **High Performance**
- Very low latency messaging
- High throughput with minimal overhead
- Efficient binary protocol
### 2. **Operational Simplicity**
- Single binary deployment
- Minimal configuration required
- Built-in monitoring and health checks
### 3. **Reliability**
- JetStream provides persistence
- Automatic message acknowledgment
- Configurable retry policies
### 4. **Scalability**
- Ready for clustering (port 6222 configured)
- Horizontal scaling support
- Load balancing across consumers
### 5. **Integration**
- Seamless integration with existing services
- Traefik routing for web UI
- Authentik authentication for monitoring
@@ -246,27 +267,30 @@ The NATS server is configured with:
## Next Steps
1. **Test the Integration**:
```bash
# Start the stack
docker-compose -f docker-compose.local.yml up -d
docker compose up -d
# Check NATS is running
docker-compose -f docker-compose.local.yml ps nats
docker compose ps nats
# View NATS logs
docker-compose -f docker-compose.local.yml logs nats
docker compose logs nats
```
2. **Switch to NATS**:
```bash
# Update environment
echo "EVENT_BUS_TYPE=nats" >> .env
# Restart services
docker-compose -f docker-compose.local.yml restart
docker compose restart
```
3. **Monitor Usage**:
- Access monitoring at `https://nats.local`
- Use NATS CLI for detailed monitoring
- Check application logs for event processing