completed local setup with compose
Some checks failed
CI/CD Pipeline / Generate SBOM (push) Has been cancelled
CI/CD Pipeline / Deploy to Staging (push) Has been cancelled
CI/CD Pipeline / Deploy to Production (push) Has been cancelled
CI/CD Pipeline / Code Quality & Linting (push) Has been cancelled
CI/CD Pipeline / Policy Validation (push) Has been cancelled
CI/CD Pipeline / Test Suite (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-coverage) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-extract) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-firm-connectors) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-forms) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-hmrc) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-ingestion) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-kg) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-normalize-map) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-ocr) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-rag-indexer) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-rag-retriever) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-reason) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-rpa) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (ui-review) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (svc-coverage) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (svc-extract) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (svc-kg) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (svc-rag-retriever) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (ui-review) (push) Has been cancelled
CI/CD Pipeline / Notifications (push) Has been cancelled
Some checks failed
CI/CD Pipeline / Generate SBOM (push) Has been cancelled
CI/CD Pipeline / Deploy to Staging (push) Has been cancelled
CI/CD Pipeline / Deploy to Production (push) Has been cancelled
CI/CD Pipeline / Code Quality & Linting (push) Has been cancelled
CI/CD Pipeline / Policy Validation (push) Has been cancelled
CI/CD Pipeline / Test Suite (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-coverage) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-extract) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-firm-connectors) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-forms) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-hmrc) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-ingestion) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-kg) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-normalize-map) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-ocr) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-rag-indexer) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-rag-retriever) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-reason) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-rpa) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (ui-review) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (svc-coverage) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (svc-extract) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (svc-kg) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (svc-rag-retriever) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (ui-review) (push) Has been cancelled
CI/CD Pipeline / Notifications (push) Has been cancelled
This commit is contained in:
@@ -8,9 +8,10 @@ Successfully integrated NATS.io message broker with JetStream support into the A
|
||||
|
||||
### 1. Added NATS Service to Docker Compose
|
||||
|
||||
**File**: `infra/compose/docker-compose.local.yml`
|
||||
**File**: `infra/compose/compose.yaml`
|
||||
|
||||
#### NATS Service Configuration:
|
||||
|
||||
```yaml
|
||||
nats:
|
||||
image: nats:2.10-alpine
|
||||
@@ -19,9 +20,9 @@ nats:
|
||||
networks:
|
||||
- backend
|
||||
ports:
|
||||
- "4222:4222" # NATS client connections
|
||||
- "8222:8222" # HTTP monitoring
|
||||
- "6222:6222" # Cluster routing (for future clustering)
|
||||
- "4222:4222" # NATS client connections
|
||||
- "8222:8222" # HTTP monitoring
|
||||
- "6222:6222" # Cluster routing (for future clustering)
|
||||
volumes:
|
||||
- nats_data:/data
|
||||
command: >
|
||||
@@ -33,7 +34,15 @@ nats:
|
||||
environment:
|
||||
NATS_LOG_LEVEL: ${NATS_LOG_LEVEL:-info}
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8222/healthz"]
|
||||
test:
|
||||
[
|
||||
"CMD",
|
||||
"wget",
|
||||
"--no-verbose",
|
||||
"--tries=1",
|
||||
"--spider",
|
||||
"http://localhost:8222/healthz",
|
||||
]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
@@ -47,6 +56,7 @@ nats:
|
||||
```
|
||||
|
||||
#### Key Features:
|
||||
|
||||
- **JetStream Enabled**: Persistent messaging with file-based storage
|
||||
- **Monitoring**: HTTP monitoring interface on port 8222
|
||||
- **Cluster Ready**: Port 6222 configured for future clustering
|
||||
@@ -63,6 +73,7 @@ Added `nats_data:` volume to the volumes section for persistent storage.
|
||||
Updated **13 application services** to include NATS configuration:
|
||||
|
||||
#### Services Updated:
|
||||
|
||||
1. `svc-ingestion`
|
||||
2. `svc-extract`
|
||||
3. `svc-kg`
|
||||
@@ -78,6 +89,7 @@ Updated **13 application services** to include NATS configuration:
|
||||
13. `svc-rpa`
|
||||
|
||||
#### Environment Variables Added to Each Service:
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
# ... existing variables ...
|
||||
@@ -95,6 +107,7 @@ depends_on:
|
||||
**File**: `infra/compose/env.example`
|
||||
|
||||
Added NATS configuration variables:
|
||||
|
||||
```bash
|
||||
# Event Bus Configuration
|
||||
EVENT_BUS_TYPE=memory
|
||||
@@ -119,18 +132,20 @@ cd infra/compose
|
||||
cp env.example .env
|
||||
|
||||
# Start all services including NATS
|
||||
docker-compose -f docker-compose.local.yml up -d
|
||||
docker compose up -d
|
||||
|
||||
# Check NATS status
|
||||
docker-compose -f docker-compose.local.yml logs nats
|
||||
docker compose logs nats
|
||||
```
|
||||
|
||||
### Using NATS in Applications
|
||||
|
||||
#### Option 1: Environment Variable Configuration
|
||||
|
||||
Set `EVENT_BUS_TYPE=nats` in your environment to use NATS instead of memory/kafka.
|
||||
|
||||
#### Option 2: Direct Configuration
|
||||
|
||||
```python
|
||||
from libs.events import create_event_bus
|
||||
|
||||
@@ -177,17 +192,18 @@ nats --server=nats://localhost:4222 stream info TAX_AGENT_EVENTS
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `NATS_SERVERS` | `nats://nats:4222` | NATS server connection string |
|
||||
| `NATS_STREAM_NAME` | `TAX_AGENT_EVENTS` | JetStream stream name |
|
||||
| `NATS_CONSUMER_GROUP` | `tax-agent` | Consumer group name |
|
||||
| `NATS_LOG_LEVEL` | `info` | NATS server log level |
|
||||
| `EVENT_BUS_TYPE` | `memory` | Event bus type (memory/kafka/nats) |
|
||||
| Variable | Default | Description |
|
||||
| --------------------- | ------------------ | ---------------------------------- |
|
||||
| `NATS_SERVERS` | `nats://nats:4222` | NATS server connection string |
|
||||
| `NATS_STREAM_NAME` | `TAX_AGENT_EVENTS` | JetStream stream name |
|
||||
| `NATS_CONSUMER_GROUP` | `tax-agent` | Consumer group name |
|
||||
| `NATS_LOG_LEVEL` | `info` | NATS server log level |
|
||||
| `EVENT_BUS_TYPE` | `memory` | Event bus type (memory/kafka/nats) |
|
||||
|
||||
### NATS Server Configuration
|
||||
|
||||
The NATS server is configured with:
|
||||
|
||||
- **JetStream**: Enabled for persistent messaging
|
||||
- **File Storage**: 10GB maximum
|
||||
- **Memory Storage**: 1GB maximum
|
||||
@@ -219,26 +235,31 @@ The NATS server is configured with:
|
||||
## Benefits
|
||||
|
||||
### 1. **High Performance**
|
||||
|
||||
- Very low latency messaging
|
||||
- High throughput with minimal overhead
|
||||
- Efficient binary protocol
|
||||
|
||||
### 2. **Operational Simplicity**
|
||||
|
||||
- Single binary deployment
|
||||
- Minimal configuration required
|
||||
- Built-in monitoring and health checks
|
||||
|
||||
### 3. **Reliability**
|
||||
|
||||
- JetStream provides persistence
|
||||
- Automatic message acknowledgment
|
||||
- Configurable retry policies
|
||||
|
||||
### 4. **Scalability**
|
||||
|
||||
- Ready for clustering (port 6222 configured)
|
||||
- Horizontal scaling support
|
||||
- Load balancing across consumers
|
||||
|
||||
### 5. **Integration**
|
||||
|
||||
- Seamless integration with existing services
|
||||
- Traefik routing for web UI
|
||||
- Authentik authentication for monitoring
|
||||
@@ -246,27 +267,30 @@ The NATS server is configured with:
|
||||
## Next Steps
|
||||
|
||||
1. **Test the Integration**:
|
||||
|
||||
```bash
|
||||
# Start the stack
|
||||
docker-compose -f docker-compose.local.yml up -d
|
||||
|
||||
docker compose up -d
|
||||
|
||||
# Check NATS is running
|
||||
docker-compose -f docker-compose.local.yml ps nats
|
||||
|
||||
docker compose ps nats
|
||||
|
||||
# View NATS logs
|
||||
docker-compose -f docker-compose.local.yml logs nats
|
||||
docker compose logs nats
|
||||
```
|
||||
|
||||
2. **Switch to NATS**:
|
||||
|
||||
```bash
|
||||
# Update environment
|
||||
echo "EVENT_BUS_TYPE=nats" >> .env
|
||||
|
||||
|
||||
# Restart services
|
||||
docker-compose -f docker-compose.local.yml restart
|
||||
docker compose restart
|
||||
```
|
||||
|
||||
3. **Monitor Usage**:
|
||||
|
||||
- Access monitoring at `https://nats.local`
|
||||
- Use NATS CLI for detailed monitoring
|
||||
- Check application logs for event processing
|
||||
|
||||
Reference in New Issue
Block a user