Initial commit
Some checks failed
CI/CD Pipeline / Code Quality & Linting (push) Has been cancelled
CI/CD Pipeline / Policy Validation (push) Has been cancelled
CI/CD Pipeline / Test Suite (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-coverage) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-extract) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-firm-connectors) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-forms) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-hmrc) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-ingestion) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-kg) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-normalize-map) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-ocr) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-rag-indexer) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-rag-retriever) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-reason) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-rpa) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (ui-review) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (svc-coverage) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (svc-extract) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (svc-kg) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (svc-rag-retriever) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (ui-review) (push) Has been cancelled
CI/CD Pipeline / Generate SBOM (push) Has been cancelled
CI/CD Pipeline / Deploy to Staging (push) Has been cancelled
CI/CD Pipeline / Deploy to Production (push) Has been cancelled
CI/CD Pipeline / Notifications (push) Has been cancelled

This commit is contained in:
harkon
2025-10-11 08:41:36 +01:00
commit b324ff09ef
276 changed files with 55220 additions and 0 deletions

396
docs/POSTMAN_SETUP.md Normal file
View File

@@ -0,0 +1,396 @@
# Postman Setup Guide
## Quick Start
### Option 1: Development Mode (Recommended for Local Testing)
Run the service with authentication disabled:
```bash
DISABLE_AUTH=true make dev-service SERVICE=svc_ingestion
```
**No authentication headers required!** Just make requests directly:
```
GET http://localhost:8000/healthz
POST http://localhost:8000/upload
```
### Option 2: Production Mode (With Authentication)
Run the service normally:
```bash
make dev-service SERVICE=svc_ingestion
```
**Authentication headers required** for all protected endpoints.
## Postman Environment Setup
### Create Environment
1. Open Postman
2. Click "Environments" in the left sidebar
3. Click "+" to create a new environment
4. Name it: **"AI Tax Agent - Development"**
### Environment Variables
Add these variables:
| Variable | Initial Value | Current Value | Description |
|----------|---------------|---------------|-------------|
| `base_url` | `http://localhost:8000` | `http://localhost:8000` | Service base URL |
| `auth_user` | `dev-user` | `dev-user` | Development user |
| `auth_email` | `dev@example.com` | `dev@example.com` | Development email |
| `auth_token` | `Bearer dev-token-12345` | `Bearer dev-token-12345` | Development token |
### JSON Export
Save this as `AI-Tax-Agent-Dev.postman_environment.json`:
```json
{
"id": "ai-tax-agent-dev",
"name": "AI Tax Agent - Development",
"values": [
{
"key": "base_url",
"value": "http://localhost:8000",
"type": "default",
"enabled": true
},
{
"key": "auth_user",
"value": "dev-user",
"type": "default",
"enabled": true
},
{
"key": "auth_email",
"value": "dev@example.com",
"type": "default",
"enabled": true
},
{
"key": "auth_token",
"value": "Bearer dev-token-12345",
"type": "default",
"enabled": true
}
],
"_postman_variable_scope": "environment"
}
```
Import this file in Postman: **Import****Upload Files** → Select the JSON file
## Request Examples
### 1. Health Check (Public Endpoint)
**No authentication required** (works in both modes)
```
GET {{base_url}}/healthz
```
**Expected Response:**
```json
{
"status": "healthy",
"service": "svc-ingestion",
"version": "1.0.0"
}
```
### 2. API Documentation (Public Endpoint)
**No authentication required** (works in both modes)
```
GET {{base_url}}/docs
```
Opens Swagger UI in browser.
### 3. Upload Document (Protected Endpoint)
#### Development Mode (DISABLE_AUTH=true)
**No headers required:**
```
POST {{base_url}}/upload
Body: form-data
- file: [Select file]
```
#### Production Mode (Authentication Required)
**Headers:**
```
X-Authenticated-User: {{auth_user}}
X-Authenticated-Email: {{auth_email}}
Authorization: {{auth_token}}
```
**Body:**
```
form-data:
- file: [Select file]
```
**Expected Response:**
```json
{
"document_id": "doc_01K6BG98T8KFF16KZ3XAJP37DX",
"filename": "invoice.pdf",
"size": 245678,
"mime_type": "application/pdf",
"checksum": "sha256:abc123...",
"storage_path": "s3://raw-documents/tenant-id/doc_01K6BG98T8KFF16KZ3XAJP37DX.pdf",
"uploaded_at": "2025-09-29T19:48:07.623900Z"
}
```
## Postman Collection
### Create Collection
1. Click "Collections" in left sidebar
2. Click "+" to create new collection
3. Name it: **"AI Tax Agent API"**
### Add Requests
#### Folder: Health & Status
**1. Health Check**
```
GET {{base_url}}/healthz
```
**2. Readiness Check**
```
GET {{base_url}}/readyz
```
**3. Liveness Check**
```
GET {{base_url}}/livez
```
**4. API Documentation**
```
GET {{base_url}}/docs
```
**5. OpenAPI Spec**
```
GET {{base_url}}/openapi.json
```
#### Folder: Document Ingestion
**1. Upload Document**
```
POST {{base_url}}/upload
Headers (Production Mode only):
X-Authenticated-User: {{auth_user}}
X-Authenticated-Email: {{auth_email}}
Authorization: {{auth_token}}
Body: form-data
file: [Select file]
```
**2. Get Document Status**
```
GET {{base_url}}/documents/{{document_id}}
Headers (Production Mode only):
X-Authenticated-User: {{auth_user}}
X-Authenticated-Email: {{auth_email}}
Authorization: {{auth_token}}
```
### Collection-Level Authorization
For Production Mode, set authorization at collection level:
1. Click on collection name
2. Go to "Authorization" tab
3. Select "Type: API Key"
4. Add three keys:
- Key: `X-Authenticated-User`, Value: `{{auth_user}}`, Add to: `Header`
- Key: `X-Authenticated-Email`, Value: `{{auth_email}}`, Add to: `Header`
- Key: `Authorization`, Value: `{{auth_token}}`, Add to: `Header`
This applies headers to all requests in the collection.
## Testing Different Services
### Change Service Port
When running multiple services locally, they use different ports:
```bash
# Terminal 1: Ingestion on port 8000
DISABLE_AUTH=true make dev-service SERVICE=svc_ingestion
# Terminal 2: Extract on port 8001
DISABLE_AUTH=true cd apps/svc_extract && uvicorn main:app --reload --host 0.0.0.0 --port 8001
# Terminal 3: KG on port 8002
DISABLE_AUTH=true cd apps/svc_kg && uvicorn main:app --reload --host 0.0.0.0 --port 8002
```
Create separate environments for each:
- **Ingestion**: `base_url = http://localhost:8000`
- **Extract**: `base_url = http://localhost:8001`
- **KG**: `base_url = http://localhost:8002`
## Pre-request Scripts
### Auto-generate Document ID
Add this pre-request script to generate unique document IDs:
```javascript
// Generate ULID for document ID
const ulid = require('ulid');
pm.environment.set('document_id', ulid.ulid());
```
### Add Timestamp
```javascript
// Add current timestamp
pm.environment.set('timestamp', new Date().toISOString());
```
## Tests
### Add Response Tests
Add these tests to verify responses:
```javascript
// Test: Status code is 200
pm.test("Status code is 200", function () {
pm.response.to.have.status(200);
});
// Test: Response time is less than 2000ms
pm.test("Response time is less than 2000ms", function () {
pm.expect(pm.response.responseTime).to.be.below(2000);
});
// Test: Response has required fields
pm.test("Response has document_id", function () {
var jsonData = pm.response.json();
pm.expect(jsonData).to.have.property('document_id');
});
// Test: Save document_id for next request
pm.test("Save document_id", function () {
var jsonData = pm.response.json();
pm.environment.set('document_id', jsonData.document_id);
});
```
## Troubleshooting
### Issue: 401 Unauthorized
**Cause**: Service running in production mode without authentication headers
**Solution 1**: Run with `DISABLE_AUTH=true`
```bash
DISABLE_AUTH=true make dev-service SERVICE=svc_ingestion
```
**Solution 2**: Add authentication headers to request
```
X-Authenticated-User: dev-user
X-Authenticated-Email: dev@example.com
Authorization: Bearer dev-token-12345
```
### Issue: Connection Refused
**Cause**: Service not running or wrong port
**Solution**:
1. Check service is running: `ps aux | grep uvicorn`
2. Verify port: Service should show `Uvicorn running on http://0.0.0.0:8000`
3. Check infrastructure: `make deploy-infra`
### Issue: 500 Internal Server Error
**Cause**: Service error (check logs)
**Solution**:
1. Check terminal where service is running for error logs
2. Verify infrastructure services are running
3. Check database connections
### Issue: File Upload Fails
**Cause**: File too large or wrong MIME type
**Solution**:
1. Check file size (max 50MB by default)
2. Verify MIME type is allowed:
- `application/pdf`
- `image/jpeg`
- `image/png`
- `image/tiff`
- `text/csv`
- `application/vnd.ms-excel`
- `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet`
## Tips & Best Practices
1. **Use Environments**: Switch between dev/staging/prod easily
2. **Use Variables**: Reference `{{base_url}}` instead of hardcoding URLs
3. **Save Responses**: Use tests to save IDs for subsequent requests
4. **Organize Collections**: Group related requests in folders
5. **Add Descriptions**: Document what each request does
6. **Use Pre-request Scripts**: Generate dynamic data
7. **Add Tests**: Verify responses automatically
8. **Export Collections**: Share with team members
## Example Workflow
### Complete Document Upload Flow
1. **Check Service Health**
```
GET {{base_url}}/healthz
```
2. **Upload Document**
```
POST {{base_url}}/upload
Body: form-data with file
```
Save `document_id` from response
3. **Check Document Status**
```
GET {{base_url}}/documents/{{document_id}}
```
4. **Verify Processing**
Check response for processing status
## Additional Resources
- [Postman Documentation](https://learning.postman.com/docs/getting-started/introduction/)
- [API Documentation](http://localhost:8000/docs) (when service is running)
- [Development Guide](DEVELOPMENT.md)
- [Infrastructure Status](INFRASTRUCTURE_STATUS.md)