ai-tax-agent/docs/GITEA_REGISTRY_DEBUG.md

# Gitea Container Registry Debugging Guide

## Common Issues When Pushing Large Docker Images

### Issue 1: Not Logged In

**Symptom**: `unauthorized: authentication required`

**Solution**:
```bash
# On remote server
docker login gitea.harkon.co.uk
# Username: blue (or your Gitea username)
# Password: <your-gitea-access-token>
```

---

### Issue 2: Upload Size Limit (413 Request Entity Too Large)

**Symptom**: Push fails with `413 Request Entity Too Large` or similar error

**Root Cause**: Traefik or Gitea has a limit on request body size

**Solution A: Configure Traefik Middleware**

1. Find your Traefik configuration directory:
```bash
docker inspect traefik | grep -A 10 Mounts
```

2. Create middleware configuration:
```bash
# Example: /opt/traefik/config/middlewares.yml
sudo tee /opt/traefik/config/middlewares.yml > /dev/null << 'EOF'
http:
  middlewares:
    large-upload:
      buffering:
        maxRequestBodyBytes: 5368709120  # 5GB
        memRequestBodyBytes: 104857600   # 100MB
        maxResponseBodyBytes: 5368709120 # 5GB
        memResponseBodyBytes: 104857600  # 100MB
EOF
```

3. Update Gitea container labels:
```yaml
labels:
  - "traefik.http.routers.gitea.middlewares=large-upload@file"
```

4. Restart Traefik:
```bash
docker restart traefik
```

**Solution B: Configure Gitea Directly**

1. Edit Gitea configuration:
```bash
docker exec -it gitea-server vi /data/gitea/conf/app.ini
```

2. Add/modify these settings:
```ini
[server]
LFS_MAX_FILE_SIZE = 5368709120  ; 5GB

[repository.upload]
FILE_MAX_SIZE = 5368709120  ; 5GB
```

3. Restart Gitea:
```bash
docker restart gitea-server
```

---

### Issue 3: Network Timeout

**Symptom**: Push hangs or times out after uploading for a while

**Root Cause**: Network instability or slow connection

**Solution**: Use chunked uploads or increase timeout

1. Configure Docker daemon timeout:
```bash
# Edit /etc/docker/daemon.json
sudo tee /etc/docker/daemon.json > /dev/null << 'EOF'
{
  "max-concurrent-uploads": 1,
  "max-concurrent-downloads": 3,
  "registry-mirrors": []
}
EOF

sudo systemctl restart docker
```

2. Or use Traefik timeout middleware:
```yaml
http:
  middlewares:
    long-timeout:
      buffering:
        retryExpression: "IsNetworkError() && Attempts() < 3"
```

---

### Issue 4: Disk Space

**Symptom**: Push fails with "no space left on device"

**Solution**:
```bash
# Check disk space
df -h

# Clean up Docker
docker system prune -a --volumes -f

# Check again
df -h
```

---

### Issue 5: Gitea Registry Not Enabled

**Symptom**: `404 Not Found` when accessing `/v2/`

**Solution**:
```bash
# Check if registry is enabled
docker exec gitea-server cat /data/gitea/conf/app.ini | grep -A 5 "\[packages\]"

# Should show:
# [packages]
# ENABLED = true
```

If not enabled, add to `app.ini`:
```ini
[packages]
ENABLED = true
```

Restart Gitea:
```bash
docker restart gitea-server
```

---

## Debugging Steps

### Step 1: Verify Gitea Registry is Accessible

```bash
# Should return 401 Unauthorized (which is good - means registry is working)
curl -I https://gitea.harkon.co.uk/v2/

# Should return 200 OK after login
docker login gitea.harkon.co.uk
curl -u "username:token" https://gitea.harkon.co.uk/v2/
```

### Step 2: Test with Small Image

```bash
# Pull a small image
docker pull alpine:latest

# Tag it for your registry
docker tag alpine:latest gitea.harkon.co.uk/harkon/test:latest

# Try to push
docker push gitea.harkon.co.uk/harkon/test:latest
```

If this works, the issue is with large images (size limit).

### Step 3: Check Gitea Logs

```bash
# Check for errors
docker logs gitea-server --tail 100 | grep -i error

# Watch logs in real-time while pushing
docker logs -f gitea-server
```

### Step 4: Check Traefik Logs

```bash
# Check for 413 or 502 errors
docker logs traefik --tail 100 | grep -E "413|502|error"

# Watch logs in real-time
docker logs -f traefik
```

### Step 5: Check Docker Daemon Logs

```bash
# Check Docker daemon logs
sudo journalctl -u docker --since "1 hour ago" | grep -i error
```

---

## Quick Fix: Bypass Traefik for Registry

If Traefik is causing issues, you can expose Gitea's registry directly:

1. Update Gitea docker-compose to expose port 3000:
```yaml
services:
  gitea:
    ports:
      - "3000:3000"  # HTTP
```

2. Use direct connection:
```bash
docker login gitea.harkon.co.uk:3000
docker push gitea.harkon.co.uk:3000/harkon/base-ml:v1.0.1
```

**Note**: This bypasses SSL, so only use for debugging!

---

## Recommended Configuration for Large Images

### Traefik Configuration

Create `/opt/traefik/config/gitea-registry.yml`:

```yaml
http:
  middlewares:
    gitea-registry:
      buffering:
        maxRequestBodyBytes: 5368709120   # 5GB
        memRequestBodyBytes: 104857600    # 100MB in memory
        maxResponseBodyBytes: 5368709120  # 5GB
        memResponseBodyBytes: 104857600   # 100MB in memory
      
  routers:
    gitea-registry:
      rule: "Host(`gitea.harkon.co.uk`) && PathPrefix(`/v2/`)"
      entryPoints:
        - websecure
      middlewares:
        - gitea-registry
      service: gitea
      tls:
        certResolver: letsencrypt
```

### Gitea Configuration

In `/data/gitea/conf/app.ini`:

```ini
[server]
PROTOCOL = http
DOMAIN = gitea.harkon.co.uk
ROOT_URL = https://gitea.harkon.co.uk/
HTTP_PORT = 3000
LFS_MAX_FILE_SIZE = 5368709120

[repository.upload]
FILE_MAX_SIZE = 5368709120
ENABLED = true

[packages]
ENABLED = true
CHUNKED_UPLOAD_PATH = /data/gitea/tmp/package-upload
```

---

## Testing the Fix

After applying configuration changes:

1. Restart services:
```bash
docker restart traefik
docker restart gitea-server
```

2. Test with a large layer:
```bash
# Build base-ml (has large layers)
cd /home/deploy/ai-tax-agent
docker build -f infra/docker/base-ml.Dockerfile -t gitea.harkon.co.uk/harkon/base-ml:test .

# Try to push
docker push gitea.harkon.co.uk/harkon/base-ml:test
```

3. Monitor logs:
```bash
# Terminal 1: Watch Traefik
docker logs -f traefik

# Terminal 2: Watch Gitea
docker logs -f gitea-server

# Terminal 3: Push image
docker push gitea.harkon.co.uk/harkon/base-ml:test
```

---

## Alternative: Use Docker Hub or GitHub Container Registry

If Gitea continues to have issues with large images, consider:

1. **Docker Hub**: Free for public images
2. **GitHub Container Registry (ghcr.io)**: Free for public/private
3. **GitLab Container Registry**: Free tier available

These are battle-tested for large ML images and have better defaults for large uploads.