ai-tax-agent/docs/GITEA_REGISTRY_DEBUG.md

Gitea Container Registry Debugging Guide

Common Issues When Pushing Large Docker Images

Issue 1: Not Logged In

Symptom: unauthorized: authentication required

Solution:

# On remote server
docker login gitea.harkon.co.uk
# Username: blue (or your Gitea username)
# Password: <your-gitea-access-token>

---

Issue 2: Upload Size Limit (413 Request Entity Too Large)

Symptom: Push fails with 413 Request Entity Too Large or similar error

Root Cause: Traefik or Gitea has a limit on request body size

Solution A: Configure Traefik Middleware

1. Find your Traefik configuration directory:

docker inspect traefik | grep -A 10 Mounts

2. Create middleware configuration:

# Example: /opt/traefik/config/middlewares.yml
sudo tee /opt/traefik/config/middlewares.yml > /dev/null << 'EOF'
http:
  middlewares:
    large-upload:
      buffering:
        maxRequestBodyBytes: 5368709120  # 5GB
        memRequestBodyBytes: 104857600   # 100MB
        maxResponseBodyBytes: 5368709120 # 5GB
        memResponseBodyBytes: 104857600  # 100MB
EOF

3. Update Gitea container labels:

labels:
  - "traefik.http.routers.gitea.middlewares=large-upload@file"

4. Restart Traefik:

docker restart traefik

Solution B: Configure Gitea Directly

1. Edit Gitea configuration:

docker exec -it gitea-server vi /data/gitea/conf/app.ini

2. Add/modify these settings:

[server]
LFS_MAX_FILE_SIZE = 5368709120  ; 5GB

[repository.upload]
FILE_MAX_SIZE = 5368709120  ; 5GB

3. Restart Gitea:

docker restart gitea-server

---

Issue 3: Network Timeout

Symptom: Push hangs or times out after uploading for a while

Root Cause: Network instability or slow connection

Solution: Use chunked uploads or increase timeout

1. Configure Docker daemon timeout:

# Edit /etc/docker/daemon.json
sudo tee /etc/docker/daemon.json > /dev/null << 'EOF'
{
  "max-concurrent-uploads": 1,
  "max-concurrent-downloads": 3,
  "registry-mirrors": []
}
EOF

sudo systemctl restart docker

2. Or use Traefik timeout middleware:

http:
  middlewares:
    long-timeout:
      buffering:
        retryExpression: "IsNetworkError() && Attempts() < 3"

---

Issue 4: Disk Space

Symptom: Push fails with "no space left on device"

Solution:

# Check disk space
df -h

# Clean up Docker
docker system prune -a --volumes -f

# Check again
df -h

---

Issue 5: Gitea Registry Not Enabled

Symptom: 404 Not Found when accessing /v2/

Solution:

# Check if registry is enabled
docker exec gitea-server cat /data/gitea/conf/app.ini | grep -A 5 "\[packages\]"

# Should show:
# [packages]
# ENABLED = true

If not enabled, add to app.ini:

[packages]
ENABLED = true

Restart Gitea:

docker restart gitea-server

---

Debugging Steps

Step 1: Verify Gitea Registry is Accessible

# Should return 401 Unauthorized (which is good - means registry is working)
curl -I https://gitea.harkon.co.uk/v2/

# Should return 200 OK after login
docker login gitea.harkon.co.uk
curl -u "username:token" https://gitea.harkon.co.uk/v2/

Step 2: Test with Small Image

# Pull a small image
docker pull alpine:latest

# Tag it for your registry
docker tag alpine:latest gitea.harkon.co.uk/harkon/test:latest

# Try to push
docker push gitea.harkon.co.uk/harkon/test:latest

If this works, the issue is with large images (size limit).

Step 3: Check Gitea Logs

# Check for errors
docker logs gitea-server --tail 100 | grep -i error

# Watch logs in real-time while pushing
docker logs -f gitea-server

Step 4: Check Traefik Logs

# Check for 413 or 502 errors
docker logs traefik --tail 100 | grep -E "413|502|error"

# Watch logs in real-time
docker logs -f traefik

Step 5: Check Docker Daemon Logs

# Check Docker daemon logs
sudo journalctl -u docker --since "1 hour ago" | grep -i error

---

Quick Fix: Bypass Traefik for Registry

If Traefik is causing issues, you can expose Gitea's registry directly:

1. Update Gitea docker-compose to expose port 3000:

services:
  gitea:
    ports:
      - "3000:3000"  # HTTP

2. Use direct connection:

docker login gitea.harkon.co.uk:3000
docker push gitea.harkon.co.uk:3000/harkon/base-ml:v1.0.1

Note: This bypasses SSL, so only use for debugging!

---

Recommended Configuration for Large Images

Traefik Configuration

Create /opt/traefik/config/gitea-registry.yml:

http:
  middlewares:
    gitea-registry:
      buffering:
        maxRequestBodyBytes: 5368709120   # 5GB
        memRequestBodyBytes: 104857600    # 100MB in memory
        maxResponseBodyBytes: 5368709120  # 5GB
        memResponseBodyBytes: 104857600   # 100MB in memory
      
  routers:
    gitea-registry:
      rule: "Host(`gitea.harkon.co.uk`) && PathPrefix(`/v2/`)"
      entryPoints:
        - websecure
      middlewares:
        - gitea-registry
      service: gitea
      tls:
        certResolver: letsencrypt

Gitea Configuration

In /data/gitea/conf/app.ini:

[server]
PROTOCOL = http
DOMAIN = gitea.harkon.co.uk
ROOT_URL = https://gitea.harkon.co.uk/
HTTP_PORT = 3000
LFS_MAX_FILE_SIZE = 5368709120

[repository.upload]
FILE_MAX_SIZE = 5368709120
ENABLED = true

[packages]
ENABLED = true
CHUNKED_UPLOAD_PATH = /data/gitea/tmp/package-upload

---

Testing the Fix

After applying configuration changes:

1. Restart services:

docker restart traefik
docker restart gitea-server

2. Test with a large layer:

# Build base-ml (has large layers)
cd /home/deploy/ai-tax-agent
docker build -f infra/docker/base-ml.Dockerfile -t gitea.harkon.co.uk/harkon/base-ml:test .

# Try to push
docker push gitea.harkon.co.uk/harkon/base-ml:test

3. Monitor logs:

# Terminal 1: Watch Traefik
docker logs -f traefik

# Terminal 2: Watch Gitea
docker logs -f gitea-server

# Terminal 3: Push image
docker push gitea.harkon.co.uk/harkon/base-ml:test

---

Alternative: Use Docker Hub or GitHub Container Registry

If Gitea continues to have issues with large images, consider:

1. Docker Hub: Free for public images
2. GitHub Container Registry (ghcr.io): Free for public/private
3. GitLab Container Registry: Free tier available

These are battle-tested for large ML images and have better defaults for large uploads.