Initial commit

docs/GITEA_REGISTRY_DEBUG.md

@@ -0,0 +1,332 @@
+# Gitea Container Registry Debugging Guide
+
+## Common Issues When Pushing Large Docker Images
+
+### Issue 1: Not Logged In
+
+**Symptom**: `unauthorized: authentication required`
+
+**Solution**:
+```bash
+# On remote server
+docker login gitea.harkon.co.uk
+# Username: blue (or your Gitea username)
+# Password: <your-gitea-access-token>
+```
+
+---
+
+### Issue 2: Upload Size Limit (413 Request Entity Too Large)
+
+**Symptom**: Push fails with `413 Request Entity Too Large` or similar error
+
+**Root Cause**: Traefik or Gitea has a limit on request body size
+
+**Solution A: Configure Traefik Middleware**
+
+1. Find your Traefik configuration directory:
+```bash
+docker inspect traefik | grep -A 10 Mounts
+```
+
+2. Create middleware configuration:
+```bash
+# Example: /opt/traefik/config/middlewares.yml
+sudo tee /opt/traefik/config/middlewares.yml > /dev/null << 'EOF'
+http:
+  middlewares:
+    large-upload:
+      buffering:
+        maxRequestBodyBytes: 5368709120  # 5GB
+        memRequestBodyBytes: 104857600   # 100MB
+        maxResponseBodyBytes: 5368709120 # 5GB
+        memResponseBodyBytes: 104857600  # 100MB
+EOF
+```
+
+3. Update Gitea container labels:
+```yaml
+labels:
+  - "traefik.http.routers.gitea.middlewares=large-upload@file"
+```
+
+4. Restart Traefik:
+```bash
+docker restart traefik
+```
+
+**Solution B: Configure Gitea Directly**
+
+1. Edit Gitea configuration:
+```bash
+docker exec -it gitea-server vi /data/gitea/conf/app.ini
+```
+
+2. Add/modify these settings:
+```ini
+[server]
+LFS_MAX_FILE_SIZE = 5368709120  ; 5GB
+
+[repository.upload]
+FILE_MAX_SIZE = 5368709120  ; 5GB
+```
+
+3. Restart Gitea:
+```bash
+docker restart gitea-server
+```
+
+---
+
+### Issue 3: Network Timeout
+
+**Symptom**: Push hangs or times out after uploading for a while
+
+**Root Cause**: Network instability or slow connection
+
+**Solution**: Use chunked uploads or increase timeout
+
+1. Configure Docker daemon timeout:
+```bash
+# Edit /etc/docker/daemon.json
+sudo tee /etc/docker/daemon.json > /dev/null << 'EOF'
+{
+  "max-concurrent-uploads": 1,
+  "max-concurrent-downloads": 3,
+  "registry-mirrors": []
+}
+EOF
+
+sudo systemctl restart docker
+```
+
+2. Or use Traefik timeout middleware:
+```yaml
+http:
+  middlewares:
+    long-timeout:
+      buffering:
+        retryExpression: "IsNetworkError() && Attempts() < 3"
+```
+
+---
+
+### Issue 4: Disk Space
+
+**Symptom**: Push fails with "no space left on device"
+
+**Solution**:
+```bash
+# Check disk space
+df -h
+
+# Clean up Docker
+docker system prune -a --volumes -f
+
+# Check again
+df -h
+```
+
+---
+
+### Issue 5: Gitea Registry Not Enabled
+
+**Symptom**: `404 Not Found` when accessing `/v2/`
+
+**Solution**:
+```bash
+# Check if registry is enabled
+docker exec gitea-server cat /data/gitea/conf/app.ini | grep -A 5 "\[packages\]"
+
+# Should show:
+# [packages]
+# ENABLED = true
+```
+
+If not enabled, add to `app.ini`:
+```ini
+[packages]
+ENABLED = true
+```
+
+Restart Gitea:
+```bash
+docker restart gitea-server
+```
+
+---
+
+## Debugging Steps
+
+### Step 1: Verify Gitea Registry is Accessible
+
+```bash
+# Should return 401 Unauthorized (which is good - means registry is working)
+curl -I https://gitea.harkon.co.uk/v2/
+
+# Should return 200 OK after login
+docker login gitea.harkon.co.uk
+curl -u "username:token" https://gitea.harkon.co.uk/v2/
+```
+
+### Step 2: Test with Small Image
+
+```bash
+# Pull a small image
+docker pull alpine:latest
+
+# Tag it for your registry
+docker tag alpine:latest gitea.harkon.co.uk/harkon/test:latest
+
+# Try to push
+docker push gitea.harkon.co.uk/harkon/test:latest
+```
+
+If this works, the issue is with large images (size limit).
+
+### Step 3: Check Gitea Logs
+
+```bash
+# Check for errors
+docker logs gitea-server --tail 100 | grep -i error
+
+# Watch logs in real-time while pushing
+docker logs -f gitea-server
+```
+
+### Step 4: Check Traefik Logs
+
+```bash
+# Check for 413 or 502 errors
+docker logs traefik --tail 100 | grep -E "413|502|error"
+
+# Watch logs in real-time
+docker logs -f traefik
+```
+
+### Step 5: Check Docker Daemon Logs
+
+```bash
+# Check Docker daemon logs
+sudo journalctl -u docker --since "1 hour ago" | grep -i error
+```
+
+---
+
+## Quick Fix: Bypass Traefik for Registry
+
+If Traefik is causing issues, you can expose Gitea's registry directly:
+
+1. Update Gitea docker-compose to expose port 3000:
+```yaml
+services:
+  gitea:
+    ports:
+      - "3000:3000"  # HTTP
+```
+
+2. Use direct connection:
+```bash
+docker login gitea.harkon.co.uk:3000
+docker push gitea.harkon.co.uk:3000/harkon/base-ml:v1.0.1
+```
+
+**Note**: This bypasses SSL, so only use for debugging!
+
+---
+
+## Recommended Configuration for Large Images
+
+### Traefik Configuration
+
+Create `/opt/traefik/config/gitea-registry.yml`:
+
+```yaml
+http:
+  middlewares:
+    gitea-registry:
+      buffering:
+        maxRequestBodyBytes: 5368709120   # 5GB
+        memRequestBodyBytes: 104857600    # 100MB in memory
+        maxResponseBodyBytes: 5368709120  # 5GB
+        memResponseBodyBytes: 104857600   # 100MB in memory
+      
+  routers:
+    gitea-registry:
+      rule: "Host(`gitea.harkon.co.uk`) && PathPrefix(`/v2/`)"
+      entryPoints:
+        - websecure
+      middlewares:
+        - gitea-registry
+      service: gitea
+      tls:
+        certResolver: letsencrypt
+```
+
+### Gitea Configuration
+
+In `/data/gitea/conf/app.ini`:
+
+```ini
+[server]
+PROTOCOL = http
+DOMAIN = gitea.harkon.co.uk
+ROOT_URL = https://gitea.harkon.co.uk/
+HTTP_PORT = 3000
+LFS_MAX_FILE_SIZE = 5368709120
+
+[repository.upload]
+FILE_MAX_SIZE = 5368709120
+ENABLED = true
+
+[packages]
+ENABLED = true
+CHUNKED_UPLOAD_PATH = /data/gitea/tmp/package-upload
+```
+
+---
+
+## Testing the Fix
+
+After applying configuration changes:
+
+1. Restart services:
+```bash
+docker restart traefik
+docker restart gitea-server
+```
+
+2. Test with a large layer:
+```bash
+# Build base-ml (has large layers)
+cd /home/deploy/ai-tax-agent
+docker build -f infra/docker/base-ml.Dockerfile -t gitea.harkon.co.uk/harkon/base-ml:test .
+
+# Try to push
+docker push gitea.harkon.co.uk/harkon/base-ml:test
+```
+
+3. Monitor logs:
+```bash
+# Terminal 1: Watch Traefik
+docker logs -f traefik
+
+# Terminal 2: Watch Gitea
+docker logs -f gitea-server
+
+# Terminal 3: Push image
+docker push gitea.harkon.co.uk/harkon/base-ml:test
+```
+
+---
+
+## Alternative: Use Docker Hub or GitHub Container Registry
+
+If Gitea continues to have issues with large images, consider:
+
+1. **Docker Hub**: Free for public images
+2. **GitHub Container Registry (ghcr.io)**: Free for public/private
+3. **GitLab Container Registry**: Free tier available
+
+These are battle-tested for large ML images and have better defaults for large uploads.
+