Initial commit
Some checks failed
CI/CD Pipeline / Code Quality & Linting (push) Has been cancelled
CI/CD Pipeline / Policy Validation (push) Has been cancelled
CI/CD Pipeline / Test Suite (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-coverage) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-extract) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-firm-connectors) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-forms) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-hmrc) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-ingestion) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-kg) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-normalize-map) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-ocr) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-rag-indexer) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-rag-retriever) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-reason) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (svc-rpa) (push) Has been cancelled
CI/CD Pipeline / Build Docker Images (ui-review) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (svc-coverage) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (svc-extract) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (svc-kg) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (svc-rag-retriever) (push) Has been cancelled
CI/CD Pipeline / Security Scanning (ui-review) (push) Has been cancelled
CI/CD Pipeline / Generate SBOM (push) Has been cancelled
CI/CD Pipeline / Deploy to Staging (push) Has been cancelled
CI/CD Pipeline / Deploy to Production (push) Has been cancelled
CI/CD Pipeline / Notifications (push) Has been cancelled

This commit is contained in:
harkon
2025-10-11 08:41:36 +01:00
commit b324ff09ef
276 changed files with 55220 additions and 0 deletions

305
docs/VM.md Normal file
View File

@@ -0,0 +1,305 @@
# VM Setup
# 0) One-time VM prep (as root just this once)
SSH to the VM your provider gave you (often only root works initially):
```bash
ssh root@<VM_IP>
```
Create a non-root deploy user with sudo, and lock down SSH:
```bash
# create user
adduser deploy
usermod -aG sudo deploy
# add your SSH key
mkdir -p /home/deploy/.ssh
chmod 700 /home/deploy/.ssh
nano /home/deploy/.ssh/authorized_keys # paste your public key
chmod 600 /home/deploy/.ssh/authorized_keys
chown -R deploy:deploy /home/deploy/.ssh
# harden SSH (optional but recommended)
sed -i 's/^#\?PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
sed -i 's/^#\?PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl reload sshd
exit
```
Now reconnect as your non-root user:
```bash
ssh deploy@<VM_IP>
```
# 1) Firewall and basics
```bash
# Ubuntu/Debian
sudo apt update
sudo apt install -y ufw
# allow SSH + web
sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable
sudo ufw status
```
# 2) Install Docker Engine + Compose plugin (non-root usage)
```bash
# Docker official repo
sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release; echo $VERSION_CODENAME) stable" \
| sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
# let your user run docker without sudo
sudo usermod -aG docker $USER
newgrp docker
# optional: limit container logs
echo '{"log-driver":"json-file","log-opts":{"max-size":"10m","max-file":"3"}}' | \
sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
```
# 3) Layout for your Compose stacks
Well keep everything under `/opt/compose`, owned by `deploy`:
```bash
sudo mkdir -p /opt/compose/{traefik,portainer,gitea,authentik}
sudo chown -R deploy:deploy /opt/compose
```
Create the shared external Docker network (once):
```bash
docker network create proxy
```
# 4) Copy your compose files (no root, via scp/rsync)
From your **local** machine:
```bash
# example: copy a whole folder into /opt/compose/portainer
scp -r ./portainer/* deploy@<VM_IP>:/opt/compose/portainer/
# or use rsync (recommended)
rsync -avz ./gitea/ deploy@<VM_IP>:/opt/compose/gitea/
```
# 5) Traefik on the VM (HTTP-01 with Lets Encrypt)
On the VM:
```bash
cd /opt/compose/traefik
```
Create `compose.yml`:
```yaml
version: "3.9"
services:
traefik:
image: traefik:v3.1
restart: unless-stopped
command:
- --providers.docker=true
- --providers.docker.exposedByDefault=false
- --entrypoints.web.address=:80
- --entrypoints.websecure.address=:443
- --entrypoints.web.http.redirections.entryPoint.to=websecure
- --entrypoints.web.http.redirections.entryPoint.scheme=https
# Let's Encrypt (HTTP-01 challenge)
- --certificatesresolvers.le.acme.email=${LE_EMAIL}
- --certificatesresolvers.le.acme.storage=/letsencrypt/acme.json
- --certificatesresolvers.le.acme.httpchallenge=true
- --certificatesresolvers.le.acme.httpchallenge.entrypoint=web
# Optional dashboard (protect later)
- --api.dashboard=true
ports:
- "80:80"
- "443:443"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./letsencrypt:/letsencrypt
networks:
- proxy
labels:
- traefik.enable=true
- traefik.http.routers.traefik.rule=Host(`traefik.YOURDOMAIN.com`)
- traefik.http.routers.traefik.entrypoints=websecure
- traefik.http.routers.traefik.tls.certresolver=le
- traefik.http.routers.traefik.service=api@internal
networks:
proxy:
external: true
```
Create the storage file and set strict perms:
```bash
mkdir -p /opt/compose/traefik/letsencrypt
touch /opt/compose/traefik/letsencrypt/acme.json
chmod 600 /opt/compose/traefik/letsencrypt/acme.json
```
Create `.env`:
```bash
echo "LE_EMAIL=you@example.com" > /opt/compose/traefik/.env
```
Bring it up:
```bash
cd /opt/compose/traefik
docker compose up -d
```
# 6) DNS records on GoDaddy
Point your domain/subdomains to the VMs **public IP**:
- `A @ -> <VM_IP>`
- `A traefik -> <VM_IP>`
- `A portainer -> <VM_IP>`
- `A git -> <VM_IP>`
- `A auth -> <VM_IP>`
(HTTP-01 will fetch per-host certs automatically the first time you visit each hostname.)
> If you want a **wildcard** (`*.example.com`), switch Traefik to **DNS-01** with your DNS providers API. GoDaddys API can be restrictive; moving DNS hosting to Cloudflare is common. But HTTP-01 works fine for named subdomains.
# 7) Example app stacks (all non-root)
## Portainer (behind Traefik)
`/opt/compose/portainer/compose.yml`
```yaml
version: "3.9"
services:
portainer:
image: portainer/portainer-ce:latest
restart: unless-stopped
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- portainer_data:/data
networks:
- proxy
labels:
- traefik.enable=true
- traefik.http.routers.portainer.rule=Host(`portainer.YOURDOMAIN.com`)
- traefik.http.routers.portainer.entrypoints=websecure
- traefik.http.routers.portainer.tls.certresolver=le
- traefik.http.services.portainer.loadbalancer.server.port=9000
volumes:
portainer_data:
networks:
proxy:
external: true
```
Deploy:
```bash
cd /opt/compose/portainer
docker compose up -d
```
## Gitea (behind Traefik)
`/opt/compose/gitea/compose.yml`
```yaml
version: "3.9"
services:
gitea:
image: gitea/gitea:1
restart: unless-stopped
environment:
- USER_UID=1000
- USER_GID=1000
volumes:
- gitea_data:/data
networks:
- proxy
labels:
- traefik.enable=true
- traefik.http.routers.gitea.rule=Host(`git.YOURDOMAIN.com`)
- traefik.http.routers.gitea.entrypoints=websecure
- traefik.http.routers.gitea.tls.certresolver=le
- traefik.http.services.gitea.loadbalancer.server.port=3000
volumes:
gitea_data:
networks:
proxy:
external: true
```
(Do the same for Authentik; keep it on `proxy` and add Traefik labels to the web service.)
# 8) Secure the Traefik dashboard (quick basic-auth)
Create a middleware once and attach it to the dashboard router.
Generate a bcrypt hash (on your laptop):
```bash
# Install apache2-utils if you have it, or use Docker to generate:
docker run --rm httpd:2.4-alpine htpasswd -nbB admin 'YOUR_STRONG_PASSWORD'
# Output looks like: admin:$2y$05$....
```
Add to Traefik labels:
```yaml
labels:
- traefik.enable=true
- traefik.http.middlewares.basicauth.basicauth.users=admin:$$2y$$05$$<HASH_REST>
- traefik.http.routers.traefik.rule=Host(`traefik.YOURDOMAIN.com`)
- traefik.http.routers.traefik.entrypoints=websecure
- traefik.http.routers.traefik.tls.certresolver=le
- traefik.http.routers.traefik.middlewares=basicauth@docker
- traefik.http.routers.traefik.service=api@internal
```
Then:
```bash
cd /opt/compose/traefik && docker compose up -d
```
# 9) Quality-of-life tips
- Containers should include `restart: unless-stopped`; Docker will auto-start them on reboot—no systemd unit needed.
- Keep everything on the `proxy` network; only Traefik publishes 80/443 to the host.
- For updates: `docker compose pull && docker compose up -d` per stack.
- Backups: snapshot `/opt/compose/*` and any named volumes (`/var/lib/docker/volumes/...`), or mount volumes to known paths you can back up.
---
If you want, paste your existing Traefik/Authentik/Gitea labels here and Ill adapt them for the VM layout (and wire Authentik as forward-auth to protect Portainer/Gitea).