Architecture & operations reference¶

Look-up material for a running deployment. The system diagram lives in the top-level README; per-service internals live in each service's doc (linked from there).

Services & ports¶

Public (exposed on master):

Service	Port	Purpose
Traefik	80 / 443	Public HTTPS front door (k3s ingress) — Let's Encrypt TLS; routes `platform.<domain>` → Platform, `app.<domain>` → Web UI
Platform	8100	API entrypoint — every SDK call goes here, downstream services proxied through it
Web UI	3000	Dashboard (served via Traefik once HTTPS is set up)
kuberay-service	8800	Worker `join-node.sh` calls `/v1/nodes/validate` pre-tunnel
K3s API	6443	Worker `k3s-agent`
WireGuard	51820/udp	Encrypted tunnel endpoint

Internal-only (not publicly exposed):

Service	Port	Purpose
Job Gateway	8200	Job submission & VRAM-aware scheduling (stateless — job/endpoint state lives in CRDs)
Metering	8300	Usage tracking & billing calc
Credit	8400	Balances, mint/deduct, transfers
Faucet	8500	Email verify, onboarding, credit refill
Postgres	5432	One instance; per-service databases: `platform`, `credits`, `metering`, `faucet`, `chain`
CometBFT	26657	Append-only audit-chain RPC (`chain_service`; active when `CHAIN_ENABLED=true`)
MinIO	9000 / 9001	S3 API / console (reached over WG from workers)
Registry	30500	Platform image registry (NodePort, reached over WG — see Image registry TLS)
Ray Head	30001 / 30002 / 30003	Ray client / dashboard / Serve (NodePort, reached over WG)
VictoriaLogs / Prometheus	9428 / 9090	Logs & metrics behind the admin Diagnostics console

Networking¶

The master needs these ports reachable publicly:

Port	Used by	Why
`6443/tcp`	Worker K3s agent	K3s API
`8800/tcp`	Worker join	`/v1/nodes/validate` is called before the WG tunnel exists, so it can't go through WG
`51820/udp`	WireGuard	Tunnel endpoint
`3000/tcp`	Users	WebUI
`8100/tcp`	Users / SDK	Platform API (every SDK call goes here; job-gateway, metering, credit, faucet, and MinIO are proxied through it)

GPU workers need no inbound ports.

deploy.sh opens the OS firewall (ufw) on every box automatically and, at the end, prints the exact host:port list to allow in your cloud firewall / security-group. In a multi-VM deploy, a public service placed on an extra machine (e.g. the dashboard) is reachable on that machine's IP — the summary tells you which.

deploy.sh brings WireGuard up on the master; join-node.sh brings it up on each worker, using the master's pubkey + the VPN IP returned by /v1/nodes/validate. K3s pod-to-pod traffic (flannel/VXLAN) is pinned to wg0 via flannel-iface.

Expected times¶

deploy.sh takes ~10–15 minutes on a fresh master:

Step	Time
K3s install + start	~1 min
Docker services build + start	~3 min
WireGuard setup	~1 min
Helm + KubeRay + device plugins	~2 min
Ray head image build (`gridweave-ray-head:latest`)	~3–5 min

GPU worker join takes ~30–40 minutes per node on a fresh install (disk-bound):

Step	NVIDIA	AMD
Prereqs, WG tunnel, K3s agent	~2 min	~2 min
Worker image build (`gridweave-worker:latest`)	~10 min (torch CUDA + HF libs, ~8 GB export)	~15–20 min (torch ROCm, slower disk)
vLLM image build (`gridweave-ray-vllm-{nvidia,amd}:latest`)	~15 min (~10 min pip install + ~5 min layer export)	~10–15 min (experimental, may fail)
Register with cluster	~30s	~30s

Re-running join-node.sh is idempotent — it skips any image already present in containerd (REBUILD=1 to force). A re-join preserves buildkit cache, so re-builds are fast (~3–5 min). AMD vLLM support is experimental (upstream vLLM ROCm is newer than CUDA).

Storage¶

GPU worker images are large: ~8 GB for gridweave-worker:latest (CUDA/ROCm + torch + HF libs), plus ~10 GB for gridweave-ray-vllm-{nvidia,amd}:latest. The BuildKit cache persists across runs (capped at 50 GB by BuildKit's own GC) so re-joins after image eviction rehydrate layers locally instead of re-downloading ~4 GB of torch. Set CLEAN_BUILDKIT=1 on a join to wipe the cache. If your GPU node has a secondary disk, mount it at /mnt/data before running join-node.sh:

sudo mkdir -p /mnt/data
sudo mount /dev/sdb1 /mnt/data
echo '/dev/sdb1 /mnt/data ext4 defaults 0 0' | sudo tee -a /etc/fstab

When /mnt/data is present and >100 GB, join-node.sh:

Adds data-dir: /mnt/data/k3s to the K3s agent config (final image storage lives there).
Uses /mnt/data/buildkit for the BuildKit cache (persistent, GC-capped at 50 GB).
Creates the model cache at /mnt/data/model-cache and symlinks /data/model-cache to it (endpoint pods hostPath-mount /data/model-cache; without a data disk it's created directly). No manual setup needed — the script handles both layouts.

The K3s data-dir choice is a one-time decision — if K3s is already installed at /var/lib/rancher/k3s, it stays there.

Bring your own S3¶

By default Gridweave runs its own MinIO for object storage. To use an external S3 instead (AWS S3, Cloudflare R2, …), set S3_ENDPOINT, S3_ACCESS_KEY, and S3_SECRET_KEY in deploy/.env before deploying — the system uses yours and skips MinIO.

Image registry TLS¶

Custom-endpoint images (spec.image) live in the platform's own container registry (see docs/image_registry). It's exposed on NodePort 30500 and reached by nodes over the WireGuard mesh at MASTER_VPN_IP:30500 — no public exposure needed for pulls. Users push to the same address over the mesh, or to a public registry.<domain> in prod.

If you put registry.<domain> behind Cloudflare, the DNS record MUST be DNS-only (grey cloud), not proxied (orange cloud). Docker pushes each image layer as one large monolithic PATCH body; Cloudflare's proxied path caps request bodies (~100 MB) and GPU image layers routinely exceed that, so a proxied record fails the push mid-layer with a 413/reset. Terminate TLS at the master (Caddy / Let's Encrypt / Traefik) on a grey-cloud record instead. This does not apply to the mesh push path (plain HTTP over WireGuard), which is already encrypted.

Tuning knobs in .env (all optional): REGISTRY_QUOTA_GB (per-user cap, default 20), REGISTRY_GC_INTERVAL_HOURS (default weekly), REGISTRY_NODEPORT, REGISTRY_S3_BUCKET. The quota is also editable live in the web UI (Configuration → Image Registry) when it isn't pinned in .env.

Files¶

core/
  deploy/
    deploy.sh                    # Master setup + multi-VM fan-out (wizard) — recreates a clean slate every run
    update.sh                    # Keep-everything re-run: pull new code + rebuild (no data loss)
    join-node.sh                 # GPU node join (single key, auto-everything)
    setup-https.sh               # Traefik + Let's Encrypt routes for app/platform domains
    .env.example                 # Config template
  docker/
    deploy/docker-compose.yml    # All services
    kuberay_service/             # KubeRay API
    ray-worker/Dockerfile        # Worker image (built on each node)
  kuberay_service/
    routers/nodes.py             # Join key + validate + register endpoints

Developer shortcut — one-container rebuild¶

For Python-only changes to job_gateway/ or platform_service/, skip the full update: copy the files and rebuild the one container.

scp job_gateway/*.py root@<master>:/tmp/core/job_gateway/
ssh root@<master> 'cd /tmp/core/docker/deploy && docker compose --env-file /tmp/core/deploy/.env up -d --build --no-deps job-gateway'

Removing Gridweave¶

Only for permanently removing Gridweave from a machine (decommissioning, repurposing). For retrying a failed join, just re-run join-node.sh — it's idempotent.

Worker — use the script. leave-node.sh deregisters the node from the master (WireGuard peer + Ray worker group), stops the k3s agent, and brings wg0 down. By default it keeps the built images so a re-join is fast; --uninstall also removes k3s, containerd, and all images:

sudo /opt/gridweave/deploy/leave-node.sh --uninstall
# BuildKit + nerdctl live outside k3s, so the uninstaller doesn't touch them.
# Scrub them too when decommissioning for good:
sudo rm -rf /var/lib/buildkit /mnt/data/buildkit /etc/buildkit /run/buildkit
sudo rm -rf /usr/local/bin/nerdctl /usr/local/bin/buildkitd /usr/local/bin/buildctl

Master. There is no master uninstall script — deploy.sh clean-slates the machine on every run, so re-deploying never needs one. To remove Gridweave from a master for good:

sudo helm uninstall kuberay-operator -n gridweave
sudo kubectl delete namespace gridweave
sudo /usr/local/bin/k3s-uninstall.sh