Skip to content

Image Registry — shared, platform-hosted, token-authed

Custom-endpoint images pushed with plain docker push (full layer dedup), served from one in-cluster registry, with no public-registry dependency and no per-user setup — your existing platform token is your registry login. Access is enforced by docker token auth, not network trust: you can only touch images under your own namespace.

Status: built and unit-tested (shared registry lifecycle + GC, token issuer, quota, node pull auth, deploy integration, SDK). Live-verify checklist in §10. File map: image_registry/.

1. Why this exists, and why it's shared

A custom endpoint (spec={"image": ...}) used to require a public registry (Docker Hub, etc.) — wrong for a self-hosted GPU platform. We want in-house images with docker-native UX (dedup, multi-arch) and zero external dependency.

An earlier design gave each user their own registry:2 pod backed by their own S3 bucket. We rejected it — the reasoning is the crux of this feature:

Per-user registry (rejected) Shared registry (this)
Storage of the multi-GB worker-base layers once per user (each bucket re-stores them) once, globally (dedup across all users)
Onboarding user must supply an S3 bucket + keys nothing — push immediately
Credentials in flight S3 keys travel laptop→platform→kuberay→Secret→pod none exist — no user S3 creds anywhere
Pods at 100 users 100+ idle registry:2 pods (eats the node pod budget) 1
Access isolation ❌ none (router proxies any namespace) ✅ token auth per namespace

The deciding insight: a registry is a stateless protocol server in front of a storage backend, and one backend can serve everyone when repos are namespaced by path (<ns_id>/repo). Dedup then makes the shared bucket cheaper in total than N private buckets, because the base layers every image is FROM are stored once instead of N times. Isolation, which per-user pods never actually gave, is provided by real registry auth.

The platform's side of the trade: it now owns image storage (bounded by a per-user quota) and garbage collection. Both are handled here.

2. Architecture

 laptop ── docker push ─▶ registry (1 shared registry:2 pod) ─▶ platform MinIO
   (TLS in prod)          NodePort MASTER_VPN_IP:30500          gridweave-registry bucket
                          token auth ▲                          global layer dedup
 GPU node ─ containerd ───pull───────┘
   (over WireGuard mesh)             │  401 + realm
                                     ▼
                          platform_service /v1/registry/token
                          (RS256 JWT scoped to <ns_id>/…, verified by the
                           registry against a cert it was handed at ensure)

Three moving parts:

  • registry (kuberay_service) — one registry:2 Deployment + NodePort Service, storage backend = platform MinIO. Runs docker token auth with a cert whose private key lives in platform_service. Scaled to zero only during GC. No per-user objects, ever.
  • token issuer (platform_service) — the docker auth realm. Verifies the caller's platform credential and signs a short-lived JWT granting exactly the scopes they're allowed (own namespace push/pull/delete, shared repos pull, nodes pull-any, admin all). Also walks the registry to compute usage for quota and drives GC.
  • platform MinIO — the one place image bytes live (gridweave-registry bucket). No lifecycle expiry; only GC deletes blobs.

Nodes trust exactly one address, set once at join, and carry a pull-only token that containerd presents automatically.

3. Data flow

3a. Push (zero setup)

  1. gridweave.push_image("myapp:v1") → SDK runs docker login <host> with the caller's platform token (via stdin), then docker tag + docker push <host>/<ns_id>/myapp:v1.
  2. Registry 401s with the realm → docker fetches a token from platform_service, which grants push,pull on <ns_id>/myapp (unless over quota → push withheld) → docker retries and streams only new layers to MinIO.

3b. Deploy

  1. serve(spec={"image": "<host>/<ns_id>/myapp:v1", "port": 8080}).
  2. Gateway checks the ref is in the caller's namespace (or a shared platform repo) and the registry is ready, then swaps in the endpoint worker (existing custom-image path, imagePullPolicy: Always).
  3. Node's containerd pulls via its pull-only token — registry → MinIO, only layers the node lacks. Everything downstream (proxy sidecar, GPU, billing) unchanged.

4. Components & responsibilities

Component Where Responsibility
Registry lifecycle + GC job kuberay_service/routers/registry.py (single k8s writer) one registry:2 Deployment+Service+Secret+ConfigMap; ensure/status/gc/delete
Token issuer + quota + GC orchestration platform_service/registry.py sign RS256 auth tokens, walk usage, prune untagged, schedule GC
Registry HTTP API platform_service/routers/registry.py /v1/registry/token (realm), /config, DELETE /image, POST /gc
Deploy integration job_gateway/serve.py rewrite/verify spec.image, ensure registry ready, pull policy
Node trust + pull auth deploy/join-node.sh write registries.yaml with the node's pull token
SDK client_sdk/registry.py push_image(), registry_login(), delete_image(), registry_status()

Single-writer rule: only kuberay_service mutates k8s. platform_service calls it over HTTP (admin token) — never touches k8s directly.

Naming

  • Registry objects: gridweave-registry (Deployment/Service), gridweave-registry-secrets, gridweave-registry-config · namespace gridweave.
  • Image ref: <host>/<ns_id>/<repo>:<tag> where ns_id = <sanitized-user>-<8 hex>.
  • Shared platform repos: <host>/gridweave/<repo> (e.g. worker-base) — every account may pull, only admin may push. ns_id always ends in an 8-hex-digest segment, so no user namespace can ever collide with gridweave/.

5. API & interfaces

HTTP (platform)

Method Path Effect
GET /v1/registry/token docker auth realm. HTTP Basic (password = any platform token); returns an RS256 JWT scoped to the caller's grants.
GET /v1/registry/config {ns_id, registry_host, state, usage_bytes, quota_bytes, push_example}. No secrets, no setup.
DELETE /v1/registry/image?ref= delete a tag's manifest from the caller's namespace (blobs reclaimed by GC).
POST /v1/registry/gc (admin) prune untagged manifests + run the blob sweep now.

kuberay-service (internal, admin-token)

Method Path Effect
POST /v1/registry/ensure create/roll the registry with the auth cert.
GET /v1/registry/status {state: absent\|starting\|ready, node_port}.
POST /v1/registry/gc scale to 0 → registry garbage-collect Job → scale to 1.
DELETE /v1/registry tear down the registry objects (S3 blobs untouched).

SDK

gridweave.auth("gw_...")                      # your normal platform token
ref = gridweave.push_image("myapp:v1")        # docker login+tag+push → returns ref
gridweave.serve(spec={"image": ref, "port": 8080}, name="myapp")
gridweave.registry_status()                   # {ns_id, state, usage_bytes, quota_bytes, ...}
gridweave.delete_image("myapp:v1")            # remove a tag; GC reclaims the bytes
push_image refuses over quota and surfaces docker errors verbatim (adds an insecure-registries hint on the HTTP/HTTPS mismatch).

Image-ref contract

spec.image accepts a normal public ref (unchanged) or a ref whose host is this platform's registry — the gateway detects the latter, checks ownership + readiness, and pulls fresh.

6. Security model

Access control — docker token auth (not network trust). The registry (auth.token in its config) rejects every request lacking a valid JWT and redirects to the realm. platform_service signs those JWTs with an RS256 key; the registry verifies them against the matching cert it was handed at ensure (mounted as rootcertbundle). Grants (platform_service/registry.py resolve_access):

Caller Grant
user / provider push+pull+delete on <ns_id>/*; pull on gridweave/*; push withheld while over quota
worker node (node-puller) pull on any repo (nodes run every user's containers)
admin everything, incl. registry:catalog (the usage walker)
foreign namespace empty grant — request denied at the registry

A cross-user pull/push is refused at the registry, at the node, and fast-failed at the gateway — three independent layers.

The JWT kid. distribution keys its cert bundle by the libtrust fingerprint of the public key; the token's kid header must equal it exactly or every login fails. Computed in registry.libtrust_kid (base32 of the first 240 bits of SHA-256 over the DER SubjectPublicKeyInfo, colon-grouped by four) and covered by an end-to-end signing test.

Credentials. - No user S3 credentials exist — the backend is the platform's own storage. The entire class of "user S3 key leaks" is deleted, not mitigated. - The signing private key lives in one Postgres row (registry_signing_key); only the cert ever leaves platform_service (to kuberay, into the pod). - The registry's S3 access uses the platform's existing root creds via secretKeyRef — never inline in a pod spec. - The node pull token is a long-lived node-puller JWT written to /etc/rancher/k3s/registries.yaml (chmod 600); a re-join refreshes it.

Transport

Leg Dev/test Prod
Push (laptop → registry) HTTP over the mesh :30500 TLS at registry.<domain> (Caddy/Traefik) — the one leg that must be encrypted
Pull (node → registry) HTTP over WireGuard mesh (already encrypted) same
Registry → MinIO in-cluster in-cluster / TLS to external S3

Prod push over a proxied CDN will break. Docker uploads layers as large monolithic PATCH bodies; Cloudflare's proxied (orange-cloud) path caps request bodies (~100 MB) and GPU layers exceed that. registry.<domain> must be a DNS-only (grey-cloud) A record with Caddy/Let's Encrypt terminating TLS — or push over the VPN. See Image registry TLS.

Multi-arch — a real registry serves manifest lists: push linux/amd64 + linux/arm64 and each node pulls its match. Prune/GC are manifest-list-aware (children are kept), so this stays safe across GC.

7. Configuration

Var Service Meaning
REGISTRY_HOST gateway host in a platform spec.image ref (MASTER_VPN_IP:30500); empty → feature off
REGISTRY_URL platform how platform reaches the registry (walk usage, prune)
REGISTRY_NODEPORT kuberay + join NodePort (default 30500); nodes trust MASTER_VPN_IP:<this>
REGISTRY_S3_BUCKET kuberay + platform + minio-init image bucket (default gridweave-registry, no ILM expiry)
REGISTRY_S3_ENDPOINT kuberay optional override. By default the registry pod's MinIO endpoint is derived from the deploy's S3_ENDPOINT so one .env works on every topology (see below) — set this only for an unusual backend
REGISTRY_TOKEN_REALM kuberay auth realm baked into the registry config (default http://MASTER_VPN_IP:8100/v1/registry/token)
REGISTRY_QUOTA_GB platform per-user referenced-bytes cap (default 20; 0 = unlimited)
REGISTRY_GC_INTERVAL_HOURS platform scheduled GC cadence (default 168 = weekly; 0 = off)

S3 creds for the registry come from the existing S3_ACCESS_KEY/S3_SECRET_KEY.

One .env, every topology. The in-cluster registry pod needs a MinIO endpoint it can route to, and that address differs by deploy shape. Rather than make you set a second value, kuberay derives it from the S3_ENDPOINT the rest of the stack already uses (resolve_s3_endpoint):

Deploy S3_ENDPOINT Registry pod uses
single-VM http://minio:9000 (docker host, not pod-resolvable) rewritten → http://MASTER_VPN_IP:9000 (master's MinIO on the mesh)
multi-VM http://10.100.0.2:9000 (data box, mesh IP) as-is
external R2/S3 https://…r2.cloudflarestorage.com as-is
real AWS S3 (empty) empty (region-addressed)

So single-VM and multi-VM deploy and test identically — no per-case flags.

8. Storage lifecycle: quota & GC

Quota is enforced at token issuance: a push scope is withheld (pull still granted) once the user's referenced bytes cross REGISTRY_QUOTA_GB. Usage is computed by walking the registry API (catalog → tags → manifests, deduping layers by digest), cached briefly, and fails open — an accounting blip never blocks a push (a broken registry blocks it anyway). Shared base layers count toward each referencing user (fairness), though storage holds them once.

GC runs weekly (and on demand via POST /v1/registry/gc): 1. Prune untagged manifests through the registry API — the leftovers of tag overwrites. Manifest-list children are kept, so this is arch-safe (we do not use distribution's buggy --delete-untagged). 2. Blob sweep — kuberay scales the registry to 0 (distribution's GC is only safe with no concurrent writes), runs a registry garbage-collect Job against the same config, and scales back to 1. Downtime is minutes; the last_gc_at timestamp is persisted so frequent redeploys can't starve it.

9. Failure modes

Situation Surfacing Handling
Registry not ready at deploy status != ready gateway 400 "registry isn't ready yet"
Push over quota token endpoint withholds push docker denied; SDK pre-check says "quota exceeded — delete_image()"
Cross-namespace pull/push empty grant → registry 403 denied at registry + gateway
Image/tag missing manifest 404 "not found in the registry — did you push_image()?"
Node can't reach registry pull timeout "node can't reach the registry over the mesh"
Push to proxied CDN 413/reset mid-layer make the DNS record grey-cloud (§6)
Registry down during GC pulls fail for minutes expected; scheduled off-hours, auto-recovers

10. Live-verify checklist (master-test)

Unit tests cover object construction, grant logic, signing, and gateway handling. These need a real cluster: 1. POST /v1/registry/ensure brings the pod Ready on the NodePort; the auth cert lands in the pod and the realm is reachable from a node. 2. docker login + push with a platform token works; a second user's token cannot push/pull the first's repo (403). 3. A node pulls a deployed image over the mesh using its registries.yaml token. 4. Dedup: push v1, change one layer, push v2 → only the changed layer uploads; redeploy → node pulls only the changed layer. 5. Quota: set a low REGISTRY_QUOTA_GB, confirm push is refused past it. 6. GC: POST /v1/registry/gc prunes untagged + reclaims blobs, registry returns to Ready.

11. Follow-ups

  • Scale-to-zero + lazy wake for the idle registry (it's one small pod, so low priority).
  • Buy-not-build line: if quota UI, retention policy, and RBAC keep growing, Harbor is the off-the-shelf answer — but it's ~6 components; don't start there.
  • Image export for data-ownership customers (skopeo copy to their bucket) — the honest replacement for the rejected per-user-S3 path.