Serving (Endpoints)

An endpoint is a model — or any container — that stays up behind a stable URL, holding a GPU until you stop it. (A job runs once and exits; an endpoint keeps serving.) This doc is how endpoints actually run on the cluster. For the SDK calls that drive them see client_sdk; for how paid calls are billed see job_gateway.

The key idea: endpoints run on the same Ray cluster as jobs — there are no separate RayService CRDs. To bring one up, the gateway swaps a plain GPU worker for a dedicated endpoint worker, pins a Ray Serve app to it, and swaps back on stop.

Three kinds of endpoint

You pass	It runs	You call it with
model="..."	a vLLM server (OpenAI-style LLM) on the worker	ep.chat(...) or the OpenAI API
spec={... "port": N ...}	your HTTP container; a ray-proxy sidecar forwards requests to it	ep.call("action", data=...)
spec={...} (no port)	your command; the sidecar writes the input, execs it, reads the output	ep.call("action", data=...)

How a deploy works (the worker-group swap)

Everything lives on the existing gridweave-cluster, where each GPU is its own Ray worker group (replicas=1). To deploy:

1. The VRAM ledger picks the GPU(s) for the requested vram / vendor / node.
2. Atomic swap — in one Kubernetes call, remove the plain worker group(s) and add the endpoint's group (named with the reserved ep- prefix, so "available" = "not ep-").
3. Delete the orphan pod KubeRay leaves behind (it doesn't clean up removed groups).
4. Wait for the endpoint pod to be Running and its custom resource to register with the Ray head.
5. ray.serve.run() the app (vLLM or proxy), pinned to that exact worker via a per-endpoint custom resource (ep_<name>).

Stop/delete reverses it — remove the endpoint group, delete its pod, add a fresh plain worker back. Multi-GPU swaps N workers into one pod. The heavy model boot runs in a background thread, so deploy returns immediately and the endpoint reports Deploying → Running.

How a call reaches the model

client → platform (auth) → gateway (proxy) → Ray Serve on the head
       → the endpoint's actor on the GPU worker
       → vLLM engine  |  your HTTP server (localhost:port)  |  your command

Loading the model (vLLM)

model= is polymorphic:

• org/name → HuggingFace Hub. Add hf_token= for gated models — the token lives only in the actor's process memory, never in a pod spec, env var, or on disk.
• s3://bucket/path → your S3/MinIO (pass s3_endpoint / s3_access_key / s3_secret_key).

Downloads always happen on the worker, into a host cache at /data/model-cache/ (huggingface/ for HF, s3/<uri-hash>/ with a .complete marker for S3). The cache is a hostPath, so it survives stop/delete/redeploy: the same model on the same host reloads instantly; a different host re-downloads. No master-side download, no shared NFS.

Custom images from the platform registry (no public registry)

spec.image takes either a public registry ref (org/name:tag, unchanged) or an image in the platform's own container registry — in-house, docker-native (pushes/pulls move only changed layers), and zero setup: your platform token is your registry login.

gridweave.auth("gw_...")
ref = gridweave.push_image("myapp:v1")     # docker login+tag+push, returns the ref
gridweave.serve(spec={"image": ref, "port": 8080}, name="myapp")
gridweave.delete_image("myapp:v1")         # storage reclaimed by scheduled GC

Under the hood: one shared registry:2 backed by the platform MinIO, nodes trust it at MASTER_VPN_IP:30500. Access is docker token auth — you can only push/pull under your own namespace (<host>/<your-ns>/…); a second user's token is refused. At deploy the gateway checks the ref is yours (or a shared gridweave/* base image) and the registry is ready, then the node pulls over the WireGuard mesh with imagePullPolicy: Always (re-pushed tags picked up; dedup keeps it cheap). No S3 credentials are involved at all. Per-user storage is bounded by a quota (registry_status() shows usage_bytes/quota_bytes).

Full design, security model, and failure modes: docs/image_registry.

Modes & billing

Every endpoint has a mode: free (public, no charge), paid (public, callers pay), or self_host (private). Paid endpoints price per call (price_per_call) or per token (price_per_1m_input/output, vLLM only) — the caller pays, the deployer keeps it minus the platform fee, and the deployer pays GPU rental while the endpoint is up. The billing mechanics live in the gateway.

Endpoint API

Reached through the platform with your token; owner-or-admin only.

Method	Path	What
POST	/v1/endpoints/deploy	Deploy (model= or spec=; vram/vendor/node, public/paid/price…)
POST	/v1/endpoints/{name}/stop	Release the GPU, keep the config
POST	/v1/endpoints/{name}/start	Reclaim a GPU, bring it back up
DELETE	/v1/endpoints/{name}	Permanent
GET	/v1/endpoints	List your endpoints (admin: all)
GET	/v1/endpoints/{name} · /logs · /health	Status, logs, health

Configuration

Variable	Description
VLLM_IMAGE_NVIDIA / VLLM_IMAGE_AMD	Per-vendor vLLM images (built on each node by join-node.sh)
PROXY_IMAGE	ray-proxy sidecar image for custom endpoints
RAY_HEAD_ADDRESS / RAY_SERVE_PORT	Ray client address + Serve HTTP port
KUBERAY_NAMESPACE / RAYCLUSTER_NAME	The cluster endpoints deploy onto

Constraints & tests

KubeRay doesn't clean up pods from removed worker groups, so the gateway deletes orphans in every path (swap, error, stop, delete). vLLM images (~12 GB) are baked into each node's containerd by join-node.sh, not pulled from a registry.

Tests: tests/serve/test_serve.py (deploy / lifecycle / queries, mocked), test_cluster.py (real KubeRay cluster), test_cluster_model_loading.py (S3 + gated-HF loading).