Skip to content

Gridweave Controller

The platform stores every job and endpoint as a Kubernetes custom resource — a GridweaveJob or GridweaveEndpoint object living in the cluster's etcd, not in an app database. This library is the typed client for those two resources: create them, patch their status, list and delete them.

Why put them in Kubernetes? It makes the cluster itself the source of truth. The gateway can stay stateless and restart freely — state lives in etcd, survives crashes, and is watchable and kubectl-inspectable like any other k8s object. (It's exactly why the gateway rehydrates in-flight work from CRDs on boot.)

The two resources

Both are gridweave.io/v1 objects in the gridweave namespace, with the usual Kubernetes shape:

  • spec — desired state: who asked, for what (GridweaveJob: the pickled function ref, VRAM, vendor, owner; GridweaveEndpoint: model/spec, pricing, mode).
  • status — observed state: phase (Pending → Allocated → Running → Completed/Failed/…), the node + GPU indices it landed on, timestamps (startedAt, billedSeconds, …), and k8s-style conditions.

The CRD definitions that register these kinds with Kubernetes live in k8s/ (gridweave-job-crd.yaml, gridweave-endpoint-crd.yaml).

The API

Plain functions over Kubernetes' CustomObjectsApi (init_k8s_client() loads in-cluster config, or a kubeconfig for local dev):

from gridweave_controller import crd

crd.create_job(job_id, user_id, function_ref, ...)
crd.set_job_running(job_id)                     # typed status transitions
crd.set_job_completed(job_id, wall_seconds=42.0)
crd.get_job(job_id); crd.list_jobs(label_selector="..."); crd.delete_job(job_id)

Endpoints mirror this (create_endpoint, set_endpoint_deploying/running/stopped/failed, set_endpoint_health, …). The typed setters own the phase machine and update conditions, so callers never hand-craft a status patch.

Two ways to reach it

Same API, two transports — because not everything can touch the K8s API:

  • Direct — code on the master with a kubeconfig (kuberay_service) imports gridweave_controller.crd and calls Kubernetes directly. Kuberay also re-exposes it over HTTP at /v1/crd.
  • Proxied — the job gateway runs in a Docker bridge with no cluster access, so it uses job_gateway/crd_client.py: a drop-in replacement with identical signatures that HTTP-proxies to kuberay's /v1/crd. Swap the import, same calls.

Configuration & tests

KUBERAY_NAMESPACE (default gridweave) selects the namespace. Unit tests use tests/mock_crd.py, an in-memory stand-in with the same signatures, so they never need a real cluster.