Job Gateway¶

The gateway is the front door for compute. Everything a user does with GPUs goes through it: run a one-off job (a pickled Python function), stand up a long-lived model endpoint, or launch distributed training. It schedules that work onto the shared Ray cluster, bills it while it runs, and streams back logs and results. Port 8200.

It's stateless. Job and endpoint state lives in Kubernetes CRDs (reached through the kuberay service, since the gateway runs in a Docker bridge and can't touch K8s directly); the live GPU booking lives in an in-memory VRAM ledger rebuilt from Ray on boot; logs, results, and model weights live in S3. On restart it re-syncs the ledger from Ray and rehydrates in-flight VRAM holds from the CRDs, so nothing double-books.

The platform proxies user requests here with X-User-Id / X-User-Role headers (it does the real auth); the gateway trusts those and enforces role on the admin and provider routes.

How it schedules — the VRAM ledger¶

Ray tells you how many whole GPUs a node has, but not how much VRAM is free on each one. That's not enough to pack fractional and multi-GPU work safely, so the gateway keeps its own per-GPU ledger (vram_ledger.py) and best-fits each request onto specific GPUs. scheduler.py is the single reserve() / release() path that jobs, training, and endpoints all funnel through.

Scheduling is owner-aware: free users are placed on org hardware only, paid users prefer provider hardware (_owner_filter_for_paid). VRAM is booked at reserve time and released on completion — and reconstructed from CRDs after a restart so a crash never leaks a booking.

How it bills¶

When CREDITS_ENABLED=true, billing (billing.py) runs the money path (all deductions go to the credit service; usage goes to metering):

Preflight — a deploy/submit is rejected if the user can't cover PREFLIGHT_HOURS of the requested compute (counting their already-running workloads).
Live tick — a background daemon deducts credits every BILLING_TICK_SECONDS (60s) while a job or endpoint runs; if the balance hits zero it flags the workload for cancellation.
Rental & revenue — provider hardware earns its custom hourly rate; paid endpoint calls split revenue between the deployer and the platform (platform_fee). Cost uses the org formula BASE_RATE + vram_gb × VRAM_RATE per hour (defaults 0 + 1 → 1 credit per VRAM-GB-hour). The same daemon also reconciles cancellations every ~5s.

Endpoints (model serving)¶

serve.py deploys vLLM and custom-image model endpoints onto the existing Ray cluster (via KubeRay worker-group swaps, not separate RayService CRDs), health-gates them, and benchmarks a live call to set per-call pricing. Endpoints can be free, paid (callers pay; deployer earns), or self_host (deployer's own hardware, no platform billing). The full endpoint lifecycle + /v1/endpoints/* API is documented in docs/serve.

HTTP API¶

Group	Endpoints	Auth
Jobs & resources	`POST /v1/jobs/submit`; `GET /v1/jobs/{id}/status` · `/logs` · `/result`; `POST /v1/jobs/{id}/cancel`; `GET /v1/jobs`; `GET /v1/resources`; `POST /v1/jobs/train`	user (owner)
Endpoints `/v1/endpoints/*`	deploy / stop / start / delete / call — see docs/serve	user
Admin `/v1/admin`	`GET /health`; `GET`·`PUT /gpu-pricing`; `GET`·`PUT /revenue-splits`	admin
Provider `/v1/provider`	`GET /status`; `GET`·`PUT /pricing`	provider
Ops	`GET /healthz`, `/version`, `/metrics` (Prometheus cluster gauges)	—

An SDK version gate rejects clients whose X-SDK-Version differs in the major version from SERVER_VERSION (426); minor/patch are compatible.

Configuration¶

Variable	Default	Description
`GATEWAY_PORT`	8200	Service port
`RAY_HEAD_ADDRESS`	`auto`	Ray cluster to submit to (lazy `ray.init` on first use)
`KUBERAY_SERVICE_URL`	`http://localhost:8800`	Kuberay — CRD state + Ray Serve ops
`RAYCLUSTER_NAME` / `KUBERAY_NAMESPACE`	— / `gridweave`	The RayCluster the gateway schedules on
`ADMIN_TOKEN`	— (required)	Admin/service bearer token
`CREDITS_ENABLED`	`false`	Master switch for balance checks + billing
`CREDIT_SERVICE_URL` / `METERING_URL`	service defaults	Billing (deducts) + usage ledger
`BASE_RATE` / `VRAM_RATE`	0 / 1	Org cost formula (1 credit per VRAM-GB-hour)
`BILLING_TICK_SECONDS` / `PREFLIGHT_HOURS` / `MAX_PRICE_PER_CALL`	60 / 1 / 0	Billing knobs (0 = no per-call ceiling)
`S3_ENDPOINT` / `S3_ACCESS_KEY` / `S3_SECRET_KEY` / `S3_BUCKET`	MinIO defaults	Logs, results, model weights
`MODEL_CACHE_DIR`	`/data/model-cache/s3`	Local vLLM model cache
`RAY_SERVE_PORT` / `PROXY_IMAGE`	—	Endpoint serving
`SERVER_VERSION`	`0.0.0`	Advertised version, for the SDK gate
`CORS_ORIGINS`	`*`	Allowed origins

Running & testing¶

# Self-contained stack (gateway + platform + a CPU-only Ray head + MinIO):
docker compose -f docker/job_gateway/docker-compose.test.yml up --build -d

# Unit tests use a FakeRay mock — no Ray cluster needed:
pytest tests/job_gateway/ -v --ignore=tests/job_gateway/test_docker.py

Coverage highlights: test_gateway (cost estimation, VRAM allocation), test_billing* (hourly cost, preflight, tick, final charge, owner routing), test_endpoint_alloc_concurrency / test_preflight_reservation / test_ledger_read_safety (the concurrency guards on GPU booking), test_owner_routing (free→org / paid→provider), test_gpu_pricing / test_provider_earnings / test_revenue_splits (pricing + payouts), test_serve*/test_endpoint_mode (endpoint lifecycle + billing routing), test_cluster_* (real GPU nodes). Interactive API docs at /docs.