Fractional & Multi-GPU Scheduling

You ask for VRAM (vram="10GB"), not "a GPU" — and the gateway figures out how to place it. This note is how that placement works; for the SDK surface see client_sdk, for the wider scheduler see job_gateway.

The problem

Ray reports how many whole GPUs a node has, not how much VRAM is free on each one. That's not enough to safely pack a 10 GB job next to a 60 GB job on the same 80 GB card. So the gateway keeps its own VRAM ledger (vram_ledger.py) — an in-memory, per-GPU account of who's using how much, built from Ray's per-GPU custom resources (VRAM_MB_GPU0, VRAM_MB_GPU1, … from gpu_support) and kept restart-safe by rehydrating from the job/endpoint CRDs.

The placement rule

Given a VRAM request, the ledger:

1. Fits on one GPU → fractional. It reserves a slice and hands Ray a fractional num_gpus (10 GB of an 80 GB card ≈ 0.125), so several small jobs share one physical GPU.
2. Bigger than any single GPU → multi-GPU. It reserves N whole GPUs on the same node and sets CUDA_VISIBLE_DEVICES to them (100 GB → 2× 80 GB).
3. Best-fit. Among the GPUs that fit, it picks the tightest one first — leaving the big, empty GPUs free for the jobs that actually need them.
4. Vendor + mixed cards. Preference is NVIDIA → AMD → Apple; and because each GPU is tracked on its own, a node with a 24 GB 4090 next to an 80 GB A100 just works.

VRAM is released when the job or endpoint finishes (or fails). The same ledger backs jobs, training, and endpoints.