Storage API

Per-user file storage on the platform, backed by S3/MinIO. Every user is boxed into a private users/{user_id}/ prefix; a user can only touch their own prefix, admins are exempt. All endpoints require a bearer token (Authorization: Bearer <jwt|gw_…>); user_id is the token's sub.

Base path: /v1/storage. Interactive schema at /docs.

Endpoints

Method	Path	Purpose
GET	/v1/storage/status	Object-store health (used %, watermark, degraded) — drives the webui banner
POST	/v1/storage/upload	Upload a file into the caller's prefix (multipart, streamed)
GET	/v1/storage/usage	Caller's used/quota bytes; admins also get the per-user table
GET	/v1/storage/list	List the caller's files (recursive); admin lists the whole bucket
POST	/v1/storage/download	Download a file by s3:// URI (own prefix only, admin exempt)
POST	/v1/storage/delete	Delete a file by s3:// URI (own prefix only, admin exempt)
POST	/v1/storage/multipart/start	Begin a chunked upload (SDK: files > 16 MiB)
PUT	/v1/storage/multipart/part	Send one part (raw body, ≤ part_size, 64 MiB)
GET	/v1/storage/multipart/parts	Parts the server holds for an upload (resume)
POST	/v1/storage/multipart/complete	Assemble the parts into the object
DELETE	/v1/storage/multipart	Discard an in-progress upload

GET /v1/storage/status

{ "used_pct": 41.2, "watermark_pct": 92.0, "degraded": false }

degraded is true once used_pct >= watermark_pct; uploads are then refused (see 507 below).

POST /v1/storage/upload

Multipart form with a file field. Optional dest form field sets a sub-path (e.g. models/qwen/weights.bin). Streamed in 8 MiB chunks.

{ "uri": "s3://gridweave-artifacts/users/alice/weights.bin", "size": 12345 }

A single request is capped by whatever proxy fronts the platform (Cloudflare: 100 MB, and ~90 s of transfer time). Anything larger must use the chunked endpoints below; gridweave.upload() switches automatically above 16 MiB.

POST /v1/storage/multipart/start

Body { "filename": "models/qwen/weights.bin", "size": 3422777952 }. The watermark and quota gates run here against the declared size, so an upload that cannot fit is refused before any bytes move.

{ "key": "users/alice/models/qwen/weights.bin", "upload_id": "…", "part_size": 67108864 }

PUT /v1/storage/multipart/part?key=…&upload_id=…&part_number=N

Raw body, at most part_size bytes (the server's ceiling); part numbers start at 1 and every part except the last must be the same size, ≥ 5 MiB (S3 rules). The SDK uses 16 MiB parts — sized so one part clears the proxy's ~90 s limit on a 0.4 MB/s uplink — growing only as far as S3's 10,000-part ceiling forces (32 MiB for a 300 GB file). Re-sending a part number replaces it. Returns { "etag": "…", "size": 16777216 }.

GET /v1/storage/multipart/parts?key=…&upload_id=…

{ "parts": [ { "part_number": 1, "etag": "…", "size": 67108864 } ] } — what MinIO holds right now; the SDK uses it to skip finished parts on resume. 404 once the upload has been completed, aborted, or reaped.

POST /v1/storage/multipart/complete

Body { "key", "upload_id", "parts": [ { "part_number", "etag" } ] }. Quota is re-checked against the part sizes the server holds (not the client's numbers); over quota → 413 and the parts are discarded. After assembly the object's size must equal the size declared at start, otherwise the object is deleted and the call returns 400 — a truncated or padded upload is never published. Returns { "uri", "size" }.

DELETE /v1/storage/multipart?key=…&upload_id=…

Discards the staged parts. Uploads nobody completes or aborts are reaped by MinIO after MINIO_API_STALE_UPLOADS_EXPIRY (24 h, docker/deploy/docker-compose.yml).

Quota is checked, not reserved: two uploads admitted concurrently can together exceed it, the same window the single-request upload has.

GET /v1/storage/usage

{ "used_bytes": 12345, "quota_bytes": 21474836480, "unlimited": false }

For an admin, unlimited is true and the response also carries a per-user table sorted by usage:

{ "used_bytes": 0, "quota_bytes": …, "unlimited": true,
  "users": [ { "user_id": "alice", "used_bytes": 999 }, … ] }

GET /v1/storage/list?prefix=<optional>

{ "files": [ { "uri": "s3://…/users/alice/a.bin", "size": 10, "last_modified": "2026-07-01T…" } ] }

Recursive (matches usage), so files in sub-folders are always listed.

POST /v1/storage/download · POST /v1/storage/delete

Body: { "uri": "s3://…" }. Download streams the object as application/octet-stream; delete removes a single object.

Error codes

Code	When
400	multipart complete: assembled size ≠ size declared at start (object discarded)
403	uri/prefix is outside the caller's users/{sub}/ prefix (non-admin)
404	file not found on download/delete; unknown upload_id for the key
413	upload would exceed the caller's quota (USER_STORAGE_QUOTA_GB); multipart part larger than part_size
507	object store is at/above the watermark — new uploads temporarily refused

Configuration

Env var	Default	Meaning
S3_ENDPOINT	http://minio:9000	Object-store endpoint
S3_ACCESS_KEY / S3_SECRET_KEY	gridweave / —	Credentials
S3_BUCKET	gridweave-artifacts	Bucket holding users/{id}/… prefixes
USER_STORAGE_QUOTA_GB	20	Per-user quota (admins exempt)
STORAGE_WATERMARK_PCT	92	Fill % at which uploads start returning 507
MINIO_API_STALE_UPLOADS_EXPIRY (minio service)	24h	Incomplete multipart uploads older than this are discarded — the resume window
MINIO_API_STALE_UPLOADS_CLEANUP_INTERVAL (minio service)	6h	How often MinIO sweeps for them

Tests

• tests/platform_service/test_routers_storage.py — upload/download/list/delete with per-user prefix isolation + admin bypass (S3 mocked via mock_s3).
• tests/platform_service/test_storage_multipart.py — chunked upload: assembly, resume listing, abort, size integrity, part cap, prefix isolation, quota at start and at complete.
• tests/platform_service/test_storage_quota.py — 413 quota enforcement.
• tests/platform_service/test_storage_watermark.py — 507 watermark / fail-open.
• tests/client_sdk/test_storage.py — SDK side: single vs chunked, part retry, resume after interruption, changed-file detection, abort on client error.

Run: pytest tests/platform_service/test_routers_storage.py tests/platform_service/test_storage_multipart.py tests/platform_service/test_storage_quota.py tests/platform_service/test_storage_watermark.py tests/client_sdk/test_storage.py. Live MinIO integration is exercised in the Layer 3 cluster pass.