Network Diagnostics¶

A self-contained, fully open-source observability stack for a Gridweave cluster — every service log, worker/job pod log, service up/down history, and GPU/host metrics — surfaced in the admin Diagnostics page so an operator never has to SSH into the master.

The browser only ever talks to platform_service; the diagnostics router (platform_service/routers/diagnostics.py, prefix /v1/diagnostics) proxies the two backends below and forces require_admin. None of these services publish a host port.

Components (all Apache-2.0 / permissive)¶

Service	Image	Role
VictoriaLogs	`victoriametrics/victoria-logs`	log store + LogsQL query + live `/tail` (7d retention)
Fluent Bit	`fluent/fluent-bit`	tails every container's JSON log → VictoriaLogs
Prometheus	`prom/prometheus`	metric store + query (15d retention)
cAdvisor	`cadvisor/cadvisor`	per-container CPU / mem / up → Prometheus
node-exporter	`prometheus/node-exporter`	master host metrics → Prometheus
DCGM exporter	`nvidia/dcgm-exporter`	per-GPU util / temp / VRAM (k8s DaemonSet on GPU nodes)

Data flow¶

 docker container logs ─┐
                        ├─ Fluent Bit (master, docker) ─┐
 k3s pod logs ──────────┘   Fluent Bit (DaemonSet) ─────┴─→ VictoriaLogs ─┐
                                                                          ├─→ platform /v1/diagnostics → admin UI
 cAdvisor / node-exporter (master) ─┐                                     │
 DCGM exporter (GPU nodes) ─────────┴────────────────────→ Prometheus ────┘

Service names enrich each log line (enrich.lua strips the gridweave- prefix / parses the pod name), so the UI can filter by service, node, or job.

Deploy — nothing extra to run¶

The five services are part of the main docker/deploy/docker-compose.yml, so the normal deploy.sh brings them up with everything else (configs here are bind-mounted in). The platform reaches victorialogs:9428 and prometheus:9090 over gridweave-network via the compose service aliases — no env vars needed (VICTORIALOGS_URL / PROMETHEUS_URL override the defaults if you relocate them).

deploy.sh also applies the k8s DaemonSets:

kubectl apply -f diagnostics/k8s/   # fluent-bit (pod logs) + dcgm (GPU metrics)

fluent-bit runs on every node, shipping pod logs to the master's mesh IP 10.100.0.1:9428.
dcgm-exporter runs only on GPU nodes (nodeSelector gridweave.io/gpu-vendor=nvidia), exposing metrics on <mesh-ip>:9400.

GPU nodes are scraped automatically: when kuberay_service enrolls an NVIDIA node (/register), it writes a Prometheus file_sd target (prometheus/targets/dcgm-<node>.json) and removes it on /disconnect — event-driven, no restart, no sweep. The targets dir is bind-mounted into both the kuberay (rw) and prometheus (ro) containers.

On rate-limited / air-gapped GPU nodes, relay the fluent-bit and dcgm-exporter images via the master (docker save | ssh … docker load); the manifests use imagePullPolicy: IfNotPresent.

Retention¶

Logs 7d (VictoriaLogs -retentionPeriod), metrics 15d (Prometheus --storage.tsdb.retention.time). Both persist in the named volumes gridweave_vlogs / gridweave_prometheus, which survive a re-deploy.