A training node is still a server, and most of what kills it is not the GPU.

How Glassmkr fits

Every GPU, read at a glance

Per-GPU temperature, power draw against limit, VRAM, ECC counters, PCIe link width and the XID event log. Everything here comes from nvidia-smi, which is present on every driver install, so there is no exporter or daemon to deploy first.

Specific alert rules that matter

GPU and ML infrastructure operators care most about:

• PSU state: redundancy loss on multi-PSU chassis (H100 boxes typically have 4-8 PSUs; losing one is degraded, losing two is at-risk)
• IPMI sensors: fan failures, temperature thresholds exceeded, SEL critical entries from the BMC indicating hardware-level issues
• ECC errors: correctable error rate trending up (often a leading indicator of impending DIMM or GPU memory failure)
• NVMe wear: high write amplification from checkpoint-heavy training workloads
• Network state: interface errors, link-down events and negotiated-speed drops on the host's Ethernet interfaces, including 100 and 400G and RoCE-capable NICs. We read the kernel's per-interface counters, so a native InfiniBand port is out of scope: it does not appear as a network interface, and its link-layer counters live somewhere we do not currently read
• Disk I/O patterns: sustained latency anomalies indicating storage degradation under shuffle/dataloader pressure
• Memory pressure: OOM kills on the host (data loader processes dying)
• Service health: DCGM exporter unhealthy, NVIDIA driver mismatches across boxes in a multi-node training cluster
• Driver survives a reboot: on an NVIDIA box where nouveau was never blacklisted, the next reboot binds nouveau first and the GPU never comes back; on a marketplace this silently de-lists the host. Glassmkr reads the loaded modules and the blacklist state and warns while the box is still up, so you fix it in a window you choose instead of finding out from lost earnings.

Full list at /docs. For the GPU vertical specifically, the IPMI sensor coverage is the differentiator: most cloud-era monitoring tools don’t ingest BMC data because cloud workloads don’t expose it.

Where we stop, and why we say so

We monitor the host. We do not monitor the fabric between hosts, and the distinction matters more on GPU infrastructure than anywhere else.

From inside a node you can see that node's NVLink state, its GPUs, its NIC endpoint and its kernel log. You cannot see switch queue depth, adaptive routing, subnet manager health or a far-end transceiver. Those need switch or controller access, and no host agent of any kind gets them. Anyone claiming otherwise is either reading a switch you have not told them about or guessing.

Two specific limits, stated plainly because you will hit them:

• We do not report NVLink bandwidth or saturation. Collective operations move data in short bursts between compute, so an average taken minutes apart says nothing useful about whether a link was saturated. We would rather report nothing than a confidently wrong number.
• We do not claim to predict NVLink failure. A published study spanning 855 days of production GPU fleets found no preceding hardware errors before its NVLink events. We detect a link that has dropped, quickly, and we can show one that is accumulating errors. Predicting the rest is not something the evidence supports, so we do not sell it.

What we do instead is cover the host completely and tell you where the boundary is. On a training node that is most of the failure surface.

The agent you can read

If you’re running ML infrastructure for research, you almost certainly have a compliance review process. Crucible passes most of them out of the box: MIT licensed, source published, no telemetry collection, non-root user, signed releases on npm.

For environments where outbound HTTPS is restricted, the agent supports HTTP proxies and air-gapped deployment with manual metric upload. Email [email protected] if you need an air-gapped variant; we’ll help.