The fleet

Not the uniform fleet we expected. Three A16 boxes (one with 4 GPUs, two with 8), fifteen L4s, one RTX A4000, one RTX A6000. GPU model and count both vary per box, which already tells you something: any monitoring that assumes a fixed hardware profile will be wrong somewhere.

Where the 88 went

Real hardware: one genuine item, not the three we first flagged.

Our first pass flagged three candidates for replacement: an L4 with an uncorrected ECC error, a drive reporting SMART failure, and a GPU on a degraded PCIe link. Then we read each one to the metal, and two did not survive scrutiny:

• The "uncorrected ECC" L4 had a single lifetime error with none since boot, no remapped rows, and the SRAM error threshold not exceeded: the signature of a one-off bit flip in on-die SRAM (the L2 cache), not in the VRAM. NVIDIA warrants a replacement only on a row-remapping failure or an exceeded SRAM error threshold, and neither was present, so this is not an RMA. (NVIDIA's GPU Memory Error Management documentation sets those thresholds: a row-remap failure, or "SRAM Threshold Exceeded" at more than four uncorrectable-error events per bank for parity-protected SRAM, more than two for SECDED.)
• The "degraded PCIe link" was a GPU sitting idle. The link narrows its width (to x4 of the full x16) to save power while the link generation stays maxed, and it retrains to full width the moment a job loads it. Expected power management, not a fault.

That left one genuine hardware item: a drive with a declining SMART health trend, which stays on the dashboard until the RMA goes out. The uncomfortable lesson, worth saying out loud: two of our own three "real hardware" calls were false positives. The fix was not to wave them away. It was to teach the rules to read the same deeper signals we had to read by hand (more below).

Required-by-design config, muted on purpose.

Three rules fired across all twenty boxes because of configuration a Vast GPU host must have:

• no_firewall: a marketplace GPU host has to keep its rental port range open (a small block from port 40000), and ufw fights Docker's iptables. Turning on a firewall would break the box.
• unattended_upgrades_disabled: enabling auto-updates would bump the NVIDIA driver out from under the running CUDA stack and de-verify the host.
• gpu_power_cap_throttling: the A16 and A6000 ship a conservative default power cap.

Muting these is not the monitor being wrong and us papering over it. It is context the monitor did not have, which is itself a product finding (below).

Self-inflicted, correctly detected.

Three unexpected_reboot alerts from our own recovery reboots during the work. They are the same three boxes from the nouveau story below: trapped on the first reboot, recovered on the second. The detection was right, and they age out on their own.

False positives we fixed.

kernel_needs_reboot fired on boxes where the running kernel was already the newest installed: a bug in the comparison, now fixed. With the two reclassified hardware calls, that is three rules an honest field test caught crying wolf. We fixed each at the source: the ECC rule now separates a benign lifetime SRAM blip from active VRAM damage, the PCIe rule ignores an idle power-capped GPU, and the kernel-reboot check compares the running kernel to the newest installed one.

The big latent one.

The single most valuable thing we found was not in the 88 at all: a reboot would have silently de-listed the whole fleet from Vast, because nouveau was never blacklisted. That one gets its own post.

The tally. The bulk of the 88 were the required-config mutes (no_firewall and unattended_upgrades_disabled on all twenty, the kernel false positives on most), with the ssh hardening and the nouveau-caused service failures fixed outright. That left 11: four gpu_power_cap_throttling, three unexpected_reboot, two disk_latency_high, one gpu_pcie_link_degraded, and one gpu_uncorrected_ecc. Two of those 11 (the PCIe and the ECC) are the ones we went on to reclassify as false positives.

What this says about monitoring

The goal was never to drive the count to zero. Plenty of monitors will happily show you a green dashboard by being quiet. The goal is that the alerts left standing are all true: real hardware to replace, or self-inflicted and aging out. Getting there meant a monitor that lets you say "this is expected on this host" without losing the ability to see it. Noise is not the alerts you mute. Noise is the alerts you cannot mute and cannot trust.

What we changed in Glassmkr because of this

A field test that does not change the product is just a status report. This one changed ours, in two rounds:

• A check for the reboot-de-list trap, so the latent fault is caught before the reboot. It is shipped and running on the exact fleet that hit it.
• Host-type profiles, so the "required config" alerts above are suppressed by host role instead of muted one rule at a time. Then we found the feature did nothing until a host opts in, so we added auto-detection: a box that looks like a marketplace GPU host now gets a one-click prompt to apply the profile.
• The false-positive fixes above: the PCIe rule is load-aware, the ECC rule reads since-boot and row-remap signals, the disk-latency rule tells saturation from a failing drive, and the kernel-reboot comparison is fixed.
• The active count was hiding worse noise underneath. Every rule that stayed true re-fired into the event log on each five-minute snapshot, so a single always-true alert logged twelve identical rows an hour, and with dozens of them across twenty boxes the log buried the real state changes. We changed it to record state transitions, not snapshots. A rule that keeps firing and clearing (flapping) is a separate failure, so we added a detector that folds it into one "this is flapping" warning instead of a stream of on-off rows.

That is the loop we care about: run the product against a real fleet, write down everything that was noise or wrong, and close the gap. The active count went from 88 to 11 in the first pass, and to a short, trustworthy list once we fixed the noise floor itself: the one drive we are replacing, plus the occasional transient that clears on its own.

What the number is for

If you operate a GPU fleet, especially on a marketplace, the number on your dashboard matters less than whether you trust it. Eighty-eight alerts you cannot reason about is worse than eleven you can. We would rather show you eleven true things than a green light we cannot defend.