A single reboot would have de-listed our entire GPU fleet. Here is the trap, and why monitoring should catch it before you hit it.
A reboot is the most routine thing a server does. On a marketplace GPU host it can also be the most expensive. While working through alerts on a 20-box fleet, we found that one restart, the kind any host eats during maintenance or a power blip, would have silently knocked every box off Vast and stopped it earning. Three had already fallen into the trap. Here is the failure mode, the one-line fix, and the case for a monitor that warns you before the reboot, not after.
The trap: nouveau wins the boot race
These are NVIDIA GPU boxes. The open-source nouveau driver was never blacklisted on them. As long as a box stays up, that is invisible: the real NVIDIA driver was loaded live at onboarding and everything works. The problem is latent, and it only springs on the next boot.
On reboot, nouveau claims the GPU first, and the real nvidia module cannot take ownership of the device. The signature is in dmesg:
NVRM: This can occur when another driver such as nouveau has already obtained ownership of the NVIDIA device(s).
nvidia: probe of 0000:01:00.0 failed with error -16nvidia-smi then fails. On a Vast host, the vast daemon cannot verify the GPUs, and the machine drops off the marketplace. The box is still powered on, still costing you, and earning nothing.
On a marketplace, downtime is not free
This is worse on a rented fleet than on your own servers. Vast.ai's documentation is blunt: hosts are told not to take their machines offline, and a machine's "reliability" is a documented measure of its historical uptime and health that feeds Vast's default search ranking. Lower reliability means your boxes rank lower and rent less. An unplanned outage, especially one that interrupts a running rental, is exactly the kind of event that hurts.
Vast does give you a real maintenance mechanism: the vastai schedule maint command, which lets you announce a window to renters. The point is not that maintenance is impossible. It is the difference between planned downtime you schedule and announce, and unplanned downtime that arrives because a latent fault detonated on a reboot you did not know was dangerous. The nouveau trap is the second kind. You reboot for something unrelated, and the box quietly de-lists.
We were already three boxes deep
When we found this, three of the twenty boxes had been rebooted a few hours earlier and were sitting off the marketplace, not earning, with no alert that said why. The other seventeen were one maintenance window, one power blip, or one kernel panic away from the same place.
The fix is one line and it is non-disruptive
echo -e "blacklist nouveau\noptions nouveau modeset=0" | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
sudo update-initramfs -uIt is idempotent and changes nothing about the running system. It only affects the next boot, which is the entire point: it makes the box reboot-safe before the reboot happens. We applied it to all twenty, recovered the three that were down (one reboot each: GPUs came back, re-listed on Vast), and baked this exact block into onboarding so future boxes are born reboot-safe.
This is a monitoring problem, and monitoring should own it
Here is the part that matters for Glassmkr. Every signal needed to catch this is already on the box: nouveau is loaded, the NVIDIA hardware is present, and no nvidia kernel module is bound to it. That combination means one thing: this host's GPU will not come back after a reboot. A monitor that can see the running kernel modules can see that, and it can tell you while the box is still up and earning, not after it has gone dark.
That is the honest version of "predictive." Not guessing at the future, just noticing a loaded gun on the table. This check now ships in Crucible and runs on the exact fleet that hit the trap. It warns in both cases: when an NVIDIA box is already running without its driver bound, and, the more useful one, while the box is still healthy but nouveau was never blacklisted, so the danger is only latent. Either way you blacklist nouveau and rebuild the initramfs in a window you chose, instead of learning about it from a drop in your earnings.
Check one thing today
If you run NVIDIA GPUs on a marketplace, check one thing: lsmod | grep -e nouveau -e nvidia. If you see nouveau and not nvidia, or you never blacklisted nouveau, you are one reboot from an outage. Fix it in a window you control. And if you want something watching for the class of latent fault that only bites on reboot, that is what we built Glassmkr to catch.