> Source: https://glassmkr.com/blog/the-reboot-that-delists-your-gpu-host

# The reboot that de-lists your GPU host - Glassmkr Blog

June 2026 · GPU · 6 min read

# A single reboot would have de-listed our entire GPU fleet. Here is the trap, and why monitoring should catch it before you hit it.

A reboot is the most routine thing a server does. On a marketplace GPU host it can also be the most expensive. While working through alerts on a [20-box fleet](/blog/20-box-gpu-fleet-88-to-11-alerts), we found that one restart, the kind any host eats during maintenance or a power blip, would have silently knocked every box off Vast and stopped it earning. Three had already fallen into the trap. Here is the failure mode, the one-line fix, and the case for a monitor that warns you before the reboot, not after.

## The trap: nouveau wins the boot race

These are NVIDIA GPU boxes. The open-source `nouveau` driver was never blacklisted on them. As long as a box stays up, that is invisible: the real NVIDIA driver was loaded live at onboarding and everything works. The problem is latent, and it only springs on the next boot.

On reboot, `nouveau` claims the GPU first, and the real `nvidia` module cannot take ownership of the device. The signature is in `dmesg`:

```
NVRM: This can occur when another driver such as nouveau has already obtained ownership of the NVIDIA device(s).
nvidia: probe of 0000:01:00.0 failed with error -16
```

`nvidia-smi` then fails. On a Vast host, the vast daemon cannot verify the GPUs, and the machine drops off the marketplace. The box is still powered on, still costing you, and earning nothing.

## On a marketplace, downtime is not free

This is worse on a rented fleet than on your own servers. Vast.ai's documentation is blunt: hosts are told not to take their machines offline, and a machine's "reliability" is a documented measure of its historical uptime and health that feeds Vast's default search ranking. Lower reliability means your boxes rank lower and rent less. An unplanned outage, especially one that interrupts a running rental, is exactly the kind of event that hurts.

Vast does give you a real maintenance mechanism: the `vastai schedule maint` command, which lets you announce a window to renters. The point is not that maintenance is impossible. It is the difference between planned downtime you schedule and announce, and unplanned downtime that arrives because a latent fault detonated on a reboot you did not know was dangerous. The nouveau trap is the second kind. You reboot for something unrelated, and the box quietly de-lists.

## We were already three boxes deep

When we found this, three of the twenty boxes had been rebooted a few hours earlier and were sitting off the marketplace, not earning, with no alert that said why. The other seventeen were one maintenance window, one power blip, or one kernel panic away from the same place.

## The fix is one line and it is non-disruptive

```
echo -e "blacklist nouveau\noptions nouveau modeset=0" | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
sudo update-initramfs -u
```

It is idempotent and changes nothing about the running system. It only affects the next boot, which is the entire point: it makes the box reboot-safe before the reboot happens. We applied it to all twenty, recovered the three that were down (one reboot each: GPUs came back, re-listed on Vast), and baked this exact block into onboarding so future boxes are born reboot-safe.

## This is a monitoring problem, and monitoring should own it

Here is the part that matters for Glassmkr. Every signal needed to catch this is already on the box: `nouveau` is loaded, the NVIDIA hardware is present, and no `nvidia` kernel module is bound to it. That combination means one thing: this host's GPU will not come back after a reboot. A monitor that can see the running kernel modules can see that, and it can tell you while the box is still up and earning, not after it has gone dark.

That is the honest version of "predictive." Not guessing at the future, just noticing a loaded gun on the table. This check now ships in Crucible and runs on the exact fleet that hit the trap. It warns in both cases: when an NVIDIA box is already running without its driver bound, and, the more useful one, while the box is still healthy but nouveau was never blacklisted, so the danger is only latent. Either way you blacklist nouveau and rebuild the initramfs in a window you chose, instead of learning about it from a drop in your earnings.

## Check one thing today

If you run NVIDIA GPUs on a marketplace, check one thing: `lsmod | grep -e nouveau -e nvidia`. If you see `nouveau` and not `nvidia`, or you never blacklisted nouveau, you are one reboot from an outage. Fix it in a window you control. And if you want something watching for the class of latent fault that only bites on reboot, that is what we built Glassmkr to catch.

Published June 30, 2026. By Simon Rybisar.

[← All posts](/blog)