Updates, guides, and operational knowledge from the team.
We ran a drive-health campaign across thirteen boxes to find out whether early warning actually works. One drive was carrying 477 reallocated sectors and reported its own health as PASSED in all 140 readings we took, a full write-and-verify pass over it came back clean, and the four healthy drives we tried hardest to destroy refused to break. Including the part we cannot yet answer.
Glassmkr now speaks the Model Context Protocol: an AI client connects to your fleet with a browser sign-in and can query it, and with the scopes you grant, manage it. The interesting part was making a delete safe to hand an LLM: a scope you grant, a two-step confirmation, and a soft delete you can undo.
Dedicated servers ship with console access and traffic graphs; the layer that predicts hardware failure is usually the customer's problem to assemble. Every step of that layer, registering servers, installing the agent, routing and testing alert channels, is an API call, and that changes who can offer it.
Most fleets onboard into monitoring by copying one powerful credential onto every box. glassmkr-crucible enroll does the opposite: one account key stays in your control plane, each host self-registers by a stable machine ID and holds only its own collector key, and running the automation twice does nothing.
We gave nine open-weight models root on real broken servers and graded every claim against the machine, not the model's own report. The thing that most decided success was not model size. It was whether our harness spoke the language the model was trained in: North scores 0 in plain text and 7 with tools, Phi-4 is the mirror.
We ran our standing validation exercise with Claude Haiku 4.5, the smallest model in the family, on a real bare-metal server with four firing alerts. It fixed what it should, correctly declined to touch healthy hardware, and surfaced a false-positive bug in our own monitoring.
Three alert rules fired on every box in our GPU fleet, and all three were correct readings of config a marketplace GPU host is supposed to have. The same signal is a finding on one host and required config on another, so we taught Glassmkr the difference with host-type profiles.
On these NVIDIA boxes nouveau was never blacklisted, so the next routine reboot would have stopped every GPU verifying and silently dropped the host off the Vast marketplace. The failure mode, the one-line fix, and why monitoring should catch the latent fault before the reboot.
A field report from twenty GPU boxes on Vast: the honest breakdown of 88 alerts down to 11. What was real hardware, what was noise we muted on purpose, what was self-inflicted, and the false positives we then fixed in the product.
We built the self-hosted-Gemma path for our ticket-draft feature, A/B'd it against a plain template on a live degraded-array alert, and shipped the template. The model was fine. Fine did not justify 22 seconds and a hallucination surface. What we kept, what we cut, and why the value was never the prose.
A customer's Ryzen 9 5950X lost its VRM and asked if Glassmkr would have caught it. The honest answer was no: voltage alone cannot watch a DVFS core rail. What we built instead (a behavioral signal and a variance-aware voltage-drift signal), what happened when we backtested it on our own MC12-LE0, and the uptime check that stopped us claiming a win we had not earned.
Four tests from the code that runs Glassmkr, and the incident that put each one there: a suppressed security alert, a temperature threshold that means different things on different boards, a 404 that has to stay a 404, and a power-supply name captured from a lying BMC. A test suite as a map of what has hurt you.
Three cards, eight rules. We tested three live on real hardware (power-cap throttling on a multi-die A16, thermal load on an L4, driver drift across all three) and deliberately did not induce the other five, because a real uncorrected-ECC or XID fault needs a power cycle to clear. The honest map of what 'GPU monitoring, tested' means.
On RHEL-family hosts, download-only dnf-automatic timers were treated as 'auto-updates configured,' silently suppressing the pending_security_updates alert while Critical patches sat unapplied. What the bug was, how dogfooding caught it, and the fix in Crucible 0.13.6.
Six specific footguns from running monitoring across Supermicro, Gigabyte, ASUS and ASRockRack on Debian, Ubuntu, Rocky, Alma and Proxmox. SEL timestamp shapes, BMC firmware that lies about vendor, distro-specific package gaps, and the Gigabyte DTS +30 °C offset.
60 minutes of stale alerts after a legitimate fix. A kernel reboot that fired its own critical alert. Two distinct bugs surfaced when we actually applied our own remediation guidance end to end on the validation fleet.
A server reported 'IPMI: Not detected' while showing ECC counts on the same screen. The spec said our vendor detection was wrong. An hour-long audit said detection was correct and three other things were broken. Map the problem before you write the code.
Furnace reads your alerts, looks at the evidence, and suggests remediation. Self-hosted Gemma 4 26B in Amsterdam. Conservative, hedging, willing to say 'I don't know'. The AI in your monitoring shouldn't be the headline.
We forbade an AI from using its training data and made it resolve real infrastructure alerts using only the guidance our own dashboard produces. Three gap patterns surfaced. All three fixed in the same week.
We retrained our drive-failure predictor on 2 years of Backblaze data (222M drive-days) on the CPU of our L4 inference server. Gemma stayed resident in VRAM. 59 minutes, no new compute, 5.8% inference overhead. Plus the feature-importance surprise: SMART 197 beat SMART 187.
Two pieces, one philosophy: Crucible (the open-source agent) plus Dashboard (the optional SaaS). Built by operators with a decade of bare metal experience across 60+ global locations.
Three-way benchmark of Gemma 4 26B-A4B, Qwen3.6 35B-A3B no-think, and Qwen3.6 35B-A3B thinking on a production infrastructure health analysis prompt. Real wall-clock numbers, VRAM footprints, and the quality-latency tradeoff that matters for narration.
A practical guide to monitoring IPMI sensors, SEL logs, and BMC health on Dell, Supermicro, and HPE servers. Covers kipmi0 CPU issues, vendor quirks, and what to alert on.
How we deployed Gemma 4 26B on an NVIDIA L4 for AI health analysis of bare metal servers. Covers model selection, why vLLM failed, quantization choices, and prompting for structured infrastructure output.
Most monitoring tools stop at the OS. Below it sits an entire hardware layer: disk firmware predicting its own failure, fans at 0 RPM, ECC memory correcting silent errors. Here is what to monitor and why.
Cloud monitoring tools were built for ephemeral workloads. They track HTTP latency and container restarts. But when you run physical servers, the failure modes are fundamentally different: drives wear out, DIMM slots develop bit errors, fans fail silently, and RAID arrays degrade without anyone noticing.
