The setup

We run a standing validation exercise against our own product. The rules are deliberately unforgiving:

• A real bare-metal server. This round: an ASUS board, an EPYC 9754, Ubuntu. Real firing alerts, the live Crucible agent, the live dashboard. No staging theater.
• A fresh AI session with zero context about our company, our codebase, or our conventions. It gets exactly three things: SSH to the box, our public docs, and a Glassmkr API key.
• The API key is read-only. That is the detail that keeps the whole thing honest. The session cannot acknowledge or resolve anything through the API; it gets a 403 if it tries. The only way an alert clears is if the underlying condition on the server actually changes and our agent observes it on its next snapshot. The scoreboard cannot be talked into a passing grade by the thing being graded.
• The session works only from what the product tells it: the alert message, the attached evidence, the fix workflow. The moment it has to reach for outside knowledge, we have found a documentation gap worth fixing.

We have run six of these rounds. The earlier ones used mid-size and large models (Sonnet, Opus). This round we used Claude Haiku 4.5, the smallest current model in the family, on purpose, running in a stock Claude Code session with native tools and no custom harness. A large model can quietly paper over a weak instruction with its own background knowledge. A small model mostly has only what you wrote down. If your guidance carries the smallest reader to the right answer, it will carry anyone.

The round

Four alerts were firing when the session connected:

• ssh_root_password (warning): root login with a password was enabled.
• ipmi_sel_critical (critical): power events logged in the BMC's system event log.
• ipmi_fan_failure (critical): 2 of 9 fans reported failed.
• memory_channels_underpopulated (warning): 8 of 12 memory channels populated, an advisory our agent derives from the board's SMBIOS DIMM topology.

Working only from the product, Haiku:

• Hardened SSH exactly per the fix workflow. It wrote the drop-in config, validated it with sshd -t, reloaded the daemon without dropping live sessions, verified the effective setting, and tested a fresh connection before trusting it. The alert auto-resolved on the next agent snapshot.
• Read the system event log, recognized the power events as transient and already deasserted, and cleared the log per the workflow. Auto-resolved.
• Checked the fans, found each one healthy, and declined to "fix" hardware that was working. (Hold that thought.)
• Read the memory advisory, confirmed the DIMM population with dmidecode, and classified it correctly as an intentional capacity tradeoff to acknowledge, not a fault to repair. That alert is designed to stay open until a human acknowledges it, and it did.

The twist, because honesty is the point

Score at the end of the session: four alerts down to two. One of the two was the memory advisory, exactly where it belonged, waiting for a human. The other was the fan alert, and it would not clear no matter what Haiku did, because clearing it was never in Haiku's power. The bug was ours.

Those two "failed" fans are discrete PSU sensors. They report a status string, not an RPM number. Our rule read the missing RPM as 0 RPM and counted the fan as stopped, even though the BMC itself reported the sensor as ok. The alert had been a false positive since the machine was onboarded, and no operator action on earth could have resolved it.

We shipped the fix the same day. The BMC's own ok verdict now wins over a missing RPM reading, and the agent-side mirror of that logic is in the public Crucible repo (crucible PR #55). The phantom alert auto-resolved across the fleet within a snapshot cycle.

Notice what did and did not happen, because it is the whole reason we trust the result. Haiku's judgment that the fans were healthy was correct. Its report that the fan alert was therefore resolved was wrong, because the alert was stuck on our bug, not on the fans. Right action, wrong conclusion about the outcome. We are not going to tell you the model was flawless; it was not. The honest version is more useful: the smallest model in the family followed our guidance to a correct action on everything it touched, and the exercise surfaced a real defect that no amount of internal review had caught.

What we believe after six rounds

• Remediation guidance should be tested like code, against an oracle that cannot be sweet-talked. Read-only credentials are a one-line way to buy that property.
• Small models make better probes than large ones. If the documentation only works when the reader is brilliant, the documentation does not work.
• The valuable output is never the pass. It is the specific, fixable gap: the wrong binary name, the missing step, the false positive that survives for months because nobody stares at an alert they have already filed under "known."

This was the small, careful end of a larger experiment. We also ran the same blind remediation across a ladder of open-weight models, from a 120B down to an 8B, and the results there were stranger and more instructive.

If you want to see what Glassmkr surfaces on a real host, the live demo is open with no signup. The Crucible agent is open source on GitHub. And for the fuller picture of how we test the product against itself, here is what our test suite looks like, and why.