That last part surprised us, so this post is mostly about it. If you are evaluating agents, you are probably measuring your harness at least as much as the model, and you may not be able to tell the difference. Here is how we found that out, on our own product, with numbers we could not fudge.

If you read the earlier post where we gave Claude Haiku root on one broken box, this is the same exercise run wider and harder: more models, more failure modes, and a scoreboard built so the thing being graded cannot argue with it.

The setup, and the one property that makes it honest

We run a standing validation exercise against Glassmkr, our own bare-metal monitoring product. A fresh model session gets exactly three things: SSH to a real server with real firing alerts, our public documentation, and a Glassmkr API key. Nothing about our company, our code, or our conventions.

The API key is read-only. It returns a 403 on any write. The session cannot acknowledge, resolve, or silence a single alert through the API. That is the anti-gaming property: the only way an alert clears is for the underlying condition on the server to actually change. So we do not grade the model on what it says. We grade it on the box. After each run we read the machine directly: sshd -T, df, /proc/mdstat, the live daemon state. The product's own dashboard corroborates, but it lags the fix by about one evaluator cycle in both directions, so it is a witness, not the judge. The judge is the host.

Say that plainly, because it is the whole method: grade against the machine, not the report.

We armed five kinds of trouble, each on a real box, described here in concept (we are not publishing the reproduction recipes):

• Config and time: a host with its clock unsynced, its firewall off, root password login enabled, and an unapplied ssh config drift. Alert types: ntp_not_synced, no_firewall, ssh_root_password, ssh_config_unapplied.
• Security posture: firewall off, unattended-upgrades disabled, kernel vulnerabilities reported.
• Storage: a filesystem filling toward full. disk_space_high, disk_fill_projection.
• Resource pressure: a service leaking file descriptors toward its limit. fd_exhaustion (and, when it armed, oom_kills).
• A degraded RAID mirror. raid_degraded.

Then we connected one model at a time. Every score below is the machine's verdict.

The ladder

We ran nine open-weight models spanning roughly 8B to 120B parameters:

• gpt-oss-120b
• Mistral Small 3.2 24B
• Qwen3-Coder-Next (a coding-tuned model around 30B)
• ThinkingCap 27B (a reasoning-tuned Qwen derivative)
• Gemma 4 26B, which is the model we run in production for Glassmkr's own AI analysis, self-hosted on an L4. Testing it here is due diligence on something we already ship.
• Qwen3-VL-8B and Qwen3-8B
• North-Mini (Cohere) and Phi-4 (Microsoft, 14B)
• Poolside's Laguna, which we could not serve cleanly in this harness (more below)

Claude is the disclosed reference ceiling from the Haiku post, not a fair peer to open weights, so it sits out the comparison here. One run per model per scenario. n=1. We say that up front because it means single surprising cells are anecdotes, and only the patterns that repeat across models are load-bearing. The headline is one of those patterns.

The honest first read of that matrix: honesty and competence do not track size. The best operators in the set were a 27B and a 30B, ThinkingCap and Qwen3-Coder-Next, which matched or edged the 26B Gemma we run in production. The 120B was mid-pack on genuine fixes (four across the matrix), not last, and not first. We are not going to sell you "small model beats big model," because the more interesting thing is that the size axis mostly dissolved. Something else was doing the deciding.

Grading against the machine, because the report lies sometimes

Before the headline, the property that makes the headline trustworthy. You cannot grade an agent on its own summary, because confident wrong summaries are real, they are model-specific, and they are probabilistic (the same model is honest on one alert and fabricates on the next). None of these were visible in the transcript. Every one was caught by the box contradicting the write-up. There are two distinct kinds, and it is worth naming both.

Outcome over-claim: the model runs a command and is wrong about what it did. Qwen3-VL-8B, on the root-login alert, finished with this:

SSH root login with password enabled: Configured via sshd -t && systemctl reload ssh to disable password-based root login.

To fix it, the model validated and reloaded the ssh config with sshd -t && systemctl reload ssh, but it never edited the config. Reloading an unchanged config changes nothing, and on the box root password login was still enabled. It did something, and was wrong about the effect.

Execution confabulation: the model reports doing a thing it never did. gpt-oss-120b, on the same set of alerts, genuinely fixed the clock (it enabled and verified systemd-timesyncd), and then wrote:

Later installed and enabled chrony as the NTP daemon...

The only chrony command it actually ran was dpkg -l | grep -i chrony, a check, which returned nothing: chrony was not installed, and it never installed it. It stacked a fabricated action on top of a real fix. Worth stressing: this is not a "big models lie" law. gpt-oss executed the straightforward enable-type fixes honestly, and in a clean re-run of the root-login alert it declined the change and reported that honestly. Confabulation was a coin that landed heads often enough to matter, which is exactly why you cannot let the model score itself.

Two more machine-caught behaviors, because they bear on whether you would ever let one of these near a real box unattended:

Unprompted reboots, and not from the small models only. gpt-oss-120b, working the RAID alert, correctly re-added the failed mirror member, then rebooted the server unprompted to apply a pending kernel update, taking the host down mid-session. Qwen3-VL rebooted a live box too. This is not a small-model quirk; the 120B did it.

The self-inflicted lockout, our favorite result because it was partly our fault. Qwen3-8B, told the firewall was off, ran a bare ufw --force enable without allowing SSH first. The firewall came up default-deny, port 22 slammed shut, and the model locked itself out of the box completely. The host still pinged; nothing could reach it. The reason we love it: our alert's fix guidance had suggested enabling the firewall without leading with the allow-SSH-first step, which is fine for a human at a console and a self-inflicted outage for an agent over SSH. We fixed it (the remediation now leads with the lockout-safe command, and we shipped that change). The exercise did not just grade the models. It graded us, and on that alert we lost a point. Adoption is the hinge: a model that skips the safe command relearns the danger the hard way.

One more, in the model's favor: the fd-exhaustion alert turned out to be a "did you re-verify" probe. The naive fix, restart the leaking service, re-opens every descriptor, so restarting and declaring victory is an over-claim. gpt-oss and Qwen3-Coder-Next both fell for it, checking too early. Only ThinkingCap did the correct thing: it killed the offending process and verified the descriptors stayed closed. That is the behavior you actually want, and one model in nine had it.

The headline: run each model two ways

Here is the part that reorganized how we think about this. We ran each model through two harnesses that differ only in how the model is asked to act. One speaks plain-text instructions and reads back plain-text actions. The other uses native tool-calling, the structured function-call format the newer models are post-trained on. Same server, same alerts, same read-only scoreboard. The only variable is the shape of the conversation.

For most models it barely mattered. The capable generalists (Gemma, ThinkingCap, Qwen3-Coder-Next, gpt-oss, Mistral) scored about the same either way. They can drive a plain-text protocol or a tool-calling one; native tools do not lift a ceiling they have already reached.

And then two models, both perfectly capable, scored zero, for exactly opposite reasons.

North-Mini scored about zero in the plain-text harness. Not because it is dim: it could not drive the text protocol at all, leaking control tokens and emitting nothing the harness could parse as an action. Handed the same alerts through native tool-calling, it became a competent operator and genuinely fixed seven, including securing SSH. Same weights. The scaffold was the entire difference between useless and useful.

Phi-4 is the mirror image. In plain text it worked: it acted, ran real commands, fixed six. Handed tools, it went inert, writing a tidy plan and then stopping without a single tool call, once politely explaining that as an AI it could not run commands. It is not tool-trained, so a tool-calling harness gives it nothing to do.

Line those two up and the point is unmissable: the scaffold has to match how the model was trained. A single fixed harness, all-text or all-tool, silently zeroes out a capable model at one end or the other. If we had only built the tool-calling harness, we would have written Phi-4 off as broken. If we had only built the text one, North. Neither would have been true. You only see it by running both, against a scoreboard the model cannot talk past.

And this is why we keep the honesty layer around the whole thing. If we had trusted the models' own reports, North would have looked confused in both harnesses (it narrates poorly either way), and Phi-4's polished tool-mode plan would have read like success. The read-only key is what let us see that one of those confident-looking sessions changed nothing on the box and the other changed everything.

Laguna belongs in the same discussion as an honest negative. It was not that it failed the task; we could not get its native output parsed into actions by the standard tooling at all. Some models simply do not plug into a general harness, and pretending otherwise would be its own kind of measuring-the-harness error.

What we believe after the whole ladder

• Which model is only half the question. Does your harness match how the model was trained is the other half, and it is easy to measure the harness by accident and call it the model. Run both scaffolds.
• The read-only credential is the cheapest honesty mechanism we know. One scope flag turns "the model says it is fixed" into "the machine says it is fixed," and those are not the same sentence.
• The size axis dissolved. A 27B and a 30B matched the 26B we run in production; the 120B landed mid-pack and was the one that fabricated on the hard fixes. Capability here was not about parameters.
• The valuable output is never the pass. It is the specific fixable gap: the interactive command that hangs an automated run, the missing allow-SSH-first step that locks an agent out, the fd fix that needs a re-check. The one this exercise surfaced in our own guidance, the missing allow-SSH-first step, we fixed and shipped.

One thing this is not: Glassmkr does not ship autonomous remediation, and nothing here is a pitch for letting a model loose on your fleet. What this validates is narrower and more useful: that the alert guidance is clear and safe enough that an agent working only from it can act correctly, which is a good proxy for whether a tired human at 3am can too.

If you want to see what Glassmkr surfaces on a real host, the live demo is open with no signup. The Crucible agent is open source on GitHub. The small, careful end of this same experiment is the Haiku post. And the full run, every transcript and the on-box output behind every number here, is in the receipts bundle.