Changelog

#2026-08-27

Crucible v1.0.1

Current. The agent is at v1.0.1 on npm (@glassmkr/crucible), MIT-licensed. Install natively: sudo npm install -g @glassmkr/crucible then sudo glassmkr-crucible init.

A mistyped ingest URL is now an error instead of a confident pass. A working ingest endpoint accepts POST only, so it answers the setup probe's GET with 405. A 404 means there is no ingest endpoint at that address at all, which is the most common self-hosting mistake. Until now any response below 500 that was not an authentication failure was reported as "api key validated", so a typo produced a successful-looking install, a running service, and telemetry sent nowhere until somebody read the journal. The message now names the address to check.

Setup no longer claims to have validated your API key. The probe is an unauthenticated request against an endpoint that does not authenticate that kind of request, so a non-rejection says the endpoint is reachable and nothing about the key. It now says exactly that.

Release binaries carry a build-provenance attestation. Each binary's exact bytes are tied to the workflow, repository and commit that produced it, and you can check a download without a GitHub account. The verification steps are on the Trust page. There is no GPG key on purpose: attestation binds provenance to the build workflow itself, so there is no private key for one maintainer to hold, rotate, or lose.

Upgrade

sudo npm install -g @glassmkr/crucible@1.0.1
sudo systemctl restart glassmkr-crucible

#2026-08-02

Crucible v0.15.1

The agent was at v0.15.1 as of this entry, superseded by v1.0.1.

Three fixes from an external review of v0.15.0. No new alert rules, no configuration changes, and no alert behavior changes. Recommended for every host, and particularly for hosts started with an explicit configuration path.

A configuration path that does not exist is now an error rather than a silent fall back to defaults. Naming a configuration file that is missing, most often a typo, was reported as "using defaults" and the agent carried on under settings you never chose. On a host configured to require a minimum ipmitool version, that meant the requirement was quietly not applied. Starting the agent with no configuration path at all is unchanged and still supported: it is only a path you name explicitly that must now exist.

The ipmitool diagnostic now reports failure in its exit status. When the configuration you named cannot be read the diagnostic declines to run, which is correct, but it still reported success, so a script wrapping it read the refusal as a clean result.

Chassis readings the controller did not supply are now recorded as unknown rather than as "no fault". A controller that answered with an error, a truncated reply, or wording the agent does not recognize produced a confident "no power event, no overload, no power fault" from data that said nothing at all. Each reading is recorded independently now, and an absent one stays unknown. A genuinely empty power event, which is the common and healthy reading, is still distinguished from a missing one.

Upgrade

sudo npm install -g @glassmkr/crucible@0.15.1
sudo systemctl restart glassmkr-crucible

#2026-08-02

Crucible v0.15.0

Superseded by v0.15.1. The agent was at v0.15.0 as of this entry, on npm (@glassmkr/crucible), MIT-licensed. Install natively: sudo npm install -g @glassmkr/crucible then sudo glassmkr-crucible init.

The agent now records what the baseboard controller says about the machine's last power transition. This is the first half of answering why a server rebooted. It collects the controller's last power event, the cause it recorded for the most recent restart, what the machine is configured to do when mains power returns, and whether a power fault or overload is present right now.

This release reports these readings as facts and does not yet draw a conclusion from them. Nothing new alerts, and no existing alert changes. Vendors word these fields differently and some populate them only partially, so the readings are being gathered across a range of hardware before anything is built on top of them. Where a value is absent, that is recorded as absent rather than guessed at, and wording the agent does not recognize is preserved verbatim instead of being discarded.

Requires glassmkr-crucible init to take effect. The readings are taken through the narrow privileged helper, which gains two new permitted actions, both read-only queries of the controller. An upgrade that skips init will run the new version without collecting them.

Hosts without a baseboard controller are unaffected and simply report nothing for this section, as do hosts where the controller is unreachable or where the agent is not permitted to query it.

Upgrade

sudo npm install -g @glassmkr/crucible@0.15.0
sudo glassmkr-crucible init
sudo systemctl restart glassmkr-crucible

#2026-07-31

Crucible v0.14.14

Superseded by v0.15.0. The agent was at v0.14.14 as of this entry, on npm (@glassmkr/crucible), MIT-licensed. Install natively: sudo npm install -g @glassmkr/crucible then sudo glassmkr-crucible init.

Further hardening of the check that decides whether the agent will run ipmitool with root privileges. A single-area release with no new alert rules and no configuration changes. Recommended for every host, and particularly for hosts that run the agent as root without the privileged helper.

The check and the execution now always name the same file. On a host with no privileged helper, where the agent runs as root directly, the check examined the binary found on a fixed system path while execution looked the command up in root's own environment, so the two could be different files. Execution now uses the exact path that was checked.

Version strings are read strictly. A prerelease of the fixed version, which comes before that version and need not contain the fix, and malformed strings such as 2vendor were both treated as new enough and skipped the origin check. Both are now treated as unverifiable, which means the binary has to come from a distribution package before the agent will run it as root. Ordinary distribution versions are unaffected.

Package-ownership checks now run with a fixed language setting. The package manager translates its output, including the notices that reveal one binary has been substituted for another, and the agent previously inherited whatever language the service was started with. Under a translated setting those notices were missed.

Also. glassmkr-crucible doctor ipmi --config <path> now stops and reports when the configuration you named cannot be read, instead of quietly reporting on default settings that may enforce differently.

Upgrade

sudo npm install -g @glassmkr/crucible@0.14.14
sudo glassmkr-crucible init
sudo systemctl restart glassmkr-crucible

#2026-07-30

Crucible v0.14.13

Superseded by v0.14.14. The agent was at v0.14.13 as of this entry, on npm (@glassmkr/crucible), MIT-licensed. Install natively: sudo npm install -g @glassmkr/crucible then sudo glassmkr-crucible init.

Hardens the check that decides whether the agent will run ipmitool with root privileges, and restores monitoring of filesystems mounted under /home. No new alert rules and no configuration changes. Recommended for every host.

Three ways the root-execution check could be bypassed are closed. The agent runs ipmitool as root through a narrow privileged helper, and since 0.14.10 it only does so when the binary comes from a distribution package. Review found three gaps in that check. An ipmitool whose version string could not be parsed, such as a vendor build reporting a name rather than a number, was treated as if it were new enough and skipped the check entirely. The version was read by searching the service's own command path, while the privileged helper resolves the command through a different, fixed path, so on a host where those disagree the check could approve one file while a different, older one ran as root. And a dpkg diversion, which is the mechanism that places one binary at another's location, was misread as proof of package ownership.

A filesystem mounted under /home is monitored again. The agent's own sandbox hid /home from it, and the tool it uses to measure disks skips anything it cannot reach, so a real filesystem mounted there was missing from reports entirely: no capacity warning, no inode warning, no read-only alert, and nothing to indicate it had been skipped. The sandbox now allows reading in those locations while still preventing the agent from writing to them. This takes effect when glassmkr-crucible init runs during the upgrade below.

Smaller fixes. Prerelease Node version strings such as 22.19.0-rc.1 were accepted as meeting the minimum supported version and are now rejected, since a prerelease comes before its final release. glassmkr-crucible doctor ipmi now accepts --config, so on a host started with a custom configuration path the diagnostic reflects the settings the running agent actually applies.

Upgrade

sudo npm install -g @glassmkr/crucible@0.14.13
sudo glassmkr-crucible init
sudo systemctl restart glassmkr-crucible

Crucible v0.14.12

Superseded by v0.14.13. The agent was at v0.14.12 as of this entry, on npm (@glassmkr/crucible), MIT-licensed. Install natively: sudo npm install -g @glassmkr/crucible then sudo glassmkr-crucible init.

Node.js 22 LTS works again, and a throttled push no longer looks like a failure. No new alert rules, no configuration changes.

The minimum Node version is now 22.19.0, lowered from 24. Versions 0.14.6 through 0.14.11 refused to start on anything older than Node 24, which excluded the whole current LTS line. The requirement only ever came from one library the agent depends on, and that library's own stated minimum is 22.19.0; the check had been set from a crash seen on Node 20 and rounded up. Node 22 was verified end to end before this change. If you upgraded Node purely to satisfy the old requirement, nothing is wrong with staying on the newer version, but it is no longer necessary. Node 22.4 and older, including Node 20, are still refused, because the agent genuinely cannot load on them.

A rate-limited push is now reported as such. The dashboard accepts one snapshot per server per 55 seconds, and the agent sends one as soon as it starts, so the first push after a restart often lands inside the previous window and is rejected. That was logged as "Push failed", which reads like a real problem; it now says the push was throttled and that the next cycle will land. No data is lost either way, since the next snapshot carries current state.

Also fixes an em-dash in one status string reported for Adaptec RAID controllers, which is a house-style matter rather than a functional one.

Upgrade

sudo npm install -g @glassmkr/crucible@0.14.12
sudo glassmkr-crucible init
sudo systemctl restart glassmkr-crucible

Crucible v0.14.11

Superseded by v0.14.12. The agent was at v0.14.11 as of this entry, on npm (@glassmkr/crucible), MIT-licensed. Install natively: sudo npm install -g @glassmkr/crucible then sudo glassmkr-crucible init.

Fixes a false "filesystem remounted read-only" CRITICAL that could appear on any host running a recent agent. Single fix, no new collectors, no configuration changes. If you saw that alert on a host whose disks were fine, this is why, and upgrading clears it on the next collection cycle with no manual action.

What went wrong. The agent runs under a hardened systemd unit that includes ProtectSystem=strict, which makes / read-only inside the agent's own sandbox as a security measure. The disk collector read the mount table from its own point of view, so it saw its sandbox rather than the machine, and reported / as read-only on hosts whose root filesystem was perfectly writable. The dashboard then raised a critical alert saying the filesystem had been remounted read-only, likely from I/O errors or corruption. No data was ever at risk on those hosts; the reading was wrong, not the disk.

The fix. The collector now reads the machine's mount table, which sits outside every sandbox. Where the host's own security settings prevent that (for example a hidepid mount option, or ProtectProc / ProcSubset restrictions), the agent marks the mount options as unreliable and the dashboard reports nothing rather than guessing, because a filesystem state it cannot actually observe should not be asserted. A genuinely read-only filesystem still raises the alert exactly as before.

Because the hardened unit is installed by glassmkr-crucible init, it arrived on hosts through ordinary upgrades, so this affected most hosts on a recent agent rather than any particular configuration.

Upgrade

sudo npm install -g @glassmkr/crucible@0.14.11
sudo glassmkr-crucible init
sudo systemctl restart glassmkr-crucible

Crucible v0.14.10

Superseded by v0.14.11. The agent was at v0.14.10 as of this entry, on npm (@glassmkr/crucible), MIT-licensed. Install natively: sudo npm install -g @glassmkr/crucible then sudo glassmkr-crucible init.

Tightens which ipmitool builds the agent will run as root, and reports a BMC that was already dead when the agent started. No new alert rules and no breaking changes. On a host using its distribution's ipmitool package, which is the normal case, nothing changes.

The CVE-2020-5208 check now looks at where the binary came from, not just its version. v0.14.9 stopped blocking any ipmitool older than 1.8.19, because Debian, Ubuntu, and Red Hat all ship the security fix inside a 1.8.18 package without changing the version number, so the old check was disabling hardware monitoring on hosts that were never exposed. That reasoning holds for a distribution's package, but not for a binary compiled from source or installed by hand: nothing backported a fix into those. Since Crucible runs ipmitool as root through its privileged wrapper, it now collects from a package-owned build below 1.8.19 as before, and declines to run one that no package owns, naming the exact file. Snapshots from those hosts also record the package version, such as ipmitool 1.8.18-11ubuntu2.2, which is where the backported fix actually lives and the part ipmitool -V never shows you.

Worth knowing if you build ipmitool yourself: /usr/local/bin comes before /usr/bin in sudo's default secure_path, so a locally built copy takes precedence over the packaged one even when both are installed. command -v ipmitool shows which file actually runs. To refuse every build below 1.8.19 regardless of origin, set collection.enforce_ipmitool_min_version: true.

A BMC that was already unreachable at agent start is now reported. IPMI capability is checked once when the agent starts and refreshed hourly. If the BMC was already silent at that moment, later snapshots recorded the check as skipped and never retried, so the "BMC present but unreadable" alert could not fire for the exact situation it exists for, and a BMC that failed mid-run had its alert clear at the next refresh while still failing. The agent now retries when the cached result says there is no BMC but the kernel is still showing an IPMI device, since those two cannot both be true. Hosts genuinely without a BMC still skip the check entirely.

Mistyped settings under collection: are now reported at startup. They were silently discarded, so a typo left the setting at its default while appearing to take effect. They are still ignored rather than rejected, so a stray key can never stop the agent from starting. glassmkr-crucible doctor ipmi also now reflects your configuration rather than probing with defaults, and explains what to do when a build is refused.

Upgrade

sudo npm install -g @glassmkr/crucible@0.14.10
sudo glassmkr-crucible init
sudo systemctl restart glassmkr-crucible

Crucible v0.14.9

Superseded by v0.14.10. The agent was at v0.14.9 as of this entry, on npm (@glassmkr/crucible), MIT-licensed. Install natively: sudo npm install -g @glassmkr/crucible then sudo glassmkr-crucible init.

Restores BMC monitoring on hosts where it had been switched off, and makes an unreachable BMC visible for the first time. No new alert rules and no breaking changes; one new optional configuration key.

IPMI monitoring is no longer disabled because of the ipmitool version. If the installed ipmitool reported a version below 1.8.19, the agent previously refused to use it at all because of CVE-2020-5208, which meant no fan, power-supply, System Event Log, or IPMI-derived memory-error monitoring on that host. That check could not tell a genuinely unpatched build from a patched one: ipmitool -V reports only the upstream version number, while Debian, Ubuntu, and Red Hat all ship the security fix inside a 1.8.18 package without changing it. Stock Ubuntu 20.04 and 22.04 and RHEL-family 9 are all in that position, so hardware monitoring was being switched off on hosts that were never exposed, with no upgrade available to satisfy the check. The agent now collects normally and reports the situation instead, so you can judge it. If you would rather keep the previous behavior, set collection.enforce_ipmitool_min_version: true in /etc/glassmkr/crucible.yaml.

An unreachable BMC can now be detected. Each snapshot reports which /dev/ipmi* device the kernel created, if any, together with the outcome of that snapshot's own collection. That distinguishes a host with no BMC, where silence is correct, from a host whose BMC has stopped answering, which previously looked identical. The earlier check ran once at agent start, so a BMC that failed later could not be reported at all.

Upgrade

sudo npm install -g @glassmkr/crucible@0.14.9
sudo glassmkr-crucible init
sudo systemctl restart glassmkr-crucible

#2026-07-29

Crucible v0.14.8

Superseded by v0.14.9. The agent was at v0.14.8 as of this entry, on npm (@glassmkr/crucible), MIT-licensed. Install natively: sudo npm install -g @glassmkr/crucible then sudo glassmkr-crucible init.

A single-fix release for kernel update detection on RPM-based hosts. No new collectors or alert rules, and no configuration changes.

Pending kernel updates are no longer missed on hosts with two kernel families. "Reboot required for kernel update" could stay silent while a genuine kernel update was pending. Oracle Linux booted on UEK is the case that matters in practice: a retained RHEL-compatible kernel-core package could be read as the installed kernel and compared against the running UEK release, so the pending UEK update went unreported. The check now recognizes Oracle's kernel-uek and compares only within the kernel family the host is actually booted into. Hosts running a stock RHEL, Rocky, Alma, or Fedora kernel are unaffected, as are Debian and Ubuntu hosts.

Upgrade

sudo npm install -g @glassmkr/crucible@0.14.8
sudo glassmkr-crucible init
sudo systemctl restart glassmkr-crucible

#2026-07-28

Crucible v0.14.7

Superseded by v0.14.8. The agent was at v0.14.7 as of this entry, on npm (@glassmkr/crucible), MIT-licensed. Install natively: sudo npm install -g @glassmkr/crucible then sudo glassmkr-crucible init.

A follow-up to the 0.14.6 hardening batch. No new collectors or alert rules: a clearer failure on unsupported Node, a startup configuration safety check, and two false-positive fixes.

Fails clearly on an unsupported runtime. On Node.js older than 24 the agent now exits immediately at startup with a message naming the required version, instead of failing deep inside a dependency and restart-looping. Node 24 or newer has been required since 0.14.6.

Kernel-reboot alert no longer misfires on mainline kernels. "Reboot required for kernel update" stayed quiet on a host running a mainline or custom kernel newer than the distribution's packaged one: the check now compares kernel versions numerically and recognizes linux-image-unsigned-* packages, while a genuine pending reboot is still reported.

Config is re-checked at every startup. The config file's ownership, permissions, and access-control list are re-verified on each start, not only during init: if they are widened after setup to expose the file, the agent refuses to read it. Update-version responses are also size-bounded, matching the other outbound calls hardened in 0.14.6.

Upgrade

sudo npm install -g @glassmkr/crucible@0.14.7
sudo glassmkr-crucible init
sudo systemctl restart glassmkr-crucible

#2026-07-24

Crucible v0.14.6

Superseded by v0.14.7. The agent was at v0.14.6 as of this entry, on npm (@glassmkr/crucible), MIT-licensed. Install natively: sudo npm install -g @glassmkr/crucible then sudo glassmkr-crucible init.

Security hardening release. No new collectors or alert rules; this batch tightens the agent's filesystem, privilege, network, and output handling. Please read the upgrade notes below.

Runs unprivileged, provably. If privileged-wrapper setup fails during init, the agent now runs unprivileged and reports the affected privileged collectors as unavailable, instead of falling back to root. Root is used only with an explicit operator opt-in.

Filesystem and secret handling. Privileged setup uses descriptor-based, no-follow, exclusive operations with ownership and ACL checks across the whole directory ancestry; secrets are kept out of subprocess environments and argv where possible; external responses are size-bounded; and notifier output escapes control characters and cannot inject headers or markup.

Network egress is validated and pinned. Enrollment, dashboard push, version checks, and notifiers require HTTPS by default, reject loopback, private, and link-local destinations unless explicitly allowed, and pin the connection to the resolved public address, mitigating SSRF and DNS-rebinding.

Failures are visible. A collector that cannot run now reports as unavailable rather than as a safe or zero value, so a failing probe no longer reads as healthy.

Upgrade notes:

• Requires Node.js 24 or newer.
• The config file becomes root:glassmkr mode 0640. Run sudo glassmkr-crucible init once after upgrading to complete the migration; it preserves your file's contents. Until you do, the agent still starts but logs a warning.
• The Prometheus metrics server now binds 127.0.0.1 by default. Remote scraping needs an explicit bind address behind an authenticated proxy or ACL.
• A self-hosted dashboard on http or a private address must opt in with --allow-insecure-endpoint.

Upgrade

sudo npm install -g @glassmkr/crucible@0.14.6
sudo glassmkr-crucible init   # re-secure the config and refresh the privileged wrapper
sudo systemctl restart glassmkr-crucible

#2026-07-18

Crucible v0.14.5

Superseded by v0.14.6. The agent was at v0.14.5 as of this entry, on npm (@glassmkr/crucible), MIT-licensed.

Install-time privilege separation is more robust. A failed privilege-separation setup now removes the sudo grant reliably and says so loudly if it cannot, so a residual escalation path is never hidden; and an upgrade restarts the running service so a change to how the agent runs takes effect immediately rather than at the next manual restart.

Hardware-RAID drive visibility. A drive behind a RAID/HBA controller that the agent enumerates but cannot read is now reported as a monitoring blind spot rather than skipped, and a genuinely unreadable direct disk is no longer hidden by a healthy controller array on the same host.

Steadier CPU-temperature alerts. Alerts from duplicate IPMI sensor names keep a stable identity as sibling sensors cross the threshold, so a hot sensor no longer resolves and re-fires when another sensor changes state.

Crucible v0.14.4

Superseded by v0.14.5. The agent was at v0.14.4 as of this entry, on npm (@glassmkr/crucible), MIT-licensed. Install natively: sudo npm install -g @glassmkr/crucible then sudo glassmkr-crucible init.

Tamper-safe install. The privilege-separation setup that runs during init was redesigned so the narrow root-owned sudo wrapper cannot be pre-planted or hijacked: group memberships are applied before the install decides the wrapper location is trustworthy, the wrapper directory and its parent are verified against the service account's final groups, the wrapper is written as root with an exclusive-create temp file then atomically put in place and re-verified, and the sudoers grant is revoked if any check fails. Default installs are unaffected; an upgrade now closes a previously-open path rather than leaving it.

Disks with unreadable SMART are no longer a silent blind spot. When a fixed disk is present but its SMART health cannot be read (smartmontools not installed, or a controller that needs a device type the agent does not probe), the agent now reports it instead of omitting it, and the dashboard raises an ack-able advisory. Previously such a host looked identical to one with no disks, so a real drive failure could go unmonitored. Removable media and the 0-byte BMC virtual-media device are excluded, and the signal is suppressed on a healthy hardware-RAID host (whose virtual disk legitimately reports no SMART while its physical drives are read through passthrough).

Drive and sensor identity fixes. Drive alerts use the drive serial only when it is present and unique in the snapshot, otherwise the device path, so two drives sharing a placeholder serial are no longer merged into one; and duplicate IPMI temperature-sensor names (for example on dual-socket boards) are disambiguated so each is tracked on its own.

Crucible v0.14.3

Superseded by v0.14.4. The agent was at v0.14.3 as of this entry.

Second-round code-review remediation. Follow-up corrections to v0.14.2: per-sensor CPU-temperature alerts now notify for each socket on dual-socket machines (identical sensor labels were collapsing into one); drive alerts key on the drive serial rather than the /dev/sdX letter, so a drive that re-enumerates after a reboot no longer emits a false "resolved" plus a fresh alert; the collection loop holds a steady cadence on slow hosts (a long cycle can no longer trip a false "unreachable"); and init now checks the sudo wrapper's parent directory, not just the file, before handing collection to the unprivileged user. The v0.14.2 change that reported disk operations as a per-second rate was reverted to per-interval counts.

#2026-07-17

Crucible v0.14.2

The agent was at v0.14.2 as of this entry (superseded by v0.14.3 above), on npm (@glassmkr/crucible). MIT-licensed.

Code-review remediation. Notifications now track each failing resource on its own (a second failing disk, drive, interface, or sensor is no longer masked by the first); the softnet drop counter reads the correct kernel column; collection cycles no longer overlap; and the installer enforces root ownership of the privileged wrapper. The Docker Compose deployment was removed: it required a privileged root container contrary to the unprivileged native model, so a native install is now the only supported path.

Crucible v0.14.1

The agent was at v0.14.1 as of this entry (superseded by v0.14.2 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed.

Hardening of the hardware-RAID passthrough. Follow-up to v0.14.0: the controller device-type check is now enforced identically by the agent and its privileged wrapper (closing a gap where the wrapper allowed more shapes than intended and rejected some valid ones), and drive de-duplication is scoped so two distinct drives that report the same blank/placeholder serial both stay visible. Reading MegaRAID drives is unchanged. As with v0.14.0, a host on the unprivileged service user needs a one-time glassmkr-crucible init re-run after upgrading.

Crucible v0.14.0

The agent was at v0.14.0 as of this entry (superseded by v0.14.1 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed.

Drive health behind hardware RAID. Physical drives behind a hardware RAID or HBA controller (MegaRAID, PERC, Smart Array, 3ware, Adaptec, Areca) were previously invisible: only the controller's virtual disk is a normal block device, and it exposes no SMART, so those hosts showed no drives at all. The agent now discovers the physical drives the way smartctl does and reads each one through the controller, so their SMART health, early-warning attributes, and self-test results are collected like any direct-attached drive. Each drive is reported with its own serial and a label naming the controller family. No third-party controller tools are required. SATA drives behind the controller in this release.

Upgrade note: this release adds privileged read actions, so a host running the unprivileged service user needs a one-time glassmkr-crucible init re-run (or wrapper refresh) after upgrading, or the new hardware-RAID reads are skipped.

Upgrade

sudo npm install -g @glassmkr/crucible@0.14.2
sudo glassmkr-crucible init   # re-run to refresh and harden the privileged wrapper
sudo systemctl restart glassmkr-crucible

#2026-07-16

Crucible v0.13.26

The agent was at v0.13.26 as of this entry (superseded by v0.14.0 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed.

Drive-health collection hardening. Three fixes to the SMART parsing added in v0.13.25, ahead of wider rollout: a self-test that reports a fatal result is now always recorded as a failure (previously it could be dropped when the drive omitted the pass/fail flag); attribute 189 is read as high fly writes only on drives that name it that way, so a SATA SSD's health-flags field is never misread; and the Seagate-specific command-timeout unpacking is scoped to Seagate drives, so a legitimately large count on any other vendor is never truncated.

Crucible v0.13.25

The agent was at v0.13.25 as of this entry (superseded by v0.13.26 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed.

Expanded SMART early-warning attributes. Every drive now reports the remaining Backblaze-five failure markers and their companions: reported uncorrectable errors (187), command timeouts (188), high fly writes (189), spin retries (10), reallocation events (196), offline uncorrectable sectors (198), and UDMA CRC errors (199, an interface/cabling signal rather than a media one). Values come from the same smartctl read the agent already performs, so no new privileged access and no wrapper refresh is needed.

SMART self-test results. Each drive carries a summary of its self-test log: the newest test's outcome, plus the newest failed test with its failing LBA kept separately, so a later passing test cannot mask a read failure. Aborted or interrupted tests are never reported as failures.

NVMe error counters. Media errors and error-log entry counts from the NVMe health log, enabling growth-over-time early warnings.

Upgrade

sudo npm install -g @glassmkr/crucible@0.13.26
sudo systemctl restart glassmkr-crucible

#2026-07-15

Crucible v0.13.24

The agent was at v0.13.24 as of this entry (superseded by v0.13.25 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed.

OS support-status awareness. The agent now reports whether the host is enrolled in extended security support past its standard end-of-life (Ubuntu Pro/ESM, or a RHEL Extended Update Support repository), read strictly unprivileged (it never runs as root). The dashboard pairs this with the release lifecycle date so a past-end-of-life host that is still receiving security fixes is not reported as unsupported.

Crucible v0.13.23

The agent was at v0.13.23 as of this entry (superseded by v0.13.24 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed.

GPU PCIe slot width. The GPU collector now records the electrical width of the PCIe slot each card sits in (the upstream port's max link width), alongside the card's own maximum. This lets the dashboard tell an x16 card seated in a physically x8 slot (which correctly negotiates x8, and is not a fault) from a link that trained below the slot's capability, so gpu_pcie_link_degraded no longer warns on the former.

#2026-07-14

Crucible v0.13.22

The agent was at v0.13.22 as of this entry (superseded by v0.13.23 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed.

SSH drop-in edits are no longer invisible on RHEL-family hosts. The agent runs unprivileged, and /etc/ssh/sshd_config.d is 0700 root by default on RHEL, so a hardening change applied only via a drop-in file looked unapplied forever. When the direct read is denied, the newest drop-in timestamp is now read through a strictly read-only, fixed-path privileged helper, restoring visibility for ssh_config_unapplied. Hosts running the unprivileged service user need a one-time glassmkr-crucible init re-run (or wrapper refresh) to grant the new read.

Dual-socket DIMM channel counts are no longer halved. A board that labels its channels A-H on both sockets was collapsing 16 channels to 8; channel totals are now qualified by socket, so the memory-channel advisory reports correct populated and available counts on dual-socket systems.

SATA wear no longer misreads a temperature attribute. SMART attribute 231 is drive-wear on some SATA SSDs but temperature on others; it has been dropped from the id-only fallback, so a temperature reading is never reported as remaining drive life (a genuine wear drive still carries a wear-named attribute the parser matches).

Privileged helper hardening. The collection wrapper's device-path check now rejects path traversal (any .. or embedded slash) to mirror the TypeScript allowlist, and its secure_path includes /usr/local/bin so hand-built tools placed there resolve for the unprivileged agent.

Crucible v0.13.21

The agent was at v0.13.21 as of this entry (superseded by v0.13.22 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed.

Hands-off fleet onboarding with enroll. A new glassmkr-crucible enroll --account-key <KEY> subcommand lets you bake one account key into an Ansible, cloud-init, or post-install script and share it across the whole fleet, instead of minting and copying a separate collector key into every host. Each server derives a stable machine identity, self-registers with the dashboard, and receives its own collector key; a re-run maps back to the same server rather than creating a duplicate. The account key is used only for that one registration call and is never written to disk.

RHEL security-update count is now accurate. On RHEL-family hosts the pending-security-update count was derived from a query that also counted kernels already installed but not yet booted, inflating it and pointing at a no-op fix. It now counts only genuinely installable security packages; an installed-but-unbooted kernel shows up (correctly) as "reboot required", not "updates pending".

Correct vendor on placeholder-firmware boards. Boards whose firmware leaves the system-manufacturer as a placeholder (e.g. "To Be Filled By O.E.M.") are now identified by their baseboard manufacturer, so vendor-specific BMC parsing and guidance work on them.

Dashboard

Idempotent server onboarding. Paired with enroll above, the dashboard now recognizes a re-provisioned host by its machine identity and re-attaches it to the existing server instead of creating a duplicate that counts against your node quota.

New advisory: a drive has disappeared. A standalone (non-RAID) disk that drops off the bus or de-enumerates used to leave no signal, since the RAID and SMART checks only see drives that are present. A new trend warning flags a disk that was consistently present and has since vanished from telemetry while the host keeps reporting, identified by serial. It is advisory (a planned hot-swap looks the same from the outside), so it never pages.

Drive-wear guidance now covers SATA SSDs. The wear alert fires on SATA SSDs as well as NVMe; its remediation is now drive-type-aware (smartctl for SATA, nvme-cli for NVMe) instead of NVMe-only. The memory-channel advisory is now informational rather than a warning, matching its "not a fault" intent.

Upgrade

sudo npm install -g @glassmkr/crucible@0.13.22
sudo systemctl restart glassmkr-crucible

#2026-07-05

Crucible v0.13.20

The agent was at v0.13.20 as of this entry (superseded by v0.13.21 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed.

Per-process file-descriptor monitoring now covers root-owned processes. The per-process file-descriptor scan runs through the same privileged helper as the rest of hardware collection, so a host running the unprivileged service user now sees descriptor counts for root-owned daemons, not just its own. Before this, a root service slowly leaking file descriptors toward its limit was invisible to the agent when it ran as glassmkr. Hosts on the unprivileged service user need a one-time glassmkr-crucible init re-run (or wrapper refresh) to grant the new read.

Fan speeds now include discrete-reading sensors. Boards that expose fan tachometers as discrete sensors rather than analog readings (some Supermicro and ASRock Rack firmware) had those fans omitted from the sensor list. They are now mirrored in alongside the analog readings, so fan coverage matches what the BMC reports.

Crucible v0.13.19

The agent was at v0.13.19 as of this entry (superseded by v0.13.20 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed.

Memory-channel population is now monitored. The agent reports every DIMM slot (populated or empty) with its channel, socket, size, and rated versus configured speed, straight from the board's firmware. The dashboard raises a new advisory when memory channels sit empty or DIMMs run below rated speed: the silent bandwidth killers on multi-channel CPUs, where an 8-channel EPYC with 4 DIMMs delivers roughly half its peak memory bandwidth. Validated on Supermicro, ASRock, Gigabyte, and ASUS boards, including dual-socket EPYC. Hosts running the unprivileged service user need a one-time glassmkr-crucible init re-run (or wrapper refresh) to grant the new read.

Fixed a phantom SMART failure on virtual devices. A device smartctl could not read (a BMC's virtual-media USB device such as "AMI Virtual HDisk0", or a USB bridge needing a device type) was reported as a failing drive. Unreadable now means no SMART data, never a failure verdict.

Upgrade

sudo npm install -g @glassmkr/crucible@0.13.19
sudo systemctl restart glassmkr-crucible

#2026-07-04

Crucible v0.13.18

The agent was at v0.13.18 as of this entry (superseded by v0.13.19 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed.

Fixed a false "reboot required" alert on RHEL-family hosts. On Rocky, AlmaLinux, RHEL, CentOS Stream, and Fedora, the kernel ships as the kernel-core package rather than kernel. The reboot check queried kernel, got "not installed", and raised kernel_needs_reboot on healthy hosts (showing "Installed kernel: package kernel is not installed"). It now reads kernel-core (and kernel-default on SUSE), so the running-versus-installed comparison is correct on the RHEL family. Debian and Ubuntu are unaffected.

Upgrade

sudo npm install -g @glassmkr/crucible@0.13.18
sudo systemctl restart glassmkr-crucible

#2026-07-03

Crucible v0.13.17

The agent was at v0.13.17 as of this entry (superseded by v0.13.18 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed.

Hardware and security collection restored on root hosts. A v0.13.16 change routed privileged checks (IPMI sensors, SMART, RAID, firewall, kernel logs) through a sudo helper that is only set up by the installer's init step. A host running the agent as root that had skipped that step collected none of them. The agent now falls back to reading those directly when it runs as root, so collection is never silently lost.

Crucible v0.13.16

The agent was at v0.13.16 as of this entry (superseded by v0.13.17 above), on npm and Docker Hub. Runs as the non-root glassmkr user.

SSH config changes are now checked against the running daemon. Crucible reads SSH settings from sshd -T, which reflects the config file on disk rather than the daemon that is actually running. So editing sshd_config to lock down root login but forgetting to reload sshd would clear the alert while the box stayed exposed. Crucible now compares the config's last-modified time against the daemon's last start-or-reload and raises a new ssh_config_unapplied alert while a change is saved but not yet live, so a host is never reported all-clear on an unapplied SSH change. It recognizes a systemctl reload, not just a full restart.

Upgrade

sudo npm install -g @glassmkr/crucible@0.13.17
sudo systemctl restart glassmkr-crucible

#2026-07-01

Crucible v0.13.15

The agent was at v0.13.15 as of this entry (superseded by v0.13.16 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed. Runs as the non-root glassmkr user.

SATA SSD endurance is now monitored. Crucible previously read write-endurance only from NVMe drives, so a worn SATA SSD reported no wear at all. It now reads a SATA SSD's wear indicator (the vendor-specific SMART life-remaining attribute) and reports percent-used the same way it does for NVMe, so an aging SATA SSD is no longer invisible. Paired with a dashboard drive-wear alert that adds a lower "plan replacement" watch level below the existing warning.

Upgrade

sudo npm install -g @glassmkr/crucible@0.13.15
sudo systemctl restart glassmkr-crucible

#2026-06-27

Crucible v0.13.14

The agent was at v0.13.14 as of this entry (superseded by v0.13.15 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed. Runs as the non-root glassmkr user.

Clean reboots are no longer misreported as unclean shutdowns. The reboot-evidence collector detected a clean shutdown with last shutdown -F, which returns nothing on modern systemd and util-linux even after a clean sudo reboot, so a deliberate planned reboot was escalated to a critical "unclean shutdown" alert. The collector now reads last -x -F (which surfaces the shutdown record) and treats a boot as clean when a shutdown record sits immediately before it. Together with the dashboard severity change above, a clean intentional reboot is now informational, not a page.

Crucible v0.13.13

The agent was at v0.13.13 as of this entry (superseded by v0.13.14 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed. Runs as the non-root glassmkr user.

GPU driver-resilience facts. On a host with an NVIDIA GPU, Crucible now reports whether the nvidia kernel module is loaded, whether nouveau is loaded, and whether nouveau is blacklisted, read from sysfs and /proc/modules even when nvidia-smi is dead. That is the dangerous case: if nouveau is not blacklisted it binds the GPU first on the next reboot, the NVIDIA driver cannot load, and a marketplace host silently de-lists itself. The new gpu_driver_unsafe_reboot alert warns before the reboot happens, while the fix is still non-disruptive.

kernel_needs_reboot false positive fixed. The Debian and Ubuntu reboot check trusted /var/run/reboot-required unconditionally, but that flag is set by any package that wants a reboot (libc, systemd), not just the kernel. A host whose running kernel was already the newest installed could raise a spurious reboot alert. The check now compares the running kernel to the newest installed kernel.

Dashboard: alert quality

Host-type profiles, now with auto-detect. Tag a server's role (for example a marketplace GPU box) and the alerts that are expected by design for that role are suppressed, with the reason recorded, while real faults still fire. A host that looks like a marketplace GPU box is now offered a one-click prompt to apply the profile, so the suppression does not depend on remembering to set it per box. Settable in the UI or via the API (the profile field on POST and PATCH /api/v1/servers).

Flapping detection. A rule that keeps firing and auto-resolving (an intermittent or expected-at-idle condition) is now rolled up into a single recurrence warning instead of a wall of identical alerts. The event log records state changes, not one row per snapshot, so a chronically-true alert no longer buries the real transitions.

False positives fixed. Several alerts were taught to tell a real fault from a benign one: gpu_pcie_link_degraded ignores an idle, power-capped GPU (the PCIe link downshifts to save power and retrains under load); disk_latency_high distinguishes I/O saturation under load from a failing drive; gpu_uncorrected_ecc treats a one-off lifetime SRAM bit flip as informational rather than an immediate replacement; and a clean, intentional reboot is no longer reported as a critical unexpected reboot.

Upgrade

sudo npm install -g @glassmkr/crucible@0.13.14
sudo systemctl restart glassmkr-crucible

#2026-06-26

Crucible v0.13.12

The agent was at v0.13.12 as of this entry (superseded by v0.13.13 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed. Runs as the non-root glassmkr user.

Free memory (MemFree). Crucible now collects MemFree from /proc/meminfo alongside MemAvailable, so the dashboard can split a host's memory headroom into reclaimable page cache versus genuinely unused RAM. Additive and backward compatible.

Upgrade

sudo npm install -g @glassmkr/crucible@0.13.12
sudo systemctl restart glassmkr-crucible

#2026-06-21

The programmatic API and all notification channels are now free

Re-gating. We moved the Pro line. The only things Pro now unlocks are the ones whose cost scales with use: more than 3 nodes, data retention beyond 7 days, and unlimited AI analysis. Everything else is free on every plan, including the full read and write programmatic API (account keys can be read, write, or admin scope), all notification channels with no cap, and predictive trend warnings. This reverses the earlier policy (see the 2026-05-12 tier-gating entry below) that kept the write API and premium channels behind Pro; we decided the API should not be the paywall.

Three new notification channels: Discord, PagerDuty, and webhooks

New. Alongside Email, Telegram, and Slack, you can now route alerts to Discord (incoming webhook), to PagerDuty (Events API v2, with automatic incident resolution when an alert clears), or to any HTTP endpoint as a structured JSON webhook. All six channel types are free, on every plan, with no cap. Add them under Channels in the dashboard.

#2026-06-16

Crucible v0.13.11

The agent was at v0.13.11 as of this entry (superseded by v0.13.13 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed. Runs as the non-root glassmkr user.

Drive serial number and firmware. Crucible now collects each disk's serial number and firmware revision, parsed from the SMART report it already reads (SATA and NVMe). These surface in the dashboard's hardware view and in drive alerts, where the serial is the identifier a hardware replacement or provider ticket needs. No configuration change; a drive whose firmware does not report them is handled gracefully.

Upgrade

sudo npm install -g @glassmkr/crucible@0.13.11
sudo systemctl restart glassmkr-crucible

#2026-06-11

Crucible v0.13.10

The agent was at v0.13.10 as of this entry (superseded by v0.13.11 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed. Runs as the non-root glassmkr user.

PSI availability notice (RHEL family). Stock CentOS, Alma, Rocky, and RHEL kernels ship Pressure Stall Information disabled, so the cpu/memory/io pressure rules cannot fire there. The agent now says so once at startup, with the remedy (the psi=1 boot parameter). Previously the gap was silent. Details in the troubleshooting entry.

Correct primary IP. On boards where the BMC's virtual USB interface enumerates first (common on Supermicro), the dashboard and notifications showed a link-local 169.254.x address as the server IP. The agent now prefers the first global-scope address.

Docker quickstart fixed. The compose file now pulls from Docker Hub, which allows anonymous pulls; the previous ghcr.io default requires authentication, so the documented docker compose up failed. Unused container plumbing the agent never read was removed, and the install docs now create the config file the agent actually reads.

Dashboard: notification dedup for event alerts

Event-type alerts (IPMI SEL, GPU XID) stack repeat occurrences into one card. Stacking previously treated every re-evaluation of a still-recent event as a new occurrence and re-sent the notification, which could repeat once per collection interval for as long as the event stayed in the rule's window. Stacking and re-notification now happen only when a genuinely new event arrives, and acknowledgements stick. Server-side fix, already live; no agent upgrade required for it.

Upgrade

sudo npm install -g @glassmkr/crucible@0.13.10
sudo systemctl restart glassmkr-crucible

#2026-06-07

Crucible v0.13.9

The agent was at v0.13.9 as of this entry (superseded by v0.13.10 above), on npm (@glassmkr/crucible) and at the registries ghcr.io/glassmkr/crucible and docker.io/glassmkr/crucible. MIT-licensed. Runs as the non-root glassmkr user.

Detection fixes (RHEL/Fedora family only). 0.13.9 broadens the dnf-automatic auto-updates check to recognize the full set of affirmative apply_updates values (yes, true, on, 1, case-insensitive) and anchors the match, and it makes /etc/os-release parsing tolerant of non-standard whitespace and quoting so the distro family always resolves correctly. Debian/Ubuntu hosts are unaffected. No change to the data collected, the CLI, the config schema, or the snapshot payload.

#2026-06-03

Crucible v0.13.8

The agent was at v0.13.8 as of this entry (superseded by v0.13.9 above), on npm (@glassmkr/crucible) and at the registries ghcr.io/glassmkr/crucible and docker.io/glassmkr/crucible. MIT-licensed. Runs as the non-root glassmkr user.

Internal refactor, no behavior change. 0.13.8 extracts two more shared helpers from per-collector code: a rate-tracker for cumulative /proc counters (used by the conntrack and softnet collectors) and key=value plus columnar /proc parsers (used by the systemd-unit and TCP-stats collectors). The rate and parse behavior is unchanged, and the data collected, the CLI, the config schema, and the snapshot payload are all unchanged; another maintainability pass, verified by the full test suite. No action required beyond a routine upgrade when convenient.

Crucible v0.13.7

The agent was at v0.13.7 as of this entry (superseded by v0.13.8 above), on npm (@glassmkr/crucible) and at the registries ghcr.io/glassmkr/crucible and docker.io/glassmkr/crucible. MIT-licensed. Runs as the non-root glassmkr user.

Internal refactor, no behavior change. 0.13.7 deduplicates repeated collector code into shared helpers (file reads, CLI and systemd-unit presence checks, kernel-log reading and timestamp parsing, and /etc/os-release parsing) and removes dead code. The data collected, the CLI, the config schema, and the snapshot payload are all unchanged; this is a maintainability and footprint pass, verified by the full test suite. No action required beyond a routine upgrade when convenient.

Upgrade

sudo npm install -g @glassmkr/crucible@latest
sudo systemctl restart glassmkr-crucible

Or via Docker: docker pull ghcr.io/glassmkr/crucible:latest.

#2026-05-29

Crucible v0.13.6

The agent was at v0.13.6 as of this entry (superseded by v0.13.7, see 2026-06-03). On npm (@glassmkr/crucible) and at the registries ghcr.io/glassmkr/crucible and docker.io/glassmkr/crucible. MIT-licensed. Runs as the non-root glassmkr user.

Security fix. 0.13.6 fixes a security false-negative on RHEL-family hosts: download-only dnf-automatic timers were treated as "auto-updates configured," which suppressed the pending_security_updates alert. Affected hosts: any RHEL-family host with dnf-automatic.timer enabled but apply_updates = no in /etc/dnf/automatic.conf. Upgrade to 0.13.6 to receive the corrected classification; the alert then fires correctly when security patches are pending and no working auto-apply mechanism is configured. Debian/Ubuntu hosts were never affected (that path already inspected the config contents).

Interim versions. 0.13.4 was a documentation sweep and 0.13.5 renamed the on-disk config file from collector.yaml to crucible.yaml (with a backwards-compatible read of the old name); neither changed alerting behavior.

Upgrade

sudo npm install -g @glassmkr/crucible@latest
sudo systemctl restart glassmkr-crucible

Or via Docker: docker pull ghcr.io/glassmkr/crucible:latest.

#2026-05-22

Crucible v0.13.3

The agent was at v0.13.3 as of this entry (superseded by v0.13.6, see 2026-05-29). On npm (@glassmkr/crucible) and at the registries ghcr.io/glassmkr/crucible and docker.io/glassmkr/crucible. MIT-licensed. Runs as the non-root glassmkr user.

Resource footprint. Validation-fleet measurement on 2026-05-21 across 7 hosts shows a median 91 MB RSS idle, near-zero CPU, and an fio delta under 1.5%. RSS ranged 65 MB to 103 MB. Updated: a later measurement against the 0.13.6 fleet shows a median around 108 MB; see the docs spec table for current numbers.

Default interval. 60 seconds (set in v0.10.0; the previous 300-second default is gone).

Rule library at 68 rules across 9 categories

Categories: storage, ZFS, filesystem, memory and CPU, network, hardware (BMC / IPMI), time and services, security and patching, GPU. 20 rules ship with deep FIX content (safe-mode, validation, rollback, impact counters); the rest are verified. GPU coverage is 9 rules across three tiers (nvidia-smi / DCGM exporter / Redfish OEM stub), validated on NVIDIA L4, A4000, and A16.

Browse all rules at /docs/rules; the machine-readable corpus is at /llms-full.txt.

Upgrade

sudo npm install -g @glassmkr/crucible@latest
sudo systemctl restart glassmkr-crucible

Or via Docker: docker pull ghcr.io/glassmkr/crucible:latest.

#2026-05-13

Crucible 0.9.4

Fixed. PSU sensor classification now works across every observed BMC vendor (Supermicro, Gigabyte, ASRockRack, ASUS). Previously the PS<N> name shape was Dell-gated and four of five PSU-having boxes were silently filtered out, so per-PSU alerts never fired regardless of state. The bitmask interpretation now follows IPMI 2.0 spec table 42-3 (Failure detected, AC lost, predictive, inactive).

Changed. When the agent cannot probe IPMI at all (no ipmitool, no /dev/ipmi0, etc.), the snapshot now emits null for ipmi.ecc_errors and sel_entries_count rather than stub zeros. The dashboard renders this as "no signal (BMC not probed)" instead of the misleading "0 / 0".

Changed. IPMI capability detection re-runs every hour. Installing ipmitool after the agent started is picked up automatically; no service restart needed.

Added. glassmkr-crucible doctor ipmi subcommand for customer self-diagnosis. See /docs/troubleshooting/ipmi.

Tier-gating policy for the programmatic API

The programmatic API surface (account keys) is now uniformly gated by Pro plan. Free customers retain full web-dashboard access and full read API access for their own data; programmatic writes (channel CRUD, alert acknowledge / resolve, server CRUD, mutes, restore endpoints, key management) and Pro features (AI analysis, trend warnings) return 402 pro_required when called via an API key on a Free plan. Web-dashboard sessions are unaffected at any tier. Full breakdown at /docs/api/tier-gating.

Unexpected-reboot alerts now auto-resolve

An unexpected_reboot alert that has been firing for at least 24 hours of continuous stable uptime now auto-resolves with resolution_reason: auto_decay_stable_24h. The original incident remains in the resolved-alerts history. Tunable per-server via config_overrides.unexpected_reboot_decay_hours.

#2026-05-12 (later still)

Rate-based ECC error detection

The ecc_errors alert rule now uses a rolling 24-hour rate window instead of a cumulative threshold. Default: a warning fires when more than 10 correctable errors are observed in 24 hours. Uncorrectable ECC errors continue to fire critical immediately on any non-zero count.

This eliminates false-positive alerts on long-running hosts where BMC error counters accumulate over months without indicating an active hardware issue.

When the BMC counter is cleared (SEL reset, BMC reboot), the rule skips one evaluation cycle and resumes on the next snapshot; no false alerts from the reset itself.

#2026-05-12 (later)

API key management UI + scopes

Pro customers can now manage API keys via the dashboard at /settings/keys. New keys can be created with one of three scopes (Read, Write, or Admin), an optional expiry date (up to 5 years), and graceful 48-hour rotation. The old key keeps working through the grace window so automation can be updated without downtime, then auto-revokes. Emergency revoke is one click away.

Reminder emails fire 7 and 1 days before key expiry, plus a notification when a key is auto-revoked on expiry. Audit log view is at /settings/audit with date-range and action filters (Pro plan; 365-day retention).

#2026-05-12

Pro-tier API gating

Programmatic API access (creating account API keys, rotating account keys, reading the audit log, plus programmatic server management via account keys: create, rename, delete, rotate collector keys) is now restricted to Pro-plan customers. Free-plan accounts can continue to ingest snapshots from their 3 free servers and use the dashboard to manage them; programmatic management requires Pro.

If you encounter a 402 response on an endpoint that previously worked, your account may have been downgraded or never had Pro access. Existing API keys created before this change keep working for ingest; manage them via the dashboard if you need to rotate or delete.

#2026-05-09

Hardware visibility on the dashboard

Server tiles and detail pages now show the hardware vendor and product detected by the monitoring agent (e.g., "GIGABYTE / R292-4S1-00"), plus an IPMI badge when sensor data is available. This helps identify what is actually on each box, especially when server names do not match the underlying hardware.

#2026-05-08

Stripe billing enforcement

Pro customers without a payment method on file now have servers beyond the 3-server free quota disabled at the end of their billing period. The oldest 3 servers stay active. Snapshot ingest continues for disabled servers, so historical data is preserved and restoration is instant once a card is added.

Affected customers receive a sequence of warning emails: when a payment method is removed, 3 days before disable, 1 day before disable, and at the moment of disable. Restoration is a single-click operation: Settings → Disabled servers → Restore all.

#2026-05-07

cpu_temperature_high reads hwmon directly, with IPMI fallback

Most customers see no change. The evaluator now reads CPU thermals from the kernel's hwmon interface (more accurate, vendor-agnostic), falling back to IPMI sensor data when hwmon is not available.

Customers running on Gigabyte AMD platforms (B650, B660, X670, EPYC) see fewer false-positive alerts. Their BMC firmware (12.61, the shipping default on these boards) reports a CPU<N>_DTS IPMI sensor that runs about 30 C hotter than the actual CPU die temperature exposed via the kernel. The agent (Crucible 0.9.1+) drops the inflated CPU<N>_DTS sensor whenever a CPU<N>_TEMP sibling exists on the same socket.

ECC alerts now fire on Dell and HPE iDRAC platforms too

Previously, ECC alerts only fired when the BMC exposed a named "correctable / uncorrectable" sensor. Dell iDRAC and some HPE iLO firmwares report ECC events only via the System Event Log (SEL), not as named sensors, so customers on those platforms were missing alerts silently. This release adds a SEL-derived counter as a second source.

install.sh simplified

The bootstrap installer at glassmkr.com/install.sh now delegates configuration and systemd setup to the glassmkr-crucible init subcommand. The --dashboard-key argument is preserved as an alias for --api-key so existing automation keeps working.