Changelog

#2026-07-05

Crucible v0.13.20

Current. The agent is at v0.13.20 on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed.

Per-process file-descriptor monitoring now covers root-owned processes. The per-process file-descriptor scan runs through the same privileged helper as the rest of hardware collection, so a host running the unprivileged service user now sees descriptor counts for root-owned daemons, not just its own. Before this, a root service slowly leaking file descriptors toward its limit was invisible to the agent when it ran as glassmkr. Hosts on the unprivileged service user need a one-time glassmkr-crucible init re-run (or wrapper refresh) to grant the new read.

Fan speeds now include discrete-reading sensors. Boards that expose fan tachometers as discrete sensors rather than analog readings (some Supermicro and ASRock Rack firmware) had those fans omitted from the sensor list. They are now mirrored in alongside the analog readings, so fan coverage matches what the BMC reports.

Crucible v0.13.19

The agent was at v0.13.19 as of this entry (superseded by v0.13.20 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed.

Memory-channel population is now monitored. The agent reports every DIMM slot (populated or empty) with its channel, socket, size, and rated versus configured speed, straight from the board's firmware. The dashboard raises a new advisory when memory channels sit empty or DIMMs run below rated speed: the silent bandwidth killers on multi-channel CPUs, where an 8-channel EPYC with 4 DIMMs delivers roughly half its peak memory bandwidth. Validated on Supermicro, ASRock, Gigabyte, and ASUS boards, including dual-socket EPYC. Hosts running the unprivileged service user need a one-time glassmkr-crucible init re-run (or wrapper refresh) to grant the new read.

Fixed a phantom SMART failure on virtual devices. A device smartctl could not read (a BMC's virtual-media USB device such as "AMI Virtual HDisk0", or a USB bridge needing a device type) was reported as a failing drive. Unreadable now means no SMART data, never a failure verdict.

Upgrade

sudo npm install -g @glassmkr/[email protected]
sudo systemctl restart glassmkr-crucible

#2026-07-04

Crucible v0.13.18

The agent was at v0.13.18 as of this entry (superseded by v0.13.19 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed.

Fixed a false "reboot required" alert on RHEL-family hosts. On Rocky, AlmaLinux, RHEL, CentOS Stream, and Fedora, the kernel ships as the kernel-core package rather than kernel. The reboot check queried kernel, got "not installed", and raised kernel_needs_reboot on healthy hosts (showing "Installed kernel: package kernel is not installed"). It now reads kernel-core (and kernel-default on SUSE), so the running-versus-installed comparison is correct on the RHEL family. Debian and Ubuntu are unaffected.

Upgrade

sudo npm install -g @glassmkr/[email protected]
sudo systemctl restart glassmkr-crucible

#2026-07-03

Crucible v0.13.17

The agent was at v0.13.17 as of this entry (superseded by v0.13.18 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed.

Hardware and security collection restored on root hosts. A v0.13.16 change routed privileged checks (IPMI sensors, SMART, RAID, firewall, kernel logs) through a sudo helper that is only set up by the installer's init step. A host running the agent as root that had skipped that step collected none of them. The agent now falls back to reading those directly when it runs as root, so collection is never silently lost.

Crucible v0.13.16

The agent was at v0.13.16 as of this entry (superseded by v0.13.17 above), on npm and Docker Hub. Runs as the non-root glassmkr user.

SSH config changes are now checked against the running daemon. Crucible reads SSH settings from sshd -T, which reflects the config file on disk rather than the daemon that is actually running. So editing sshd_config to lock down root login but forgetting to reload sshd would clear the alert while the box stayed exposed. Crucible now compares the config's last-modified time against the daemon's last start-or-reload and raises a new ssh_config_unapplied alert while a change is saved but not yet live, so a host is never reported all-clear on an unapplied SSH change. It recognizes a systemctl reload, not just a full restart.

Upgrade

sudo npm install -g @glassmkr/[email protected]
sudo systemctl restart glassmkr-crucible

#2026-07-01

Crucible v0.13.15

The agent was at v0.13.15 as of this entry (superseded by v0.13.16 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed. Runs as the non-root glassmkr user.

SATA SSD endurance is now monitored. Crucible previously read write-endurance only from NVMe drives, so a worn SATA SSD reported no wear at all. It now reads a SATA SSD's wear indicator (the vendor-specific SMART life-remaining attribute) and reports percent-used the same way it does for NVMe, so an aging SATA SSD is no longer invisible. Paired with a dashboard drive-wear alert that adds a lower "plan replacement" watch level below the existing warning.

Upgrade

sudo npm install -g @glassmkr/[email protected]
sudo systemctl restart glassmkr-crucible

#2026-06-27

Crucible v0.13.14

The agent was at v0.13.14 as of this entry (superseded by v0.13.15 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed. Runs as the non-root glassmkr user.

Clean reboots are no longer misreported as unclean shutdowns. The reboot-evidence collector detected a clean shutdown with last shutdown -F, which returns nothing on modern systemd and util-linux even after a clean sudo reboot, so a deliberate planned reboot was escalated to a critical "unclean shutdown" alert. The collector now reads last -x -F (which surfaces the shutdown record) and treats a boot as clean when a shutdown record sits immediately before it. Together with the dashboard severity change above, a clean intentional reboot is now informational, not a page.

Crucible v0.13.13

The agent was at v0.13.13 as of this entry (superseded by v0.13.14 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed. Runs as the non-root glassmkr user.

GPU driver-resilience facts. On a host with an NVIDIA GPU, Crucible now reports whether the nvidia kernel module is loaded, whether nouveau is loaded, and whether nouveau is blacklisted, read from sysfs and /proc/modules even when nvidia-smi is dead. That is the dangerous case: if nouveau is not blacklisted it binds the GPU first on the next reboot, the NVIDIA driver cannot load, and a marketplace host silently de-lists itself. The new gpu_driver_unsafe_reboot alert warns before the reboot happens, while the fix is still non-disruptive.

kernel_needs_reboot false positive fixed. The Debian and Ubuntu reboot check trusted /var/run/reboot-required unconditionally, but that flag is set by any package that wants a reboot (libc, systemd), not just the kernel. A host whose running kernel was already the newest installed could raise a spurious reboot alert. The check now compares the running kernel to the newest installed kernel.

Dashboard: alert quality

Host-type profiles, now with auto-detect. Tag a server's role (for example a marketplace GPU box) and the alerts that are expected by design for that role are suppressed, with the reason recorded, while real faults still fire. A host that looks like a marketplace GPU box is now offered a one-click prompt to apply the profile, so the suppression does not depend on remembering to set it per box. Settable in the UI or via the API (the profile field on POST and PATCH /api/v1/servers).

Flapping detection. A rule that keeps firing and auto-resolving (an intermittent or expected-at-idle condition) is now rolled up into a single recurrence warning instead of a wall of identical alerts. The event log records state changes, not one row per snapshot, so a chronically-true alert no longer buries the real transitions.

False positives fixed. Several alerts were taught to tell a real fault from a benign one: gpu_pcie_link_degraded ignores an idle, power-capped GPU (the PCIe link downshifts to save power and retrains under load); disk_latency_high distinguishes I/O saturation under load from a failing drive; gpu_uncorrected_ecc treats a one-off lifetime SRAM bit flip as informational rather than an immediate replacement; and a clean, intentional reboot is no longer reported as a critical unexpected reboot.

Upgrade

sudo npm install -g @glassmkr/[email protected]
sudo systemctl restart glassmkr-crucible

#2026-06-26

Crucible v0.13.12

The agent was at v0.13.12 as of this entry (superseded by v0.13.13 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed. Runs as the non-root glassmkr user.

Free memory (MemFree). Crucible now collects MemFree from /proc/meminfo alongside MemAvailable, so the dashboard can split a host's memory headroom into reclaimable page cache versus genuinely unused RAM. Additive and backward compatible.

Upgrade

sudo npm install -g @glassmkr/[email protected]
sudo systemctl restart glassmkr-crucible

#2026-06-21

The programmatic API and all notification channels are now free

Re-gating. We moved the Pro line. The only things Pro now unlocks are the ones whose cost scales with use: more than 3 nodes, data retention beyond 7 days, and unlimited AI analysis. Everything else is free on every plan, including the full read and write programmatic API (account keys can be read, write, or admin scope), all notification channels with no cap, and predictive trend warnings. This reverses the earlier policy (see the 2026-05-12 tier-gating entry below) that kept the write API and premium channels behind Pro; we decided the API should not be the paywall.

Three new notification channels: Discord, PagerDuty, and webhooks

New. Alongside Email, Telegram, and Slack, you can now route alerts to Discord (incoming webhook), to PagerDuty (Events API v2, with automatic incident resolution when an alert clears), or to any HTTP endpoint as a structured JSON webhook. All six channel types are free, on every plan, with no cap. Add them under Channels in the dashboard.

#2026-06-16

Crucible v0.13.11

The agent was at v0.13.11 as of this entry (superseded by v0.13.13 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed. Runs as the non-root glassmkr user.

Drive serial number and firmware. Crucible now collects each disk's serial number and firmware revision, parsed from the SMART report it already reads (SATA and NVMe). These surface in the dashboard's hardware view and in drive alerts, where the serial is the identifier a hardware replacement or provider ticket needs. No configuration change; a drive whose firmware does not report them is handled gracefully.

Upgrade

sudo npm install -g @glassmkr/[email protected]
sudo systemctl restart glassmkr-crucible

#2026-06-11

Crucible v0.13.10

The agent was at v0.13.10 as of this entry (superseded by v0.13.11 above), on npm (@glassmkr/crucible) and on Docker Hub (docker.io/glassmkr/crucible, anonymous pulls). MIT-licensed. Runs as the non-root glassmkr user.

PSI availability notice (RHEL family). Stock CentOS, Alma, Rocky, and RHEL kernels ship Pressure Stall Information disabled, so the cpu/memory/io pressure rules cannot fire there. The agent now says so once at startup, with the remedy (the psi=1 boot parameter). Previously the gap was silent. Details in the troubleshooting entry.

Correct primary IP. On boards where the BMC's virtual USB interface enumerates first (common on Supermicro), the dashboard and notifications showed a link-local 169.254.x address as the server IP. The agent now prefers the first global-scope address.

Docker quickstart fixed. The compose file now pulls from Docker Hub, which allows anonymous pulls; the previous ghcr.io default requires authentication, so the documented docker compose up failed. Unused container plumbing the agent never read was removed, and the install docs now create the config file the agent actually reads.

Dashboard: notification dedup for event alerts

Event-type alerts (IPMI SEL, GPU XID) stack repeat occurrences into one card. Stacking previously treated every re-evaluation of a still-recent event as a new occurrence and re-sent the notification, which could repeat once per collection interval for as long as the event stayed in the rule's window. Stacking and re-notification now happen only when a genuinely new event arrives, and acknowledgements stick. Server-side fix, already live; no agent upgrade required for it.

Upgrade

sudo npm install -g @glassmkr/[email protected]
sudo systemctl restart glassmkr-crucible

#2026-06-07

Crucible v0.13.9

The agent was at v0.13.9 as of this entry (superseded by v0.13.10 above), on npm (@glassmkr/crucible) and at the registries ghcr.io/glassmkr/crucible and docker.io/glassmkr/crucible. MIT-licensed. Runs as the non-root glassmkr user.

Detection fixes (RHEL/Fedora family only). 0.13.9 broadens the dnf-automatic auto-updates check to recognize the full set of affirmative apply_updates values (yes, true, on, 1, case-insensitive) and anchors the match, and it makes /etc/os-release parsing tolerant of non-standard whitespace and quoting so the distro family always resolves correctly. Debian/Ubuntu hosts are unaffected. No change to the data collected, the CLI, the config schema, or the snapshot payload.

#2026-06-03

Crucible v0.13.8

The agent was at v0.13.8 as of this entry (superseded by v0.13.9 above), on npm (@glassmkr/crucible) and at the registries ghcr.io/glassmkr/crucible and docker.io/glassmkr/crucible. MIT-licensed. Runs as the non-root glassmkr user.

Internal refactor, no behavior change. 0.13.8 extracts two more shared helpers from per-collector code: a rate-tracker for cumulative /proc counters (used by the conntrack and softnet collectors) and key=value plus columnar /proc parsers (used by the systemd-unit and TCP-stats collectors). The rate and parse behavior is unchanged, and the data collected, the CLI, the config schema, and the snapshot payload are all unchanged; another maintainability pass, verified by the full test suite. No action required beyond a routine upgrade when convenient.

Crucible v0.13.7

The agent was at v0.13.7 as of this entry (superseded by v0.13.8 above), on npm (@glassmkr/crucible) and at the registries ghcr.io/glassmkr/crucible and docker.io/glassmkr/crucible. MIT-licensed. Runs as the non-root glassmkr user.

Internal refactor, no behavior change. 0.13.7 deduplicates repeated collector code into shared helpers (file reads, CLI and systemd-unit presence checks, kernel-log reading and timestamp parsing, and /etc/os-release parsing) and removes dead code. The data collected, the CLI, the config schema, and the snapshot payload are all unchanged; this is a maintainability and footprint pass, verified by the full test suite. No action required beyond a routine upgrade when convenient.

Upgrade

sudo npm install -g @glassmkr/crucible@latest
sudo systemctl restart glassmkr-crucible

Or via Docker: docker pull ghcr.io/glassmkr/crucible:latest.

#2026-05-29

Crucible v0.13.6

The agent was at v0.13.6 as of this entry (superseded by v0.13.7, see 2026-06-03). On npm (@glassmkr/crucible) and at the registries ghcr.io/glassmkr/crucible and docker.io/glassmkr/crucible. MIT-licensed. Runs as the non-root glassmkr user.

Security fix. 0.13.6 fixes a security false-negative on RHEL-family hosts: download-only dnf-automatic timers were treated as "auto-updates configured," which suppressed the pending_security_updates alert. Affected hosts: any RHEL-family host with dnf-automatic.timer enabled but apply_updates = no in /etc/dnf/automatic.conf. Upgrade to 0.13.6 to receive the corrected classification; the alert then fires correctly when security patches are pending and no working auto-apply mechanism is configured. Debian/Ubuntu hosts were never affected (that path already inspected the config contents).

Interim versions. 0.13.4 was a documentation sweep and 0.13.5 renamed the on-disk config file from collector.yaml to crucible.yaml (with a backwards-compatible read of the old name); neither changed alerting behavior.

Upgrade

sudo npm install -g @glassmkr/crucible@latest
sudo systemctl restart glassmkr-crucible

Or via Docker: docker pull ghcr.io/glassmkr/crucible:latest.

#2026-05-22

Crucible v0.13.3

The agent was at v0.13.3 as of this entry (superseded by v0.13.6, see 2026-05-29). On npm (@glassmkr/crucible) and at the registries ghcr.io/glassmkr/crucible and docker.io/glassmkr/crucible. MIT-licensed. Runs as the non-root glassmkr user.

Resource footprint. Validation-fleet measurement on 2026-05-21 across 7 hosts shows a median 91 MB RSS idle, near-zero CPU, and an fio delta under 1.5%. RSS ranged 65 MB to 103 MB. Updated: a later measurement against the 0.13.6 fleet shows a median around 108 MB; see the docs spec table for current numbers.

Default interval. 60 seconds (set in v0.10.0; the previous 300-second default is gone).

Rule library at 65 rules across 9 categories

Categories: storage, ZFS, filesystem, memory and CPU, network, hardware (BMC / IPMI), time and services, security and patching, GPU. 20 rules ship with deep FIX content (safe-mode, validation, rollback, impact counters); the rest are verified. GPU coverage is 8 rules across three tiers (nvidia-smi / DCGM exporter / Redfish OEM stub), validated on NVIDIA L4, A4000, and A16.

Browse all rules at /docs/rules; the machine-readable corpus is at /llms-full.txt.

Upgrade

sudo npm install -g @glassmkr/crucible@latest
sudo systemctl restart glassmkr-crucible

Or via Docker: docker pull ghcr.io/glassmkr/crucible:latest.

#2026-05-13

Crucible 0.9.4

Fixed. PSU sensor classification now works across every observed BMC vendor (Supermicro, Gigabyte, ASRockRack, ASUS). Previously the PS<N> name shape was Dell-gated and four of five PSU-having boxes were silently filtered out, so per-PSU alerts never fired regardless of state. The bitmask interpretation now follows IPMI 2.0 spec table 42-3 (Failure detected, AC lost, predictive, inactive).

Changed. When the agent cannot probe IPMI at all (no ipmitool, no /dev/ipmi0, etc.), the snapshot now emits null for ipmi.ecc_errors and sel_entries_count rather than stub zeros. The dashboard renders this as "no signal (BMC not probed)" instead of the misleading "0 / 0".

Changed. IPMI capability detection re-runs every hour. Installing ipmitool after the agent started is picked up automatically; no service restart needed.

Added. glassmkr-crucible doctor ipmi subcommand for customer self-diagnosis. See /docs/troubleshooting/ipmi.

Tier-gating policy for the programmatic API

The programmatic API surface (account keys) is now uniformly gated by Pro plan. Free customers retain full web-dashboard access and full read API access for their own data; programmatic writes (channel CRUD, alert acknowledge / resolve, server CRUD, mutes, restore endpoints, key management) and Pro features (AI analysis, trend warnings) return 402 pro_required when called via an API key on a Free plan. Web-dashboard sessions are unaffected at any tier. Full breakdown at /docs/api/tier-gating.

Unexpected-reboot alerts now auto-resolve

An unexpected_reboot alert that has been firing for at least 24 hours of continuous stable uptime now auto-resolves with resolution_reason: auto_decay_stable_24h. The original incident remains in the resolved-alerts history. Tunable per-server via config_overrides.unexpected_reboot_decay_hours.

#2026-05-12 (later still)

Rate-based ECC error detection

The ecc_errors alert rule now uses a rolling 24-hour rate window instead of a cumulative threshold. Default: a warning fires when more than 10 correctable errors are observed in 24 hours. Uncorrectable ECC errors continue to fire critical immediately on any non-zero count.

This eliminates false-positive alerts on long-running hosts where BMC error counters accumulate over months without indicating an active hardware issue.

When the BMC counter is cleared (SEL reset, BMC reboot), the rule skips one evaluation cycle and resumes on the next snapshot; no false alerts from the reset itself.

#2026-05-12 (later)

API key management UI + scopes

Pro customers can now manage API keys via the dashboard at /settings/keys. New keys can be created with one of three scopes (Read, Write, or Admin), an optional expiry date (up to 5 years), and graceful 48-hour rotation. The old key keeps working through the grace window so automation can be updated without downtime, then auto-revokes. Emergency revoke is one click away.

Reminder emails fire 7 and 1 days before key expiry, plus a notification when a key is auto-revoked on expiry. Audit log view is at /settings/audit with date-range and action filters (Pro plan; 365-day retention).

#2026-05-12

Pro-tier API gating

Programmatic API access (creating account API keys, rotating account keys, reading the audit log, plus programmatic server management via account keys: create, rename, delete, rotate collector keys) is now restricted to Pro-plan customers. Free-plan accounts can continue to ingest snapshots from their 3 free servers and use the dashboard to manage them; programmatic management requires Pro.

If you encounter a 402 response on an endpoint that previously worked, your account may have been downgraded or never had Pro access. Existing API keys created before this change keep working for ingest; manage them via the dashboard if you need to rotate or delete.

#2026-05-09

Hardware visibility on the dashboard

Server tiles and detail pages now show the hardware vendor and product detected by the monitoring agent (e.g., "GIGABYTE / R292-4S1-00"), plus an IPMI badge when sensor data is available. This helps identify what is actually on each box, especially when server names do not match the underlying hardware.

#2026-05-08

Stripe billing enforcement

Pro customers without a payment method on file now have servers beyond the 3-server free quota disabled at the end of their billing period. The oldest 3 servers stay active. Snapshot ingest continues for disabled servers, so historical data is preserved and restoration is instant once a card is added.

Affected customers receive a sequence of warning emails: when a payment method is removed, 3 days before disable, 1 day before disable, and at the moment of disable. Restoration is a single-click operation: Settings → Disabled servers → Restore all.

#2026-05-07

cpu_temperature_high reads hwmon directly, with IPMI fallback

Most customers see no change. The evaluator now reads CPU thermals from the kernel's hwmon interface (more accurate, vendor-agnostic), falling back to IPMI sensor data when hwmon is not available.

Customers running on Gigabyte AMD platforms (B650, B660, X670, EPYC) see fewer false-positive alerts. Their BMC firmware (12.61, the shipping default on these boards) reports a CPU<N>_DTS IPMI sensor that runs about 30 C hotter than the actual CPU die temperature exposed via the kernel. The agent (Crucible 0.9.1+) drops the inflated CPU<N>_DTS sensor whenever a CPU<N>_TEMP sibling exists on the same socket.

ECC alerts now fire on Dell and HPE iDRAC platforms too

Previously, ECC alerts only fired when the BMC exposed a named "correctable / uncorrectable" sensor. Dell iDRAC and some HPE iLO firmwares report ECC events only via the System Event Log (SEL), not as named sensors, so customers on those platforms were missing alerts silently. This release adds a SEL-derived counter as a second source.

install.sh simplified

The bootstrap installer at glassmkr.com/install.sh now delegates configuration and systemd setup to the glassmkr-crucible init subcommand. The --dashboard-key argument is preserved as an alias for --api-key so existing automation keeps working.