IPMI troubleshooting

#How Crucible detects IPMI

Detection is capability-based, not vendor-allowlist. Crucible does not look at your BMC vendor string and decide whether to support you; it asks "can I actually talk to the BMC?" and uses the answer.

The probe chain at agent start, and on every re-check:

1. ipmitool binary check. ipmitool -V, which needs no BMC access and so works as the unprivileged service user. A missing binary surfaces as no_ipmitool_binary.
2. Version note. A version below 1.8.19 is recorded as an advisory (CVE-2020-5208) and does not stop collection when the binary comes from a distro package. An unowned build below that version is refused. See below.
3. Reachability probe. The privileged sensor read. Non-empty output means the BMC answered. An empty result surfaces as no_bmc_device; a thrown error surfaces as execution_failed.

The result is a structured detection.reason field, with one of five values: no_ipmitool_binary, permission_denied, no_bmc_device, execution_failed, or ipmitool_cve_2020_5208. The Dashboard surfaces the reason under "IPMI: Not detected" so you know which fix to apply.

Note what that chain cannot tell you, and what 0.14.9 adds. Because the binary is checked before any BMC contact, no_ipmitool_binary says nothing about whether a BMC exists; and because no_bmc_device is produced whenever the reachability probe comes back empty, it covers a genuinely absent BMC and a BMC that has stopped answering alike. Agent 0.14.9 therefore reports two additional facts on every snapshot: which /dev/ipmi* node the kernel created, if any, and whether that snapshot's own collection succeeded. A device node present with a failed collection is a BMC that exists and is not responding, which is the condition ipmi_monitoring_unavailable alerts on.

The startup capability is re-checked periodically, so installing ipmitool or loading the kernel modules after the agent started is picked up without a restart. The device-node and collection-status facts above are re-evaluated on every snapshot rather than periodically, which is what lets a BMC failing mid-life be noticed.

#Detection vs collection: they can disagree, by design

It is normal for the Dashboard to report "IPMI: Not detected" on a host where some hardware metrics still appear. This is not a bug: detection and collection use different data sources.

The header IPMI verdict reflects Crucible's BMC probe. The dashboard's CPU temperature, fan, and ECC blocks can also be populated from non-BMC sources:

• CPU temperature often comes from hwmon (kernel-side, no BMC needed) or lm-sensors.
• ECC counters can come from kernel EDAC (/sys/devices/system/edac/mc/mc*/{ce,ue}_count) on systems where the BIOS exposes them, completely separate from the BMC.
• SMART, RAID, network, disk usage are kernel-side and do not depend on IPMI at all.

When the agent cannot probe IPMI at all, the snapshot emits null for ECC and SEL counters; the Dashboard renders that as "no signal (BMC not probed)" instead of the misleading "0 / 0" reading.

#Self-diagnose with glassmkr-crucible doctor ipmi

The doctor subcommand runs the same probes the agent uses and prints actionable guidance for each failure mode. It is read-only and does not modify system state.

sudo glassmkr-crucible doctor ipmi

The available case looks like:

IPMI capability check:
  Result:        [OK] IPMI detected via ipmitool_in_band
  ipmitool:      1.8.19

Crucible will collect:
  - Sensor readings (temperature, fan, voltage, power)
  - SEL events (recent + cumulative ECC counters)
  - PSU redundancy state (per-PSU + aggregate)

Failure cases print the matching detection.reason plus a fix recipe.

#no_ipmitool_binary

Meaning: the /dev/ipmi0 device exists, but ipmitool is not installed.

Fix: install the package:

• Debian / Ubuntu: sudo apt install ipmitool
• RHEL / Rocky / Alma: sudo dnf install ipmitool
• Arch: sudo pacman -S ipmitool
• Alpine: sudo apk add ipmitool

No restart needed. The next collection cycle (within ~5 minutes at the default interval) sees the binary, and the next hourly re-check flips detection.available to true. The Dashboard updates on the following ingest.

#permission_denied

Meaning: Crucible cannot open /dev/ipmi0. The device node is mode 0600 owned by root.

Fix: Crucible runs as the non-root glassmkr user; the install script provisions a udev rule granting that user read access. If you customized the service unit, confirm:

systemctl cat glassmkr-crucible | grep '^User='
ls -l /dev/ipmi0

The default install ships a udev rule at /etc/udev/rules.d/99-glassmkr-ipmi.rules that grants the glassmkr group access. If you removed it, restore via the install script or run the agent as root (less preferred).

#no_bmc_device

Meaning: ipmitool is installed and runs, but the kernel has no IPMI device node and the in-band ipmitool probe could not open one. Usually the kernel modules are not loaded.

sudo modprobe ipmi_si ipmi_devintf ipmi_msghandler
ls -l /dev/ipmi0    # should appear after the modules load

If /dev/ipmi0 still does not appear, the host may genuinely have no BMC. This is common on consumer hardware, Raspberry Pi, laptops, and virtual machines without IPMI passthrough. In that case set collection.ipmi: false in /etc/glassmkr/crucible.yaml (legacy installs: /etc/glassmkr/collector.yaml; the agent reads either) to silence the snapshot field; the dashboard stops trying to render IPMI for this host.

#execution_failed

Meaning: ipmitool ran, but the call returned an error other than "could not open device". The BMC is reachable in some sense but not responding the way Crucible expected.

Fix: reproduce by hand and read the error:

sudo ipmitool mc info

Common causes:

• The BMC is in a degraded state and dropped the request. Retry; if it persists, escalate via the support path below.
• The in-band interface (KCS or SSIF) is busy. Sustained busy state usually means firmware is mid-task; wait a few minutes and retry.
• The installed ipmitool is too old for the BMC's IPMI 2.0 dialect. Upgrade ipmitool via the distribution package manager.

Do not run sudo ipmitool mc reset cold without first confirming with your hardware vendor. Some BMCs do not recover cleanly from a cold reset and hang past the operation, which on a remote machine is much worse than the original failure.

#ipmitool_cve_2020_5208

Meaning: ipmitool is installed and working, but its version reads older than 1.8.19. CVE-2020-5208 is a set of buffer overflows in ipmitool's parsers that a malicious or compromised BMC could reach.

From agent 0.14.9 a distro-packaged build no longer stops monitoring. Versions 0.14.6 through 0.14.8 refused to run any ipmitool below 1.8.19, which switched off fan, PSU, SEL, and IPMI-derived memory-error monitoring entirely. That check could not distinguish a patched build from an unpatched one, because ipmitool -V reports only the upstream version number while Debian, Ubuntu, and Red Hat all ship the security fix inside a 1.8.18 package without changing it. Stock Ubuntu 20.04 and 22.04 and RHEL-family 9 are all in that position, so monitoring was being disabled on hosts that were never exposed, with no upgrade available to satisfy the check.

Agent 0.14.10 and newer decide on where the binary came from, because "the distro patched it" is a claim only a distro package can make:

• Owned by a dpkg or rpm package. Crucible collects normally and records the package version alongside the reading, so you see ipmitool 1.8.18-11ubuntu2.2 rather than the bare 1.8.18. That release suffix is where the backported fix lives, and it is the part ipmitool -V does not show you. This is the ordinary case on a supported distribution.
• Owned by nothing. A build from source, a vendor tarball, or a hand-installed binary carries no backport guarantee, so it really may be unpatched. Crucible runs ipmitool as root through its privileged wrapper, so it declines to run that one and reports this reason code. The detection.detail field names the exact path.

Watch out for shadowing. /usr/local/bin comes before /usr/bin in sudo's default secure_path, so a locally built ipmitool wins over the packaged one even when both are installed. Run command -v ipmitool to see which file actually runs. Removing the unowned copy is usually the fix.

To refuse every below-floor build regardless of origin, set collection.enforce_ipmitool_min_version: true in /etc/glassmkr/crucible.yaml. Note that unknown keys under collection are ignored rather than rejected, so a typo leaves the setting off; from 0.14.10 the agent logs a warning naming any key it dropped.

If you want to confirm your build carries the fix, and it very likely does on a supported distribution:

# Debian / Ubuntu: is there a newer candidate, and is the installed one patched?
apt-cache policy ipmitool
zcat /usr/share/doc/ipmitool/changelog.Debian.gz | grep -i 5208

# RHEL family: same two questions
dnf -q list --available ipmitool
rpm -q --changelog ipmitool | grep -i 5208
dnf updateinfo list --cve CVE-2020-5208

A non-zero count means your build already carries the fix, even though its version string still reads 1.8.18, so the host is not exposed. Stock Ubuntu 20.04 and 22.04 and RHEL-family 9 are all in this position. On RHEL-family hosts the changelog cites internal bug ids rather than CVE numbers, so also try dnf updateinfo list --cve CVE-2020-5208, which returns nothing when no security update is outstanding.

If a newer package genuinely exists, installing it is still worthwhile for its own sake: confirm ipmitool -V reports 1.8.19 or later and restart the agent. Do not build an unpatched ipmitool from source merely to satisfy a version string.

Nothing about this state requires touching the BMC. If a remediation suggests resetting it, that guidance is wrong for this reason code.

#Per-vendor notes

Crucible's detection is capability-based, so any BMC that responds to standard IPMI 2.0 commands works. These notes are vendor-specific quirks observed on real hardware, not detection-gating rules.

Supermicro

Usually clean. The BMC reports vendor strings cleanly via ipmitool mc info (Manufacturer Name: Supermicro or Super Micro Computer Inc.). PSU sensors typically appear as PS1 Status / PS2 Status with the discrete-state bitmask in the Reading column.

Gigabyte

The BMC sometimes reports Manufacturer Name: Unknown (0x3C0A) in ipmitool mc info output, even though the IANA manufacturer ID (15370) resolves to Gigabyte. This is a Gigabyte BMC firmware quirk; Crucible does not gate detection on the manufacturer string, so no customer action is needed. PSU sensors typically appear as PS1_Status with an underscore separator.

ASUS

Validated on RS700-E10-RS4U. Detection works correctly when ipmitool is installed; the most common issue is that distributions sometimes ship without ipmitool by default, which surfaces as no_ipmitool_binary in the doctor output. Install via the per-distro command above.

ASRockRack

DMI sys_vendor may read "To Be Filled By O.E.M." on some boards (a known firmware default), but the BMC itself reports vendor cleanly via ipmitool mc info (Manufacturer Name: ASRock Rack Incorporation). PSU sensors appear as PSU1 Status / PSU2 Status.

Dell PowerEdge (iDRAC)

In-band IPMI through iDRAC works without an iDRAC Enterprise license. The license gates out-of-band IPMI over LAN, not the in-band KCS path Crucible uses. PSU sensors appear as PS1 Status / PS2 Status, and iDRAC also exposes an aggregate PS Redundancy sensor that Crucible reads for whole-pair redundancy state.

Dell iDRAC compatibility has not been validated on real hardware in our validation fleet. If you hit a detection or collection issue specific to iDRAC, file a support request with the output of sudo ipmitool mc info and sudo glassmkr-crucible doctor ipmi.

HP ProLiant (iLO)

In-band IPMI via KCS usually works without an iLO Advanced license. The license gates out-of-band iLO features, not in-band IPMI. Some older iLO firmware revisions require ipmitool 1.8.18 or later for IPMI 2.0 compatibility.

HP iLO compatibility has not been validated on real hardware in our validation fleet. Same support-request convention as Dell above.

#A note on PSU monitoring

The isPsuSensor classifier covers Supermicro, Gigabyte, ASRockRack, and ASUS naming conventions, and interprets discrete states as IPMI 2.0 spec table 42-3 hex bitmasks (Failure detected, AC lost, predictive, inactive) in addition to text-status strings.

If a multi-PSU box previously showed two healthy PSUs in the dashboard but one was actually failed or unplugged, that is the bug shape that current Crucible releases catch.

#When to file a support request

Email [email protected] when:

• Your BMC vendor is not in the validated list above, and detection works (the doctor output shows [OK]) but a specific collection path (sensors, SEL, PSU) returns unexpected values.
• Detection fails (doctor output shows [FAIL]) but sudo ipmitool mc info works fine when you run it interactively.
• The doctor subcommand returns execution_failed with an error message not covered above.

Attach:

• The doctor output: sudo glassmkr-crucible doctor ipmi 2>&1
• A successful raw probe: sudo ipmitool mc info 2>&1
• One hour of agent logs: sudo journalctl -u glassmkr-crucible --since "1 hour ago" --no-pager > crucible.log
• Your server ID from the Dashboard.

Last verified: 2026-05-22 against Crucible v0.13.3.