Docs Blog Trust Pricing Live demo GitHub
Self-host
View as Markdown

DOCS / TROUBLESHOOTING

Troubleshooting

Common issues with the Crucible agent and the Glassmkr Dashboard, with step-by-step solutions.

#Topic pages

  • IPMI: how Crucible detects IPMI, why "Not detected" can be correct behavior, using glassmkr-crucible doctor ipmi, per-vendor notes.

#Crucible service fails to start

Symptom: systemctl status glassmkr-crucible shows failed or inactive (dead).

  1. Check the service logs: 
    journalctl -u glassmkr-crucible --no-pager -n 50
  2. If you see a YAML parse error, re-run the init wizard with the same key to rewrite the config from scratch: 
    sudo glassmkr-crucible init --api-key <your_collector_key>
    The wizard validates the key against the Dashboard before writing the config, so a typo surfaces immediately. Common YAML mistakes include tabs instead of spaces, missing quotes around strings with special characters, and incorrect indentation.
  3. If you see permission denied, ensure the configuration file is readable: 
    ls -la /etc/glassmkr/crucible.yaml /etc/glassmkr/collector.yaml 2>/dev/null
    The file should be owned by root with mode 0600. Pre-0.13.5 installs have the file at the legacy /etc/glassmkr/collector.yaml path; the agent reads either.
  4. If you see bind: address already in use, another instance may be running: 
    pgrep -a glassmkr-crucible
    Kill the stale process and try again.

#Server shows "offline" in the dashboard

Symptom: The server card shows a gray status indicator and "last seen" is more than 2 minutes ago (the agent pushes roughly every five minutes by default; the server_unreachable rule fires after 2 missed check-ins).

  1. Check that Crucible is running: 
    systemctl status glassmkr-crucible
  2. Check network connectivity to the API: 
    curl -s -o /dev/null -w "%{http_code}" https://app.glassmkr.com/api/v1/health
    You should get 200. If not, check DNS resolution, firewall rules, and proxy settings.
  3. Check whether the collector key is valid: 
    sudo journalctl -u glassmkr-crucible --since "5 min ago" --no-pager
    If you see auth error: 401, rotate the key in the Dashboard and update /etc/glassmkr/crucible.yaml (legacy installs: /etc/glassmkr/collector.yaml; the agent reads either).
  4. Check for network-level blocks: 
    nc -zv app.glassmkr.com 443
  5. If you are behind a proxy, configure it in crucible.yaml: 
    proxy:
  https: http://proxy.internal:3128

#Metrics are delayed or missing

Symptom: The dashboard shows gaps in charts or data arrives minutes late.

  1. Check the agent's push timing: 
    sudo journalctl -u glassmkr-crucible --since "5 min ago" --no-pager
    The "Last push" value should be close to the configured interval (default 300 seconds).
  2. If pushes are slow, check the agent log for timeout errors: 
    grep -i "timeout\|retry" /var/log/glassmkr/crucible.log | tail -20
  3. If the server's clock is significantly off, snapshots may be dropped. Verify NTP is working: 
    timedatectl status
    If not synchronized: 
    sudo timedatectl set-ntp true
  4. If specific collectors are slow (e.g., SMART queries on many disks), they can delay the entire push. Inspect collector timing: 
    sudo journalctl -u glassmkr-crucible -f
    Consider increasing the interval or disabling slow collectors.

#SMART data is not appearing

Symptom: The Disk tab in the dashboard shows no SMART information.

  1. Ensure smartmontools is installed: 
    # Debian / Ubuntu
sudo apt install smartmontools

# RHEL / Rocky / Alma
sudo dnf install smartmontools
  2. Verify that smartctl can read your drives: 
    sudo smartctl -a /dev/sda
    If this fails with a permission error, Crucible's glassmkr service user needs read access (the default install handles this via udev rules).
  3. For hardware RAID controllers, drives behind the controller are not visible to smartctl without the -d flag: 
    sudo smartctl -a /dev/sda -d megaraid,0
  4. Verify the SMART collector is enabled: 
    collectors:
  smart:
    enabled: true

#IPMI, thermal, or fan data is missing

Symptom: The Hardware tab shows no temperature, fan, or PSU data.

  1. Install lm-sensors for hwmon data: 
    # Debian / Ubuntu
sudo apt install lm-sensors
sudo sensors-detect --auto
  2. For IPMI data, install ipmitool and verify it works: 
    sudo apt install ipmitool
sudo ipmitool sdr list
  3. Run the IPMI self-diagnostic: 
    sudo glassmkr-crucible doctor ipmi
    See the IPMI troubleshooting page for the full per-reason fix guide.
  4. If IPMI is not available (common on consumer hardware, cloud VMs without passthrough, laptops, Raspberry Pi), Crucible reads thermal data from hwmon directly.
  5. Confirm the thermal collector is not disabled: 
    collectors:
  thermal:
    enabled: true
    source: auto

#ZFS module not loaded

Symptom: the Storage tab shows no ZFS pools even though zpool list works on the host, or the zfs_* rules never fire.

  1. Check that the ZFS kernel module is loaded: 
    lsmod | grep zfs
    On many distributions the module is loaded on-demand by the first zpool or zfs call. If Crucible starts before that happens, it sees no ZFS surface.
  2. Force-load the module at boot: 
    echo zfs | sudo tee /etc/modules-load.d/zfs.conf
sudo systemctl restart glassmkr-crucible
  3. If lsmod | grep zfs shows nothing and you expected ZFS, install the package set for your distribution (zfsutils-linux on Debian/Ubuntu, zfs on Rocky/Alma with EPEL).
  4. If you have a kernel update pending, ZFS DKMS sometimes lags behind the running kernel; reboot or rebuild the module against the new kernel before assuming Crucible is at fault.

#GPU tier-1 (nvidia-smi) unavailable

Symptom: a server with NVIDIA GPUs reports no GPU data even though nvidia-smi works interactively.

Crucible's GPU collector probes three tiers in order: nvidia-smi (most common), DCGM exporter (preferred when present), and Redfish OEM stub (BMC-side, vendor-dependent). Validated on L4, A4000, and A16 in the validation fleet.

  1. Confirm nvidia-smi is on the PATH that systemd sees: 
    sudo systemd-run --pty --uid=glassmkr nvidia-smi
    Some distributions install nvidia-smi to /usr/lib/nvidia/current/ rather than /usr/bin/; the systemd unit's PATH may differ from your interactive shell.
  2. If the binary is found but exits non-zero, check the driver state: 
    nvidia-smi --query-gpu=name,driver_version,pstate --format=csv
    A driver loaded against a different kernel than the running one will fail here.
  3. If DCGM is installed and you want the richer dataset, ensure the exporter is running: 
    systemctl status nvidia-dcgm
  4. For BMC-side Redfish GPU telemetry (rare; vendor-specific OEM extension), confirm the BMC has the GPU sensor model populated: 
    curl -k -u user:pass https://<bmc>/redfish/v1/Systems/1/Oem/

#Pressure (PSI) alerts never fire

Symptom: cpu_pressure_high, mem_pressure_high, and io_pressure_high never fire on a CentOS, Alma, Rocky, or RHEL host, even under heavy load.

These three rules read the kernel's Pressure Stall Information. RHEL-family kernels compile PSI in but ship it disabled (CONFIG_PSI_DEFAULT_DISABLED=y), so /proc/pressure does not exist until you opt in at boot. Debian and Ubuntu enable PSI by default.

  1. Confirm PSI is the gap: 
    ls /proc/pressure
    "No such file or directory" means the kernel is not exporting PSI; the agent omits the data and the three pressure rules stay inactive on this host.
  2. Enable it with the psi=1 boot parameter: 
    sudo grubby --update-kernel=ALL --args="psi=1"
sudo reboot
  3. Verify after the reboot: 
    cat /proc/pressure/cpu
    The next Crucible snapshot picks PSI up automatically; no agent restart or config change is needed.

#Telegram notifications are not arriving

Symptom: Alerts fire in the dashboard but no Telegram messages are received.

  1. Test the channel from the dashboard or API: 
    curl -X POST https://app.glassmkr.com/api/v1/channels/CHANNEL_ID/test \
  -H "Authorization: Bearer $ACCT_KEY"
  2. If the test fails with 401 Unauthorized, the bot token is invalid. Re-create the bot via BotFather or regenerate the token.
  3. If the test fails with 400 Bad Request: chat not found, the chat ID is wrong. Common mistakes: missing the -100 prefix for supergroups, the bot was removed from the group, the bot never received any message in the chat (send a message to the bot first).
  4. If the test succeeds but real alerts do not arrive, check the channel routing. Go to Settings → Alert Defaults and confirm your Telegram channel is listed.
  5. Check the alert cooldown. By default, Glassmkr sends one notification per active alert per hour. Acknowledged or recently-notified alerts are suppressed.

#Email notifications go to spam

Symptom: Test emails arrive in the spam folder.

  1. Check the spam folder and mark messages as "not spam" to train your provider.
  2. Add [email protected] to your contacts or safe senders list.
  3. If you control the recipient domain, allow Glassmkr's SPF record. Contact support for the current IP ranges.
  4. For better deliverability, route through a custom SMTP server in your own domain. See the Channels page for setup.

#High CPU usage by Crucible

Symptom: the Crucible process uses more than 1-2% CPU consistently.

For reference, the Crucible 0.13.6 measurement across all 10 validation hosts shows a median RSS of 108 MB at steady state (range 81 to 116 MB, varies with the bundled Node version), under 1% of host RAM on every host, ~0% CPU, and fio delta under 1.5%. Sustained higher usage is unusual.

  1. Check which collectors are running: 
    sudo journalctl -u glassmkr-crucible -f
  2. SMART queries on many disks can be expensive. If you have more than 20 disks, narrow the device list or increase the interval: 
    collectors:
  smart:
    devices:
      - /dev/sda
      - /dev/sdb
  3. Per-core CPU metrics on machines with 64+ cores generate a lot of data. Disable per-core reporting if you do not need it: 
    collectors:
  cpu:
    per_core: false
  4. If the collection interval is set very low (e.g., 10 seconds), increase it: 
    collectors:
  interval_seconds: 300

#Registration fails with "server limit reached"

Symptom: + Add Server returns an error about the server limit.

  1. Hosted accounts have a 10-node per-account cap (a capacity protection, not a tier).
  2. If you have decommissioned servers still registered, delete them from the dashboard to free up slots.
  3. Need more than 10 nodes? Self-host the stack; self-hosted instances have no node limits. See /docs/self-hosting.

#My servers are disabled (lock icon, "no payment method on file")

Symptom: some server tiles show a lock-icon overlay and "Manage in Settings". Notifications stopped firing for those servers.

Why: this state is historical. Before August 2026, servers beyond a paid quota could be disabled when no payment method was on file. Glassmkr no longer charges for anything, and no server is disabled for payment reasons anymore. Disabled servers continued to ingest snapshots, so historical data is preserved.

  1. Restore in bulk: Settings → Disabled servers → Restore all. Restoration is instant; no payment method is required.
  2. Servers you no longer want count against the hosted 10-node cap; delete them from the same screen.

#Configuration changes are not taking effect

Symptom: you edited crucible.yaml (or legacy collector.yaml) but Crucible still uses the old settings.

  1. Restart the service after any configuration change: 
    sudo systemctl restart glassmkr-crucible
  2. Verify the running config by inspecting the agent's startup banner: 
    sudo journalctl -u glassmkr-crucible --since "1 min ago" --no-pager
    The first lines after restart print the resolved interval, enabled collectors, and Dashboard URL.
  3. Check that you edited the correct file. The systemd unit may pin a non-default config path: 
    systemctl show glassmkr-crucible -p Environment
  4. Environment variables override the config file. Check for any GLASSMKR_* or CRUCIBLE_* variables in the systemd unit or shell environment.

#Per-core CPU data is not showing

Symptom: the per-core CPU chart does not appear, or per-core data is missing from AI analysis.

  1. Per-core monitoring requires Crucible 0.3.0 or later. Check: 
    glassmkr-crucible --version
  2. Enable per-core in the config: 
    collectors:
  cpu:
    per_core: true
  3. Restart Crucible: 
    sudo systemctl restart glassmkr-crucible
  4. Wait for the next collection interval (default 300 seconds) for data to appear.

#Muted rules are still firing

Symptom: you muted a rule but it continues to fire alerts or send notifications.

  1. Muting takes effect on the next ingest cycle. Wait at least one collection interval after muting.
  2. If you muted via the configuration file, restart Crucible: 
    sudo systemctl restart glassmkr-crucible
  3. If you muted via the dashboard, no restart is needed; the change applies on the next push from that server.
  4. Verify the rule is muted in the dashboard under the server's Alerts tab. Muted rules show a mute icon.

#Getting help

If your issue is not covered here:

  • Capture an hour of agent logs: sudo journalctl -u glassmkr-crucible --since "1 hour ago" --no-pager > crucible.log. Attach it when contacting support.
  • Email [email protected] with your server ID and a description of the issue.

Last verified: 2026-05-31 against Crucible v0.13.6. Resource footprint figures are from a 10-host validation-fleet measurement on 2026-05-31.

We use a single authentication cookie. No tracking, no analytics. Privacy policy