# Health checks

Health checks find problems on a server that a chart does not show well. Examples are a failed systemd service, a process killed for lack of memory, and a pending security update. Spectate runs the checks on every report from the Spectate agent and can fire an alert for each one. Health checks need agent 2.1 or later.

## Where you see health checks

- The **Overview** tab of a host shows the health panel. It lists the checks that need attention first. It then shows the passing checks in one line, for example "All 7 checks passing".
- The **Hosts** page has the filter **Needs attention**. It shows the hosts with at least one check in the `warning` or `critical` status.

Each check has one of these statuses:

| Status | Meaning |
|---|---|
| `ok` | No problem. |
| `warning` | A problem that needs attention soon. |
| `critical` | A problem that needs attention now. |
| `unknown` | The host does not report this data yet. |

If the agent is older than 2.1, the checks show `unknown` with the message "Needs agent 2.1.0 or later: sudo spectate-agent update". See [update the agent](/hosts/install-the-agent#update-the-agent).

## The checks

| Check | Label | Warning | Critical | Collector |
|---|---|---|---|---|
| `services` | **Services** | | A systemd service failed | Services |
| `oom` | **Out of memory** | The kernel killed a process for lack of memory in the last 24 hours | A kill in the last hour | Pressure & out of memory |
| `reboot` | **Reboot** | The server needs a reboot | | Updates & reboot |
| `updates` | **Updates** | Security updates are pending | | Updates & reboot |
| `clock` | **Clock** | The clock is not synchronised with NTP, or it is more than 30 seconds off | | Clock |
| `files` | **File handles** | More than 80% of the maximum open files are in use | More than 95% | Connections & file handles |
| `pressure` | **Pressure** | Tasks stalled on memory or I/O more than 20% of the time, for 5 minutes | | Pressure & out of memory |

More details per check:

- **Services** names up to four failed units, for example `nginx.service failed`.
- **Updates** is `ok` when only normal updates are pending. It shows the number of pending updates.
- **Reboot** shows the packages that need the reboot, if the package manager reports them.
- **Clock** uses two sources. The agent reports whether NTP is synchronised. Spectate also measures the clock offset on every report. Spectate corrects the timestamps of the samples, but you must fix the server's clock.
- **Pressure** only uses memory and I/O pressure. CPU pressure shows in the detail, but does not change the status.

## Which hosts have which checks

Spectate leaves out a check in these cases:

- Its collector is off for the host. See [collectors](/hosts/metrics#collectors).
- The check is Linux only and the host runs another system. Only **Clock** and **File handles** run on macOS.

The **Updates** and **Reboot** checks need apt, dnf or yum. The agent runs a separate package check every hour, which only reads the package lists on the server. It never runs `apt-get update` and never installs anything.

## Health alerts

Spectate can fire one alert per check. The alert fires when the problem has lasted for the time in the table. These are the defaults:

| Alert | Default | Severity | Fires after |
|---|---|---|---|
| Services | On | Critical | At once |
| Out of memory | On | High | At once. The alert stays open until one hour after the last kill. |
| Clock | On | Medium | 30 minutes without NTP synchronisation |
| Reboot | On | Low | 1 day |
| Updates | On | Low | 1 day of pending security updates |
| File handles | Off | High | 5 minutes above 95% |
| Pressure | Off | Medium | 5 minutes above 20% |

The **Clock** alert only fires when NTP is not synchronised. A clock offset alone changes the check to `warning`, but does not fire the alert.

A health alert resolves when the problem is over. It also resolves when you turn the alert off, or when you turn off its collector. Health alerts go to the same [escalation paths](/alerting/escalation-paths) as the other alerts of the host.

The alert text tells you where to look. For example, the Services alert says `Check them with: systemctl --failed`.

## Change the health alerts of a host

Owners, managers and admins can change the health alerts.

1. Open the host and go to **Settings & alerts**.
2. Under **Health alerts**, turn each alert on or off.
3. Select a severity for each alert: **Critical**, **High**, **Medium** or **Low**.
4. Select **Save changes**.

You cannot change the time before an alert fires. If the collector of an alert is off, the dashboard says that the alert cannot fire.

## Related

- [Metric conditions](/hosts/metric-conditions): alert on a metric threshold, or when a host stops reporting.
- [Alerts](/alerting/alerts): how Spectate acknowledges and resolves alerts.