Host metrics
The Spectate agent reports metrics for CPU, memory, disk, network and the system. The host page in the dashboard shows them as charts, and metric conditions use them to fire alerts. The agent collects a sample every 15 seconds and sends the samples once a minute.
The host page
Open a host from the Hosts page. The host page has three tabs: Overview, Metrics and Settings & alerts.
The header shows the platform, the kernel, the virtualization, the agent version and Last seen. If a newer agent release exists, the header shows it next to the agent version. If the server's clock differs from Spectate's clock by more than 30 seconds, the header also shows the difference under Clock.
Overview tab
The Overview tab shows, from top to bottom:
- The health checks of the host (agent 2.1 or later).
- System uptime, the number of CPU cores, and the total memory and swap.
- Charts for CPU, Memory, Load, Network and Disk. Select a period of 6h, 24h, 7d or 30d. Select a network interface for the network chart and a mountpoint for the disk chart.
- A table of the disks, with the space used per mountpoint.
- CPU inventory: the model and cores of each CPU.
Metrics tab
The Metrics tab shows all charts of the host, in five sections. Select a period of 6h, 24h (default), 7d or 30d.
| Section | Charts |
|---|---|
| CPU | CPU use, load, and CPU pressure |
| Memory | RAM and swap, Memory pressure, and Out of memory kills |
| Disk | Space per mountpoint, Throughput, Operations, Busy, Latency and I/O pressure |
| Network | Traffic, and TCP connections (established, time wait and orphaned) |
| System | File handles: the share of the system-wide maximum in use |
The disk I/O charts show all disks together. If the host has more than one disk, select one disk in the list next to the Disk heading.
Charts of agent 2.1 metrics show only when the host reports them. A host with an older agent, or a collector that is off, has fewer charts. On macOS, the agent does not report pressure, out of memory kills or TCP connections in the time wait state.
Collectors
A collector is the part of the agent that reads one group of metrics. The agent always runs the base collectors: CPU, memory, load, disk, network, uptime and the agent's own metrics.
With agent 2.1 or later, you can turn off these collectors per host:
| Collector | Label in the dashboard | Systems |
|---|---|---|
diskio | Disk I/O | Linux, macOS |
pressure | Pressure & out of memory | Linux |
services | Services | Linux with systemd |
updates | Updates & reboot | Linux with apt, dnf or yum |
sockets | Connections & file handles | Linux, macOS |
clock | Clock | Linux |
To turn a collector off or on:
- Open the host and go to Settings & alerts.
- Under Collection, clear or select the collector.
- Select Save changes.
The change applies at the agent's next report. A collector that is off also hides its health checks and resolves their alerts.
A collector that has nothing to report on a server sends nothing, not zeros. For example, Linux before kernel 4.20 has no pressure data.
Metrics and units
These are the metrics that you can see in charts and use in metric conditions. The key is the name in the public API and in alert data.
Summary metrics
Spectate calculates these from the agent's raw values:
| Key | Label | Unit |
|---|---|---|
cpu_usage | CPU usage | % (100 minus idle) |
memory_usage | Memory usage | % (100 minus available) |
disk_usage | Disk usage | % per mountpoint |
load_average | Load average | The 1-minute load average |
network_in | Network received | KB per 60 seconds, per interface |
network_out | Network sent | KB per 60 seconds, per interface |
CPU
| Key | Label | Unit |
|---|---|---|
system.cpu.user | CPU user | % |
system.cpu.system | CPU system | % |
system.cpu.iowait | CPU I/O wait | % |
system.cpu.stolen | CPU steal | % |
system.cpu.idle | CPU idle | % |
system.cpu.guest | CPU guest | % |
system.cpu.interrupt | CPU interrupt | % |
system.pressure.cpu.some | CPU pressure | % of time that tasks waited (agent 2.1) |
Memory and swap
| Key | Label | Unit |
|---|---|---|
system.memory.total | Memory total | MB |
system.memory.used | Memory used | MB |
system.memory.free | Memory free | MB |
system.memory.available | Memory available | MB |
system.memory.available_percent | Memory available | % |
system.swap.total | Swap total | MB |
system.swap.used | Swap used | MB |
system.swap.free | Swap free | MB |
system.swap.free_percent | Swap free | % |
system.swap.in | Swap in | MB |
system.swap.out | Swap out | MB |
system.pressure.memory.some | Memory pressure | % of time that tasks waited (agent 2.1) |
Disk
| Key | Label | Unit |
|---|---|---|
system.disk_usage.total | Disk total | KB per mountpoint |
system.disk_usage.used | Disk used | KB per mountpoint |
system.disk_usage.free | Disk free | KB per mountpoint |
system.disk_usage.inodes_total | Disk inodes total | Inodes |
system.disk_usage.inodes_used | Disk inodes used | Inodes |
system.disk_usage.inodes_free | Disk inodes free | Inodes |
system.disk_usage.inodes_used_percent | Disk inodes used | % |
system.io.busy_percent | Disk busy | % of time, per disk (agent 2.1) |
system.io.await_ms | Disk latency | Milliseconds per operation, per disk (agent 2.1) |
system.pressure.io.some | I/O pressure | % of time that tasks waited (agent 2.1) |
The disk I/O charts also show bytes read and written per second and operations per second.
Network
| Key | Label | Unit |
|---|---|---|
system.net.packets_recv | Packets received | Packets per 60 seconds |
system.net.packets_sent | Packets sent | Packets per 60 seconds |
system.net.err_in | Network receive errors | Per 60 seconds |
system.net.err_out | Network send errors | Per 60 seconds |
system.net.drop_in | Network receive drops | Per 60 seconds |
system.net.drop_out | Network send drops | Per 60 seconds |
system.net.fifo_in | Network receive FIFO | Per 60 seconds |
system.net.fifo_out | Network send FIFO | Per 60 seconds |
system.net.tcp.established | TCP connections | Connections (agent 2.1) |
system.net.tcp.time_wait | TCP connections in TIME_WAIT | Connections (agent 2.1) |
Network values are amounts per 60 seconds, also when the agent collects every 15 seconds. The v1 agent used the same unit, so thresholds keep their meaning after you move a host to v2.
System
| Key | Label | Unit |
|---|---|---|
system.load.1 | Load 1 minute | Load average |
system.load.5 | Load 5 minutes | Load average |
system.load.15 | Load 15 minutes | Load average |
system.uptime | System uptime | Seconds |
system.files.used_percent | File handles used | % of the system-wide maximum (agent 2.1) |
The agent also reports metrics that only the health checks use: failed services, out of memory kills, pending updates, reboot required and clock sync.
Disk and network filters
The agent skips some disks and network interfaces by default:
- File systems: network and virtual file systems, such as
nfs,nfs4,cifs,smbfs,squashfs,vfat,fuse.sshfs,cephand9p. - Network interfaces: loopback (
lo), bridges and virtual interfaces, such asbr*,docker*,veth*,virbr*,cni*,flannel*andcali*. - Disk I/O: virtual and stacked devices, such as
loop*,ram*,zram*,dm-*andmd*. Their I/O is already counted on the disks below them.
How long Spectate keeps metrics
| Data | Kept for |
|---|---|
| Samples, one every 15 seconds | 30 days |
| Hourly minimum, maximum, average and last value per metric | 400 days (about 13 months) |
When you delete a team, Spectate deletes its metrics. See data and privacy.