Copy page

Servers

Metric conditions

A metric condition is an alert rule for host metrics, for example "disk usage above 90 for 10 minutes". You add it to one host or to a project. When the condition holds for its duration, Spectate fires an alert with the condition's severity. A separate setting fires an alert when a host stops reporting.

How a condition works

A metric condition has these parts:

Spectate checks the conditions each time the agent sends data, once a minute:

• If the metric has more than one series, one series is enough. For example, one full mountpoint fires a disk_usage condition. The alert title names the series, such as mountpoint=/data.
• A breach that is shorter than For does not fire. If the host sends no data for 5 minutes, the count starts again.
• equals matches values within 0.001 of the threshold.
• Spectate does not check data that the agent sends late from its disk buffer, if the newest sample is more than 5 minutes old. This prevents alerts about old data.

The alert title has the metric, the threshold and the host, for example Disk usage above 90 on web-01 (mountpoint=/). When the value is back on the right side of the threshold, Spectate resolves the alert. Spectate fires one alert per condition and host at a time.

The alert goes to the host's escalation paths. Set them on the host under Settings & alerts → Alerting.

Add a condition to a host

Owners, managers and admins can add conditions.

1. Open the host and go to Settings & alerts.
2. Under Metric conditions, select Add condition.
3. Select the Metric.
4. Select the Condition.
5. Type the Threshold.
6. Under For, select how long the condition must hold.
7. Select the Severity.
8. Select Create condition.

You cannot edit a condition. To change it, delete it and add a new one. In the row of the condition, select the delete icon (Delete condition).

Add a condition to a project

A condition on a project applies to every host in that project, also to hosts that join later.

1. Go to Projects and open the project.
2. Under Metric conditions, select Add condition.
3. Fill in the fields as for a host, then select Create condition.

The Metric conditions list on a host shows only the conditions of that host. The host also uses the conditions of its project. If a host has a condition and its project has the same condition, Spectate evaluates both.

Alert when a host stops reporting

The "Host stops reporting" alert fires when Spectate gets no data from the agent for a set time. For example, the server is down, the agent stopped, or the network is down.

To change it:

1. Open the host and go to Settings & alerts.
2. Under Alerting, find Host stops reporting.
3. Turn the alert on or off.
4. In Silent for, type the number of minutes.
5. Select Save changes.

Spectate checks every minute. The alert title is No data received from followed by the host's label. The alert resolves as soon as the agent reports again.

The alert does not fire in these cases:

• The agent never reported. The host is still pending.
• Spectate paused the host, for example because the team is over its plan limit.
• An alert of this type is already open for the host.

If you turn the alert off or make the time longer, Spectate resolves a firing alert.

Use the API

The public API can list, create, get and delete metric conditions. It needs a team API token with hosts:read to read and hosts:write to create. Deleting needs hosts:delete.

Code
curl https://api.spectate.net/v1/metric-conditions \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "metric": "disk_usage",
    "condition": "greater_than",
    "threshold": 90,
    "severity": "high",
    "duration_minutes": 10,
    "project_ids": ["YOUR_PROJECT_ID"]
  }'

duration_minutes is 0 to 60. Use host_ids, project_ids or both. To change the "Host stops reporting" time, update the host with no_data_alert_minutes. Use null to turn the alert off.