Alerts and on-call

Alerts

An alert is a signal that something is wrong with one of your resources, for example a monitor that is down. Spectate fires the alert, sends it through the escalation paths of the resource, and resolves it when the problem is over. An alert is not an incident. An incident is the work your team does on a problem, with updates, roles and a postmortem.

Where alerts come from

Every alert has a source type (source_type in the API). The source type also sets the default severity.

Source	`source_type`	Spectate fires an alert when	Severity
Uptime monitors	`uptime`	A monitor is down: a check failed, and a second check from the same region about 5 seconds later also failed.	`critical`
Performance	`performance`	The response time is over the threshold, or deviates from normal.	`high` over the threshold, `medium` for a deviation
Certificates	`certificate`	A TLS certificate is invalid, expires soon, or its issuer changed.	Depends on the problem, `low` for an issuer change
DNS monitors	`dns`	The DNS answer changed, if the monitor alerts on changes.	`low`
Hosts	`host_metric`	A metric condition holds, a health check fails, or the host stopped sending data.	Set per condition and per health check. `high` for no data.
Domains	`domain`	A domain expires soon, has expired, or its registration changed.	`medium`, `critical` when expired, `low` for a change
Status pages	`status_page`	Visitors report problems over your threshold, or traffic spikes.	`high` for reports, `medium` for a traffic spike
Alert sources	`external`	An external tool, such as Grafana or Sentry, sends an alert.	From the tool, else the source's default (`high` unless you change it)

States and severity

An alert has two statuses (status):

Status	Meaning
`firing`	The problem is ongoing. The alert escalates until someone acknowledges it.
`resolved`	The problem is over, or someone resolved the alert by hand.

Acknowledged is not a separate status. An acknowledged alert is still firing, with a value in acknowledged_at. The dashboard shows it as acknowledged.

Severity (severity) is one of four values:

Severity	Use
`critical`	Customers are affected right now.
`high`	Serious, needs attention soon.
`medium`	Should be looked at.
`low`	Informational.

One alert per problem

Spectate does not fire a second alert for a problem that already has a firing alert. For alerts from Spectate itself, a resource has one firing alert per source type and title. For example, a monitor that stays down keeps one uptime alert, whatever the number of failed checks.

For alert sources, the key is the dedupe key (dedupe_key) that the tool sends. One firing alert exists per alert source and dedupe key. A repeat of the same key keeps that alert firing.

Find alerts

Select Alerts in the dashboard. The tabs Firing, Resolved and All filter by status. You can also filter by source and by severity.

Open an alert to see what fired it, the labels and links that an external tool sent, and the Escalation panel. The Escalation panel shows each step of the escalation path, when it ran, and why a step was skipped.

Acknowledge an alert

Acknowledge an alert to tell your team that you work on it. Acknowledging stops the escalation: steps that did not run yet do not run. The alert keeps firing until its source recovers or someone resolves it.

Open the alert in the dashboard.
Select Acknowledge.

You can also acknowledge an alert in these places:

The Acknowledge button on an alert message in Slack or Telegram.
A voice call from Spectate: press 1. See Telegram and phone.
The public API and the MCP server.
Acknowledging the incident that the alert opened.

Spectate records who acknowledged the alert. Slack and Telegram messages about the alert change to show it.

Resolve an alert

Most alerts resolve on their own. When the monitor is up again, the metric is back under its threshold, or the alert source sends a resolve, Spectate resolves the alert.

To resolve an alert by hand:

Open the alert in the dashboard.
Select Resolve.
Select Resolve alert to confirm.

If the problem is still there, the next check fires a new alert. An alert about a change, for example DNS records changed for api.example.com, has no recovery. Resolve it by hand after you examine the change.

When an alert resolves, by hand or on its own, Spectate does these things:

It stops the escalation.
It sends a resolved notification to every enabled channel that got the alert.
It sends an email to each on-call member that Spectate paged for the alert.
It updates the alert messages in Slack and Telegram.

Spectate records who resolved an alert by hand (resolved_by). For an alert that resolved on its own, resolved_by is null.

Delete an alert

You can delete resolved alerts only. Resolve a firing alert first.

Open the resolved alert.
Select Delete.
Select Delete alert to confirm.

Spectate deletes the alert and its escalation history. Incidents that the alert opened stay. Messages that Spectate posted in Slack or Telegram stay in their channels.

Who can do what

Action	Owner	Manager	Admin	Member	Billing
See alerts	Yes	Yes	Yes	Yes	No
Acknowledge and resolve	Yes	Yes	Yes	Yes	No
Delete resolved alerts	Yes	Yes	Yes	No	No

See teams and roles for all permissions.

Escalation paths: who gets an alert, in which order.
Channels: where Spectate sends alerts.
Alerts API: list, get, acknowledge, resolve and delete alerts.

Domain checks Escalation paths