Alerts and on-call

Alerts

An alert is a signal that something is wrong with one of your resources, for example a monitor that is down. Spectate fires the alert, sends it through the escalation paths of the resource, and resolves it when the problem is over. An alert is not an incident. An incident is the work your team does on a problem, with updates, roles and a postmortem.

Where alerts come from

Every alert has a source type (source_type in the API). The source type also sets the default severity.

Sourcesource_typeSpectate fires an alert whenSeverity
Uptime monitorsuptimeA monitor is down: a check failed, and a second check from the same region about 5 seconds later also failed.critical
PerformanceperformanceThe response time is over the threshold, or deviates from normal.high over the threshold, medium for a deviation
CertificatescertificateA TLS certificate is invalid, expires soon, or its issuer changed.Depends on the problem, low for an issuer change
DNS monitorsdnsThe DNS answer changed, if the monitor alerts on changes.low
Hostshost_metricA metric condition holds, a health check fails, or the host stopped sending data.Set per condition and per health check. high for no data.
DomainsdomainA domain expires soon, has expired, or its registration changed.medium, critical when expired, low for a change
Status pagesstatus_pageVisitors report problems over your threshold, or traffic spikes.high for reports, medium for a traffic spike
Alert sourcesexternalAn external tool, such as Grafana or Sentry, sends an alert.From the tool, else the source's default (high unless you change it)

States and severity

An alert has two statuses (status):

StatusMeaning
firingThe problem is ongoing. The alert escalates until someone acknowledges it.
resolvedThe problem is over, or someone resolved the alert by hand.

Acknowledged is not a separate status. An acknowledged alert is still firing, with a value in acknowledged_at. The dashboard shows it as acknowledged.

Recovering is not a separate status either. A recovering alert is still firing, with a value in recovering_since. The dashboard shows it as Recovering. See recovery wait.

Severity (severity) is one of four values:

SeverityUse
criticalCustomers are affected right now.
highSerious, needs attention soon.
mediumShould be looked at.
lowInformational.

One alert per problem

Spectate does not fire a second alert for a problem that already has a firing alert. For alerts from Spectate itself, a resource has one firing alert per source type and title. For example, a monitor that stays down keeps one uptime alert, whatever the number of failed checks.

For alert sources, the key is the dedupe key (dedupe_key) that the tool sends. One firing alert exists per alert source and dedupe key. A repeat of the same key keeps that alert firing.

Find alerts

Select Alerts in the dashboard. The tabs Firing, Resolved and All filter by status. You can also filter by source and by severity.

Open an alert to see what fired it, the labels and links that an external tool sent, and the Escalation panel. The Escalation panel shows each step of the escalation path, when it ran, and why a step was skipped.

The list and the alert page link to the monitor, host, domain or status page that the alert is about. If you deleted that resource, the alert has no link.

Acknowledge an alert

Acknowledge an alert to tell your team that you work on it. Acknowledging stops the escalation: steps that did not run yet do not run. The alert keeps firing until its source recovers or someone resolves it.

  1. Open the alert in the dashboard.
  2. Select Acknowledge.

You can also acknowledge an alert in these places:

Spectate records who acknowledged the alert. Slack and Telegram messages about the alert change to show it.

Snooze an alert

Snooze pauses the escalation of an alert for 15 minutes to 24 hours. You can add a reason for the team.

  1. Open the alert.
  2. Select Snooze.
  3. Select a time and select Snooze.

The alert counts as acknowledged while it sleeps, and shows Snoozed. If it still fires at the end of that time, the acknowledgement clears and the escalation starts again from the first step. Repeat limits count again from that restart. If the alert resolves first, the snooze ends and nothing else happens.

Slack and Discord messages have snooze options. The public API has POST /v1/alerts/{alert}/snooze, and the MCP server has snooze_alert.

Say "not me"

If Spectate paged you for an alert and you can't take it, decline the page. Spectate does not page you again for this alert, and drops any delayed page to you. If everyone paged on the current step declined, the next step runs at once instead of after its delay.

You can decline with Not me on the alert page, in Slack, in Discord and in Telegram. On a phone call, press 2. Only a person that Spectate paged for this alert can decline, so the public API and the MCP server have no decline.

Page someone by hand

Page someone pulls in a member, whoever is on call for a schedule, or an escalation path. They hear it by their own notification rules. Text messages and calls use the credits of the team.

  • On the alert page, the page goes on that alert, also if the alert is acknowledged.
  • On the alert list, the page creates a new alert with source manual. Enter what it is about. The alert stays open until someone resolves it.
  • On an incident, the page goes on the alert of the incident while that alert fires. Otherwise Spectate creates a manual alert that resolves with the incident.

If you page a member by name who declined this alert, the page goes out and the decline ends.

In Slack, use /spectate page @someone message, or the name of an on-call schedule or escalation path instead of @someone. The public API has POST /v1/alerts/page and POST /v1/alerts/{alert}/page. The MCP server has page_someone, which needs confirm. The limit is 10 manual pages a minute per team for the dashboard and public API together. Slack and the MCP server each have the same limit.

Resolve an alert

Most alerts resolve on their own. When the metric is back under its threshold, or the alert source sends a resolve, Spectate resolves the alert. When a monitor is up again, Spectate resolves its uptime alert after the recovery wait.

To resolve an alert by hand:

  1. Open the alert in the dashboard.
  2. Select Resolve.
  3. Select Resolve alert to confirm.

If the problem is still there, the next check fires a new alert. An alert about a change, for example DNS records changed for api.example.com, has no recovery. Resolve it by hand after you examine the change.

When an alert resolves, by hand or on its own, Spectate does these things:

  • It stops the escalation.
  • It sends a resolved notification to every enabled channel that got the alert.
  • It sends an email to each on-call member that Spectate paged for the alert.
  • It updates the alert messages in Slack and Telegram.

Spectate records who resolved an alert by hand (resolved_by). For an alert that resolved on its own, resolved_by is null.

Recovery wait

A monitor can go down and up several times in a few minutes. The recovery wait keeps this as one alert with one set of notifications. It applies to uptime alerts only. The default is 5 minutes.

When a monitor with a firing uptime alert passes a check, Spectate does these things:

  • It shows the monitor as up at once. Uptime and the downtime report come from the checks, so the wait does not change them.
  • It keeps the alert firing and sets recovering_since.
  • It holds the escalation. Steps that did not run yet wait.
  • It edits the alert messages in Slack, Discord and Telegram to show that the monitor is back up. An edit sends no new notification.

The next event decides the result:

EventResult
The monitor stays up until the wait endsSpectate resolves the alert and sends the resolved notifications. resolved_at is the time that the monitor came back.
The monitor goes down againThe same alert goes on, and so does its escalation. Spectate fires no second alert. The wait starts again at the next passed check.
Someone resolves the alert by handThe alert resolves at once. resolved_at is the time of the action.

A failed check with a successful recheck does not end the wait, because the monitor did not go down.

To change the wait, open Team → Settings → Alert recovery. The value is 0 to 60 minutes. With 0, the alert resolves at the first passed check. Owners and managers can change it.

The first notification of an outage does not wait. To prevent notifications about short outages, add a delay to the first step of the escalation path.

Muted and grouped alerts

An alert that fires during a maintenance window shows Muted and pages nobody until the window ends. An alert in a group waits with the alert that leads the group. The alert list shows Grouped on it, and +3 on a leader with 3 more firing alerts.

Delete an alert

You can delete resolved alerts only. Resolve a firing alert first.

  1. Open the resolved alert.
  2. Select Delete.
  3. Select Delete alert to confirm.

Spectate deletes the alert and its escalation history. Incidents that the alert opened stay. Messages that Spectate posted in Slack or Telegram stay in their channels.

Who can do what

ActionOwnerManagerAdminMemberBilling
See alertsYesYesYesYesNo
Acknowledge and resolveYesYesYesYesNo
Delete resolved alertsYesYesYesNoNo

See teams and roles for all permissions.