Alerts and on-call

Escalation paths

An escalation path decides who Spectate notifies about an alert, in which order, and when it opens an incident. A path is a list of steps. For example: notify the Slack channel, then page the person on call after 5 minutes. Page the whole team 15 minutes later, then create an incident. An acknowledgement stops the paging. A resolved alert stops the escalation.

Which paths an alert uses

When an alert fires, Spectate looks for escalation paths in this order and uses the first match:

  1. The paths of the resource itself. These are the monitor, host, domain, status page or alert source that fired the alert.
  2. The paths of the resource's project.
  3. The team default path.

If the resource has more than one path, every enabled path runs, each on its own. A disabled path does not run. If a resource has only disabled paths, nobody gets the alert, because Spectate does not fall back to the project or the team default.

If no path matches, the alert fires but nobody gets a notification. The dashboard warns about this in the Alerting section of the resource settings.

Attach a path to a resource

Each monitor, host, domain, status page and alert source has an Alerting section in its settings.

  1. Open the settings of the resource.
  2. In the Alerting section, select Specific paths.
  3. Select one or more escalation paths.
  4. Save the settings.

Default routing uses the project paths, or else the team default path. Projects have the same choice of paths.

Create an escalation path

  1. In the dashboard, select Alerting.
  2. Select New path.
  3. Enter a Name. The Description is optional.
  4. To make this the fallback for resources without a path, select Use as team default.
  5. Select Create and open builder.
  6. Select a step to edit it, or select + to add a step.
  7. Select Save flow.

A + at the end of a path, or on an empty side of a branch, adds a step there. A + on the connection between two steps adds a step between them. To change the order, select a step, then select Move up or Move down. The step changes places with the step before or after it. Branches and Repeat steps do not move.

A new path starts with the alert trigger and one notify step for your first channel. A team has one default path. If you make a path the team default, the previous default path stops being the default.

Steps

Step in the buildertype in the APIWhat it does
Notify channelnotify_channelSends the alert to a channel: email, Slack, a webhook, SMS and more.
Notify on-callnotify_on_callPages people on an on-call schedule: whoever is on call, whoever is on call next, or everyone on the schedule.
Notify peoplenotify_usersPages members that you choose, or the whole team.
Create incidentcreate_incidentDeclares an incident with the alert's title and severity.
BranchbranchSplits the path. Alerts that meet all conditions continue on the Yes side, other alerts on the No side.
RepeatrepeatGoes back to an earlier step, 1 to 10 times. Then the escalation ends.

Each step runs one time per alert, unless the step pages again or a Repeat step goes back to it. A Create incident step does not open a second incident for an alert that already has one.

Page people

The Notify on-call and Notify people steps page members. Spectate pages each member by their own notification rules. For example, one member gets a push in the Spectate app at once and a phone call after 2 minutes. Another member gets an email only.

Under Page, a Notify on-call step pages one of these on the selected schedule:

OptiontargetWho Spectate pages
Whoever is on callcurrentThe member on call when the step runs. This is the default.
Whoever is on call nextnextThe first other member to go on call after the current one, in the next 28 days. Use this as a backup when the first member does not respond.
Everyone on the scheduleeveryoneAll participants of the schedule's rotations, and the member on call by override.

Under Rotation, you can limit the step to one rotation (rotation_id). The default is All rotations. With one rotation, Spectate uses the shifts of that rotation only, without overrides. For example, page the Secondary rotation after the Primary rotation.

A Notify people step pages the members that you select under People (user_ids), or every member of the team with The whole team (everyone). Spectate skips members who left the team.

Both steps have these settings:

SettingAPI fieldValues
Urgencyurgencyhigh (default) or low. It selects which list of each member's notification rules Spectate uses.
Page again, number of timesrepage_count0 to 9 more pages to the same members while nobody acknowledges. The default is 0 (No).
Page again, every N minrepage_interval_minutes1 to 60 minutes between those pages. The default is 5.

If a step pages again, the next step waits until the last page of the step. Its own delay starts then. Paging steps are part of the Team and Business plans.

Repeat

A Repeat step goes back to the step in Go back to, Times times. That step must come before the repeat, on the way from the start. Use it to page the same people again when nobody acknowledges, for example 3 times. Each time, the steps after that earlier step run again, and Spectate decides each branch again.

A Repeat step ends its side of the path: nothing can follow it. It has a delay like other steps. If someone acknowledges the alert, the repeat does not go back. After the last time, Spectate writes "nobody acknowledged" in the alert's Escalation panel and the escalation ends.

Delays

Each step has a Wait before this step setting. For a branch it is Wait before checking, for a repeat Wait before going back:

Optiondelay_untilBehaviour
TimenullWaits a number of minutes, 0 to 1440, after the step before it. The first step waits from the moment the alert fires. The default is 0.
Hours beginworking_hours_startOutside working hours, waits until they begin. During working hours, the step runs at once.
Hours endworking_hours_endDuring working hours, waits until they end. Outside working hours, the step runs at once.

If a step reaches nobody, Spectate logs it as skipped and runs the next step at once, without its delay. A step reaches nobody when its channel is disabled or deleted, or when nobody is on call. A step that waits for working hours still waits.

When escalation stops

Spectate checks the alert before each step:

  • If the alert resolved, the escalation stops.
  • If the alert is in the recovery wait, the step waits. So do the later methods of a member's notification rules. Spectate looks again each minute. The step runs if the monitor goes down again.
  • If someone acknowledged the alert, Spectate stops paging and notifying. If a branch on Alert status comes later in the path, Spectate goes on to that branch and skips the steps before it. Without such a branch, the escalation stops.

See alerts for the places where you can acknowledge an alert.

Branches

A branch has 1 to 10 conditions. The alert takes the Yes side only if all conditions hold. Spectate decides the branch when the escalation reaches it and after the branch's wait, not when the alert fires. A side without steps ends the escalation.

FieldOperatorsValues
Severity (severity)is one of (is_one_of), is not one of (is_not_one_of)critical, high, medium, low
Working hours (working_hours)within (within), outside (outside)None. Spectate uses the path's working hours.
Source (source)is one of, is not one ofAn alert kind, or a specific external tool. See the list below.
Label (label)is one of, is not one of, is set (exists), is not set (not_exists)The value of a label that an alert source sent, for example env is production.
Alert status (alert_status)is one of, is not one ofacknowledged, not_acknowledged

Source values:

  • Alert kinds: uptime, performance, certificate, dns, domain, host_metric, status_page, and external for any external tool.
  • External tools: external:alertmanager, external:grafana, external:sentry, external:webhook, external:email.

Labels come from alert sources that send labels: Prometheus Alertmanager and Grafana. An alert without the label does not match "is one of", and does match "is not one of".

Check the alert status after a time

A branch on Alert status answers the question "after this time, is the alert still not acknowledged, or acknowledged but not resolved?". A resolved alert never gets to the branch, because the escalation stopped. For example:

  1. Notify on-call: page whoever is on call.
  2. Branch, wait 60 minutes: Alert status is Acknowledged.
  3. Yes side: Notify people: page the team lead, because the alert is acknowledged but still not resolved after 1 hour.
  4. No side: Notify people: page the whole team, because nobody acknowledged.

This branch also runs after an acknowledgement. If someone acknowledges the alert before the branch, Spectate skips the steps before the branch. The branch's wait then starts at the moment Spectate skips to it. The steps after the branch page members even if the alert is acknowledged. They stop when the alert resolves. A Repeat after the branch can page again, at most its number of times.

Working hours

Each path has its own working hours. Branches on working hours and the Hours begin and Hours end delays use them. The default is Monday to Friday, 09:00 to 17:00, in the team's time zone.

  1. Open the path in the builder.
  2. Open Path settings.
  3. Under Working hours, select the days.
  4. Enter From and Until, and select the Time zone.
  5. Select Save flow.

If Until is earlier than From, the working hours run past midnight. They belong to the day on which they start.

Group alerts

Grouping makes one outage page once. The first alert on the path leads and escalates. Alerts that follow within the window wait with it.

  1. Open the escalation path.
  2. Open the path settings.
  3. Select Group alerts.
  4. Enter the minutes in Within, 1 to 1440. The first value is 5.
  5. Select what to Group by.
Group byValueAlerts group when they
Everything on this pathallUse this path. This is the default.
Same projectprojectBelong to one project. Alerts without a project page by themselves.
Same monitor, host or domainresourceCome from one resource, for example several health checks of one host.
Same labellabelHave the same value for a label that an alert source sent. Alerts without the label page by themselves.

Grouping is off by default. These rules apply:

  • Each alert that joins extends the window. A group takes alerts for up to 24 hours, and up to 1,000 alerts.
  • An alert with a higher severity than the leader never joins. It pages by itself, and alerts that follow join that alert.
  • An acknowledged leader takes no new alerts. The next alert pages and leads a new group.
  • If you acknowledge the leader, Spectate acknowledges the whole group. If you acknowledge a member, only that member changes.
  • If the leader resolves while members still fire, the oldest unacknowledged member becomes the leader within a minute. Its escalation starts from the first step.
  • Chat channels get only the leader's message. That message shows +3 more alerts when it updates.
  • Status page automation runs for each alert. A create_incident step runs only for the leader.
  • A muted alert does not lead or join. See maintenance windows.

An alert has one place in one group. If an alert uses two paths with different leaders, it joins the first leader and pages on the second path. Two alerts that fire in the same moment can both page. Neither of them can get lost.

In the public API, the fields are group_window_minutes, group_by and group_label.

Changes apply to new alerts only

When an alert fires, Spectate takes a copy of the path's steps and working hours. The alert runs from that copy until it stops. If you change or delete the path, alerts that already fired keep their copy. New alerts use the new version.

Spectate checks two things when a step runs, not when the alert fires:

  • The channel. Spectate skips a channel that is disabled or deleted at that time.
  • The people to page. A Notify on-call step pages whoever is on call when the step runs. A Notify people step pages the members of the team at that time.

Warnings

The dashboard and the API (warnings) show a warning when a path cannot reach everyone:

CodeMeaning
no_deliverable_stepsEvery notify step targets a disabled channel or a deleted schedule. Alerts on this path reach nobody.
disabled_channelsSome channels are disabled. Spectate skips their steps.
missing_schedulesAn on-call step points to a deleted schedule. Spectate skips the step.

Channels can be disabled automatically after repeated delivery failures. See channels for how to turn them on again.

Plans and limits

LimitValue
Escalation paths on the Starter plan1, the team default path
Escalation paths on the Team and Business plansNo limit
Notify on-call and Notify people stepsTeam and Business plans
Steps per path99
Conditions per branch10
Values per condition20
Delay per step0 to 1440 minutes
Pages again per paging step0 to 9, every 1 to 60 minutes
Times a Repeat step goes back1 to 10
Members per Notify people step50, or the whole team

See plans and billing and pricing.

Owners, managers and admins can create and change escalation paths. Members can see them. See teams and roles.

Create a path with the API

The escalation paths API shows a path as a list of steps. A branch is the last step of its list, and continues in its then and else lists. A repeat is also the last step of its list. Its go_to is the key of an earlier step. This path notifies a channel. For critical alerts, it then pages the person on call after 5 minutes, two more times if nobody acknowledges. After 15 minutes it pages the whole team, and repeats from the first page 2 times. Other alerts go to a channel when working hours begin.

JSONCode
{ "name": "Production", "is_default": true, "working_hours": { "timezone": "Europe/Amsterdam", "days": [1, 2, 3, 4, 5], "start_time": "09:00", "end_time": "17:00" }, "steps": [ { "type": "notify_channel", "channel_id": "CHANNEL_ID" }, { "type": "branch", "conditions": [ { "field": "severity", "operator": "is_one_of", "values": ["critical"] } ], "then": [ { "key": "page", "type": "notify_on_call", "schedule_id": "SCHEDULE_ID", "target": "current", "delay_minutes": 5, "repage_count": 2, "repage_interval_minutes": 3 }, { "type": "notify_users", "everyone": true, "urgency": "high", "delay_minutes": 15 }, { "type": "repeat", "go_to": "page", "times": 2, "delay_minutes": 10 } ], "else": [ { "type": "notify_channel", "channel_id": "OTHER_CHANNEL_ID", "delay_until": "working_hours_start" } ] } ] }

A path that you make with the API opens in the dashboard builder, and the other way around.