Copy page

Alerts and on-call

Escalation paths

An escalation path decides who Spectate notifies about an alert, in which order, and when it opens an incident. A path is a list of steps. For example: notify the Slack channel, wait 5 minutes, page the person on call, then create an incident. Escalation stops when someone acknowledges or resolves the alert.

Which paths an alert uses

When an alert fires, Spectate looks for escalation paths in this order and uses the first match:

1. The paths of the resource itself. These are the monitor, host, domain, status page or alert source that fired the alert.
2. The paths of the resource's project.
3. The team default path.

If the resource has more than one path, every enabled path runs, each on its own. A disabled path does not run. If a resource has only disabled paths, nobody gets the alert, because Spectate does not fall back to the project or the team default.

If no path matches, the alert fires but nobody gets a notification. The dashboard warns about this in the Alerting section of the resource settings.

Attach a path to a resource

Each monitor, host, domain, status page and alert source has an Alerting section in its settings.

1. Open the settings of the resource.
2. In the Alerting section, select Specific paths.
3. Select one or more escalation paths.
4. Save the settings.

Default routing uses the project paths, or else the team default path. Projects have the same choice of paths.

Create an escalation path

1. In the dashboard, select Alerting.
2. Select New path.
3. Enter a Name. The Description is optional.
4. To make this the fallback for resources without a path, select Use as team default.
5. Select Create and open builder.
6. Select a step to edit it, or select + to add a step.
7. Select Save flow.

A new path starts with the alert trigger and one notify step for your first channel. A team has one default path. If you make a path the team default, the previous default path stops being the default.

Steps

Each step runs at most one time per alert. A Create incident step does not open a second incident for an alert that already has one.

Delays

Each step except a branch has a Wait before this step setting:

If a step reaches nobody, Spectate logs it as skipped and runs the next step at once, without its delay. A step reaches nobody when its channel is disabled or deleted, or when nobody is on call. A step that waits for working hours still waits.

When escalation stops

Spectate checks the alert before each step. If someone acknowledged the alert, or the alert resolved, the remaining steps do not run. See alerts for the places where you can acknowledge an alert.

Branches

A branch has 1 to 10 conditions. The alert takes the Yes side only if all conditions hold. Spectate decides the branch when the escalation reaches it, not when the alert fires. A side without steps ends the escalation.

Source values:

• Alert kinds: uptime, performance, certificate, dns, domain, host_metric, status_page, and external for any external tool.
• External tools: external:alertmanager, external:grafana, external:sentry, external:webhook, external:email.

Labels come from alert sources that send labels: Prometheus Alertmanager and Grafana. An alert without the label does not match "is one of", and does match "is not one of".

Working hours

Each path has its own working hours. Branches on working hours and the Hours begin and Hours end delays use them. The default is Monday to Friday, 09:00 to 17:00, in the team's time zone.

1. Open the path in the builder.
2. Open Path settings.
3. Under Working hours, select the days.
4. Enter From and Until, and select the Time zone.
5. Select Save flow.

If Until is earlier than From, the working hours run past midnight. They belong to the day on which they start.

Changes apply to new alerts only

When an alert fires, Spectate takes a copy of the path's steps and working hours. The alert runs from that copy until it stops. If you change or delete the path, alerts that already fired keep their copy. New alerts use the new version.

Spectate checks two things when a step runs, not when the alert fires:

• The channel. Spectate skips a channel that is disabled or deleted at that time.
• The person on call. A Notify on-call step pages whoever is on call when the step runs.

Warnings

The dashboard and the API (warnings) show a warning when a path cannot reach everyone:

Channels can be disabled automatically after repeated delivery failures. See channels for how to turn them on again.

Plans and limits

See plans and billing and pricing.

Owners, managers and admins can create and change escalation paths. Members can see them. See teams and roles.

Create a path with the API

The escalation paths API shows a path as a list of steps. A branch is the last step of its list, and continues in its then and else lists. This path notifies a channel, then pages the person on call after 5 minutes for critical alerts. Other alerts go to a channel when working hours begin.

Code
{
  "name": "Production",
  "is_default": true,
  "working_hours": {
    "timezone": "Europe/Amsterdam",
    "days": [1, 2, 3, 4, 5],
    "start_time": "09:00",
    "end_time": "17:00"
  },
  "steps": [
    { "type": "notify_channel", "channel_id": "CHANNEL_ID" },
    {
      "type": "branch",
      "conditions": [
        { "field": "severity", "operator": "is_one_of", "values": ["critical"] }
      ],
      "then": [
        { "type": "notify_on_call", "schedule_id": "SCHEDULE_ID", "delay_minutes": 5 }
      ],
      "else": [
        { "type": "notify_channel", "channel_id": "OTHER_CHANNEL_ID", "delay_until": "working_hours_start" }
      ]
    }
  ]
}

A path that you make with the API opens in the dashboard builder, and the other way around.