# Escalation paths

An **escalation path** decides who Spectate notifies about an alert, in which order, and when it opens an incident. A path is a list of steps. For example: notify the Slack channel, wait 5 minutes, page the person on call, then create an incident. Escalation stops when someone acknowledges or resolves the alert.

## Which paths an alert uses

When an alert fires, Spectate looks for escalation paths in this order and uses the first match:

1. The paths of the resource itself. These are the monitor, host, domain, status page or alert source that fired the alert.
2. The paths of the resource's [project](/account/projects).
3. The team default path.

If the resource has more than one path, every enabled path runs, each on its own. A disabled path does not run. If a resource has only disabled paths, nobody gets the alert, because Spectate does not fall back to the project or the team default.

If no path matches, the alert fires but nobody gets a notification. The dashboard warns about this in the **Alerting** section of the resource settings.

### Attach a path to a resource

Each monitor, host, domain, status page and alert source has an **Alerting** section in its settings.

1. Open the settings of the resource.
2. In the **Alerting** section, select **Specific paths**.
3. Select one or more escalation paths.
4. Save the settings.

**Default routing** uses the project paths, or else the team default path. Projects have the same choice of paths.

## Create an escalation path

1. In the dashboard, select **Alerting**.
2. Select **New path**.
3. Enter a **Name**. The **Description** is optional.
4. To make this the fallback for resources without a path, select **Use as team default**.
5. Select **Create and open builder**.
6. Select a step to edit it, or select **+** to add a step.
7. Select **Save flow**.

A new path starts with the alert trigger and one notify step for your first channel. A team has one default path. If you make a path the team default, the previous default path stops being the default.

## Steps

| Step in the builder | `type` in the API | What it does |
|---|---|---|
| **Notify channel** | `notify_channel` | Sends the alert to a [channel](/alerting/channels): email, Slack, a webhook, SMS and more. |
| **Notify on-call** | `notify_on_call` | Pages the person who is on call in an [on-call schedule](/alerting/on-call) at the time the step runs. |
| **Create incident** | `create_incident` | Declares an [incident](/incidents/introduction) with the alert's title and severity. |
| **Branch** | `branch` | Splits the path. Alerts that meet all conditions continue on the **Yes** side, other alerts on the **No** side. |

Each step runs at most one time per alert. A **Create incident** step does not open a second incident for an alert that already has one.

### Delays

Each step except a branch has a **Wait before this step** setting:

| Option | `delay_until` | Behaviour |
|---|---|---|
| **Time** | `null` | Waits a number of minutes, 0 to 1440, after the step before it. The first step waits from the moment the alert fires. The default is 0. |
| **Hours begin** | `working_hours_start` | Outside working hours, waits until they begin. During working hours, the step runs at once. |
| **Hours end** | `working_hours_end` | During working hours, waits until they end. Outside working hours, the step runs at once. |

If a step reaches nobody, Spectate logs it as skipped and runs the next step at once, without its delay. A step reaches nobody when its channel is disabled or deleted, or when nobody is on call. A step that waits for working hours still waits.

### When escalation stops

Spectate checks the alert before each step. If someone acknowledged the alert, or the alert resolved, the remaining steps do not run. See [alerts](/alerting/alerts) for the places where you can acknowledge an alert.

## Branches

A branch has 1 to 10 conditions. The alert takes the **Yes** side only if all conditions hold. Spectate decides the branch when the escalation reaches it, not when the alert fires. A side without steps ends the escalation.

| Field | Operators | Values |
|---|---|---|
| Severity (`severity`) | is one of (`is_one_of`), is not one of (`is_not_one_of`) | `critical`, `high`, `medium`, `low` |
| Working hours (`working_hours`) | within (`within`), outside (`outside`) | None. Spectate uses the path's working hours. |
| Source (`source`) | is one of, is not one of | An alert kind, or a specific external tool. See the list below. |
| Label (`label`) | is one of, is not one of, is set (`exists`), is not set (`not_exists`) | The value of a label that an alert source sent, for example `env` is `production`. |

Source values:

- Alert kinds: `uptime`, `performance`, `certificate`, `dns`, `domain`, `host_metric`, `status_page`, and `external` for any external tool.
- External tools: `external:alertmanager`, `external:grafana`, `external:sentry`, `external:webhook`, `external:email`.

Labels come from [alert sources](/alerting/alert-sources) that send labels: Prometheus Alertmanager and Grafana. An alert without the label does not match "is one of", and does match "is not one of".

## Working hours

Each path has its own working hours. Branches on working hours and the **Hours begin** and **Hours end** delays use them. The default is Monday to Friday, 09:00 to 17:00, in the team's time zone.

1. Open the path in the builder.
2. Open **Path settings**.
3. Under **Working hours**, select the days.
4. Enter **From** and **Until**, and select the **Time zone**.
5. Select **Save flow**.

If **Until** is earlier than **From**, the working hours run past midnight. They belong to the day on which they start.

## Changes apply to new alerts only

When an alert fires, Spectate takes a copy of the path's steps and working hours. The alert runs from that copy until it stops. If you change or delete the path, alerts that already fired keep their copy. New alerts use the new version.

Spectate checks two things when a step runs, not when the alert fires:

- The channel. Spectate skips a channel that is disabled or deleted at that time.
- The person on call. A **Notify on-call** step pages whoever is on call when the step runs.

## Warnings

The dashboard and the API (`warnings`) show a warning when a path cannot reach everyone:

| Code | Meaning |
|---|---|
| `no_deliverable_steps` | Every notify step targets a disabled channel or a deleted schedule. Alerts on this path reach nobody. |
| `disabled_channels` | Some channels are disabled. Spectate skips their steps. |
| `missing_schedules` | An on-call step points to a deleted schedule. Spectate skips the step. |

Channels can be disabled automatically after repeated delivery failures. See [channels](/alerting/channels) for how to turn them on again.

## Plans and limits

| Limit | Value |
|---|---|
| Escalation paths on the Starter plan | 1, the team default path |
| Escalation paths on the Team and Business plans | No limit |
| **Notify on-call** steps | Team and Business plans |
| Steps per path | 99 |
| Conditions per branch | 10 |
| Values per condition | 20 |
| Delay per step | 0 to 1440 minutes |

See [plans and billing](/account/plans-and-billing) and [pricing](https://spectate.net/pricing).

Owners, managers and admins can create and change escalation paths. Members can see them. See [teams and roles](/account/teams-and-roles).

## Create a path with the API

The [escalation paths API](/api-reference/escalation-paths) shows a path as a list of steps. A `branch` is the last step of its list, and continues in its `then` and `else` lists. This path notifies a channel, then pages the person on call after 5 minutes for critical alerts. Other alerts go to a channel when working hours begin.

```json
{
  "name": "Production",
  "is_default": true,
  "working_hours": {
    "timezone": "Europe/Amsterdam",
    "days": [1, 2, 3, 4, 5],
    "start_time": "09:00",
    "end_time": "17:00"
  },
  "steps": [
    { "type": "notify_channel", "channel_id": "CHANNEL_ID" },
    {
      "type": "branch",
      "conditions": [
        { "field": "severity", "operator": "is_one_of", "values": ["critical"] }
      ],
      "then": [
        { "type": "notify_on_call", "schedule_id": "SCHEDULE_ID", "delay_minutes": 5 }
      ],
      "else": [
        { "type": "notify_channel", "channel_id": "OTHER_CHANNEL_ID", "delay_until": "working_hours_start" }
      ]
    }
  ]
}
```

A path that you make with the API opens in the dashboard builder, and the other way around.