Alerts and on-call

Channels

A channel is a destination for alert notifications, for example an email address, a Slack channel or a phone number. Each Notify channel step of an escalation path sends the alert to one channel. A channel does nothing until an escalation path uses it.

Add a channel

  1. In the dashboard, select Alerting.
  2. Select New channel.
  3. Under Integration, select the channel type.
  4. Enter a Name, for example Primary on-call.
  5. Fill in the fields for the type. The table below lists them.
  6. Select Create channel.

Then add the channel to an escalation path with a Notify channel step. Owners, managers and admins can add, change and delete channels. Members can see them.

Channel types

TypetypeWhat you enterWhat the alert looks like
EmailemailEmail addressAn email with the subject [CRITICAL] Title. It explains what failed, shows a chart when there is data, and has a View alert button.
SlackslackSlack channel, after you connect your Slack workspaceA message with the colour of the severity and the buttons Acknowledge, Declare incident and View in Spectate. See Slack.
DiscorddiscordDiscord webhook URLAn embed in the colour of the severity, with a link to the alert, the cause and what to check, the facts, a chart and the times in the reader's own time zone. A resolved alert posts a green message with the duration. The message never mentions people or roles.
Microsoft Teamsmicrosoft_teamsMicrosoft Teams connector URLAn adaptive card with the alert fields, a chart and an Open alert button.
TelegramtelegramSelect Connect Telegram and open the linkA message with an Acknowledge button. See Telegram and phone.
WebhookwebhookWebhook URL and optional headersA JSON POST to your URL. See webhooks.
PushoverpushoverUser key, Application token and PriorityA push notification with the chart attached.
SMSsmsPhone number in international format, for example +31612345678A text message: Title (source: Uptime). Uses 1 credit.
Voice callvoicePhone number in international formatA phone call that reads the alert. Press 1 to acknowledge. Uses 2 credits. See Telegram and phone.

Discord, Microsoft Teams and webhook URLs must resolve to a public address. Spectate checks this when you save the channel and again each time it sends. See webhooks for the rules.

Pushover priority

Priority is a Pushover priority from -2 to 2:

OptionValue
Lowest-2
Low-1
Normal (default)0
High1
Emergency2

With Emergency, Pushover repeats the notification every 30 seconds for up to 1 hour, until you acknowledge it in Pushover. This does not acknowledge the alert in Spectate.

What an alert notification contains

Every notification about a firing alert has the alert title and these fields:

  • Severity, source and resource.
  • The time the alert fired.
  • Fields for the source type, for example the monitor URL and the region for an uptime alert.

Uptime, performance and host metric alerts also get a chart of the data around the alert. Spectate leaves the chart out when there is no data.

When the alert resolves, Spectate sends a resolved notification to every enabled channel that got the alert. In Slack and Telegram, it is a reply to the alert message. A voice channel gets the resolved notification as an SMS, which uses 1 credit.

Test a channel

Send a test to make sure that the channel delivers:

  1. Select Alerting.
  2. In the channel's row, select Send test.

On the channel's own page, the Test delivery panel has the same action and shows the result of each test. Each member can send 10 tests a minute.

A test to an SMS channel sends a test SMS and uses 1 credit. A test to a voice channel places a short test call and uses 2 credits. The test call has nothing to acknowledge.

A failed test does not count towards disabling the channel. A successful test resets the failure counter.

When Spectate disables a channel

Spectate tries each delivery 3 times: at once, after 30 seconds and after 2 more minutes. If all tries fail, the delivery counts as one failure for the channel. Spectate disables the channel in these cases:

  • 5 failed deliveries in a row.
  • 2 failed deliveries in a row, where the last one is a permanent error. Permanent errors are HTTP 401, 403, 404 and 410, a revoked token, or an unknown chat. Spectate does not retry permanent errors.
  • At least 2 failed deliveries in a row over more than 24 hours.

A successful delivery resets the count. When Spectate disables a channel, it sends an email to the team's owners, managers and admins. Escalation steps for a disabled channel are skipped, and the next step runs at once. The escalation path shows a warning.

To turn the channel on again:

  1. Fix the destination, for example a new webhook URL.
  2. Select Test & re-enable on the Alerting page.

If the test arrives, Spectate turns the channel on again. You can also open the channel, select Edit, and select Enabled.

Turn a channel off

  1. Open the channel.
  2. Select Edit.
  3. Clear Enabled.
  4. Select Save changes.

Escalation paths skip a disabled channel. In the dashboard, you can change the name of a channel, but not its delivery details. To send to another address or URL, add a new channel, or change the channel with the API. A Slack channel can move to another Slack channel, and a Telegram channel to another chat.

SMS and call credits

SMS, voice calls and on-call pages by phone use notification credits:

WhatCredits
One SMS1 per recipient
One voice call2 per number

Spectate takes credits only after the SMS or the call went out. A retry does not use credits again.

Each paid plan includes credits every month. Unused monthly credits do not carry over. The trial includes no monthly credits.

PlanCredits a month
Starter10
Team50
Business200

You can buy more in packs of 100 under Billing → SMS and call credits. Bought credits never expire, and Spectate uses them after the monthly credits. Owners and billing members can buy credits. Owners, admins and billing members get an email when the balance drops to 20% of the monthly credits, and when it reaches zero.

Without credits, SMS and voice channels send nothing, and on-call pages go out by email only. Other channel types do not use credits.

See plans and billing and pricing.