Copy page

Alerts and on-call

Alert sources

An alert source receives alerts from another tool and turns them into Spectate alerts. The alerts go through escalation paths, on-call, Slack, phone calls and incidents, the same as alerts from Spectate's own monitors. Each alert source has its own URL, or its own email address for email sources.

Supported tools

Add an alert source

Owners, managers and admins can add alert sources.

1. In the dashboard, select Alerting.
2. Under Alert sources, select Add alert source.
3. Select the tool.
4. Enter a Name, for example Grafana production. Spectate shows the name on the alerts.
5. Optional: select a Project.
6. Select who gets the alerts: Default routing or Specific paths.
7. Optional: open Severity and auto-resolve and change the Default severity or Resolve automatically.
8. Select Create and connect.
9. Follow the setup steps for the tool. Spectate fills in the URL.

The Live check panel shows the first request that arrives and what Spectate made of it.

Default routing uses the project's escalation paths, or else the team default path.

The URL and the email address

Each alert source gets a secret URL:

Code
https://api.spectate.net/integrations/alerts/YOUR_SOURCE_TOKEN

An email source gets a secret address:

Code
YOUR_SOURCE_TOKEN@alerts.spectate.net

The token in the URL or address is the credential. Anyone with the URL can send alerts to the source. Keep it secret.

Webhook format

Send a POST request with a JSON body. Only title is required.

Code
curl -X POST "https://api.spectate.net/integrations/alerts/YOUR_SOURCE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Disk almost full on db-1",
    "severity": "high",
    "dedupe_key": "db-1/disk",
    "description": "Less than 5% free on /var/lib/postgresql",
    "links": [{ "label": "Runbook", "url": "https://wiki.example.com/disk" }],
    "metadata": { "host": "db-1" }
  }'

To resolve the alert, send the same dedupe_key with "status": "resolved":

Code
curl -X POST "https://api.spectate.net/integrations/alerts/YOUR_SOURCE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "dedupe_key": "db-1/disk", "title": "Disk almost full on db-1", "status": "resolved" }'

To send more than one alert in one request, put them in an alerts list, up to 100 alerts:

Code
{ "alerts": [ { "title": "Disk almost full on db-1" }, { "title": "Backup failed", "severity": "critical" } ] }

To check the connection without alerting anyone, send {"test": true}.

Response

Spectate answers a valid request with 202 Accepted and what it did:

Code
{
  "outcome": "fired",
  "summary": "Fired 1",
  "error": null,
  "alerts": [{ "action": "fired", "dedupe_key": "db-1/disk" }]
}

Rejected requests fire nothing.

Prometheus Alertmanager

Add a receiver to alertmanager.yml, then reload Alertmanager:

Code
receivers:
  - name: spectate
    webhook_configs:
      - url: "https://api.spectate.net/integrations/alerts/YOUR_SOURCE_TOKEN"
        send_resolved: true

route:
  receiver: spectate

• Each alert becomes its own Spectate alert. The dedupe key is the alert's fingerprint.
• The title is the summary annotation. Without it, the title is the alertname label, followed by on and the instance label.
• The severity comes from the severity label, or else the priority label.
• The labels are kept. Escalation path branches can test them, for example env is production.

If you set a secret on the source and Alertmanager sends an Authorization: Bearer header, the token must be the secret.

Grafana

1. In Grafana, go to Alerting → Contact points → + Add contact point.
2. Name it Spectate and select the Webhook integration.
3. Paste the alert source URL.
4. Under Optional Webhook settings → HMAC signature, paste a secret.
5. Select Test in Grafana. Spectate shows Grafana test notification received.
6. In Alerting → Notification policies, use the Spectate contact point.

Generate the secret in Spectate with Generate a secret in the setup steps. With a secret, Spectate rejects requests without a valid X-Grafana-Alerting-Signature header. The Grafana Test button fires no alert. Grafana alerts use the Alertmanager format, so titles, severity and labels work as above.

Sentry

1. In Sentry, go to Settings → Developer Settings → Custom Integrations.
2. Select Create New Integration → Internal Integration and name it Spectate.
3. Paste the alert source URL as the webhook URL, and turn on Alert Rule Action.
4. Give Issue & Event read access, and under Webhooks select issue.
5. Save, then paste the Client Secret in the Spectate setup steps.
6. In your Sentry alert rules, add the action Send a notification via Spectate.

With the client secret saved, Spectate checks the Sentry-Hook-Signature header of each request. Very old self-hosted Sentry versions can use the legacy WebHooks plugin with the same URL. That plugin does not sign requests.

Email

Send or forward alert emails to the source's address. Backup jobs, NAS and UPS alerts, cron, Zabbix and Nagios can all send email.

• The subject is the alert title. The body is the description.
• The same subject again keeps the alert firing.
• A subject that starts with RESOLVED, RECOVERED, RECOVERY, OK or CLEARED resolves the alert with the same subject.
• A word such as [CRITICAL] or Warning: at the start of the subject sets the severity.

Email has no standard way to say that a problem is over. Email sources resolve their alerts after 60 minutes without a repeat by default.

Dedupe keys

One alert source has one firing alert per dedupe key. A repeat of a firing key keeps the alert firing and does not notify anyone again. A resolve resolves the alert with that key.

Severity

If the tool sends a severity, Spectate maps it to its own four values. If the tool sends none, or a word not in this table, the alert gets the source's Default severity. The default is high.

Resolve automatically

Some tools never say that a problem is over. For these, set Resolve automatically. Spectate then resolves an alert when the tool has not repeated it for that time.

With the API, any value from 5 to 10080 minutes is allowed.

Recent events

The source page shows Recent events: every request of the last 7 days, with the outcome and the request itself. Look here first when an alert did not arrive. Spectate stores the first 32 KB of each request.

Send a test alert

1. Open the alert source.
2. Select Send test alert.
3. Confirm.

Spectate fires a real alert, Test alert from SOURCE NAME, through the source's escalation paths. The test alert resolves itself after 5 minutes.

Make a new URL

If the URL or address leaked, make a new one:

1. Open the alert source.
2. Select New URL, or New address for an email source.
3. Select Create new URL or Create new address to confirm.

Turn off or delete a source

To stop a source without deleting it, open its Settings and clear Receiving alerts. Spectate still logs the requests, but fires nothing. When you delete a source, Spectate resolves its firing alerts first.

Related

• Alert sources API: create sources, rotate the URL, send a test alert and read events.
• Escalation paths: route alerts by source and label.