# Alert sources

An **alert source** receives alerts from another tool and turns them into Spectate alerts. The alerts go through [escalation paths](/alerting/escalation-paths), on-call, Slack, phone calls and incidents, the same as alerts from Spectate's own monitors. Each alert source has its own URL, or its own email address for email sources.

## Supported tools

| Tool | `type` | Spectate reads | Resolves when |
|---|---|---|---|
| Webhook | `webhook` | Spectate's own JSON format, for scripts, CI and anything that can call a URL | The request has `"status": "resolved"` |
| Email | `email` | The subject is the alert, the body is the description | The subject starts with `RESOLVED`, `RECOVERED`, `RECOVERY`, `OK` or `CLEARED`, or after the auto-resolve time |
| Prometheus Alertmanager | `alertmanager` | The `webhook_configs` payload, version 4 | Alertmanager sends `resolved`, with `send_resolved: true` |
| Grafana | `grafana` | A Grafana Alerting webhook contact point, Grafana Cloud or self-hosted | Grafana sends `resolved` |
| Sentry | `sentry` | An internal integration on sentry.io or self-hosted Sentry, or the legacy WebHooks plugin | The issue is resolved, ignored or archived, or the metric alert resolves |

## Add an alert source

Owners, managers and admins can add alert sources.

1. In the dashboard, select **Alerting**.
2. Under **Alert sources**, select **Add alert source**.
3. Select the tool.
4. Enter a **Name**, for example `Grafana production`. Spectate shows the name on the alerts.
5. Optional: select a **Project**.
6. Select who gets the alerts: **Default routing** or **Specific paths**.
7. Optional: open **Severity and auto-resolve** and change the **Default severity** or **Resolve automatically**.
8. Select **Create and connect**.
9. Follow the setup steps for the tool. Spectate fills in the URL.

The **Live check** panel shows the first request that arrives and what Spectate made of it.

**Default routing** uses the project's escalation paths, or else the team default path.

### The URL and the email address

Each alert source gets a secret URL:

```text
https://api.spectate.net/integrations/alerts/YOUR_SOURCE_TOKEN
```

An email source gets a secret address:

```text
YOUR_SOURCE_TOKEN@alerts.spectate.net
```

The token in the URL or address is the credential. Anyone with the URL can send alerts to the source. Keep it secret.

## Webhook format

Send a `POST` request with a JSON body. Only `title` is required.

```bash
curl -X POST "https://api.spectate.net/integrations/alerts/YOUR_SOURCE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Disk almost full on db-1",
    "severity": "high",
    "dedupe_key": "db-1/disk",
    "description": "Less than 5% free on /var/lib/postgresql",
    "links": [{ "label": "Runbook", "url": "https://wiki.example.com/disk" }],
    "metadata": { "host": "db-1" }
  }'
```

To resolve the alert, send the same `dedupe_key` with `"status": "resolved"`:

```bash
curl -X POST "https://api.spectate.net/integrations/alerts/YOUR_SOURCE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "dedupe_key": "db-1/disk", "title": "Disk almost full on db-1", "status": "resolved" }'
```

| Field | Required | Description |
|---|---|---|
| `title` | Yes | What is wrong, up to 250 characters. Spectate also reads `summary` or `name`. |
| `status` | No | `firing` (default) or `resolved`. Spectate also reads `ok`, `recovered` and `closed` as resolved. |
| `dedupe_key` | No | Your ID for the problem. Requests with the same key keep one alert firing. The default is the title. Spectate also reads `id`. |
| `severity` | No | `critical`, `high`, `medium` or `low`, or a word from the severity table below. Spectate also reads `priority`. |
| `description` | No | More about the problem, up to 4000 characters. Spectate also reads `message`. |
| `links` | No | Up to 5 links, each `{"label": "...", "url": "https://..."}`. A single URL string also works. |
| `metadata` | No | Up to 20 keys with text, number or boolean values. |

To send more than one alert in one request, put them in an `alerts` list, up to 100 alerts:

```json
{ "alerts": [ { "title": "Disk almost full on db-1" }, { "title": "Backup failed", "severity": "critical" } ] }
```

To check the connection without alerting anyone, send `{"test": true}`.

### Response

Spectate answers a valid request with `202 Accepted` and what it did:

```json
{
  "outcome": "fired",
  "summary": "Fired 1",
  "error": null,
  "alerts": [{ "action": "fired", "dedupe_key": "db-1/disk" }]
}
```

| Status | Meaning |
|---|---|
| `202` | Received. `outcome` says what happened. |
| `401` | The signature or bearer token does not match the source's secret. |
| `404` | No alert source has this URL. The token is wrong, or someone made a new URL. |
| `413` | The body is larger than 512 KB. |
| `422` | Spectate cannot read the body, for example JSON without a title. |
| `429` | More than 300 requests in a minute for this URL. |

Rejected requests fire nothing.

## Prometheus Alertmanager

Add a receiver to `alertmanager.yml`, then reload Alertmanager:

```yaml
receivers:
  - name: spectate
    webhook_configs:
      - url: "https://api.spectate.net/integrations/alerts/YOUR_SOURCE_TOKEN"
        send_resolved: true

route:
  receiver: spectate
```

- Each alert becomes its own Spectate alert. The dedupe key is the alert's `fingerprint`.
- The title is the `summary` annotation. Without it, the title is the `alertname` label, followed by `on` and the `instance` label.
- The severity comes from the `severity` label, or else the `priority` label.
- The labels are kept. Escalation path [branches](/alerting/escalation-paths#branches) can test them, for example `env` is `production`.

If you set a secret on the source and Alertmanager sends an `Authorization: Bearer` header, the token must be the secret.

## Grafana

1. In Grafana, go to **Alerting → Contact points → + Add contact point**.
2. Name it `Spectate` and select the **Webhook** integration.
3. Paste the alert source URL.
4. Under **Optional Webhook settings → HMAC signature**, paste a secret.
5. Select **Test** in Grafana. Spectate shows **Grafana test notification received**.
6. In **Alerting → Notification policies**, use the Spectate contact point.

Generate the secret in Spectate with **Generate a secret** in the setup steps. With a secret, Spectate rejects requests without a valid `X-Grafana-Alerting-Signature` header. The Grafana **Test** button fires no alert. Grafana alerts use the Alertmanager format, so titles, severity and labels work as above.

## Sentry

1. In Sentry, go to **Settings → Developer Settings → Custom Integrations**.
2. Select **Create New Integration → Internal Integration** and name it `Spectate`.
3. Paste the alert source URL as the webhook URL, and turn on **Alert Rule Action**.
4. Give **Issue & Event** read access, and under **Webhooks** select **issue**.
5. Save, then paste the **Client Secret** in the Spectate setup steps.
6. In your Sentry alert rules, add the action **Send a notification via Spectate**.

| Sentry sends | Spectate does |
|---|---|
| An issue alert rule fires | Fires an alert for the issue |
| A metric alert is `critical` or `warning` | Fires an alert for the metric alert |
| A metric alert resolves | Resolves its alert |
| An issue is resolved, ignored or archived | Resolves the issue's alert |
| A new issue | Fires an alert only if you select **Also alert on every new issue** |

With the client secret saved, Spectate checks the `Sentry-Hook-Signature` header of each request. Very old self-hosted Sentry versions can use the legacy **WebHooks** plugin with the same URL. That plugin does not sign requests.

## Email

Send or forward alert emails to the source's address. Backup jobs, NAS and UPS alerts, cron, Zabbix and Nagios can all send email.

- The subject is the alert title. The body is the description.
- The same subject again keeps the alert firing.
- A subject that starts with `RESOLVED`, `RECOVERED`, `RECOVERY`, `OK` or `CLEARED` resolves the alert with the same subject.
- A word such as `[CRITICAL]` or `Warning:` at the start of the subject sets the severity.

Email has no standard way to say that a problem is over. Email sources resolve their alerts after 60 minutes without a repeat by default.

## Dedupe keys

One alert source has one firing alert per dedupe key. A repeat of a firing key keeps the alert firing and does not notify anyone again. A resolve resolves the alert with that key.

| Tool | Dedupe key |
|---|---|
| Webhook | `dedupe_key`, else `id`, else the title |
| Email | The subject, without `RESOLVED`, `[PROBLEM]` and similar words |
| Alertmanager and Grafana | The alert's `fingerprint` |
| Sentry | The issue, or the metric alert |

## Severity

If the tool sends a severity, Spectate maps it to its own four values. If the tool sends none, or a word not in this table, the alert gets the source's **Default severity**. The default is `high`.

| Spectate severity | Words that map to it |
|---|---|
| `critical` | critical, crit, fatal, emergency, alert, page, p1, sev1, disaster |
| `high` | high, error, err, major, p2, sev2 |
| `medium` | medium, warning, warn, average, p3, sev3 |
| `low` | low, info, information, informational, minor, notice, debug, p4, p5, sev4, sev5 |

## Resolve automatically

Some tools never say that a problem is over. For these, set **Resolve automatically**. Spectate then resolves an alert when the tool has not repeated it for that time.

| Option | `auto_resolve_minutes` |
|---|---|
| **Never** | `null` (default, except for email) |
| **After 30 minutes** | `30` |
| **After 1 hour** | `60` (default for email) |
| **After 4 hours** | `240` |
| **After 24 hours** | `1440` |

With the API, any value from 5 to 10080 minutes is allowed.

## Recent events

The source page shows **Recent events**: every request of the last 7 days, with the outcome and the request itself. Look here first when an alert did not arrive. Spectate stores the first 32 KB of each request.

| Outcome | Meaning |
|---|---|
| `fired` | It fired at least one new alert. |
| `resolved` | It resolved at least one alert. |
| `updated` | It repeated alerts that were already firing. |
| `test` | A test from the tool's own test button. No alert. |
| `ignored` | Valid, but nothing to do. For example, a resolve for an alert that is not firing, or a request to a source that is turned off. |
| `rejected` | Refused: a bad signature or a body that Spectate cannot read. The event shows the error. |

## Send a test alert

1. Open the alert source.
2. Select **Send test alert**.
3. Confirm.

Spectate fires a real alert, `Test alert from SOURCE NAME`, through the source's escalation paths. The test alert resolves itself after 5 minutes.

## Make a new URL

If the URL or address leaked, make a new one:

1. Open the alert source.
2. Select **New URL**, or **New address** for an email source.
3. Select **Create new URL** or **Create new address** to confirm.

:::caution
The old URL or address stops working at once. Update the tool with the new one, or its alerts do not arrive.
:::

## Turn off or delete a source

To stop a source without deleting it, open its **Settings** and clear **Receiving alerts**. Spectate still logs the requests, but fires nothing. When you delete a source, Spectate resolves its firing alerts first.

## Related

- [Alert sources API](/api-reference/alert-sources): create sources, rotate the URL, send a test alert and read events.
- [Escalation paths](/alerting/escalation-paths): route alerts by source and label.