# Heartbeat monitors

A heartbeat monitor watches a job that runs on a schedule, such as a backup or a queue worker. No check goes out from Spectate. The job calls a secret URL each time it runs, and that call is a **ping**. If a ping does not arrive in time, Spectate fires an alert through the [escalation paths](/alerting/escalation-paths) of the monitor.

## Add a heartbeat monitor

1. In the sidebar, select **Monitors**.
2. Select **New monitor**.
3. Under **Type**, select **Heartbeat**.
4. Set the **Name** and **The job runs every**.
5. Select **Create and start**.
6. On the page of the monitor, copy the ping URL.
7. Add a call to the ping URL at the end of the job.

```bash
./backup.sh && curl -fsS -m 10 --retry 3 https://api.spectate.net/ping/YOUR_TOKEN
```

The monitor is **Pending** until the first ping arrives. A pending monitor does not fire alerts, so you can configure the job without a page. Monitoring starts with the first ping. If the job never sends a ping, the monitor stays **Pending** and you get no alert.

## Report a failure

The job can report a failure. The monitor then goes down at once, without a wait for the deadline. There are two ways.

Send the exit code of the job in the URL. Exit code `0` is a normal ping. Spectate treats any other code as a failure and shows the code in the alert.

```bash
output=$(./backup.sh 2>&1); curl -fsS -m 10 --retry 3 --data-raw "$output" https://api.spectate.net/ping/YOUR_TOKEN/$?
```

Or add `/fail` to the URL:

```bash
curl -fsS -m 10 --retry 3 --data-raw "disk full" https://api.spectate.net/ping/YOUR_TOKEN/fail
```

With a `POST` request, the body is the output of the job. Spectate shows the first 1,000 characters in the alert. The monitor is down until the next normal ping.

## Ping URLs

| Request | Result |
|---|---|
| `GET`, `HEAD` or `POST /ping/YOUR_TOKEN` | A normal ping |
| `GET` or `POST /ping/YOUR_TOKEN/0` | A normal ping |
| `GET` or `POST /ping/YOUR_TOKEN/CODE`, with `CODE` not `0` | A failure with that exit code |
| `GET` or `POST /ping/YOUR_TOKEN/fail` | A failure |

Spectate answers `200` to a known URL, also when the monitor is paused. It ignores pings to a paused monitor. An unknown URL answers `404`. Spectate accepts 30 requests per minute per URL, and answers `429` above that.

:::caution
Anyone with the ping URL can report for the job, and can hide a failure with normal pings. Do not put the URL in a public repository. Spectate does not show the URL in notifications.
:::

To replace the URL, select **New URL** on the page of the monitor. The old URL stops working at once. Then update the job.

## When a monitor goes down

The deadline for the next ping is the time of the last ping, plus the period, plus the grace period. Spectate looks for passed deadlines every minute, so an alert can come up to 1 minute after the deadline.

If no ping arrives by the deadline, the monitor goes down. Spectate fires one alert, also if the job misses more runs. With **Alert after** above 1 missed run, the monitor goes down only after that many misses in a row. A ping sets the count back to 0.

When pings return, the monitor is up at once. The alert stays open for the [recovery wait](/alerting/alerts#recovery-wait) of the team, then resolves.

## Settings

| Setting | API field | Values | Default |
|---|---|---|---|
| **The job runs every** | `ping_period` | 30 seconds to 366 days | 1 day in the dashboard. Required in the API. |
| **Grace period** | `ping_grace` | 0 seconds to 30 days | 5 minutes |
| **Time zone of the job's clock** | `ping_timezone` | An IANA time zone, for example `Europe/Amsterdam` | The time zone of the team |
| **Alert after** | `ping_alert_after` | 1 to 20 missed runs in a row | 1 |
| **Alert severity** | `ping_severity` | `critical`, `high`, `medium`, `low` | `critical` |

A heartbeat monitor counts as one monitor towards the plan limit. See the [pricing page](https://spectate.net/pricing).

If a job runs once a month, set the period to the longest month. The dashboard option **1 month** is 31 days. A job on the 1st of each month is then on time after a month of 31 days.

## Daylight saving time

A job that runs at 02:30 on a local clock runs 25 hours after the previous run on the night that clocks go back. Spectate does not treat that run as late, and you do not need a longer grace period.

For a period of 1 hour or more, the deadline is the later of these two times, plus the grace period:

- The time of the last ping plus the period.
- The same time on the clock in **Time zone of the job's clock**.

Set the time zone to the time zone of the server that runs the job. If the server uses UTC, any time zone is safe, because the first time still applies. A wrong alert is possible only if the server uses a different time zone with daylight saving time.

On the two nights a year that clocks change, an alert about a job that really stopped can come up to 1 hour late.

If clocks go forward and skip the time of a job, the behaviour depends on your scheduler. Some schedulers run the job after the change, and others skip that run. If the job does not run, Spectate fires an alert, because the job did not run.

## Uptime of a heartbeat monitor

Each ping counts as a passed check, and each missed period as a failed check. A daily job that misses 1 run in 90 days shows an uptime of about 97.8%. The figure includes one extra failed check per outage, so that the outage shows the correct duration.

Misses before **Alert after** is reached do not count as downtime.

If Spectate itself cannot receive pings because of an outage, Spectate does not fire these alerts. A job that really stopped during the outage is reported one period later.

## In the API

Create a monitor with `protocol` set to `heartbeat` and a `ping_period`. It does not need a `uri` or an `interval`. The `ping_url` field of the [monitor](/api-reference/monitors) has the URL. A monitor cannot change to or from `heartbeat`.

## Related

- [Uptime monitors](/monitors/introduction)
- [Escalation paths](/alerting/escalation-paths)
- [Pause and delete a monitor](/monitors/pause-and-delete)