Heartbeat monitors
A heartbeat monitor watches a job that runs on a schedule, such as a backup or a queue worker. No check goes out from Spectate. The job calls a secret URL each time it runs, and that call is a ping. If a ping does not arrive in time, Spectate fires an alert through the escalation paths of the monitor.
Add a heartbeat monitor
- In the sidebar, select Monitors.
- Select New monitor.
- Under Type, select Heartbeat.
- Set the Name and The job runs every.
- Select Create and start.
- On the page of the monitor, copy the ping URL.
- Add a call to the ping URL at the end of the job.
Code
The monitor is Pending until the first ping arrives. A pending monitor does not fire alerts, so you can configure the job without a page. Monitoring starts with the first ping. If the job never sends a ping, the monitor stays Pending and you get no alert.
Report a failure
The job can report a failure. The monitor then goes down at once, without a wait for the deadline. There are two ways.
Send the exit code of the job in the URL. Exit code 0 is a normal ping. Spectate treats any other code as a failure and shows the code in the alert.
Code
Or add /fail to the URL:
Code
With a POST request, the body is the output of the job. Spectate shows the first 1,000 characters in the alert. The monitor is down until the next normal ping.
Ping URLs
| Request | Result |
|---|---|
GET, HEAD or POST /ping/YOUR_TOKEN | A normal ping |
GET or POST /ping/YOUR_TOKEN/0 | A normal ping |
GET or POST /ping/YOUR_TOKEN/CODE, with CODE not 0 | A failure with that exit code |
GET or POST /ping/YOUR_TOKEN/fail | A failure |
Spectate answers 200 to a known URL, also when the monitor is paused. It ignores pings to a paused monitor. An unknown URL answers 404. Spectate accepts 30 requests per minute per URL, and answers 429 above that.
Anyone with the ping URL can report for the job, and can hide a failure with normal pings. Do not put the URL in a public repository. Spectate does not show the URL in notifications.
To replace the URL, select New URL on the page of the monitor. The old URL stops working at once. Then update the job.
When a monitor goes down
The deadline for the next ping is the time of the last ping, plus the period, plus the grace period. Spectate looks for passed deadlines every minute, so an alert can come up to 1 minute after the deadline.
If no ping arrives by the deadline, the monitor goes down. Spectate fires one alert, also if the job misses more runs. With Alert after above 1 missed run, the monitor goes down only after that many misses in a row. A ping sets the count back to 0.
When pings return, the monitor is up at once. The alert stays open for the recovery wait of the team, then resolves.
Settings
| Setting | API field | Values | Default |
|---|---|---|---|
| The job runs every | ping_period | 30 seconds to 366 days | 1 day in the dashboard. Required in the API. |
| Grace period | ping_grace | 0 seconds to 30 days | 5 minutes |
| Time zone of the job's clock | ping_timezone | An IANA time zone, for example Europe/Amsterdam | The time zone of the team |
| Alert after | ping_alert_after | 1 to 20 missed runs in a row | 1 |
| Alert severity | ping_severity | critical, high, medium, low | critical |
A heartbeat monitor counts as one monitor towards the plan limit. See the pricing page.
If a job runs once a month, set the period to the longest month. The dashboard option 1 month is 31 days. A job on the 1st of each month is then on time after a month of 31 days.
Daylight saving time
A job that runs at 02:30 on a local clock runs 25 hours after the previous run on the night that clocks go back. Spectate does not treat that run as late, and you do not need a longer grace period.
For a period of 1 hour or more, the deadline is the later of these two times, plus the grace period:
- The time of the last ping plus the period.
- The same time on the clock in Time zone of the job's clock.
Set the time zone to the time zone of the server that runs the job. If the server uses UTC, any time zone is safe, because the first time still applies. A wrong alert is possible only if the server uses a different time zone with daylight saving time.
On the two nights a year that clocks change, an alert about a job that really stopped can come up to 1 hour late.
If clocks go forward and skip the time of a job, the behaviour depends on your scheduler. Some schedulers run the job after the change, and others skip that run. If the job does not run, Spectate fires an alert, because the job did not run.
Uptime of a heartbeat monitor
Each ping counts as a passed check, and each missed period as a failed check. A daily job that misses 1 run in 90 days shows an uptime of about 97.8%. The figure includes one extra failed check per outage, so that the outage shows the correct duration.
Misses before Alert after is reached do not count as downtime.
If Spectate itself cannot receive pings because of an outage, Spectate does not fire these alerts. A job that really stopped during the outage is reported one period later.
In the API
Create a monitor with protocol set to heartbeat and a ping_period. It does not need a uri or an interval. The ping_url field of the monitor has the URL. A monitor cannot change to or from heartbeat.