Uptime monitors

Locations and checks

Spectate runs uptime checks from checkers in four regions. Each monitor has one primary region, and every check of the monitor runs there. Before Spectate marks a monitor down, a second check from the same region, about 5 seconds later, must confirm the failure.

Regions

Regionprimary_location in the API
Europeeu
North Americaus
Asiaas
Oceaniaoc

The checks of a region come from its own IP addresses. IP addresses has the current list, also as plain text for firewalls.

To change the region of a monitor, open the monitor and select the Settings tab. Under Region, select the region, then select Save changes. Uptime, response times, charts and alerts of the monitor all come from this region. Select the region that is nearest to your users, or nearest to the server, to measure without long network paths.

How often Spectate checks

A checker asks Spectate for the monitors that are due, about every 10 seconds. A monitor is due when its Interval has passed since its last check. One checker in the region gets the monitor, so each check runs one time.

Two kinds of monitors do not wait for the interval:

  • Spectate checks a new monitor as soon as a checker asks.
  • Spectate checks a monitor that is down every time a checker asks, until it is up again. So Spectate sees a recovery soon after it happens.

The plan sets the shortest interval: 60 seconds on Starter, 30 seconds on Team and Business. See Uptime monitors.

How Spectate confirms a failure

  1. A check fails, for example with a timeout or an unexpected status code.
  2. The checker waits about 5 seconds.
  3. The checker runs the check again, from the same region. This is the recheck.
  4. If the recheck succeeds, the monitor is up. Spectate fires no alert.
  5. If the recheck also fails, the monitor is down. Spectate fires the alert "NAME is down" with the severity critical.

A failed check with a successful recheck is not downtime. The uptime, the downtime report and the status pages count it as up. The waterfall of the check on the Performance tab still shows the failure, marked "not counted as down".

A failed check during a maintenance window is also not downtime, unless the window counts it. The waterfall marks it the same way.

The recheck does not run from another region. The other regions never change the uptime or the alerts of a monitor. With Also measure response time from the other regions (Team and Business plans), they only measure its response time for comparison.

For HTTP and HTTPS monitors, the recheck that makes the monitor down also records a failure snapshot. One successful check makes the monitor up again. Spectate resolves the down alert after the recovery wait, 5 minutes by default.

While the alert fires, Spectate does not fire a second down alert for the same monitor. This includes a monitor that goes down again during the recovery wait. The escalation path of the alert decides who gets notified, and when.

Route of a failed check

When a monitor goes down without an HTTP response, the checker traces the route to the target. It does this once per outage, from the region of the recheck. The alert does not wait for it. The route shows some seconds later on the alert page, under What we saw.

The checker sends 5 ICMP echo requests to each hop, up to 30 hops. This is the same method as the mtr tool. For each hop, the table shows the name and address, the Loss, and the Best, Avg and Worst round-trip times.

  • If the target answered, the network path works. The problem is on the target, for example a service that stopped.
  • If the target did not answer, the last hop that answered shows how far the probes got. The problem is likely after that hop.
  • A hop with No answer does not always mean a problem. Many routers do not answer these probes.
  • Loss at one hop that does not continue to later hops is usually harmless. Routers often limit these answers.

Spectate does not trace a route for an HTTP response with an error status, because the response proves that the route works. DNS monitors and heartbeat monitors have no route.

A firewall can block ICMP on the way to a target that accepts your traffic. The route then ends early, also if the path for your traffic works.

Recognise the checks

HTTP and HTTPS checks send this User-Agent header, unless the monitor sets its own:

Code
User-Agent: Spectate/1.0 (+https://docs.spectate.net/faq/uptime-monitor-bot)

If a firewall, a WAF or a bot filter blocks the checks, allow the user agent or the IP address of the monitor's region. See The uptime monitor bot and IP addresses.

Each HTTP check opens a new connection. It does not keep cookies between checks. The check measures DNS lookup, connection, TLS handshake and time to first byte. The Heartbeats tab shows these times per check in Check log · 24h.

Targets Spectate does not check

Spectate checks only public addresses. It refuses targets that are, or resolve to, private, loopback, link-local, carrier-grade NAT, multicast or reserved addresses. Examples are 10.0.0.5, 192.168.1.1, 127.0.0.1 and 169.254.169.254. The checker tests the address again before each connection, also after a redirect.

To monitor a server on a private network, give it a public address or a public URL. For metrics from inside the network, install the Spectate host agent.