Hosted checks

A check is a source of events, not a second alerting path. It names an addon and the parameters to run it with, never a command. Each run’s result arrives as an ordinary event carrying the tags addon:<name>, check:<name> and status:<status>. Every alerting decision about it is a rule’s.

On pingtower.com, the regions a check runs from are Pingtower’s own servers. A self-hosted tower has no fleet. There, a check names your own sources, and the addons agent on each one runs it. The agent holds that source’s agent credential (ptp_, minted with POST /v1/projects/{id}/sources/{name}/agent-token), asks the tower what the source should run, and posts the results back. The same check, rule and quorum work in both places. A hosted project can mix the two: a check can name regions and your own sources together.

What runs where

UDP and mail checks are agent-only. Pingtower’s servers share their addresses across every customer, and UDP payloads and mail-server probes from shared addresses are what get an address range listed. A check of either kind that names a Pingtower region is refused with a 422 on sources, under the rule agent_only:

udpcheck runs only on your own agent; source <region> is a fleet region

The other addons, such as hostmetrics and the privileged ones, are not on the fleet either. A check naming a region for one of them is refused with addon <name> is not available on the fleet; run it from your own agent.

Three addons have tighter rules on Pingtower’s servers than on your own agent:

• tcpcheck may name only well-known service ports: 21, 22, 25, 53, 80, 110, 143, 443, 465, 587, 993, 995, 3306, 5432, 6379, 8080 and 8443.
• grpccheck may name only ports 80, 443, 8080, 8443, 9090 and 50051.
• dnscheck must leave resolver unset or name 1.1.1.1, 1.0.0.1, 8.8.8.8, 8.8.4.4 or 9.9.9.9.

A target that resolves to a private, loopback, link-local or cloud-metadata address is refused when you save the check. The refusal says the target was refused, never what it resolved to.

Create one

Create a check in the web app or the iOS app, each with a form per kind. The iOS app creates and edits every kind above except browser checks, which it shows read-only.

Through the API, POST /v1/projects/{id}/checks creates one, and GET PUT DELETE /v1/projects/{id}/checks/{name} read, replace and remove it. A check’s name is its identity and can’t be changed, because it is what the event’s check key carries and what a rule groups on.

curl -s https://api.pingtower.com/v1/projects/web/checks \
  -H "Authorization: Bearer ptk_…" \
  -d '{
    "name": "api-health",
    "addon": "httpcheck",
    "with": {"url": "https://api.acme.example/healthz"},
    "interval_seconds": 60,
    "sources": ["<region>", "<region>", "<region>"],
    "alert_if": {}
  }'

GET /v1/regions lists the regions your plan can name, and /v1/regions?pool=browser the regions with browser runners. Naming a region on a check adds that region as a source in the project, along with a <region>-quiet rule (see When a region goes quiet). A region’s name is reserved: you can’t create a source under it by hand.

A check write answers with the stored check, the name of the rule alert_if wrote in alert_rule, and any warnings.

Intervals

A browser check runs at most every 300 seconds on every plan. The plan also caps how many regions one check may name, and holds every check to a capacity budget in runs per minute: each check costs 60 ÷ interval × weight, once per region it names. Most kinds weigh 1. An httpcheck with a body weighs 2, an icmpping weighs its count (3 by default), a multi-step check one per step plus one per OAuth step, and a browser check 20. A write past any of these limits is refused with 429, naming the limit (max_checks, max_check_sources, min_check_interval_seconds or slot_budget) and both numbers:

min_check_interval_seconds: interval_seconds 60 is below this plan's floor of 300 seconds

On Free, the budget fits one browser check only at an interval of 400 seconds or more from one region, and only while your other checks leave room for it.

Regions and quorum

A check that runs from several regions gets several independent answers, and one region’s own network failing is not your service down. alert_if.down_from is how many regions must report the check down inside one window before it pages. Leave it out and it is a majority of the check’s sources:

A down_from above the number of sources is refused, naming both numbers. Setting it to 1 on a check with several sources is allowed, and the write carries this warning, which the web app and the iOS app show as written:

This check alerts when any 1 of its 3 locations reports it down, including a location whose own network failed. Requiring 2 of 3 is recommended.

Instant confirmation

When a check runs from two or more regions and one of them reports it down, Pingtower asks each of the check’s other regions to run it once more straight away, rather than at their next scheduled run. The quorum is then met, or not, within seconds instead of an interval.

That happens once per failure streak. The streak opens with the first region to report down and stays open while any region is still reporting down. A region that reports up leaves the streak, and so does one whose last down is more than about two intervals old, so a region that stopped reporting can’t hold it open. Further failures inside a streak don’t ask again. A region that can’t take the extra run skips it and answers at its next scheduled run. That happens when the check isn’t placed there, when the region is shedding load, or when the target has used up its share of requests from Pingtower’s servers. The extra run’s result is an ordinary result: it counts toward the quorum and the history like any other.

A check that runs from one region, and a domain expiry check, has no other region to ask.

Domain expiry runs from one place

A domain’s expiry date is the same from everywhere, and registries rate-limit by address, so a domainexpiry check runs from exactly one source. Name one, or name none and Pingtower picks one. Naming more is refused with a 422 under the rule single_placement. Its alert_if.down_from is always 1, its interval defaults to once a day and can be no shorter than an hour (3,600 seconds; shorter is a 400 with the rule interval_floor on interval_seconds), and its timeout defaults to 30 seconds. On a self-hosted tower, Pingtower picks a live agent of yours that runs the addon. When none does, the check’s placement_note says so.

Alerting

alert_if writes an ordinary rule, named <check> is down, that you can see in the rule list and edit like any other. Send "alert_if": {} for the default majority, or {"down_from": 2}. Leave alert_if out for no rule. The rule matches the check’s own tag at level 400 or above, which is a down result:

{
  "name": "api-health is down",
  "match": { "tags": ["check:api-health"], "min_level": 400 },
  "identifier_keys": ["check"],
  "fire_after": { "count": 2, "window_minutes": 2, "distinct_sources": true }
}

The window is twice the interval, rounded up to whole minutes and at least one: 1 minute at 15 seconds, 2 at 60, 10 at 300. Every region re-sends a down result on each run, so two regions that are both down always have two votes in the window.

The check owns match, identifier_keys and fire_after, and rewrites them when you edit the check. Everything else is yours from the first write: escalations, renotify, actions, auto-resolve. Removing alert_if, deleting the check, or deleting the rule by hand detaches the rule rather than deleting it.

A degraded result, such as an httpcheck over its max_ttfb, is level 300 and does not trip this rule. Write your own rule on the measurements for that, as on each addon’s page.

A run that could not happen at all, such as a probe that could not open a socket or a browser sandbox that would not start, is tagged status:error. It says nothing about your target. It is kept at level 300, so it shows in the log but never votes toward an alert.

When a region goes quiet

The <region>-quiet rule Pingtower writes beside each region is a source_silent rule on that region’s source. Its horizon is twice the check’s interval plus 30 seconds, rounded up to whole minutes. Each region reports every run, so a working region never trips it and a stopped one always does. It is your rule from the start: give it a ladder, change the horizon, or delete it. It counts toward the watched-pairs cap but is always created, so a check or a new region is never refused over it.

Secrets

Some kinds take secrets: an httpcheck’s authentication secret, a multi-step check’s secret variables, a mail check’s PASSWORD, a browser check’s environment variables and the values a WebSocket check’s headers reference as {{NAME}}. They go in the check’s env, never in with. They are sealed at rest and write-only: reading the check back shows which names are set (env_keys for a browser or WebSocket check), never their values. Leaving env out of an update keeps the stored set, and {} clears it, unless the update changes where a secret goes: an HTTP check’s url, oauth_token_url or method, a browser check’s or WebSocket check’s url, a mail check’s address, the URL or method of a multi-step check’s step that uses a secret, even only a path or a plain variable that URL uses, or of a step whose extraction that URL uses, or that step’s extraction rule, headers, body, authentication or redirect setting, or sources gaining a region or source the env was not sealed for. That update must send env again, or it is refused with a 422 such as env: with.url now points at another destination; re-supply env (or send {} to clear it) or env: sources now include <names> the env was not sealed for; re-supply env (or send {} to clear it). Removing or reordering sources keeps the stored set. A value of env that turns up in a run’s result, as written, JSON-escaped or URL-encoded, is stored as [redacted] in its message, keys and step errors; a browser check also catches more encodings, listed on its page. A value under 4 bytes is left as it is, and so is one encoded some other way. Each kind’s page has its own rules for which names it takes. A create or update that finds a credential in plain text in a parameter, such as a token written as it is in a WebSocket check’s headers, still saves, and the answer carries secret_warnings: [{"field": "with.headers.Authorization", "kind": "…"}], naming the parameter and what was found, never the value. Where the kind takes one, use its sealed secret instead; a value that only references env as {{NAME}} is not flagged.

A new agent token withholds secrets

Secrets are sealed for each source as its agent token stood when they were set. Minting an agent token for a source, with POST /v1/projects/{id}/sources/{name}/agent-token, withholds every check’s env from that source: a replacement token, the source’s first token, and a token for a source deleted and created again under the same name all count, whoever mints it, an owner included. A check with env stops running from that source until you send its env again. The source records no results for it, never reports it down, and the agent is not sent the check at all. Reading the check lists those sources in env_withheld_sources, and the web app and the iOS app show the check as not running there. An update that sends env again, as on any update, clears the list, and the check runs from those sources again from the agent’s next poll; an update that leaves env out keeps it. So a project admin can’t read a check’s secrets by taking over a source’s agent: they would have to know the secrets to send them again.

History

Every run is recorded as it arrives, per source, in five-minute buckets kept for 8 days and hourly buckets kept for a year. The web app and the apps draw each check’s success rate and response time over the last 24 hours, 7 days, 30 days or 90 days, for all its regions together or one at a time, and show each region’s latest result.

The success rate is the share of judged runs that passed: up and degraded runs, over all runs less the status:error runs and the runs inside a maintenance window. A probe that could not run is not a judgement of your target.

Disabled and paused

"enabled": false stops a check without deleting it or its history. A browser check can also be paused by the plan: when the tenant’s browser minutes for the month are used up, reading the check shows paused with the reason browser_minutes_exhausted and the time it resumes. See Browser checks. An sshcheck or logwatch check stored before Release 1 shows paused with the reason local_only and resumes_at 0: Pingtower no longer assigns it, and it belongs in the agent’s own config. See Upgrading to Release 1.