Rules

{
  "name": "gateway-timeouts",
  "match": {
    "min_level": 400,
    "sources": ["api", "worker"],
    "tags": ["billing"],
    "template_contains": "gateway timeout"
  },
  "conditions": [
    { "key": "latency_ms", "op": "gte", "value": 10000 }
  ],
  "fire_after": { "count": 5, "window_minutes": 10, "distinct_sources": true },
  "identifier_keys": ["gateway"],
  "dedup_window_minutes": 30,
  "renotify_minutes": 10,
  "max_renotify": 6,
  "escalations": [
    { "after_minutes": 15, "schedules": ["backend-secondary"], "repeat_minutes": 10 },
    { "after_minutes": 45, "accounts": ["[email protected]"] }
  ],
  "actions": ["restart-gateway-worker"],
  "auto_resolve_quiet_minutes": 60,
  "investigate": true
}

POST /v1/projects/{id}/rules creates one. GET PUT DELETE /v1/projects/{id}/rules/{name} read, replace and remove it. A rule’s name is its identity. A project holds at most 500 rules, and every …_minutes field is capped at 525,600, a year; the rest are in Limits.

Match

All fields optional, combined with AND. An empty match matches every line in the project.

Conditions

Compare a key from the line. op is one of eq, neq, gte, lte. Numbers compare numerically. value is a string of at most 1024 bytes, a number, a boolean or null; up to 32 conditions. A line without the key fails the condition. This is how addon measurements become alerts: {"key": "days_until_expiry", "op": "lte", "value": 14} pages you two weeks before a certificate expires.

Fire after

Without fire_after, the first matching line opens the alert. With it, the rule needs count matches inside window_minutes. distinct_sources requires them to come from different sources, which is how “down from more than one probe” works.

Identity

By default one rule holds one firing alert, and every matching line bumps its count. identifier_keys splits that by key value, so a rule watching gateway holds a separate alert for stripe and for adyen, each with its own count and its own ack. The values appear on the alert as identifiers.

dedup_window_minutes is how long after an alert resolves a recurrence still folds into it rather than opening a new one.

Renotify and escalation

A firing, unacked alert re-pages every renotify_minutes, up to max_renotify times. Each escalation level, once after_minutes has passed, adds who it names to the people being paged: an on-call schedule by name, or specific accounts by email. repeat_minutes on a level re-pages that level on its own cadence. Up to ten levels. Acking stops all of it.

Opening and escalating are the events delivered as critical pushes, which sound through Silent mode on phones that allow it. A bump, an ack or a resolve is an ordinary notification.

Actions

actions names runbook actions in the same project, by id or name. They fire when the alert opens, and again on escalation for actions that opted into that event. The reference is checked when the rule is written; a rule cannot point at an action that does not exist, and an action a rule points at cannot be deleted.

Resolution

Anyone can resolve from the app or the API. auto_resolve_quiet_minutes resolves on your behalf once the rule has been silent that long. A resolved alert that matches again inside the dedup window reopens; after it, a fresh alert opens with a fresh count.

Muting a rule

PUT /v1/projects/{id}/rules/{name}/mute with {"minutes": 60} or {"until": <unix seconds>} (exactly one, at most 90 days out) mutes it: its alerts still open and show, but nothing pages until the mute ends. DELETE on the same path ends it now, and anything it held back pages at the next tick. GET /v1/projects/{id}/mutes lists the active ones. Responders and above can mute or unmute; a viewer cannot. The project’s channels hear about every mute and unmute as a rule.muted or rule.unmuted event naming the rule, who did it and until when, so paging never goes quiet without anyone knowing.

Sources that go quiet

A normal rule is evaluated when a line arrives, so it can’t notice a line that never comes. A rule with a source_silent trigger is evaluated by time instead:

{
  "name": "backup-job-quiet",
  "trigger": { "type": "source_silent", "sources": ["nightly-backup"], "silent_for_minutes": 1500 },
  "escalations": [
    { "after_minutes": 10, "schedules": ["ops-oncall"] }
  ]
}

• It opens one alert per named source that has sent nothing for silent_for_minutes (1 minute to a year), and pages through the rule’s ladder and actions like any alert.
• The first line that source sends again resolves the alert. You can also ack or resolve it by hand.
• A new source gets one full quiet interval to send its first line before it counts as silent. A source with a heartbeat URL is different: it is pending until its first ping and never pages before it.
• It names up to 64 sources and carries no match, conditions, fire_after, identifier_keys, dedup_window_minutes or auto_resolve_quiet_minutes. Omitting trigger, or sending {"type": "event"}, is an ordinary rule.

On a cron schedule

A job that runs at set times rather than at a steady interval can give the trigger a schedule instead of silent_for_minutes:

{
  "name": "backup-job-missed",
  "trigger": {
    "type": "source_silent",
    "sources": ["nightly-backup"],
    "cron": "5 2 * * *",
    "timezone": "Europe/Oslo",
    "grace_minutes": 10
  },
  "escalations": [
    { "after_minutes": 10, "schedules": ["ops-oncall"] }
  ]
}

• cron is a standard 5-field expression (minute, hour, day of month, month, day of week) in Vixie cron’s dialect (lists, ranges, /step, month and weekday names, 0 or 7 for Sunday; when both day fields are set, a day matching either runs), with no seconds field and no @daily-style macros. timezone is an IANA name and is required with cron. grace_minutes is 1 minute to a year.
• A run is missed when nothing arrives by its expected time plus grace_minutes. The alert opens then, not before, and the next line resolves it. A line counts for a run when it lands within grace_minutes either side of the run’s expected time; one that arrives far from any run doesn’t count for the next.
• The schedule is read in local time, so 5 3 * * * stays 03:05 in Oslo on both sides of a daylight-saving change. A run due in the hour the clocks skip is expected when the gap ends (02:30 on the spring-forward night in Oslo is expected at 03:00), and several times inside one gap are one run. A time the clocks repeat is expected once, at its first instance.
• cron and silent_for_minutes are mutually exclusive: sending both is a 400 on trigger.silent_for_minutes. timezone or grace_minutes without cron is a 400 on that field.

Heartbeat runs

A heartbeat URL can also mark when a job starts. Call /start first and ping as usual when the job finishes:

curl -fsS https://api.pingtower.com/v1/heartbeat/pth_.../start && ./backup.sh && curl -fsS https://api.pingtower.com/v1/heartbeat/pth_...

• /start takes GET, HEAD or POST and answers 200 ({"ok": true} for GET and POST). The finishing ping, or a /fail, records the run’s duration on its event as duration_ms.
• On a cron schedule, a start with no finishing ping within grace_minutes counts as a missed run, so when a job calls /start, set grace_minutes longer than the job takes. With silent_for_minutes, a start doesn’t reset the quiet clock: a run that never finishes is caught once silent_for_minutes has passed since the last finished ping.

Referential checks

A rule’s sources, actions and escalation schedules are checked at write time; a name that does not exist is a 422. Deleting a source, action or schedule a rule still names is refused with 409 and the list of rules that reference it. See Errors.