Pingtower
Docs
Rules
A rule says what to look for, how much of it is a problem, and what to do about it. Rules are per project, declarative, and editable from the phone.
{
"name": "gateway-timeouts",
"match": {
"min_level": 400,
"sources": ["api", "worker"],
"tags": ["billing"],
"template_contains": "gateway timeout"
},
"conditions": [
{ "key": "latency_ms", "op": "gte", "value": 10000 }
],
"fire_after": { "count": 5, "window_minutes": 10, "distinct_sources": true },
"identifier_keys": ["gateway"],
"dedup_window_minutes": 30,
"renotify_minutes": 10,
"max_renotify": 6,
"escalations": [
{ "after_minutes": 15, "schedules": ["backend-secondary"], "repeat_minutes": 10 },
{ "after_minutes": 45, "accounts": ["[email protected]"] }
],
"actions": ["restart-gateway-worker"],
"auto_resolve_quiet_minutes": 60,
"investigate": true
}
POST /v1/projects/{id}/rules creates one. GET PUT DELETE /v1/projects/{id}/rules/{name} read, replace and remove it. A rule’s name is its identity.
Match
All fields optional, combined with AND. An empty match matches every line in the project.
| Field | Matches when |
|---|---|
min_level | The line’s level is at least this. |
sources | The line came from one of these source names. |
tags | The line carries every tag listed. |
template_contains | The recovered template contains this substring, up to 500 characters. Matched on the shape, so variable parts never break it. |
Conditions
Compare a key from the line. op is one of eq, neq, gte, lte. Numbers compare numerically. A line without the key fails the condition. This is how addon measurements become alerts: {"key": "days_until_expiry", "op": "lte", "value": 14} pages you two weeks before a certificate expires.
Fire after
Without fire_after, the first matching line opens the alert. With it, the rule needs count matches inside window_minutes. distinct_sources requires them to come from different sources, which is how “down from more than one probe” works.
Identity
By default one rule holds one firing alert, and every matching line bumps its count. identifier_keys splits that by key value, so a rule watching gateway holds a separate alert for stripe and for adyen, each with its own count and its own ack. The values appear on the alert as identifiers.
dedup_window_minutes is how long after an alert resolves a recurrence still folds into it rather than opening a new one.
Renotify and escalation
A firing, unacked alert re-pages every renotify_minutes, up to max_renotify times. Each escalation level, once after_minutes has passed, adds who it names to the people being paged: an on-call schedule by name, or specific accounts by email. repeat_minutes on a level re-pages that level on its own cadence. Up to ten levels. Acking stops all of it.
An escalation is what makes a push critical. The rule’s own recipients get an ordinary notification; the ladder’s recipients get one that sounds through Silent mode.
Actions
actions names runbook actions in the same project, by id or name. They fire when the alert opens, and again on escalation for actions that opted into that event. The reference is checked when the rule is written; a rule cannot point at an action that does not exist, and an action a rule points at cannot be deleted.
Resolution
Anyone can resolve from the app or the API. auto_resolve_quiet_minutes resolves on your behalf once the rule has been silent that long. A resolved alert that matches again inside the dedup window reopens; after it, a fresh alert opens with a fresh count.
investigate: true the alert is flagged for the app to lead with the log tail filtered to its template, so the first tap shows the actual lines rather than the summary. Needs retain_logs on the source.Referential checks
A rule’s sources, actions and escalation schedules are checked at write time; a name that does not exist is a 422. Deleting a source, action or schedule a rule still names is refused with 409 and the list of rules that reference it. See Errors.