Docs The service Alerts & escalation

Alerts & escalation

An alert has three states and two verbs. Everything about paging follows from that.

firing→acked→resolved

Firing means nobody has claimed it. It re-pages on its rule’s cadence and climbs its escalation ladder. In the app this is the Open filter.
Acked means someone is on it. Paging stops. The count keeps climbing if lines keep matching, so you can watch whether your fix worked.
Resolved means done. The alert leaves the open feed, stops counting toward the project’s open_alerts, and stays in history for thirty days by default.

What an alert carries

Field	Meaning
`id`	Stable id, used by ack and resolve.
`project`, `rule`	Which rule opened it.
`state`	`firing`, `acked` or `resolved`.
`level`	The highest level seen among its lines.
`count`	Matching lines since it opened.
`escalation_level`	Which rung of the ladder it has reached. 0 is the rule’s own recipients.
`template`	The recovered shape that matched.
`last_message`	The most recent raw line, so the notification says something useful.
`first_seen`, `last_seen`	When it opened and when the last line matched.
`identifiers`	The key values that split this alert from its siblings, when the rule uses `identifier_keys`.
`page_accounts`	Who the current escalation rung is paging, when it narrows the audience.
`investigate`	The rule asked the app to lead with the log tail.
`kind`	`event` for an alert a rule opened from incoming lines, `source_health` for a source gone quiet, `secret_warning` for a secret warning.
`source`	The source a `source_health` or `secret_warning` alert is about.
`reported_by`	Which sources sent the lines it absorbed. See below.
`recent_counts`	The last hour at a glance. See below.
`last_notified_at`	Unix seconds of its last notification: when it opened, or its latest escalation or renotify. Repeats count from it.
`next_escalation_at`	Unix seconds of when it next escalates or renotifies, or `0` when nothing more is due.

These come on every alert in the feed and in the ack and resolve responses.

Who reported it

reported_by lists each source whose lines folded into the alert, oldest first:

"reported_by": [
  { "source": "eu-fra", "count": 3, "first_seen": 1791072000, "last_seen": 1791072240 },
  { "source": "us-iad", "count": 2, "first_seen": 1791072060, "last_seen": 1791072180 }
]

count is how many of that source’s lines the alert absorbed, and first_seen and last_seen are when the first and latest arrived. For a rule with fire_after, the sources whose hits met the threshold are listed from the moment it opens. This is what separates “one box is noisy” from “every box says this”, and for a check run from several places, which ones saw it fail. Up to 64 sources are named; a source past that still bumps the alert. The field is left out when no source is recorded.

The last hour

recent_counts is twelve numbers, one per five minutes over the last hour, oldest first: how many matching lines landed in each. It is what the apps draw as the alert’s sparkline. An alert with nothing in the last hour reads twelve zeros. It is [] when the counts are not known, for example on an alert that opened before the service started keeping them.

When it escalates next

next_escalation_at folds the rule’s three clocks into one time. A ladder’s next rung is due after_minutes past first_seen. The rung it rests on repeats repeat_minutes after its last notification. A rule without a ladder renotifies renotify_minutes after its last notification, up to max_renotify. The earliest that applies wins. Only a firing alert has one: acking it, or reaching a top rung that does not repeat, makes it 0. The escalation tick runs about once a minute, so the page lands up to a minute after this time.

Reading the feed

GET /v1/pull?cursor=N&wait=S returns every alert changed since cursor N and the next cursor. wait parks the request for up to that many seconds if nothing has changed, which is how the app keeps a live feed without polling in a loop. wait=0 returns immediately.

curl -s "$PT/v1/pull?cursor=0" -H "Authorization: Bearer ptk_…"

Ack and resolve

POST /v1/alerts/{id}/ack and POST /v1/alerts/{id}/resolve. Both return the alert as it now stands. Acking an alert that is already acked, or resolving one already resolved, returns 200 with the unchanged alert; only an unknown id is a 404. The app uses exactly these routes, so anything it can do a script can do.

Both are written to the audit log with who did it.

Who gets paged

Every device of every member of the tenant gets a push for every alert event. An escalation rung that names schedules or accounts narrows that event’s audience to those people. alert.opened and alert.escalated are delivered as critical alerts to devices that allow them; every other event is an ordinary notification. Webhook, Slack and Telegram integrations receive the same events, filtered however each integration was configured.

Event	When
`alert.opened`	The rule fired and no matching alert was firing.
`alert.bumped`	Another line folded into a firing alert.
`alert.escalated`	A ladder rung’s `after_minutes` passed without an ack, or a renotify fired.
`alert.acked`	Someone acked.
`alert.resolved`	Someone resolved, or quiet time ran out.

A muted rule or a maintenance window holds all of these back: the alert still opens and shows in the feed, and pages when the mute or window ends if it is still firing and unacked.

Secret warnings

When a source sends something shaped like a secret, Pingtower opens an alert for the finding: kind secret_warning, source naming the source, level 300, and the rule name secret-warning. Its message names the source and the kind of secret, never the value. Its identifiers carry the finding’s finding_id and pattern, so the apps link the alert to the finding.

It travels the same path as any alert: push, integrations, the feed, ack and resolve. It belongs to no rule, so it never escalates or renotifies. It resolves on its own when the finding is marked safe or its value is blocked. One finding has at most one such alert open at a time.