# Incidents

Source: https://www.pingtower.com/docs/service/incidents/

An incident is the team's record of one outage. It collects the alerts the outage raised and the notes people wrote while working it, and it closes with what caused it and what fixed it. It stays internal until someone publishes an update.

An alert says one rule fired. An incident says what was actually wrong. Many alerts attach to one incident. People, and AI assistants if you let them, write on its timeline next to the events Pingtower records itself. It closes with a cause and a fix, not just "resolved". Months after its alerts have been pruned, it still says which alerts fired, when, and why.

Incidents belong to the team, not to a project, because one outage often spans several. An incident can have a home project, and the [projects](/docs/getting-started/concepts/#per-project-roles) of its alerts decide who can see it. Deleting a project keeps its incidents: they become team-wide, with their history and their share links.

In the [web app](https://app.pingtower.com) it is **Incidents**, which lists them newest first, filters them by state, and has a **Needs a cause** list. In the iOS app it is Account → Incidents.

## Numbers, states and stages

Each incident gets the team's next number, written `INC-42`. A number is never handed out again, even after its incident is deleted.

| State | |
| --- | --- |
| `triage` | Suggested by an AI assistant and not yet accepted. Only an AI-opened incident starts here. It pages nobody, cannot be published, and opens no issues. A person **accepts** it, which makes it open, or **declines** it. |
| `open` | Being worked. |
| `resolved` | Closed, with a close reason: `resolved`, `declined` (not an incident) or `duplicate` (of another incident). A resolved incident can be **reopened**. |

The **stage** is what the outside world is told: `investigating`, `identified`, `monitoring` or `resolved`. It is set on a public update, and the newest public update with a stage is what [status pages](/docs/service/status-pages/) show. A public update with stage `resolved`, or of type `resolution`, also resolves an open incident, with no cause recorded.

An incident also has:

- a **title** of up to 256 bytes;
- an internal **description** of up to 4096 bytes, which is never published;
- an optional **severity**, `sev1`, `sev2` or `sev3`, which describes the incident and does not change how its alerts page;
- an optional **lead**, a member of the team, who is the person running it;
- a **start**, the opening time of its earliest attached alert, or the time it was opened if it has none.

Severity and lead can be changed or cleared at any time, and each change is recorded on the timeline.

## Opening one

- **From an alert.** On an alert in the web app, **Open incident** opens one with the alert attached. **Add to incident** adds the alert to an open incident instead.
- **By hand.** Give a title and, optionally, a description, severity, lead and home project. A member limited to some projects opens incidents in a project they hold.
- **By resolving an alert with its cause.** See below. This keeps a record of a one-alert outage in one step.
- **By an AI assistant.** It opens the incident in triage. See [AI assistants](#ai-assistants).
- **Through the older project report routes.** These open a numbered incident too. See [Reports](/docs/service/reports/).

Resolving an alert can carry what caused it and what fixed it, each up to 2000 bytes:

<span class="pill post">POST</span> `/v1/alerts/{id}/resolve`

```json
{ "cause": "Gateway TLS certificate rotated without the intermediate", "fix": "Redeployed the full chain" }
```

If the alert is on no incident, this opens a resolved incident with the alert attached and the cause and fix recorded. The incident is titled after the rule and when the alert opened, never after its log line. If the alert is already on an incident, the cause and fix are added to that incident as an internal note. With no body, the resolve works as it always has. In the web app, resolving an alert offers **What caused it?** for this.

## Alerts on an incident

Attach alerts up to 50 at a time, and up to 500 per incident. An alert belongs to at most one incident at a time. Attaching one that is already on another incident is refused with a `409` naming that incident, unless you say to move it. A move is recorded on both incidents' timelines. Detaching an alert takes it off. A [project-limited member](/docs/getting-started/concepts/#per-project-roles) needs `incidents:write` on the alert's own project to attach, move or detach it, and on every project of the incident a moved alert comes off. When a person attaches an alert that opened before the incident's start, the start moves back to it. An attach by an AI never moves the start.

Each attached alert keeps a snapshot of its project, rule, message, and when it opened and resolved. Resolved alerts are pruned after 30 days by default, but the snapshot stays, so the incident still shows "api-health, 03:12 to 03:40" long after the alert itself is gone.

An alert carries the incident it is on wherever alerts are read: in [`GET /v1/pull`](/docs/service/alerts/) and in the answers to ack and resolve.

```json
"incident": { "number": 42, "reference": "INC-42", "title": "Checkout errors", "state": "open" }
```

## Grouping

An open incident takes in new alerts from the rules already on it, so one failure stays one incident rather than a pile of alerts to attach by hand.

- **What joins.** A newly opened alert joins an open incident that already holds an alert from the same project and rule. For a check, that is the check's alerting rule.
- **For how long.** The window runs the team's grouping window past the incident's latest alert activity, meaning the later of an alert's opening and its attachment. Each alert that joins extends it. The window never runs past 24 hours after the incident's first alert.
- **The window.** 30 minutes by default. An owner or admin, or any role with `incident_settings:write`, sets it from 1 to 1440 minutes under Settings → Incidents in the web app.
- **Which incident.** When several could take the alert, the one with the most recent activity does, and a tie goes to the higher number. An incident at 500 alerts takes no more.
- **It still pages.** A grouped alert notifies and escalates exactly as any other alert. Grouping files it; it does not quiet it.
- **Off per incident.** Grouping is on for every new incident and can be turned off on any one of them. Triage and resolved incidents never take in alerts.

A grouped alert shows on the timeline as joined by the system, not by a person.

## The timeline

The timeline merges three things by time: what people and assistants wrote, what was done to the incident, and what its alerts did.

**Written entries** are Markdown (CommonMark, plus GitHub tables, strikethrough, autolinks and task lists; lists and quotes nested more than 16 deep on one line stay plain text), up to 64 KiB, with an optional title of up to 256 bytes. A table wider than 64 columns, or one that would take an entry past 10,000 table cells, shows as plain text, and so do link, image, emphasis, strikethrough and code-span markers past a per-paragraph budget that only very long, marker-packed paragraphs reach. An entry whose rendered HTML would pass 1 MiB is refused with `400` on `content_md` (rule `rendered_size`) and is not stored. Each has a type (`update`, `analysis` or `resolution`) and an optional stage. Each is labelled with whether a human or an AI wrote it.

- **Internal by default.** An entry stays inside the team unless someone makes it public. A public entry, the "public update", is the only text that ever leaves the team.
- **Backdating.** An entry's time defaults to when it was written and can be set to when the thing actually happened, up to 400 days back and never in the future.
- **Public updates are for members and above.** A responder writes, edits and deletes internal notes, but cannot write a public update or change or delete one. For a member [limited to some projects](/docs/getting-started/concepts/#per-project-roles), the highest grant role among the incident's projects decides.
- **Mentions.** Writing `INC-40` in an entry records a reference on INC-40's timeline that points back.

**Recorded events** are written by Pingtower and never edited:

- opened, accepted, resolved (with the reason, and whether a cause and fix were recorded) and reopened;
- edits to the title, description, severity, lead, cause, fix or grouping;
- alerts attached, detached, moved and grouped;
- marked as a duplicate, on the incident it duplicates;
- every alert resolved;
- published, and the share link replaced;
- an AI's drafted cause and fix;
- issues opened, linked, failed, closed and reopened, and pull or merge requests merged (see [GitHub and GitLab issues](#github-and-gitlab-issues)).

**Alert events** come from the attached alerts themselves: each one's opening, acknowledgement and resolution. Escalations are not on the timeline.

## When every alert resolves

An incident does not resolve itself. An alert is a symptom, and only a person knows the outage is over. When every alert on an open incident has resolved, the timeline records it once. If the incident has a lead who can still see its project, the lead gets one push notification saying so. Nobody else is notified. A triage incident, or one with no lead, gets the timeline event only. Attaching another alert resets this, so the event and the push can come again once that alert resolves too.

## Resolving

Resolving asks for what caused the incident and what fixed it, up to 2000 bytes each. Both are optional. An incident resolved as `resolved` without both is marked **cause not recorded** and listed under **Needs a cause**. Either can be added later, on a resolved incident too, without reopening it.

| Reason | |
| --- | --- |
| `resolved` | It was an incident and it is over. |
| `declined` | It was not an incident. This is also how a triage suggestion is turned down. |
| `duplicate` | Name the incident it duplicates. Its alerts move to that incident, and both timelines record it. |

If an AI assistant drafted a cause and fix, the web app's resolve dialog offers them for a person to accept or change. Resolving ends the incident's phases on status pages. A declined or duplicate incident is removed from them entirely, because it was never an outage worth showing. Reopening keeps the cause and fix already recorded.

There is no way to delete a team-wide incident: decline it instead. An incident with a home project can be deleted through the project's [report routes](/docs/service/reports/#deleting-one).

## Publishing

An incident is internal. It reaches the outside world in two ways, and both show only its public updates, never internal notes, the description, the cause and fix, the alerts or the people who worked on it. Publishing needs a member or above, or any role with `incident_publishing:write`. Turning either kind on is refused (`409`) until the incident has at least one public update, and while it is in triage.

**Share link.** A page at `/public/reports/{uuid}` on the service's host shows the incident's title and its public updates. The link is off for an incident opened in the web app, the iOS app or by an assistant, until someone turns it on.

- **Password.** A view password of 8 to 72 bytes gates the page. Wrong passwords are [rate limited](/docs/reference/limits/#routes-that-take-a-password). Setting a password also takes the incident off every status page.
- **Rotate.** Replacing the link mints a new address. While the link is on, the old one answers 404 at once.
- **Off.** Turning the link off makes it answer 404.

**Status pages.** Turn the status page on and name the affected components, at most 50, each with an impact: `degraded`, `partial` or `major`. The incident shows on every page that holds one of those components, with its stage and public updates, and each named component shows the impact from when the incident was opened until it resolves. [Status pages](/docs/service/status-pages/#incidents-on-the-page) has the details.

## GitHub and GitLab issues

An incident is where the outage is handled. An issue is where the engineering follow-up lives. Linking the two lets "what fixed it" point at real code.

### Trackers

A tracker is a repository and a token that may write issues there. Trackers belong to the team. Owners and admins, or any role with `incident_settings:write`, manage them in the web app under Integrations → Issue trackers. A team holds at most 10.

| Field | |
| --- | --- |
| Provider | `github` or `gitlab`. |
| Repository | GitHub's `owner/name`, or GitLab's project path. A leading or trailing `/` is dropped. A path with an empty, `.` or `..` segment is a `400` `repo is a repository path: no segment may be empty, "." or ".."`. |
| Token | For GitHub, a fine-grained personal access token with Issues read and write on that repository. For GitLab, a project or group access token with the `api` scope. It is sealed at rest and never shown again. |
| Base URL | Optional, for GitHub Enterprise Server or a self-managed GitLab: its `https` API base. Empty means github.com or gitlab.com. |
| Labels | Up to 10, each up to 50 characters, put on every issue it opens. |
| Trigger | **On open**: open an issue for every incident that opens, optionally only at or above a severity. **Manual**: only when someone asks on an incident. |
| Close on resolve | Close the issue when the incident resolves. Off, the issue gets the cause and fix and stays open for the follow-up work. |

Saving checks the token by reading the repository. If the repository is public, saving is refused until you confirm it is. A public repository's issues then carry only the title and a link back. Calls to the provider go out through the same guard as [webhooks](/docs/service/integrations/), so a tracker cannot be pointed at a private network address.

Editing a tracker keeps its stored token unless you change the provider, the base URL or the repository in any way, even only the letter case of the repository. Those changes need the token again, because the stored one is never sent to a new address or repository. Without it, the save is a `400` `token must be re-supplied when the provider, the base_url or the repo changes; the stored value is sealed and is not sent to a new destination`, and the token stays tied to the old repository.

### What an issue says

The title is `INC-42: <incident title>`. In a private repository, the body gives the reference, severity and start time, a link to the incident in the web app, and each attached alert's rule, project and opening time. It never includes internal notes or log lines, because a repository has its own audience. In a public repository, the body is only a link back.

- **Opening.** A tracker set to **On open** opens its issue when an incident opens, or when a triage suggestion is accepted. A severity minimum leaves out incidents below it, and incidents with no severity. **Create issue** on an incident opens one in any tracker, manual ones included. Each tracker opens at most one issue per incident.
- **Already resolved.** The incident opened by [resolving an alert with its cause](#opening-one) opens no issue: it is a record, already resolved. If an incident resolves before its issue is opened, a tracker with **Close on resolve** opens none. One without it still opens the issue for the follow-up work, with the cause and fix in a private repository. **Create issue** on a resolved incident works the same way: a tracker with **Close on resolve** refuses it with a `409` `the incident is resolved and this tracker closes issues on resolve`. A declined or duplicate incident gets no issue from any tracker, and **Create issue** on one is a `409` `the incident was declined or closed as a duplicate: it opens no issues`.
- **Resolving** comments on the issue. In a private repository the comment carries the cause and fix. With **Close on resolve**, it also closes the issue as completed.
- **Declining**, or marking a duplicate, comments and closes the issue as not planned. GitLab has no close reason, so there it just closes.
- **Reopening** the incident reopens the issue.
- **Public updates are not copied** to the issue. The issue is not a status page.

None of this waits on the provider. Each call is queued and made in the background, normally within 30 seconds. A call that fails is tried 8 times in all, with waits that double from 1 minute to 64 minutes, about two hours altogether. If every attempt fails, the failure shows on the incident's timeline and on the tracker, and **Retry** on the incident queues it again.

### Linking issues and pull requests

Paste a GitHub issue or pull request URL from github.com, or a GitLab issue or merge request URL from gitlab.com or from the host of a self-managed GitLab your team has connected as a tracker, to link it to an incident. A GitLab link on any other host is refused with `400` on `url` (rule `host`). A GitHub Enterprise link is refused with `400` on `url` (rule `format`). Each link is a timeline event. Links you paste can be removed. The issues a tracker opened stay as history.

### The tracker's webhook

With a webhook, the issue's side comes back into the incident's timeline. Set it up from the tracker in the web app, which gives the URL, the steps for your provider, and a webhook secret. The secret is shown once, when it is made. Generating a new one stops the old one at once. On pingtower.com the URL is

```text
https://api.pingtower.com/v1/issue-webhooks/{tenant}/{tracker}
```

and on a self-hosted box it is the same path on the box's address.

| | GitHub | GitLab |
| --- | --- | --- |
| Secret | **Secret**, checked as `X-Hub-Signature-256` | **Secret token**, checked as `X-Gitlab-Token` |
| Content type | `application/json` (not GitHub's form default) | JSON |
| Events | Issues, and Pull requests | Issues events, and Merge request events |

Three things are recorded on the incidents that link the issue:

- **an issue closed**, with GitHub's reason (completed, not planned or duplicate);
- **an issue reopened**;
- **a pull or merge request merged**, when it is linked itself, or when its title or description names a linked issue as `#12`, `owner/name#12` or the issue's URL. <!-- public-ok -->

Webhook events only add to the timeline. Closing the issue does not resolve the incident. Comments are not imported, because they are the repository's conversation.

A delivery that fails its signature check gets a `401` and writes nothing. A delivery about another repository, or about an issue no incident links, gets a `200` and writes nothing, so the provider does not mark the hook as broken. A repeated delivery is counted once. If Pingtower cannot store a delivery, it answers with an error, so the provider's own retry is accepted rather than dropped. A tracker made before webhooks existed has no secret, and its webhook refuses every delivery until one is generated.

## AI assistants

An assistant connected over [MCP](/docs/service/ai-assistants/) can read incidents. With the incidents permission and the team's switch both on, it can also suggest an incident, which opens in triage. It can attach alerts that are on no incident, add internal notes labelled as AI, draft a cause and fix, and link an issue. It never resolves, declines, accepts, publishes, shares, or writes anything public. [AI assistants](/docs/service/ai-assistants/#what-it-can-change) has the details.

## Who can do what

| Role | Can |
| --- | --- |
| Viewer | Read incidents, their timelines, the grouping window and the trackers. |
| Responder | Everything a viewer can, and open and edit incidents, attach and detach alerts, write internal notes, accept, resolve and reopen, create issues and link issues and pull requests. |
| Member | Everything a responder can, and write public updates, publish to status pages, and manage the share link and its password. |
| Owner, admin | Everything, including the grouping window and issue trackers. Deleting an incident is theirs alone, through the [report routes](/docs/service/reports/#deleting-one). |

A [custom role](/docs/getting-started/concepts/#custom-roles) gets these through `incidents:write`, `incident_publishing:write` and `incident_settings:write`.

## Limits

| | |
| --- | --- |
| Team-wide incidents per team | 5000. An incident with a home project counts toward that project's 200 reports instead. |
| Alerts per incident | 500 |
| Alerts per attach | 50 |
| Entries per incident | 1000 |
| Entry | 64 KiB of Markdown, rendering to at most 1 MiB of HTML |
| Cause, fix | 2000 bytes each |
| Status page components per incident | 50 |
| Issue trackers per team | 10 |
| Grouping window | 1 to 1440 minutes, never more than 24 hours after the first alert |