Troubleshooting

Lines arrive but no alert opens

Check the ingest response: "alerted": false means no rule matched. The usual causes, in order:

1. The rule’s min_level is above the line’s level. min_level: 400 ignores a 300.
2. match.sources names a source and the line came from another.
3. template_contains is matched against the template, not the message. If the substring you wrote contains a value that is in keys, it has been replaced with {{ .name }} in the template.
4. fire_after is waiting for more lines.

GET /v1/projects/{id}/rules/{name} shows the rule as stored.

The alert is there but the phone did not ring

• The app is signed into the right tenant. The tenant switcher is under Account.
• Notifications are allowed for Pingtower in iOS Settings.
• Only an alert opening or escalating is critical, and only where the phone allows it. A bump, ack or resolve is ordinary, and Silent mode or a Focus can hold it. If the phone is signed into several accounts, one registration that declined critical alerts makes it ordinary everywhere. See Critical alerts.
• On a paired box, the relay integration’s delivery log shows whether the page left the box: GET /v1/projects/{id}/integrations/{iid}/deliveries. Account → Who gets paged flags a project whose relay row is missing, disabled or pointing elsewhere; PUT /v1/relay again fixes it.

A test notification arrived silently, or late

Expected when the phone’s ordinary notifications are quieted. Send test event always sends an ordinary notification, never a critical one, so Deliver Quietly, a Scheduled Summary or a Focus holds it like any other app’s. Real alert openings and escalations are the critical ones. That the test arrived at all is the proof it was sent for.

Slack or the webhook stopped receiving

GET /v1/projects/{id}/integrations/{iid}/deliveries shows the last error. After ten consecutive failures the integration is disabled; fix the far end, then POST …/{iid}/enable. Test with POST …/{iid}/test and read the response the far end gave.

refusing to connect to blocked address means the URL resolved to an address outbound calls may not reach: private, loopback, link-local, multicast, carrier-grade NAT, or a range that tunnels into one, such as NAT64 (64:ff9b::/96), 6to4 (2002::/16) and Teredo (2001::/32). Webhooks, runbook actions and the checks pingtower.com runs from its own probe boxes share the list; checks in your own agent’s config file do not. It applies on your own box too, so a webhook to a machine on your LAN needs a public address.

401 on ingest

The token is not a pti_ token, or belongs to a rotated source. Rotation invalidates the old token immediately. Mint a new one with rotate-token and redeploy.

429

rate limit exceeded: back off for Retry-After seconds. In the app, outside sign-in, it reads “Too many requests right now”; wait a minute and try again. storage quota exceeded: the tenant is at its retained-rows cap; alerting still works, history does not. See Rate limits.

409 on delete

Something still references it. The body lists the rules, or the schedules, escalations and overrides for a member. Edit those first. See Errors.

The addons agent starts and then nothing shows up

journalctl -u addons -f
addons agent --config /etc/addons/agent.yaml --check
addons check httpcheck --param url=https://example.com --report

The first shows send errors. The second validates the config without running it. The third proves the token and the tower URL by sending one event and printing whether a rule fired. A 401 means the token in /etc/addons/agent.env is wrong; a connection error means agent.tower.url is wrong or the box has no outbound route.

The agent is dead and nobody noticed

Add a rule with a source_silent trigger naming the agent’s source. When the agent stops reporting for longer than the horizon you set, the rule opens an alert and pages through its ladder like any other; the first line the agent sends again resolves it. See Sources that go quiet.

Self-hosted: the daemon will not start after a redeploy

FATAL: tenant … holds secrets sealed under a master key, but no --secret-key (PINGTOWER_SECRET_KEY) is set means the key did not reach the unit. Put back the exact key the data was sealed under; a new one opens nothing. See Useful flags.

FATAL: --secret-key (PINGTOWER_SECRET_KEY) is N characters; it must be at least 16 means the key is too short. If nothing was sealed under it yet, set a key of 32 or more random characters. If secrets were, re-create the integrations, actions and signed webhooks under the new key, because the old key’s secrets cannot be opened by a new one.

Self-hosted: /v1/pull returns 504

The reverse proxy’s read timeout is shorter than the poll’s wait. Raise it above 30 seconds. See Reverse proxy & TLS.