Troubleshooting

Lines arrive but no alert opens

Check the ingest response: "alerted": false means no rule matched. The usual causes, in order:

1. The rule’s min_level is above the line’s level. min_level: 400 ignores a 300.
2. match.sources names a source and the line came from another.
3. template_contains is matched against the template, not the message. If the substring you wrote contains a value that is in keys, it has been replaced with {{ .name }} in the template.
4. fire_after is waiting for more lines.

GET /v1/projects/{id}/rules/{name} shows the rule as stored.

The alert is there but the phone did not ring

• The app is signed into the right tenant. The tenant switcher is under Account.
• Notifications are allowed for Pingtower in iOS Settings.
• The first notification is ordinary; only an escalation is critical. If Silent mode swallowed it, see Critical alerts.
• On a paired box, the relay integration’s delivery log shows whether the page left the box: GET /v1/projects/{id}/integrations/{iid}/deliveries.

Slack or the webhook stopped receiving

GET /v1/projects/{id}/integrations/{iid}/deliveries shows the last error. After ten consecutive failures the integration is disabled; fix the far end, then POST …/{iid}/enable. Test with POST …/{iid}/test and read the response the far end gave.

401 on ingest

The token is not a pti_ token, or belongs to a rotated source. Rotation invalidates the old token immediately. Mint a new one with rotate-token and redeploy.

429

rate limit exceeded: back off for Retry-After seconds. storage quota exceeded: the tenant is at its retained-rows cap; alerting still works, history does not. See Rate limits.

409 on delete

Something still references it. The body lists the rules, or the schedules, escalations and overrides for a member. Edit those first. See Errors.

The addons agent starts and then nothing shows up

journalctl -u addons -f
addons agent --config /etc/addons/agent.yaml --check
addons check httpcheck --param url=https://example.com --report

The first shows send errors. The second validates the config without running it. The third proves the token and the tower URL by sending one event and printing whether a rule fired. A 401 means the token in /etc/addons/agent.env is wrong; a connection error means agent.tower.url is wrong or the box has no outbound route.

The agent is dead and nobody noticed

Nothing in the service alerts on silence. Give the agent’s heartbeat a rule from a second vantage point, or run two agents with distinct_sources so one going quiet is visible as the other’s lone report.

Self-hosted: /v1/pull returns 504

The reverse proxy’s read timeout is shorter than the poll’s wait. Raise it above 30 seconds. See Reverse proxy & TLS.