Addons — the probe agent

Watch from outside. And now from inside.

A single Go binary. On a box beyond your infrastructure it runs checks on a schedule, so an outage your app can't log is still an alert that finds a human. And new in 0.2.0, the same agent on the box itself runs continuous kernel-level sensors.

Download Run one check

1.0External monitoring — the addons

Nineteen scheduled checks. One agent.

On a box beyond your infrastructure, the agent runs HTTP, TCP, DNS, certificate-expiry and ping checks on a schedule — and from agent 0.4.0, multi-step API, WebSocket, UDP, mail-server and domain-expiry checks, with gRPC from 0.5.0.

Addons docs →

httpcheck

DNS, connect, TLS, and first-byte timed separately — "the port is slow" and "your resolver is slow" wake different people. Asserts status and body — and from agent 0.4.0, up to 20 assertions on status, headers, JSONPath, timing and a body hash, signing in with basic, bearer, mTLS or OAuth.

synthetic agent 0.4.0+

An API journey in up to ten requests: log in, carry the token into the next call, assert at every step. A failure names the step and the assertion that broke.

grpccheck agent 0.5.0+

The standard grpc.health.v1 health call, with connect, TLS, and the call timed apart — not_serving and "can't connect" are different alerts.

websocket agent 0.4.0+

Open the socket, optionally send one message, and check the first reply. The upgrade and the reply are timed separately.

domainexpiry agent 0.4.0+

Reads the registry's RDAP record (WHOIS for the few registries without one) and warns 30 days before your domain lapses — once a day is plenty.

udpcheck agent 0.4.0+

Send a datagram, wait for the reply, retry when one is lost — for DNS servers, game servers, and anything else that won't speak TCP.

smtpcheck · imapcheck · popcheck agent 0.4.0+

The greeting, STARTTLS or implicit TLS, a warning 21 days before the certificate runs out, and an optional login over TLS.

tcpcheck

Is the port answering, and how fast? Connect and DNS timings for anything that speaks a socket.

dnscheck

Resolution time and answers, checked against what the record is supposed to say. Catches hijacks and expired zones.

tlscert

Reports days_until_expiry on every run, so the warning threshold lives in a tower rule. An untrusted certificate is reported down — with issuer, subject, and expiry as the diagnosis.

icmpping

Round-trip min/avg/max and packet loss. "We lack permission" and "the host is unreachable" are opposite conclusions — it never confuses them.

sshcheck opt-in

Log in, run a command, assert its exit code and output — with connect, handshake, and auth timed separately.

portscan opt-in

Scans against a baseline: a port that opened without being declared can page you before someone else finds it.

traceroute opt-in

Hop count, per-hop RTTs, and a path_change signal when the route to your box quietly rewires itself.

logwatch opt-in

Tails a host log for a pattern, rotation-aware — for the one machine whose software will never POST to a tower itself.

securityupdates opt-in

Pending updates and security updates from apt, plus reboot_required and the age of the package metadata — a stale mirror and an unpatched box are both a rule away from paging.

hostmetrics

CPU time, load averages, memory and swap gauges, paging and fault rates — read from /proc, no privileges needed. Thresholds live in tower rules, where they can match the workload.

The checks marked with an agent version, such as agent 0.4.0+, need that addons agent or later; an older agent reports them as unknown addons. gRPC checks need 0.5.0: on 0.4.0 a run can hang. The download below is agent 0.5.0. On pingtower.com, synthetic, grpccheck, websocket and domainexpiry already run as hosted checks from Pingtower's own servers. udpcheck and the mail checks run only on your own agents.

The five marked opt-in need a credential, an external binary, or host file access — each is off until the agent's own config lists it in enable_addons, and a config naming one without that fails at startup instead of silently running. sshcheck and logwatch go further: they run only from that local config, and the agent refuses one assigned from the tower.

2.0Internal monitoring — the sensors

Seven continuous host sensors. New in 0.2.0.

Checks run on a schedule; sensors never stop. Installed on the box you're watching, the agent subscribes to the kernel and turns what it sees into the same events your rules already threshold — Linux-only, off until agent.enable_sensors names one, and built privacy-first: payloads, file contents, usernames, and exact remote addresses never enter an event.

Sensors →

nftwatch

Subscribes to the host's nftables ruleset and reports the moment it changes — a rule added at 3 a.m. is an alert, not a surprise in next week's audit. Sends only ruleset hashes and object counts; your rules themselves never leave the box.

network

Sees every inbound TCP connection attempt with a pass-only eBPF/XDP program — it can observe, never drop or rewrite. Aggregated by port and redacted remote prefix before anything is sent.

authwatch

Reads SSH and sudo authentication events from the system journal, strictly matched against the distribution's exact message forms. Usernames and raw messages are never sent; remote addresses are reduced to a /24 or /64 before they leave the box.

filewatch

Watches an explicit set of files and shallow directories — and because every inotify wake-up settles into a full reconciliation, an atomic editor replace is a confirmed change, not a missed one. Events carry type, owner, mode, and SHA-256 — never contents.

listeningports

The kernel's own socket inventory against an explicit baseline: a listener that appeared without being declared is an event, so is one that vanished. Needs no added capability; exact addresses and PIDs never enter events.

servicewatch

An explicit list of systemd units, watched for failures, recoveries, restart loops, changed unit files, and a service whose command quietly became a different binary. No journal text, command lines, or unit-file contents are ever sent.

pressure

Kernel PSI triggers for CPU, memory, and I/O stall: the kernel itself wakes the sensor only when workloads were actually stalled past your threshold — one event per crossing, one per recovery, no polling.

Each sensor takes only what it needs: several run under the agent's stock unit with an empty capability set, and the kernel-attached ones get one narrow capability via a local systemd drop-in (nftwatch CAP_NET_ADMIN; network CAP_BPF + CAP_NET_ADMIN) — the release's own VM suite proves each is necessary by removing them one at a time.

3.0No new concepts

Measurements are just events. Rules do the rest.

Every metric travels as a JSON number on a normal ingest event. Thresholds live in your tower rules — not in agent config — so changing "page me at 500 ms" never means touching a probe box.

Rules →

A working latency alert

Match the addon's events by tag, put the condition on the measurement, and the same rule engine — dedup, ladders, on-call — takes it from there.

Behaves like everything else

Down at level 400 — critical if it should page. Flapping collapses into one alert with a count; quiet when healthy, spooling through outages, the same against hosted and self-hosted towers.

Download

Version 0.5.0. Releases are signed with the addons release key (addons-release.pub — signatures live in their own pingtower-addons-release namespace, so an addons signature can never pass as a daemon release or vice versa): one signed SHA256SUMS (.sig) covers every tarball.

ArtifactArchSizeSHA-256
addons_0.5.0_linux_amd64.tar.gzamd645.2 MB2c13deb98c300cc9c618d72d80f35a5b6a6c70ce5a7724aaaeeaecd5b19b0f97
addons_0.5.0_linux_arm64.tar.gzarm644.6 MB29688707ba15bda96f0abd83dbb7fb4fdbf6cf95d50f4495b654167a29bd8dfe

Verify before unpacking. First check the key: this site serves the tarballs, the signed sums and the key, so a tampered site could swap them all. The key's fingerprint is also published in the DNS TXT record _release-key.pingtower.com, which is set apart from the site, so changing the site alone can't change it. That record is the authority: the ssh-keygen -lf and dig lines must both show SHA256:tOBYukuZW+6pJBQFDd2Hw/nWSHQa4UdLijNayRHzgrQ, and if the record is empty or differs, stop. Then check the sums file was signed with that key, and that your tarball matches it. Skipping either lets a tampered tarball and a tampered checksum file agree with each other:

ssh-keygen -lf addons-release.pub   # the key
dig +short TXT _release-key.pingtower.com   # must match; stop if not

printf '[email protected] namespaces="pingtower-addons-release" %s\n' \
  "$(cut -d' ' -f1,2 addons-release.pub)" > allowed_signers
ssh-keygen -Y verify -f allowed_signers -I [email protected] \
  -n pingtower-addons-release -s SHA256SUMS.sig < SHA256SUMS
sha256sum --ignore-missing -c SHA256SUMS

Install

tar xzf addons_<version>_linux_<arch>.tar.gz
cd addons_<version>_linux_<arch>
sudo ./install.sh

The installer creates a dedicated addons system user, installs the binary to /usr/local/bin/addons, and sets up /etc/addons and a systemd unit. Each tarball's contents are covered by a signed manifest that install.sh verifies before it runs anything as root.

Run one check first

addons check runs a single check once and prints the event it would send. Add --report to send it — the ingest token comes from the environment, never a flag, so it stays out of shell history:

export PINGTOWER_INGEST_TOKEN=pti_<tenant>_…
export PINGTOWER_URL=https://api.pingtower.com   # or your own box
addons check httpcheck --param url=https://example.com --report

Then let addons agent run a whole config on a schedule — spooling through outages, heartbeating on itself:

# /etc/addons/agent.yaml
agent:
  id: probe-fra-01
  heartbeat: 5m
  tower:
    url: https://api.pingtower.com
    token_env: PINGTOWER_INGEST_TOKEN

defaults:
  interval: 60s
  timeout: 10s
  report: on_change

checks:
  - id: site
    addon: httpcheck
    params: { url: "https://example.com", expect_status: 200 }
  - id: cert
    addon: tlscert
    params: { address: "example.com:443" }

The agent-readable version of this page lives at /addons/index.md — hand it to your AI agent along with the setup guide and it can do all of the above for you.