Addons — the probe agent

Watch from outside. And now from inside.

A single Go binary. On a box beyond your infrastructure it runs HTTP, TCP, DNS, certificate-expiry, and ping checks on a schedule — so an outage your app can't log is still an alert that finds a human. And new in 0.2.0, the same agent on the box itself runs continuous kernel-level sensors: a firewall change, a burst of failed logins, an undeclared listener become events the moment they happen, not a minute later.

Download Run one check

External monitoring — the addons

Eleven scheduled checks. One agent.

httpcheck

DNS, connect, TLS, and first-byte timed separately — "the port is slow" and "your resolver is slow" wake different people. Asserts status and body.

tcpcheck

Is the port answering, and how fast? Connect and DNS timings for anything that speaks a socket.

dnscheck

Resolution time and answers, checked against what the record is supposed to say. Catches hijacks and expired zones.

tlscert

Reports days_until_expiry on every run, so the warning threshold lives in a tower rule. An untrusted certificate is reported down — with issuer, subject, and expiry as the diagnosis.

icmpping

Round-trip min/avg/max and packet loss. "We lack permission" and "the host is unreachable" are opposite conclusions — it never confuses them.

sshcheck opt-in

Log in, run a command, assert its exit code and output — with connect, handshake, and auth timed separately.

portscan opt-in

Scans against a baseline: a port that opened without being declared can page you before someone else finds it.

traceroute opt-in

Hop count, per-hop RTTs, and a path_change signal when the route to your box quietly rewires itself.

logwatch opt-in

Tails a host log for a pattern, rotation-aware — for the one machine whose software will never POST to a tower itself.

securityupdates opt-in

Pending updates and security updates from apt, plus reboot_required and the age of the package metadata — a stale mirror and an unpatched box are both a rule away from paging.

hostmetrics

CPU time, load averages, memory and swap gauges, paging and fault rates — read from /proc, no privileges needed. Thresholds live in tower rules, where they can match the workload.

The five marked opt-in need a credential, an external binary, or host file access — each is off until the agent's own config lists it in enable_addons, and a config naming one without that fails at startup instead of silently running.

Internal monitoring — the sensors

Seven continuous host sensors. New in 0.2.0.

Checks run on a schedule; sensors never stop. Installed on the box you're watching, the agent subscribes to the kernel and turns what it sees into the same events your rules already threshold — Linux-only, off until agent.enable_sensors names one, and built privacy-first: payloads, file contents, usernames, and exact remote addresses never enter an event.

nftwatch

Subscribes to the host's nftables ruleset and reports the moment it changes — a rule added at 3 a.m. is an alert, not a surprise in next week's audit. Sends only ruleset hashes and object counts; your rules themselves never leave the box.

network

Sees every inbound TCP connection attempt with a pass-only eBPF/XDP program — it can observe, never drop or rewrite. Aggregated by port and redacted remote prefix before anything is sent.

authwatch

Reads SSH and sudo authentication events from the system journal, strictly matched against the distribution's exact message forms. Usernames and raw messages are never sent; remote addresses are reduced to a /24 or /64 before they leave the box.

filewatch

Watches an explicit set of files and shallow directories — and because every inotify wake-up settles into a full reconciliation, an atomic editor replace is a confirmed change, not a missed one. Events carry type, owner, mode, and SHA-256 — never contents.

listeningports

The kernel's own socket inventory against an explicit baseline: a listener that appeared without being declared is an event, so is one that vanished. Needs no added capability; exact addresses and PIDs never enter events.

servicewatch

An explicit list of systemd units, watched for failures, recoveries, restart loops, changed unit files, and a service whose command quietly became a different binary. No journal text, command lines, or unit-file contents are ever sent.

pressure

Kernel PSI triggers for CPU, memory, and I/O stall: the kernel itself wakes the sensor only when workloads were actually stalled past your threshold — one event per crossing, one per recovery, no polling.

Each sensor takes only what it needs: several run under the agent's stock unit with an empty capability set, and the kernel-attached ones get one narrow capability via a local systemd drop-in (nftwatch CAP_NET_ADMIN; network CAP_BPF + CAP_NET_ADMIN) — the release's own VM suite proves each is necessary by removing them one at a time.

No new concepts

Measurements are just events. Rules do the rest.

Every metric travels as a JSON number on a normal ingest event. Thresholds live in your tower rules — not in agent config — so changing "page me at 500 ms" never means touching a probe box.

  • Down at level 400 — critical if it should page
  • Flapping collapses into one alert with a count
  • Quiet when healthy, spools through outages
  • Same against hosted and self-hosted towers
a working latency alert
curl -s $PT/v1/projects/probes/rules \
  -H "Authorization: Bearer $API_KEY" \
  -d '{
    "name": "slow-ttfb",
    "match": { "tags": ["addon:httpcheck"] },
    "conditions": [
      { "key": "ttfb_ms", "op": "gte", "value": 500 }
    ]
  }'

Download

Version 0.3.1. Releases are signed with the addons release key (addons-release.pub — signatures live in their own pingtower-addons-release namespace, so an addons signature can never pass as a daemon release or vice versa): one signed SHA256SUMS (.sig) covers every tarball.

ArtifactArchSizeSHA-256
addons_0.3.1_linux_amd64.tar.gzamd644.6 MBb3560963a7c0f46bc7ebe0ad832009ed508eca024a56813fddcb1ab704caf474
addons_0.3.1_linux_arm64.tar.gzarm644.1 MBa41c882003ab34e35126a7249fb4a5a4a40c9fa555aa63dbe935b5e56d9f2b72

Verify before unpacking — the first command checks the sums file was signed by this project, the second checks your tarball matches it. Skipping either lets a tampered tarball and a tampered checksum file agree with each other:

printf '[email protected] namespaces="pingtower-addons-release" %s\n' \
  "$(cut -d' ' -f1,2 addons-release.pub)" > allowed_signers
ssh-keygen -Y verify -f allowed_signers -I [email protected] \
  -n pingtower-addons-release -s SHA256SUMS.sig < SHA256SUMS
sha256sum --ignore-missing -c SHA256SUMS

Install

tar xzf addons_<version>_linux_<arch>.tar.gz
cd addons_<version>_linux_<arch>
sudo ./install.sh

The installer creates a dedicated addons system user, installs the binary to /usr/local/bin/addons, and sets up /etc/addons and a systemd unit. Each tarball's contents are covered by a signed manifest that install.sh verifies before it runs anything as root.

Run one check first

addons check runs a single check once and prints the event it would send. Add --report to send it — the ingest token comes from the environment, never a flag, so it stays out of shell history:

export PINGTOWER_INGEST_TOKEN=pti_<tenant>_…
export PINGTOWER_URL=https://api.pingtower.com   # or your own box
addons check httpcheck --param url=https://example.com --report

Then let addons agent run a whole config on a schedule — spooling through outages, heartbeating on itself:

# /etc/addons/agent.yaml
agent:
  id: probe-fra-01
  heartbeat: 5m
  tower:
    url: https://api.pingtower.com
    token_env: PINGTOWER_INGEST_TOKEN

defaults:
  interval: 60s
  timeout: 10s
  report: on_change

checks:
  - id: site
    addon: httpcheck
    params: { url: "https://example.com", expect_status: 200 }
  - id: cert
    addon: tlscert
    params: { address: "example.com:443" }

The agent-readable version of this page lives at /addons/index.md — hand it to your AI agent along with the setup guide and it can do all of the above for you.