Go

Upgrading a self-hosted box to Release 1

The install guide says an upgrade is a binary swap and a restart, and that a release needing more says so on this blog. Release 1 needs more. This post is the whole list.

1. Take a backup first

Startup runs Release 1’s schema migrations, and they only go forward. Back up before you replace the binary:

  • SQLite: stop the pingtower unit and copy the data directory, or keep a recent backup that -backup-dir wrote.
  • PostgreSQL: back it up the way you back up any PostgreSQL server, for example with pg_dump.

See Storage for what lives where.

2. Migrations run once, forward only

When the new pingtower binary starts, it migrates the control database and every tenant’s storage. For a box coming from the previous release, the migrations add room for:

  • Per-project access: which projects a member, or the invite that brings them in, is limited to.
  • Custom roles.
  • Cleanup owed to a team that an account left while that team could not be reached.
  • A synthetic check’s last run, kept step by step.
  • Whether a browser check had sealed env when each run was recorded, so that run’s artifacts stay behind checks:write after the env is cleared.
  • The addons agent release each source last reported.
  • Who last pointed each integration and action at its destination, and a review flag on older ones.
  • The project behind an incident’s resolve cause, so a member limited to other projects doesn’t read it. Causes written before the upgrade record no project and stay visible to everyone who reads the incident.

One more migration changes no schema: it hands every project that holds a stored sshcheck or logwatch check a fresh assignment, so a running agent drops that check on its next poll (see step 4).

These migrations, the per-project access and custom role ones among them, run by themselves at startup. They add tables, columns and flags, and bump a version counter. They delete nothing and don’t change your checks, members or settings, and they need nothing from you beyond the backup in step 1. Adding is not the same as reversible, though, as the next paragraph explains.

There is no downgrade path other than restoring the backup you took in step 1. An older binary does not refuse a database it has never seen. It ignores the migrations it doesn’t know and starts. Once per-project access and custom roles are in place that is not safe: it gives a project-limited member their team role on every project, and it locks out members who hold a custom role. To go back, stop the unit, put the old binary back, and restore the backup.

3. Upgrade the addons agent

Upgrade every addons agent to 0.4.0 or later. It matters for two reasons: synthetic checks, and sshcheck and logwatch checks stored in the tower.

Synthetic checks

Synthetic checks, the multi-step API checks that carry values from one request into the next, need addons agent 0.4.0 or later.

An older agent doesn’t know the addon:

  • For a check the tower assigns it, it rejects the check as an unknown addon and never runs it. Its log shows assignment rejected with unknown addon "synthetic". The rest of its checks keep running.
  • With a synthetic check in its local agent.yaml, it fails validation with addon: unknown addon "synthetic" and does not start.

The tower warns when you save a synthetic check that names a source whose agent is older than 0.4.0, or reports no version (every release before 0.4.0 is one of those). The save still succeeds. The sources API reports each source’s agent_version, so you can see which boxes still need the new agent.

Stored sshcheck and logwatch checks

An agent of 0.4.0 or later refuses an sshcheck or logwatch check that the tower assigns to it, because their parameters name a command, a key or a file on the box. Its log shows assignment rejected. An older agent doesn’t know to refuse: it keeps running whatever command or path a stored check names, for as long as the tower hands the check out.

The Release 1 tower stops handing them out, whatever the agent’s version (step 4). Upgrade the agents anyway: from 0.4.0 on, the refusal is on the box itself, so it doesn’t depend on the tower.

Order

We recommend upgrading the tower first, then its agents. That order isn’t enforced. A new agent also works against an older tower. The tower turns away an agent that reports a version older than 0.4.0 with 426 client_outdated (Errors). Agents from before 0.4.0 report no version at all, so they are never turned away. They keep running every other check.

4. Move stored sshcheck and logwatch checks into the agent’s config

Before Release 1, a check of either addon could be stored in the tower and assigned to an agent. The Release 1 tower keeps every such check, with its configuration, but no longer assigns it to any agent, and answers a result for it with 404 unknown assignment. Until you move it, that check runs nowhere.

Find them. List each project’s checks (GET /v1/projects/{id}/checks, or the project’s checks in the web app) and look for addon sshcheck or logwatch. After the upgrade, the web app and the iOS, macOS and Android apps mark each one local-only, and reading one through the API shows it as paused:

"paused": {
  "reason": "local_only",
  "message": "local-only: the tower no longer assigns this check; move it into the agent's YAML (enable_addons + a checks entry) on the box that runs it, then delete it here",
  "resumes_at": 0
}

resumes_at is 0 because it never resumes by itself.

Move each one to the box that should run it:

  1. Copy the check’s addon, with, interval and timeout into a checks entry in /etc/addons/agent.yaml, with the check’s name as its id. See Configure.
  2. Add the addon to agent.enable_addons, for example enable_addons: [sshcheck].
  3. Validate with addons agent --config /etc/addons/agent.yaml --check, then restart the agent (systemctl restart addons).
  4. Once the agent reports the check, delete the stored copy in the tower.

The local check and the paused stored one stay separate: the agent’s reports never attach to the stored check. Alerting moves with it. The stored check’s own rule only heard results the tower assigned, so it goes quiet once the check is paused. Alert on the local check with a rule on its events, as for any check in the agent’s config.

5. Custom roles and per-project access

Release 1 adds custom roles and per-project grants. On pingtower.com, a team whose data plane is your own box can’t use them. pingtower.com answers every custom-role or grant write for that team with 409 membership is managed on the data plane. That team’s membership is managed on your box.

Your box’s own API serves both. An owner on the box can create custom roles there and limit members to projects, as on a hosted team.

6. Changes your team will notice

  • Adding an admin asks for a password. Making someone an owner, or giving them any role that holds members:write (admin or a custom role), by role change, invite or pairing code, needs the current password of whoever does it. See Members and roles.
  • Secrets are needed again when a destination changes. Any change to an action’s URL or method, a check’s URL, token URL, step URLs, method or mail server address, or an issue tracker’s base URL or provider, even only a path, needs its headers and body, env or token sent again.
  • Secrets are needed again when a check’s env would reach somewhere new. Adding a region or agent source to any check with sealed env is refused with a 422 unless env is sent again: env: sources now include <names> the env was not sealed for; re-supply env (or send {} to clear it). Removing or reordering sources keeps it. On a synthetic check, changing a plain variable that a secret step’s URL or token URL uses moves where that secret goes, so it is also refused with a 422 unless env is sent again: env NAME would reach a destination it was not sealed for; re-supply env (or send {} to clear it). A variable those URLs don’t use changes freely. Changing the URL or method of a step whose extraction such a URL uses, or of an earlier step in that chain, is refused the same way, and so is changing that step’s extraction rule, headers, body, authentication or follow_redirects.
  • Domain-expiry checks run at most hourly. A shorter interval is refused.
  • Some destinations ask for a review. Channels, webhooks and actions saved before this release show a review pill ("pointer_review": true) until a manager checks and saves them.
  • Maintenance windows are announced. Scheduling, changing or deleting one posts a maintenance.* notice to the project’s webhooks and chat integrations, as a rule mute does. See Maintenance notices.
  • Project-limited admins edit less. A member limited to projects, admin included, can no longer change issue trackers, the relay or status pages.

7. A -secret-key shorter than 16 characters stops the box

Release 1 refuses to start with a -secret-key (PINGTOWER_SECRET_KEY) shorter than 16 characters, and warns at startup when it is shorter than 32. Check yours before the upgrade. A short key can’t simply be swapped: a different key opens none of the secrets sealed under the old one, so every integration, action and signed webhook has to be re-created under the new key. See the flag’s entry.

#release#self-hosting

← All posts