Go

Upgrading a self-hosted box to 1.2.0

The install guide says an upgrade is a binary swap and a restart, and that a release needing more says so on this blog. 1.2.0, coming from 1.1.0, needs more. This post is the whole list.

1. Take a backup first

Startup runs 1.2.0’s schema migrations, and they only go forward. Its first start with your secret key also reseals stored secrets in a form 1.1.0 can’t read (step 3). Back up before you replace the binary:

  • SQLite: stop the pingtower unit and copy the data directory, or keep a recent backup that -backup-dir wrote.
  • PostgreSQL: back it up the way you back up any PostgreSQL server, for example with pg_dump.

See Storage for what lives where.

2. Migrations run once, forward only

When the new pingtower binary starts, it migrates the control database and every tenant’s storage. For a box coming from 1.1.0, the migrations add room for:

  • A team’s IP allowlist, off until an owner adds a range. See IP allowlist.
  • The tools each connected AI app may use, per member. See Limit an app’s tools.
  • Whether each push registration is verified. A registration that never verified still gets pages as before.
  • Who added each member, and how, and the member reviews a password change raises. See Member reviews.
  • When each source’s agent token was minted, against when each check’s env was last saved. Checks and tokens from before the upgrade keep running as they did. See A new agent token withholds secrets.
  • A log-query rule’s settings.
  • A per-minute history of what each log-query rule counted, for the change and anomaly modes. It starts empty: those rules learn from the upgrade on.

These migrations run by themselves at startup. They add tables and columns and delete nothing, and they need nothing from you beyond the backup in step 1.

3. Your secret key reseals secrets: a binary-only downgrade breaks them

The first time 1.2.0 starts with your -secret-key (PINGTOWER_SECRET_KEY), it reseals every action’s headers and body and every typed channel’s config (Slack, Discord, Telegram and the other integrations that aren’t a plain webhook), each bound to its own record. It does this for each team the first time it opens that team’s data.

1.1.0 can’t read the resealed values. Putting the 1.1.0 binary back on a box that has run 1.2.0 breaks every action with headers or a body and every typed channel until you restore the backup you took in step 1. Their deliveries fail. To go back, stop the unit, put the old binary back, and restore the backup.

4. Keep the secret key set

A pingtower started without a -secret-key refuses to start when the data it serves holds sealed secrets. 1.2.0 also counts sealed issue-tracker tokens and webhook secrets, so a box that holds only those now refuses a keyless start too. It logs which team holds them and exits:

pingtower: FATAL: tenant <id> holds secrets sealed under a master key, but no --secret-key (PINGTOWER_SECRET_KEY) is set; every sealed tenant would refuse to open. Restart with the key this data was sealed under.

Start it with the key the data was sealed under. A key under 16 characters still stops the box, and under 32 logs a warning.

5. Move your agents to addons 0.6.0

Upgrade the tower to 1.2.0 first, then upgrade each addons agent to addons agent 0.6.0. The tower still accepts agents from 0.4.0 on, so nothing warns you about an older one: this note is the warning.

  • A 0.5.0 agent refuses a WebSocket check that carries env. Sealed headers on a WebSocket check need 0.6.0. A 0.5.0 agent rejects that check and runs the rest of its checks, so the check silently stops running from that agent. The agent’s heartbeat counts it in assignments_rejected.
  • 0.6.0 refuses a plain http:// tower.url whose host is not localhost or a loopback IP, and does not start with it. Before you upgrade, put TLS in front of the tower (Reverse proxy & TLS) and switch url to https://. To keep plain http://, add allow_insecure_http: true under agent.tower after installing 0.6.0, never before: 0.5.0 doesn’t know the key and refuses to start with it. See tower.url.
  • 0.6.0 never follows a redirect from the tower. A 3xx is an error naming its status and Location. If your agent reaches the tower through a redirect, point url at the final address.

6. New: moving between SQLite and PostgreSQL

1.2.0 adds pingtower move-data, which copies every database of a stopped install from SQLite to PostgreSQL or back, checks the copy, and prints the database line to set in config.yaml. The source is only read. Start 1.2.0 on the source once first, so it is on 1.2.0’s schema, then stop it. The target must be empty. See Moving between SQLite and PostgreSQL.

So that move-data can tell a stopped install from a running one, a running 1.2.0 holds the lock file pingtower.lock in its data directory. A second pingtower started on the same directory now refuses with another pingtower process is using this directory. A one-shot -backup and pingtower pair take no lock, so a cron’d backup of the running box still works. See Storage.

#release#self-hosting

← All posts