#12 - CD phase 2 — host deployer with AUTO_DEPLOY=off - Nectenda/nectenda

cruelacid commented

2026-09-15 08:59:23 +01:00

Owner

deploy/lib/deployer.sh (tick/apply/rollback/status/backup, --dry-run) and deployer-decide.sh; nectenda-deploy.service/timer; deploy.env per host; Kuma push monitor and a per-host manual maintenance (status page banner during the gap, incident on failure) toggled via a bundled socket.io client run with the image's Node; install-deployer.sh for existing hosts; cloud-init for new ones; docs. Checkpoint: hand apply on accounts then eu1 with gap logged, banner visible during and gone after, one rollback exercised on eu1, push monitor green.

cruelacid referenced this issue from a commit

2026-09-15 09:38:38 +01:00

Give every host a deployer that follows :stable on its own

cruelacid commented

2026-09-15 10:00:10 +01:00

Author

Owner

Done and verified live. Commits 23db287 (the deployer) and 69a8317 (the banner fix and the measured numbers).

What is on the hosts. deploy/lib/deployer.sh plus nectenda-deploy.timer on both accounts and eu1, ticking every 60 s, AUTO_DEPLOY=off as this phase specifies. Config in /srv/<app>/deploy.env, root-only 0600. Installed and upgraded by deploy/install-deployer.sh; new hosts get everything but the Kuma bundle from cloud-init with AUTO_DEPLOY=on.

Checkpoint. All four deploys run by hand, readiness probed through Caddy at 2/s throughout each one.

Deploy	Host	stop→ready	Readiness	Banner
`4446a33` → `23db287`	identity	6.3 s	no failure, slowest held 1.4 s	up 4 s, then gone
`4446a33` → `23db287`	eu1	7.3 s	no failure, slowest held 2.9 s	up 4 s, then gone
`23db287` → `4446a33` (`rollback`)	eu1	7.4 s	no failure	up, then gone
`4446a33` → `23db287` (back)	eu1	7.2 s	no failure	up, then gone

/api/health confirmed the version each way round, including the rollback. Both push monitors are green and report the running sha and :stable unresolved, which is correct until phase 3.

One real bug, found by doing this rather than by the tests. The first live deploy raised no banner and logged only "kuma maintenance on failed (not fatal)". The deployer was mounting the bundle directory over the container's /lib, which shadows the dynamic linker and libc, so node died before reading the bundle. Two things made it survive review: the output was sent to /dev/null, and the failure was non-fatal by design. Now mounted at /opt/nectenda-deploy, the client's output is logged, and a test asserts the bundle is never mounted at /lib.

Tests. deploy/ is a workspace package; 13 tests covering the decision truth table, the dry-run plan and its ordering, the .env pin, the rollback, the refusal to roll back after a migration, and the refusal to deploy without a backup. Five mutations each fail the suite: ignore AUTO_DEPLOY, ignore the identity gate, raise the banner after the restart instead of before, roll back after a migration, mount the bundle at /lib.

Also fixed along the way. The cloud-init template's header comment spelled out its own placeholder names, so every payload was substituted twice and the rendered user-data was 16 KB over Hetzner's 32 KiB limit; both provisioning scripts now measure it. kuma-monitor.mjs gained push, maintenance and push --rotate.

Next: phase 3 (#13) creates the :stable tag, which is what turns this from a hand tool into a pipeline.

Done and verified live. Commits `23db287` (the deployer) and `69a8317` (the banner fix and the measured numbers). **What is on the hosts.** `deploy/lib/deployer.sh` plus `nectenda-deploy.timer` on both `accounts` and `eu1`, ticking every 60 s, `AUTO_DEPLOY=off` as this phase specifies. Config in `/srv/<app>/deploy.env`, root-only 0600. Installed and upgraded by `deploy/install-deployer.sh`; new hosts get everything but the Kuma bundle from cloud-init with `AUTO_DEPLOY=on`. **Checkpoint.** All four deploys run by hand, readiness probed through Caddy at 2/s throughout each one. | Deploy | Host | stop→ready | Readiness | Banner | |---|---|---|---|---| | 4446a33 → 23db287 | identity | 6.3 s | no failure, slowest held 1.4 s | up 4 s, then gone | | 4446a33 → 23db287 | eu1 | 7.3 s | no failure, slowest held 2.9 s | up 4 s, then gone | | 23db287 → 4446a33 (`rollback`) | eu1 | 7.4 s | no failure | up, then gone | | 4446a33 → 23db287 (back) | eu1 | 7.2 s | no failure | up, then gone | `/api/health` confirmed the version each way round, including the rollback. Both push monitors are green and report the running sha and `:stable unresolved`, which is correct until phase 3. **One real bug, found by doing this rather than by the tests.** The first live deploy raised no banner and logged only "kuma maintenance on failed (not fatal)". The deployer was mounting the bundle directory over the container's `/lib`, which shadows the dynamic linker and libc, so `node` died before reading the bundle. Two things made it survive review: the output was sent to `/dev/null`, and the failure was non-fatal by design. Now mounted at `/opt/nectenda-deploy`, the client's output is logged, and a test asserts the bundle is never mounted at `/lib`. **Tests.** `deploy/` is a workspace package; 13 tests covering the decision truth table, the dry-run plan and its ordering, the `.env` pin, the rollback, the refusal to roll back after a migration, and the refusal to deploy without a backup. Five mutations each fail the suite: ignore `AUTO_DEPLOY`, ignore the identity gate, raise the banner after the restart instead of before, roll back after a migration, mount the bundle at `/lib`. **Also fixed along the way.** The cloud-init template's header comment spelled out its own placeholder names, so every payload was substituted twice and the rendered user-data was 16 KB over Hetzner's 32 KiB limit; both provisioning scripts now measure it. `kuma-monitor.mjs` gained `push`, `maintenance` and `push --rotate`. Next: phase 3 (#13) creates the `:stable` tag, which is what turns this from a hand tool into a pipeline.

cruelacid closed this issue

2026-09-15 10:00:10 +01:00

cruelacid referenced this issue

2026-09-15 10:00:10 +01:00

CD phase 1 — foundations for automatic deploys #11

Rows
Columns

CD phase 2 — host deployer with AUTO_DEPLOY=off #12