CD phase 3 — CI gate, promotion, switch on #13
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
ci.yml: e2e job on main after build (concurrency e2e), promote job that smoke-starts both :sha images, asserts the version literal, and retags :stable; caddy validate step; cloud-init default IMAGE_TAG=stable; delete e2e-not-run-here. Flip AUTO_DEPLOY=on on identity, watch a merge land, then eu1. Checkpoint: a merge to main reaches both hosts with no human step; first three gaps recorded in docs/releasing.md.
Before this phase turns
pnpm testinto the deploy gate, the gate needs to mean something. It does not yet.Running every package's vitest at once oversubscribes the machine, and several server tests measure real elapsed time, so a different one fails on each run. Three consecutive full runs failed in three different files, and each of those files passed six times out of six when run on its own:
health-routes.test.ts—saturatedis the p99 event-loop delay againstLOOP_LAG_ALERT_MS, which defaults to 100 ms. Fixed in7110b19by pinning the threshold in the shape test; the test that proves the flag flips on a real stall is untouched.awareness-coalescing.test.ts— wall-clock sleeps with tight margins around the presence coalescing window.loop-stats.test.ts > sees a blocked loop.Also in
7110b19: the suites now run one package at a time. The whole tree is 899 tests in about 26 s serialised, so it costs nothing.That is not a full fix. Serialised, roughly one run in three still fails somewhere in the same family. Those tests assert wall-clock behaviour, and on a loaded box they are marginal by construction. Options, in rough order of preference: give the window a configurable length so the test can set a generous one; drive them on fake timers; or scope
retry: 2to that family only. The last is the cheapest and the least honest.Worth settling before the promote job exists, because a gate that fails at random trains everyone to re-run it until green, and then it is not a gate.
Written and half verified, on branch
cd-phase-3(through4f507b5). Not merged, because it cannot be switched on yet — see below.What the branch does.
build→e2e→promote. A push to main builds both images, runs the real-Obsidian suite against that commit, starts each published image and waits for/api/healthto report the literal sha, then retags it:stablewithbuildx imagetools create— a manifest copy, so the digest a host pulls is the one that was tested and both platforms move together. The e2e steps moved into.forgejo/actions/obsidian-e2eand are shared with the nightly so the two cannot drift. New hosts now default toIMAGE_TAG=stable.Green on the branch: install, build, typecheck, lint, the serialised unit suites (20 s), the deployer suite, the mirror and design-asset checks, both Caddyfiles, and the two-platform image build.
e2eandpromoteare main-only and so remain unverified until merge.Three bugs found by pushing the first branch in weeks, each of which passed on a developer machine and failed only on the runner:
$PWD, but the job runs inside a container, so the daemon resolved that path against the host's filesystem, mounted an empty directory, and reported "no such file or directory" for a file plainly present. Piped in on stdin instead, and mutation-checked that it still rejects an unknown directive, an unclosed brace and a bad duration.--output=type=cacheonly, which is what its comment already claimed.test:coverage, which since phase 2 meantdeploy— the host deployer, whose suite decides whether a machine may restart production. It was the one package CI never ran.I spent two attempts changing the Caddyfile command before reading the log. The log was the whole answer; this instance exposes no log API, so it has to come from the run page in the browser. Worth knowing next time.
What blocks switching on: #15. The e2e suite has been red on main since the nightly of 14 September, and the failing scenario is recovery after a password reset restoring a note as an empty file. Merging this gate would correctly refuse to promote anything, so
:stablewould never be created and no host would move. Fix #15 first, then merge, then flipAUTO_DEPLOY=onon identity and watch one merge land before eu1.Both hosts stay safe in the meantime:
AUTO_DEPLOY=off, both running23db287, push monitors green.Done, and switched on. Both hosts run
AUTO_DEPLOY=onand are deploying themselves.The checkpoint. A push to main, and 70 seconds after the promote job retagged
:stable, both hosts were running it. Nobody touched a host.23db287→32cfbc523db287→32cfbc532cfbc5→b3b8c4c32cfbc5→b3b8c4cReadiness was probed through Caddy at 2/s across every one of them and never returned anything but 200. The upgrade banner appeared on status.nectenda.com for three to four seconds each time and cleared afterwards. Identity went first every time, because eu1's gate held it until accounts reported the build — the ordering rule working rather than being masked by timing. Both push monitors are green and carry the running sha.
Two bugs the first real run of the gate found, both of which had been failing quietly:
identity.test.tsasserted the sign-in button's fill as the literalrgb(217, 182, 90), which is--nectenda-action-primaryin Obsidian's dark theme. A developer's Obsidian is dark and the runner's is light (#B08A2E), so the assertion was reporting which machine ran the suite. It now reads the token from the button and requires the fill to match it, and is mutation-checked: forcing the fill to grey still fails. Read from the button rather thandocument.body, because the settings pane is a popout window and the property comes back empty from the wrong document.What the gate did correctly while it was red. On the first merge, build passed, e2e failed, promote was skipped,
:stablewas never created, and no host moved. That is the whole point of it, and it behaved that way before I had proved it would.Also worth knowing. This Forgejo exposes no Actions log API at all — the swagger spec has none. Job logs come only from the run page in the browser, and the log viewer renders lazily, so jumping to the end shows a blank pane and you have to scroll back up a little. Two fixes were wasted guessing at a failure before I read one.
Everything else from the plan is in: shared composite action for the e2e steps,
caddy validateon both files, the deployer's own suite now running in CI,IMAGE_TAG=stablefor new hosts, ande2e-not-run-heredeleted. Measured gaps are indocs/releasing.md, which stops there deliberately — every deploy logs its own gap and pushes it to its monitor.Next is #14, the plugin version skew.