NEC-383: Server logs are lost on every deploy: keep them across container recreation #350

Merged
nectenda-agent merged 6 commits from worktree-nec-383-server-logs-are-lost-on-every-deploy-kee into main 2026-11-10 01:06:35 +00:00
Collaborator

Task: NEC-383 — https://projectron.nerchure.com/tasks/405

Every deploy recreated the server container and deleted its json-file log, so docker logs reached back only to the last deploy; on 8 Nov a KPI-only merge took the line recording when a paying customer's connection closed. Every service on the shard, identity and ops hosts now logs to the host journal, tagged with the container's stable name. A journald drop-in (a host-files.txt row for every kind) keeps the journal persistent, capped at 3 GB and 30 days and rotated daily; the installer restarts journald when it changes. The deployer's deploy/failed/ tails are deleted at 30 days too, and docs/privacy.md states the 30 days, with a test holding it to the drop-in. Ride-alongs, each approved by the user: NEC-385 (provision-identity.sh rendered invalid cloud-config) and NEC-388 (shard user-data was over Hetzner's 32 KiB; shard and identity cloud-init are now a bootstrap and provisioning runs install-deployer.sh itself, then a first start). Rollout is manual (production ssh): see the spec's Rollout section. The journald probe already passed on all three hosts.

Spec: docs/changes/NEC-383-server-logs-across-deploys/spec.md

Also fixes NEC-385: provision-identity.sh renders invalid cloud-config: $(seq 1 30) expands inside its heredoc
Also fixes NEC-388: provision-shard.sh cannot provision: cloud-init user-data is over Hetzner's 32 KiB

Changelog

Server logs are now kept for 30 days across updates, so problems you report can still be traced after we ship a fix, and the privacy policy now says how long they are kept.

Task: NEC-383 — https://projectron.nerchure.com/tasks/405 Every deploy recreated the server container and deleted its json-file log, so docker logs reached back only to the last deploy; on 8 Nov a KPI-only merge took the line recording when a paying customer's connection closed. Every service on the shard, identity and ops hosts now logs to the host journal, tagged with the container's stable name. A journald drop-in (a host-files.txt row for every kind) keeps the journal persistent, capped at 3 GB and 30 days and rotated daily; the installer restarts journald when it changes. The deployer's deploy/failed/ tails are deleted at 30 days too, and docs/privacy.md states the 30 days, with a test holding it to the drop-in. Ride-alongs, each approved by the user: NEC-385 (provision-identity.sh rendered invalid cloud-config) and NEC-388 (shard user-data was over Hetzner's 32 KiB; shard and identity cloud-init are now a bootstrap and provisioning runs install-deployer.sh itself, then a first start). Rollout is manual (production ssh): see the spec's Rollout section. The journald probe already passed on all three hosts. Spec: `docs/changes/NEC-383-server-logs-across-deploys/spec.md` Also fixes NEC-385: provision-identity.sh renders invalid cloud-config: $(seq 1 30) expands inside its heredoc Also fixes NEC-388: provision-shard.sh cannot provision: cloud-init user-data is over Hetzner's 32 KiB ## Changelog Server logs are now kept for 30 days across updates, so problems you report can still be traced after we ship a fix, and the privacy policy now says how long they are kept.
provision-identity.sh renders its cloud-config through an unquoted heredoc,
so bash expanded `$(seq 1 30)` on the operator's machine into thirty lines and
the document stopped being YAML; backticks in two comments ran their words as
commands. The shard template and the identity script also ended a runcmd entry
in `echo 'WARNING: …'`, which YAML reads as a mapping, not a command. All from
16 September, and latent only because no host has been built since.

Escaped and quoted, and deploy/test/cloud-config.test.ts now renders all three
cloud-configs the way their scripts do and requires each to parse, to run no
command while rendering, and to give runcmd strings only.

Task: NEC-383
Also-fixes: NEC-385

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EV8ztYUxfVZQVeRDjDZ4hu
Every service logged through Docker's json-file driver, whose log belongs to
the container and is deleted with it. The deployer recreates the service
container on every promotion, and every merge to main promotes, so `docker
logs` reached back only to the last deploy. On 8 November a KPI-only merge
redeployed eu1 and took the line saying when a paying customer's connection
had closed.

Every service on the shard, identity and ops hosts now logs to the host's
journal, tagged with the container's name so the identifier survives each
recreation. A journald drop-in, a host-files.txt row for every kind, keeps the
journal on disk, capped at 1 GB and 30 days and rotated daily; the installer
restarts journald when it changes, since journald reads it only at start. New
hosts get the same, and journald as Docker's daemon default. docs/privacy.md
now says server logs are kept for 30 days, and a test fails if it and the
drop-in part. README, runbook and the manual check say how to read across
deploys and how a live host is verified.

Task: NEC-383

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EV8ztYUxfVZQVeRDjDZ4hu
provision-shard.sh could not build a shard: carrying every host file in
cloud-init put its user-data at about 48 KB with the example env files, over
Hetzner's 32 KiB, and its own check refused it. Each of those files was also a
second copy that nothing kept current, which is how the identity host went on
running the NEC-96 backup bug after it was fixed.

The shard's and identity's cloud-init now carry a bootstrap only: the host's
env files, deploy.env held at AUTO_DEPLOY=off, .env, Docker with journald as
its default log driver, the registry login and the data directory. That is
about 11 KB. Once the box exists the script, through deploy/lib/first-start.sh,
waits for cloud-init (forgetting any key held for the name or address and
filing the new one under the name), adds the host to hosts.txt, runs
install-deployer.sh --at <ip> --install-only, does the first start under the
deployer's lock (pull, up, metrics, restic init, then AUTO_DEPLOY=on), and runs
--check. A failed step stops with the command to resume it. The installer skips
the Caddy reload when Caddy is not running yet.

The ops box keeps its full cloud-init: no deployer, and a small one.

Task: NEC-383
Also-fixes: NEC-388

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EV8ztYUxfVZQVeRDjDZ4hu
Review found the new privacy line ("server logs are kept for 30 days, then
deleted") untrue for deploy/failed/: capture_failed_logs keeps a 200-line
tail of each failed container, which can hold connection lines with account
names and failed sign-ins with addresses, and nothing ever deleted them. Every
tick now deletes those 30 whole days old, not only at the next failure, which
may never come; a dry run deletes nothing.

Also escapes the pipe in the runbook's journalctl example, which split its
table cell and dropped the rest of the row when rendered.

Task: NEC-383

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EV8ztYUxfVZQVeRDjDZ4hu
From review of the bootstrap:

- The shard's A record was made only after the first start, so Caddy asked
  for its certificate for a name that did not resolve and backed off for
  minutes, long enough for --register's readiness wait to give up. The
  record is made, or asked for, as soon as the address is known.
- A failed step printed only its own command, so resuming after a failed
  wait skipped add_host and the installer then refused the unknown host. The
  sequence is now provision_rest in first-start.sh, and a failure prints that
  step and every one after it, each in `bash -c` so they work pasted into zsh,
  which would not split the old string-held ssh command into words (now a
  function). provision_rest is tested end to end against the fake host.
- `compose up -d` only creates containers; the first start now waits for
  them to be running and healthy before AUTO_DEPLOY can turn on. Tests now
  defend the lock, and the restic step coming before AUTO_DEPLOY.
- The installer read a failing `compose ps` as "no Caddy" and skipped the
  reload silently; it now fails the install instead.
- The ops box's user-data is checked against 32 KiB too.

The README's Logs section also says deploy/failed/ is kept 30 days (NEC-383).

Task: NEC-383
Also-fixes: NEC-388

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EV8ztYUxfVZQVeRDjDZ4hu
Cap each host's journal at 3 GB, not 1, so 30 days is what limits it
All checks were successful
Release note / release-note (pull_request) Successful in 17s
CI / build (pull_request) Successful in 3m57s
CI / e2e (pull_request) Successful in 17m6s
CI / promote (pull_request) Has been skipped
Deploy site / deploy (push) Successful in 57s
CI / build (push) Successful in 4m6s
CI / e2e (push) Successful in 23m18s
CI / promote (push) Successful in 32s
39fb9b66a8
Measured on 10 November: eu1 and accounts each write about 36 MB a day to the
journal before any container logs to it, most of it sshd lines from port 22
being open to everyone (about 40,000 a day per host). A 1 GB cap would have
held under 30 days, so size rather than age would have decided how long the
server logs are kept. 3 GB holds 30 days with the containers' lines added, and
is still under journald's own default on those disks (10% of the disk, up to
4 GB). The sshd noise is its own Backlog card.

Also records the journald probe run on all three hosts on 9 November.

Task: NEC-383

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EV8ztYUxfVZQVeRDjDZ4hu
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
Nectenda/nectenda!350
No description provided.