NEC-383: Server logs are lost on every deploy: keep them across container recreation #350
No reviewers
Labels
No labels
area:docs
area:identity
area:ops
area:plugin
area:server
channel:community
channel:direct
channel:owned
channel:press
channel:social
dependencies
e2ee-constrained
gate:at-ga
gate:pre-ga
marketing
parity
relay:absent
relay:planned
relay:requested
relay:supported
risk:additive
risk:contract
risk:none
security
usability
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
Nectenda/nectenda!350
Loading…
Reference in a new issue
No description provided.
Delete branch "worktree-nec-383-server-logs-are-lost-on-every-deploy-kee"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Task: NEC-383 — https://projectron.nerchure.com/tasks/405
Every deploy recreated the server container and deleted its json-file log, so docker logs reached back only to the last deploy; on 8 Nov a KPI-only merge took the line recording when a paying customer's connection closed. Every service on the shard, identity and ops hosts now logs to the host journal, tagged with the container's stable name. A journald drop-in (a host-files.txt row for every kind) keeps the journal persistent, capped at 3 GB and 30 days and rotated daily; the installer restarts journald when it changes. The deployer's deploy/failed/ tails are deleted at 30 days too, and docs/privacy.md states the 30 days, with a test holding it to the drop-in. Ride-alongs, each approved by the user: NEC-385 (provision-identity.sh rendered invalid cloud-config) and NEC-388 (shard user-data was over Hetzner's 32 KiB; shard and identity cloud-init are now a bootstrap and provisioning runs install-deployer.sh itself, then a first start). Rollout is manual (production ssh): see the spec's Rollout section. The journald probe already passed on all three hosts.
Spec:
docs/changes/NEC-383-server-logs-across-deploys/spec.mdAlso fixes NEC-385: provision-identity.sh renders invalid cloud-config: $(seq 1 30) expands inside its heredoc
Also fixes NEC-388: provision-shard.sh cannot provision: cloud-init user-data is over Hetzner's 32 KiB
Changelog
Server logs are now kept for 30 days across updates, so problems you report can still be traced after we ship a fix, and the privacy policy now says how long they are kept.
Review found the new privacy line ("server logs are kept for 30 days, then deleted") untrue for deploy/failed/: capture_failed_logs keeps a 200-line tail of each failed container, which can hold connection lines with account names and failed sign-ins with addresses, and nothing ever deleted them. Every tick now deletes those 30 whole days old, not only at the next failure, which may never come; a dry run deletes nothing. Also escapes the pipe in the runbook's journalctl example, which split its table cell and dropped the rest of the row when rendered. Task: NEC-383 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EV8ztYUxfVZQVeRDjDZ4hu