NEC-390: Container log lines are copied to /var/log/syslog and kept ~33 days, past the 30 privacy.md promises #353

Merged
nectenda-agent merged 4 commits from worktree-nec-390-container-log-lines-are-copied-to-varlog into main 2026-11-10 11:52:52 +00:00
Collaborator

Task: NEC-390 — https://projectron.nerchure.com/tasks/412

Container log lines were copied to /var/log/syslog. Ubuntu's own journald drop-in, /usr/lib/systemd/journald.conf.d/syslog.conf, turns on forwarding to rsyslog, and it sorts after our nectenda.conf. So since NEC-383 every container line also landed in /var/log/syslog, where logrotate keeps it about 33 days. That is outside the journal's caps and past the 30 days docs/privacy.md promises. The first copied line passes 30 days on 10 December. A drop-in of the same name in /etc, deploy/journald/syslog.conf, replaces Ubuntu's and turns the forward off. It is installed as a host-files.txt row on all three kinds and written by the ops box's cloud-init. host-logs.test.ts now merges the drop-ins the way systemd does, Ubuntu's included, and tests what is in force rather than our files alone. docs/manual-testing.md corrects the NEC-383 entry and records its run. A new entry holds the host steps: a read-only check that nothing reads syslog or auth.log, install, verify, and a cleanup of the copies already written. Each file is replaced only when its whole pipeline succeeded, which was rehearsed on synthetic logs in Ubuntu 24.04. Ride-along NEC-394: install-deployer.sh --check now reads systemd-analyze cat-config on every host. It fails unless our journald settings have the last word, reading values as systemd does (whitespace trimmed, [Journal] only), and fails a host it cannot read. That gap is how this bug went unreported. The fix and the read were rehearsed against a real systemd 255. Nothing changes on the fleet until deploy/install-deployer.sh is run after merge.

Spec: docs/changes/NEC-390-container-lines-out-of-syslog/spec.md

Also fixes NEC-394: install-deployer.sh --check does not see what journald reads, so a vendor drop-in can turn the syslog forward back on unreported

Changelog

NONE

Task: NEC-390 — https://projectron.nerchure.com/tasks/412 Container log lines were copied to /var/log/syslog. Ubuntu's own journald drop-in, /usr/lib/systemd/journald.conf.d/syslog.conf, turns on forwarding to rsyslog, and it sorts after our nectenda.conf. So since NEC-383 every container line also landed in /var/log/syslog, where logrotate keeps it about 33 days. That is outside the journal's caps and past the 30 days docs/privacy.md promises. The first copied line passes 30 days on 10 December. A drop-in of the same name in /etc, deploy/journald/syslog.conf, replaces Ubuntu's and turns the forward off. It is installed as a host-files.txt row on all three kinds and written by the ops box's cloud-init. host-logs.test.ts now merges the drop-ins the way systemd does, Ubuntu's included, and tests what is in force rather than our files alone. docs/manual-testing.md corrects the NEC-383 entry and records its run. A new entry holds the host steps: a read-only check that nothing reads syslog or auth.log, install, verify, and a cleanup of the copies already written. Each file is replaced only when its whole pipeline succeeded, which was rehearsed on synthetic logs in Ubuntu 24.04. Ride-along NEC-394: install-deployer.sh --check now reads systemd-analyze cat-config on every host. It fails unless our journald settings have the last word, reading values as systemd does (whitespace trimmed, [Journal] only), and fails a host it cannot read. That gap is how this bug went unreported. The fix and the read were rehearsed against a real systemd 255. Nothing changes on the fleet until deploy/install-deployer.sh is run after merge. Spec: `docs/changes/NEC-390-container-lines-out-of-syslog/spec.md` Also fixes NEC-394: install-deployer.sh --check does not see what journald reads, so a vendor drop-in can turn the syslog forward back on unreported ## Changelog NONE
Ubuntu's own journald drop-in turns on forwarding to syslog, so since the
switch to journald (NEC-383) rsyslog wrote every container line to
/var/log/syslog too, where logrotate keeps it about 33 days: past the 30
docs/privacy.md promises, and outside the journal's caps. A drop-in of the
same name in /etc replaces the vendor's and turns the forward off; it cannot
live in nectenda.conf, which sorts before it.

host-logs.test.ts now merges the drop-ins the way systemd does, the vendor's
included, and tests the configuration in force rather than our files alone.
The manual-testing entry for NEC-383 is corrected and its run recorded, and a
new entry holds the host steps: install, check, and delete the copies already
written before 10 December.

Task: NEC-390

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EV8ztZmwACNFQRfJ3eiY7m
From review. The cleanup in docs/manual-testing.md moved a filtered file into
place whatever happened to grep or zcat, so a full disk or a damaged archive
left a truncated log, and plain grep reads a file with a NUL byte in it as
binary and prints none of its lines. Each file is now moved only when its
whole pipeline succeeded, grep runs as LC_ALL=C grep -a, and the pattern is
anchored to the program name. Run against synthetic logs on Ubuntu 24.04.

The test's merge counted rows systemd ignores (a name not ending in .conf),
so syslog.conf set to yes plus a zz-off row saying no passed it. A host with
only nectenda.conf, every host's state today, now has a test of its own, and
NEC-383's settings check asks for the last assignment, not the file's.

Task: NEC-390

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EV8ztZmwACNFQRfJ3eiY7m
install-deployer.sh --check compared each host's copy of our files byte for
byte, and that was all it knew of journald. journald merges every drop-in
directory in file-name order, last assignment winning, so a vendor file
sorting after ours overrides it with every file still matching. That is how
the forward to syslog stayed on with every host "verified" (NEC-390), and how
it would come back if an Ubuntu shipped it under another name.

--check now reads `systemd-analyze cat-config systemd/journald.conf` on every
host and fails unless the last value of each setting our journald drop-ins
make is ours; a host that cannot answer fails as journald(unread). The fake
host gains a systemd-analyze serving test/vendor/, Ubuntu's drop-in as the
hosts carry it, which host-logs.test.ts now reads too. Rehearsed against a
real systemd 255 in an Ubuntu 24.04 container: the forward on with
nectenda.conf alone, off with syslog.conf, back on under a later vendor file.

The README's cat-config grep no longer matches a comment line in
journald.conf.

Task: NEC-390
Also-fixes: NEC-394

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EV8ztZmwACNFQRfJ3eiY7m
Read journald's settings as systemd does, so a spaced override is seen
All checks were successful
Release note / release-note (pull_request) Successful in 18s
CI / build (pull_request) Successful in 4m18s
CI / e2e (pull_request) Successful in 19m49s
CI / promote (pull_request) Has been skipped
CI / build (push) Successful in 4m39s
CI / e2e (push) Successful in 17m12s
CI / promote (push) Successful in 43s
75ab8188d1
From the second review. --check read only `Key=Value` at column 0, but systemd
trims the whitespace round a key and its value, so `ForwardToSyslog = yes` or
an indented line in a later drop-in turned the forward back on with --check
saying verified. Confirmed against real systemd 255.

One awk program now reads both the host's cat-config merge and our own files:
whitespace trimmed, [Journal] only, the section reset at each file, comments
skipped, empty values kept (an empty assignment resets a setting). Booleans
compare as booleans, so `false` is `no`; a key the merge never sets is still
reported. The verdict tells the operator to restart journald after fixing a
foreign drop-in, since --check reads the configuration on disk. Each rule is
mutation-checked, and the parser was rerun against real cat-config output.

The manual greps tolerate the same whitespace.

Task: NEC-390
Also-fixes: NEC-394

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EV8ztZmwACNFQRfJ3eiY7m
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
Nectenda/nectenda!353
No description provided.