NEC-389: Every host's journal is mostly sshd noise from port 22 open to the internet #359

Merged
nectenda-agent merged 5 commits from worktree-nec-389-every-hosts-journal-is-mostly-sshd-noise into main 2026-11-12 20:21:31 +00:00
Collaborator

Task: NEC-389 — https://projectron.nerchure.com/tasks/411

Port 22 is open to the internet on every host. On eu1 sshd wrote about 28,500 journal lines a day, almost all of them scanners failing to log in, and all of it counts against the 3 GB cap that holds 30 days of the server's own logs. Reading eu1 also showed password login was on (passwordauthentication yes). Root was safe only because PermitRootLogin defaults to without-password.

Two drop-ins are added to host-files.txt for every kind:

  • deploy/ssh/00-nectenda.conf allows keys only. sshd keeps the first value it reads, so the file is named to sort before cloud-init's. That ends the password guessing, about two thirds of eu1's sshd lines.
  • deploy/ssh/log-filter.conf is a LogFilterPatterns= deny-list on ssh.service. It is written from eu1's 40 most common line shapes and drops the lines from connections that never logged in. Logins, sessions and anything never measured are kept. systemd 255 matches against the line with its sshd[pid]: prefix still on, so the patterns anchor after that prefix.

install-deployer.sh runs sshd -t before reloading sshd, and puts the previous file back if sshd rejects the new one or the reload fails. --check reads sshd -T -C, for root and for a user that does not exist, and the filter systemd has loaded. The ops box's cloud-init writes both files, gzipped to stay under Hetzner's 32 KiB. All of this was rehearsed on real systemd 255 and OpenSSH 9.6p1, and mutation-checked.

Merging changes no host. The production steps, with what each prints and how to undo it, are in the spec under "After merge". deploy/README.md has a new section, "ssh on the hosts".

🤖 Generated with Claude Code

https://claude.ai/code/session_01EV8ztZCfAWqDxDxgtz8LMp

Spec: docs/changes/NEC-389-ssh-log-noise/spec.md

Changelog

NONE

Task: NEC-389 — https://projectron.nerchure.com/tasks/411 Port 22 is open to the internet on every host. On eu1 sshd wrote about 28,500 journal lines a day, almost all of them scanners failing to log in, and all of it counts against the 3 GB cap that holds 30 days of the server's own logs. Reading eu1 also showed **password login was on** (`passwordauthentication yes`). Root was safe only because `PermitRootLogin` defaults to `without-password`. Two drop-ins are added to `host-files.txt` for every kind: - `deploy/ssh/00-nectenda.conf` allows keys only. sshd keeps the first value it reads, so the file is named to sort before cloud-init's. That ends the password guessing, about two thirds of eu1's sshd lines. - `deploy/ssh/log-filter.conf` is a `LogFilterPatterns=` deny-list on ssh.service. It is written from eu1's 40 most common line shapes and drops the lines from connections that never logged in. Logins, sessions and anything never measured are kept. systemd 255 matches against the line with its `sshd[pid]: ` prefix still on, so the patterns anchor after that prefix. `install-deployer.sh` runs `sshd -t` before reloading sshd, and puts the previous file back if sshd rejects the new one or the reload fails. `--check` reads `sshd -T -C`, for root and for a user that does not exist, and the filter systemd has loaded. The ops box's cloud-init writes both files, gzipped to stay under Hetzner's 32 KiB. All of this was rehearsed on real systemd 255 and OpenSSH 9.6p1, and mutation-checked. **Merging changes no host.** The production steps, with what each prints and how to undo it, are in the spec under "After merge". `deploy/README.md` has a new section, "ssh on the hosts". 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01EV8ztZCfAWqDxDxgtz8LMp Spec: `docs/changes/NEC-389-ssh-log-noise/spec.md` ## Changelog NONE
Port 22 is open to the whole internet, and sshd's lines about scanners
failing to log in were most of every host's journal: about 40,000 a day
on eu1 and on accounts, sharing the 3 GB cap that keeps 30 days of the
server's own logs. It stays open, since the operator's address rotates.

Two drop-ins, rows in host-files.txt for every kind:

- 00-nectenda.conf turns password and keyboard-interactive login off and
  keeps root by key. sshd keeps the first value it reads, so the name
  sorts ahead of cloud-init's 50-cloud-init.conf.
- log-filter.conf is a LogFilterPatterns= deny-list on ssh.service that
  drops lines about connections that never logged in, and keeps logins,
  sessions and anything unmeasured.

The installer runs sshd -t before reloading sshd and puts the previous
file back if sshd rejects the new one. --check reads sshd -T -C for root
and for a user that does not exist, and the filter systemd has loaded.
The ops box gets both through its cloud-init, gzipped to stay under
Hetzner's 32 KiB.

Rehearsed on systemd 255 and OpenSSH 9.6p1. journald matches a syslog
line with its "sshd[123]: " prefix still on, so the patterns anchor
after it. A line from a process that has already exited reaches journald
with no unit and is not filtered; deploy/README.md says how to read both.

Task: NEC-389

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EV8ztZCfAWqDxDxgtz8LMp
The install verifies the host once its files are in place, and the
verification now asks systemd which ssh log filter it has loaded.

Task: NEC-389

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EV8ztZCfAWqDxDxgtz8LMp
- A failed sshd reload now puts the previous files back, as a rejection
  does. Left in place, the new file would pass --check, which reads the
  files, while the running sshd went on with the old ones, and no later
  install would reload.
- sshd -t is given /run/sshd first. Under socket activation an idle
  ssh.service has none, and sshd -t then fails whatever the files say
  (confirmed in the rehearsal). On a new ops box that would have removed
  the drop-in at first boot.
- A drop-in is handed to sshd only once it is actually in place, and the
  ssh steps run straight after the moves, the filter's daemon-reload first.
- Tests for a failed reload, a failed daemon-reload, a previous file that
  cannot be kept, and an idle ssh.service.
- The README and the spec no longer claim more than was shown. --check
  reads sshd's files, not the running daemon: the README gives the probe
  that asks the daemon itself. A Match block is caught only for root and for
  an unknown user. Nobody has measured yet how much a real host's noise the
  filter removes. And the filter drops the operator's own failed logins
  too, with a note on how to see them during a lockout.
- The spec records what changed after approval and why, and its
  production steps measure the same 24 hours before and after.

Task: NEC-389

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EV8ztZCfAWqDxDxgtz8LMp
Write the ssh log filter from a day of eu1's sshd lines
Some checks failed
Release note / release-note (pull_request) Successful in 18s
CI / build (pull_request) Failing after 2m40s
CI / promote (pull_request) Has been cancelled
CI / e2e (pull_request) Has been cancelled
60d15aceb9
Read on eu1 on 12 November 2026: 28,483 sshd lines in 24 hours, all but
21 of them attributed to ssh.service, so the filter can reach nearly
everything. About 18,800 were password guessing. Passwords were on there:
sshd -T read passwordauthentication yes, and sshd_config.d was empty. The
key-only drop-in ends those lines, so the fixture keeps them, as the sign
that passwords are back on.

Two measured shapes the filter missed are added: kex_exchange_identification
errors, and "Failed none" for invalid users only. The fixture now holds
the day's 40 commonest shapes.

Task: NEC-389

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EV8ztZCfAWqDxDxgtz8LMp
Make the cannot-keep test fail cp for root too
All checks were successful
Release note / release-note (pull_request) Successful in 19s
CI / build (pull_request) Successful in 4m44s
CI / e2e (pull_request) Successful in 17m15s
CI / promote (pull_request) Has been skipped
CI / build (push) Successful in 5m40s
CI / e2e (push) Successful in 17m37s
CI / promote (push) Successful in 45s
1080d956f0
CI runs the deploy suite as root, and root writes into a 0555 directory,
so the copy the test meant to make fail succeeded there. A directory where
the copy would land fails cp whoever runs it.

Task: NEC-389

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EV8ztZCfAWqDxDxgtz8LMp
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
Nectenda/nectenda!359
No description provided.