NEC-346: A degraded dependency took down all sync: attachment storage must not stop the shard #323

Merged
nectenda-agent merged 20 commits from worktree-nec-346-a-degraded-dependency-took-down-all-sync into main 2026-10-27 07:01:32 +00:00
Collaborator

Task: NEC-346 — https://projectron.nerchure.com/tasks/367

On 26 October 2026 the attachment bucket answered 503, a deploy restarted eu1, the startup probe threw, and every note's sync stopped; the deployer then rolled back to an identical build and posted incidents nobody had checked. The shard now starts with attachments degraded on a transient store failure (still refusing on misconfiguration), refuses attachment calls fast with 503 ATTACHMENTS_UNAVAILABLE, re-probes and recovers without a restart, and says so under degraded on /api/ready (identity does the same for a shard it cannot pull). The plugin retries refused uploads on a timer. The deployer keeps the failed container's log, recognises a rollback that fails the same way as not the build (unless the rollback refused a migrated schema), names only its own service in incidents, and no longer redeploys a failed sha every minute. Reviewed by a fresh-context agent; its two high findings (boot backup skipped, fs backend guarded) and four others are fixed with mutation-checked tests. Not run on a real host: the docker logs and Kuma checks listed in docs/manual-testing.md.

Spec: docs/changes/NEC-346-attachment-store-degraded/spec.md

Also fixes NEC-347: A failed attachment put leaves a charged row with no object, and reconcile never corrects it
Also fixes NEC-348: Object deletes that fail at the provider are dropped, leaving orphaned objects in the bucket
Also fixes NEC-349: deleteRemote forgets a pending attachment delete on any HTTP answer, 5xx included
Also fixes NEC-350: attestFolder's re-upload of ids the server lost is a no-op for unchanged files
Also fixes NEC-351: After a successful rollback the deployer redeploys the same failed sha every minute

Changelog

Notes keep syncing when the server's attachment storage is down: attachments wait and upload on their own once it is back.

Task: NEC-346 — https://projectron.nerchure.com/tasks/367 On 26 October 2026 the attachment bucket answered 503, a deploy restarted eu1, the startup probe threw, and every note's sync stopped; the deployer then rolled back to an identical build and posted incidents nobody had checked. The shard now starts with attachments degraded on a transient store failure (still refusing on misconfiguration), refuses attachment calls fast with 503 ATTACHMENTS_UNAVAILABLE, re-probes and recovers without a restart, and says so under `degraded` on /api/ready (identity does the same for a shard it cannot pull). The plugin retries refused uploads on a timer. The deployer keeps the failed container's log, recognises a rollback that fails the same way as not the build (unless the rollback refused a migrated schema), names only its own service in incidents, and no longer redeploys a failed sha every minute. Reviewed by a fresh-context agent; its two high findings (boot backup skipped, fs backend guarded) and four others are fixed with mutation-checked tests. Not run on a real host: the docker logs and Kuma checks listed in docs/manual-testing.md. Spec: `docs/changes/NEC-346-attachment-store-degraded/spec.md` Also fixes NEC-347: A failed attachment put leaves a charged row with no object, and reconcile never corrects it Also fixes NEC-348: Object deletes that fail at the provider are dropped, leaving orphaned objects in the bucket Also fixes NEC-349: deleteRemote forgets a pending attachment delete on any HTTP answer, 5xx included Also fixes NEC-350: attestFolder's re-upload of ids the server lost is a no-op for unchanged files Also fixes NEC-351: After a successful rollback the deployer redeploys the same failed sha every minute ## Changelog Notes keep syncing when the server's attachment storage is down: attachments wait and upload on their own once it is back.
The startup probe was one HeadBucket and one throw. On 26 October 2026 a
provider's 503 on it stopped the shard, and with it every note's sync,
although only attachments live in that bucket. The backend now answers
`probe()` with what kind of unusable it found: a definite 4xx is
misconfiguration and still refuses to start; a 5xx, a timeout or a socket
error is transient (STORE-070). The classification reads what the store
answered, never its message, and errs toward transient, since a shard that
starts with attachments visibly degraded costs less than the outage.

The S3 client also gets a connection timeout, an idle timeout and two
attempts; it had none, so a provider that hung would have hung the probe.
The handler's request timeout only warns unless `throwOnRequestTimeout` is
set, which the test against a bucket that never answers found on its first
run.

The fake S3 gains a fault switch (`{ status }` or `'hang'`) for the tests of
what the shard does when its store misbehaves.

Task: NEC-346

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7
On 26 October 2026 the attachment bucket answered 503, a routine deploy
restarted eu1, the startup probe threw, and every note's sync stopped for a
dependency the document path never touches. The shard now keeps one current
opinion of its store (blob-health.ts): the probe at startup decides only what
it is evidence of — misconfiguration still refuses to start, a provider
failing starts the shard with attachments degraded (STORE-070) — and every
remote call reports its outcome, so a store that dies later is noticed too.

While the store is down, uploads and streamed downloads answer `503
ATTACHMENTS_UNAVAILABLE` with `Retry-After`, before the body is read and
before a row or usage is reserved, and without a round trip to the provider
(STORE-071); the document relay is untouched. A timer re-probes the store and
it comes back without a restart (STORE-072); the SDK's idle and connection
timeouts bound how long a hung provider can hold any call.

Readiness learns the difference between the service being broken and a
dependency being down (STORE-073): `/api/ready` answers 503 only for
`database` and `disk`, and names a down dependency under `degraded` with 200,
`{}` when nothing is, so a keyword monitor can watch for the healthy bytes.
The identity service's "no shard manifest pulled recently" moves from a 503
with `reason: database` — the only reason the type allowed — to
`degraded.directory`, per shard, with the error: sign-in works without the
manifest, and the status page said sign-in was down for a shard outage.

A startup failure is now written with `log.fatal`, a synchronous write to
stderr (STORE-074). The logger already was synchronous on a Linux pipe; the
guarantee is now the call's rather than the platform's.

Task: NEC-346

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7
A client's DELETE, stage two of the sweep, a folder delete and an account
purge all remove the row first and the object second, so the accounting is
right if the second step fails — and then the object stayed in the bucket for
ever, because the reconcile reads rows and the row was gone. During a store
outage that was every delete.

A failed object delete is now written to `blob_object_deletes` (migration 12;
an empty blob id is a whole folder prefix) and tried again on each maintenance
tick, at startup, and the moment the store comes back. An entry whose row
exists again is dropped rather than run: a re-upload under the same id owns
the object now. A put that failed after its reservation queues its key too,
since the object may have landed on the provider's side.

Task: NEC-346
Also-fixes: NEC-348

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7
`storeBlob` reserves the usage and inserts the row before `backend.put`, and
left both in place when the put failed, on the reasoning that the reconcile
would correct the over-count. It never did: `reconcileUsage` sums rows, and
the row was there. The plugin mints a fresh id for every attempt, so a store
outage charged the account once per retry until the quota refused real
uploads.

A failed put now takes the reservation back in one transaction: a fresh id
loses its row and its charge, a re-upload of an existing id goes back to the
row it replaced. The object that may have landed is already queued for
deletion (NEC-348), which answers the old objection to un-reserving.

Task: NEC-346
Also-fixes: NEC-347

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7
An upload the server refused or never answered went to the persisted queue
and was tried again on the provider's next `connected` status, which is a
WebSocket reconnect. With notes syncing and the server's attachment store down
— the shape of 26 October 2026 — the socket never dropped, and every upload
waited for it to. The queue is now retried on a ladder (30 s, 2 min, 10 min),
one timer for the whole queue, then on each reconnect as before; a success or
a reconnect starts the ladder over (SAFE-A34). The server's `503
ATTACHMENTS_UNAVAILABLE` is logged as the expected answer it is.

SAFE-A51 writes down what the code already did and this outage made worth
stating: no HTTP answer to an upload or a download is deletion. The settings
list of waiting uploads stops calling every wait "waiting for space".

STORE-070 to STORE-074 state the server's side of the same day, defended by
the tests of the previous commits.

Task: NEC-346

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7
`deleteRemote` never read the response status: any HTTP answer, a 500 or the
store-down 503 included, cleared the pending delete, and only a thrown network
error kept it. A delete made during an outage was forgotten and its object
left in the bucket. It is forgotten now only on a 2xx or a 404.

Task: NEC-346
Also-fixes: NEC-349

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7
`attestFolder` called `upload()` for every id the server said it had no bytes
for, and `upload()` returned early when the file's hash matched the listed
entry — which for a lost object it always does. The repair for a listing
entry whose bytes the server lost was a no-op, and no test said so because the
test attested with nothing missing. The attest path now forces the upload past
the short-circuit, and the test names a missing id.

Task: NEC-346
Also-fixes: NEC-350

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7
On 26 October 2026 a shard's container failed to start because a dependency
was down. The deployer read "never reported its sha" as a bad build and
rolled back to one with identical server code, which failed the same way. It
then fell through to the schema-advanced branch: switched itself off, pinned
a second incident about sign-ins on the sync host, and left the host on a
build that could not start either. The `die` meant for that case was
unreachable.

- **The failed container's log is kept** in deploy/failed/ and its tail in
  the journal, before the rollback's `compose up` recreates the container
  and deletes the json-file log with it. That is the likeliest reason the
  startup failure's own line never turned up in `docker logs`. compose's
  output is kept too, no longer sent to /dev/null.
- **A rollback that fails the same way is evidence the build is not the
  cause.** The new build is re-pinned (it comes up by itself when the
  dependency does), auto-deploy stays on, history records `not-the-build`,
  and the incident says both versions fail to start.
- **Incidents name this host's service and say what the deployer knows**:
  "is being rolled back" before the rollback, not "was rolled back"; no
  "Sync is expected to be working" from a process that had not looked.
  `PUBLIC_NAME` in deploy.env names the service, defaulting from `SERVICE`,
  and the Kuma client takes a title per case.
- **A deploy that comes up with a dependency degraded is a deploy that
  worked** (STORE-073): it is not rolled back, and the degraded names ride
  on the Kuma push and the history line.

Task: NEC-346

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7
After a deploy failed and the rollback succeeded, AUTO_DEPLOY stayed on and
nothing remembered the sha, so the next tick a minute later deployed it
again: fail, roll back, post an incident, clear it, every minute for as long
as the build or the thing it needed stayed broken. `decide` now takes
whether history's last word on the stable sha is `failed` or
`not-the-build`, and answers `blocked:failed-before`; an `ok` for the sha
clears it. `deployer.sh apply <sha>` by hand does not ask, as no blocked
state stops it.

Task: NEC-346
Also-fixes: NEC-351

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7
`/api/ready` answers 200 while the shard's attachment store, or the identity
service's view of a shard, is down, and names the dependency under
`degraded` (STORE-073). The `ready` monitors and the public rows stay green,
which is right: notes sync and sign-in works. Two keyword monitors for the
healthy bytes `"degraded":{}` say what is actually down: `sync <shard>
attachments` and `service accounts directory`. They are not public rows, and
they need registering by hand with a re-run of `kuma-monitor.mjs add`.

Task: NEC-346

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7
The afternoon of 26 October 2026, end to end: a real shard on a real
S3Backend against the fake bucket, started while the bucket answers 503. A
note typed in one vault reaches the other; the image written beside it stays
on disk where it was, queued, and reaches nobody; `/api/ready` stays 200 and
names `attachments` under `degraded`. Then the bucket answers again and the
image arrives byte-identical, with no restart of the shard or either vault.

The server's test harness re-exports the fake bucket, the S3 backend and the
health pieces, since the e2e package reaches the server only through it. The
port block grows to 36 for the suite's own port.

Also gives the degraded-readiness stub in health-routes.test.ts the
annotation the typecheck asked for.

Task: NEC-346

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7
The change spec, with its five ride-alongs and the audit of every startup
dependency both services await. `docs/sync-limitations.md` gains the
outage and what is still open. The deploy runbook, the support runbook and
the release notes on rollback describe the kept log, the not-the-build
outcome and the failed-before hold. `docs/manual-testing.md` lists what only
a real host can show, including the eu1 log reading this session could not
make.

Task: NEC-346

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7
The destructured `{ …, title }` made `title` required to the deploy
package's typecheck, and every existing call omits it.

Task: NEC-346

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7
Three findings from review, each confirmed by running the code:

- **The boot backup was skipped.** The new drain of queued object deletes
  went through maintenance's one single-flight slot and took it on boot
  before the backup owed there asked for it, and `run` skips a job that finds
  the slot taken. The drain now has its own single-flight, awaited by stop.
- **The filesystem backend was guarded too.** One missing file there — the
  sweep's expected case — classified as transient, marked the whole store
  degraded, and every later `setAside` in the pass failed fast, leaving
  quarantined rows whose files never moved and so could not be restored.
  Only a remote store is guarded now.
- **The startup decision had no test.** It moves to `openBlobStore`, tested
  against the fake bucket, and `startup.test.ts` runs the real entrypoint as
  a process: it listens with `degraded.attachments` on a 503 and clears it
  when the bucket answers, and on a 403 it exits 1 with a `[FATAL]` line on
  stderr (STORE-070, STORE-074).

The startup probe also gets its own bound (10 s): the client's two 30 s
attempts against a bucket that never answers would have held listening for
a minute on every restart. A refusal no longer logs "attachments degraded"
on its way out.

Each was mutation-checked: undone, its test fails.

Task: NEC-346

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7
Found in review: a build that migrates the database and then dies before it
answers leaves no schema to compare, so the rollback looks safe; the previous
build then refuses the newer database and fails too. That read exactly like
a dependency both builds need being down, and the not-the-build path re-pinned
the crash-looping build, left the timer on, and told the status page the
update was not the cause. The rollback's kept log now decides: if the older
build refused a newer schema, the new build is pinned again and the host
stops as it does after any migration, with the restore hint.

Keeping the log is also `|| true` throughout, so it can never be what ends a
deploy under pipefail before its rollback.

Task: NEC-346

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7
Two findings from review:

- A forced re-upload (NEC-350) puts back bytes the listing already names.
  If another vault listed a newer version while this one was sending, the
  old entry was written over it, and that vault would then download the old
  bytes over its own newer file. The forced upload now lists only if the
  listing still names the entry it started from.
- An upload in flight when sync stopped failed afterwards and armed the
  retry timer on a stopped BlobSync. It would have uploaded nothing, since
  no folder is connected, but it outlived the object. The ladder now arms
  only while a folder is connected.

Both mutation-checked.

Task: NEC-346

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7
The STORE-071 tests could not tell "refused before the body was read" from
"reserved, then released": both end with no row and no charge. The new test
sends ten bytes of a declared hundred thousand and never finishes; only the
route's early refusal answers it, and with that refusal removed it hangs.

The health-routes assertion that a detail names no endpoint ran against a
string the test wrote itself, and is dropped; blob-backend.test.ts checks
the real detail against a failing bucket.

The limitation entry gains the review's two same-id races, which the plugin
cannot cause, and a pending delete answered 403 staying queued.

Task: NEC-346

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7
Task: NEC-346

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7
Drop an unused import lint refused
Some checks failed
CI / promote (pull_request) Has been cancelled
CI / e2e (pull_request) Has been cancelled
CI / build (pull_request) Has been cancelled
Release note / release-note (pull_request) Successful in 16s
5723073669
Task: NEC-346

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7
Add NEC-346's release notes, one line per change
All checks were successful
Release note / release-note (pull_request) Successful in 16s
CI / build (pull_request) Successful in 5m50s
CI / e2e (pull_request) Successful in 14m52s
CI / promote (pull_request) Has been skipped
CI / build (push) Successful in 6m0s
CI / e2e (push) Successful in 14m44s
CI / promote (push) Successful in 33s
043615546c
Task: NEC-346

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
Nectenda/nectenda!323
No description provided.