NEC-346: A degraded dependency took down all sync: attachment storage must not stop the shard #323
No reviewers
Labels
No labels
area:docs
area:identity
area:ops
area:plugin
area:server
channel:community
channel:direct
channel:owned
channel:press
channel:social
dependencies
e2ee-constrained
gate:at-ga
gate:pre-ga
marketing
parity
relay:absent
relay:planned
relay:requested
relay:supported
risk:additive
risk:contract
risk:none
security
usability
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
Nectenda/nectenda!323
Loading…
Reference in a new issue
No description provided.
Delete branch "worktree-nec-346-a-degraded-dependency-took-down-all-sync"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Task: NEC-346 — https://projectron.nerchure.com/tasks/367
On 26 October 2026 the attachment bucket answered 503, a deploy restarted eu1, the startup probe threw, and every note's sync stopped; the deployer then rolled back to an identical build and posted incidents nobody had checked. The shard now starts with attachments degraded on a transient store failure (still refusing on misconfiguration), refuses attachment calls fast with 503 ATTACHMENTS_UNAVAILABLE, re-probes and recovers without a restart, and says so under
degradedon /api/ready (identity does the same for a shard it cannot pull). The plugin retries refused uploads on a timer. The deployer keeps the failed container's log, recognises a rollback that fails the same way as not the build (unless the rollback refused a migrated schema), names only its own service in incidents, and no longer redeploys a failed sha every minute. Reviewed by a fresh-context agent; its two high findings (boot backup skipped, fs backend guarded) and four others are fixed with mutation-checked tests. Not run on a real host: the docker logs and Kuma checks listed in docs/manual-testing.md.Spec:
docs/changes/NEC-346-attachment-store-degraded/spec.mdAlso fixes NEC-347: A failed attachment put leaves a charged row with no object, and reconcile never corrects it
Also fixes NEC-348: Object deletes that fail at the provider are dropped, leaving orphaned objects in the bucket
Also fixes NEC-349: deleteRemote forgets a pending attachment delete on any HTTP answer, 5xx included
Also fixes NEC-350: attestFolder's re-upload of ids the server lost is a no-op for unchanged files
Also fixes NEC-351: After a successful rollback the deployer redeploys the same failed sha every minute
Changelog
Notes keep syncing when the server's attachment storage is down: attachments wait and upload on their own once it is back.
The startup probe was one HeadBucket and one throw. On 26 October 2026 a provider's 503 on it stopped the shard, and with it every note's sync, although only attachments live in that bucket. The backend now answers `probe()` with what kind of unusable it found: a definite 4xx is misconfiguration and still refuses to start; a 5xx, a timeout or a socket error is transient (STORE-070). The classification reads what the store answered, never its message, and errs toward transient, since a shard that starts with attachments visibly degraded costs less than the outage. The S3 client also gets a connection timeout, an idle timeout and two attempts; it had none, so a provider that hung would have hung the probe. The handler's request timeout only warns unless `throwOnRequestTimeout` is set, which the test against a bucket that never answers found on its first run. The fake S3 gains a fault switch (`{ status }` or `'hang'`) for the tests of what the shard does when its store misbehaves. Task: NEC-346 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7On 26 October 2026 the attachment bucket answered 503, a routine deploy restarted eu1, the startup probe threw, and every note's sync stopped for a dependency the document path never touches. The shard now keeps one current opinion of its store (blob-health.ts): the probe at startup decides only what it is evidence of — misconfiguration still refuses to start, a provider failing starts the shard with attachments degraded (STORE-070) — and every remote call reports its outcome, so a store that dies later is noticed too. While the store is down, uploads and streamed downloads answer `503 ATTACHMENTS_UNAVAILABLE` with `Retry-After`, before the body is read and before a row or usage is reserved, and without a round trip to the provider (STORE-071); the document relay is untouched. A timer re-probes the store and it comes back without a restart (STORE-072); the SDK's idle and connection timeouts bound how long a hung provider can hold any call. Readiness learns the difference between the service being broken and a dependency being down (STORE-073): `/api/ready` answers 503 only for `database` and `disk`, and names a down dependency under `degraded` with 200, `{}` when nothing is, so a keyword monitor can watch for the healthy bytes. The identity service's "no shard manifest pulled recently" moves from a 503 with `reason: database` — the only reason the type allowed — to `degraded.directory`, per shard, with the error: sign-in works without the manifest, and the status page said sign-in was down for a shard outage. A startup failure is now written with `log.fatal`, a synchronous write to stderr (STORE-074). The logger already was synchronous on a Linux pipe; the guarantee is now the call's rather than the platform's. Task: NEC-346 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7`/api/ready` answers 200 while the shard's attachment store, or the identity service's view of a shard, is down, and names the dependency under `degraded` (STORE-073). The `ready` monitors and the public rows stay green, which is right: notes sync and sign-in works. Two keyword monitors for the healthy bytes `"degraded":{}` say what is actually down: `sync <shard> attachments` and `service accounts directory`. They are not public rows, and they need registering by hand with a re-run of `kuma-monitor.mjs add`. Task: NEC-346 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7The destructured `{ …, title }` made `title` required to the deploy package's typecheck, and every existing call omits it. Task: NEC-346 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AJMmbLhYmGG5gCsNiXxSo7