The comment-debt clock — a fault that STOPS still gets its last update

MeshWeaver.Plugins#1836. A log incident keeps a ledger of what its GitHub issue has been told: Occurrences is what the incident has seen, OccurrencesAtLastComment is what the issue has heard, and LastCommentedAt is when. The difference is a comment debt, and the recurrence comment (LogIncidentFiler.RecurrenceComment) reports exactly that difference — "Still occurring: N more since the last update".

What was wrong

The decision to post that comment lived in one place: LogIncidentIngestService.NextRequest, evaluated inside Merge, which runs only while folding a fresh report for that fingerprint. Every rule was right — the issue link outranks the status, CommentDue rate-limits to one comment per CommentInterval, a Failed incident with an issue is re-asked, never re-triaged — and the comment still never happened for the incident that mattered, because no new occurrence ⇒ no ingest ⇒ the decision is never taken.

Measured on memex.systemorph.com, 2026-09-14: Admin/_LogIncident/0b1877ae6c51353f, the incident that owns core#3780, sat at occurrences: 69, occurrencesAtLastComment: 2, lastCommentedAt: null, status: Failed, with an error recording a 401 from a one-second credential wobble. The 401 only created the debt — a comment that was due, refused. The debt's only settlement trigger was the next report, and the fault had stopped. Core#3780's body still described occurrence 2 of 69; the five-burst crash loop that would have settled which of its two candidate causes was right never reached the thread.

Generalised: the incidents most likely to be left stale are exactly the ones that stopped — every fault that was fixed, rolled past, or took its pod with it. Those are the ones whose final count is the evidence for closing or escalating.

What the fix is — and why it is not a watchdog

The ledger already exists on the node. Only the trigger was missing, and for a fault that has stopped the only remaining evidence that time has passed is a clock. So LogIncidentControlPlane now carries one (OpenCommentDebtClock, fifteen minutes by default): on every tick it re-reads the incident snapshot the live watch already holds — no second query — and for every incident where

RequestedStatus == None
&& Status != Suppressed
&& IssueNumber is not null
&& Occurrences > OccurrencesAtLastComment
&& CommentDue(incident, options, now)

(NeedsCommentReconcile) it queues one reconcile. That is the same rule NextRequest applies on ingest, plus the one check ingest gets for free — a repeat report always just grew Occurrences, so "there is something new to say" never needed spelling out there. Off the clock it does: without it a ticketed incident sitting quietly past its window would be re-asked every tick to report "0 more".

The reconcile does not talk to GitHub. It writes exactly what a fresh report would have written — RequestedStatus = Comment — recomputed against the LIVE node inside the update lambda (CommentDebtOutcome), so a debt that a real recurrence has already paid off by the time the work item runs is left alone. The ordinary live watch then picks the changed node up, and Claim → Perform → LogIncidentFiler.Comment → the ledger write-back run unchanged. That reuse is the point: the claim consumes the request on the live node before the outward call, so two replicas ticking at once (the same field, the same value — one ask), or a tick racing a real recurrence, resolve to exactly one comment — at most one performer per ask, the same guarantee an ingest-triggered comment already has, not a second, independently written path to GitHub.

What that guarantee does not cover, stated so nobody reads more into it. The Comment leg has no durable in-flight state: the claim clears the ask, and the ledger moves only when GitHub has answered. Inside that one round-trip the node reads "nothing requested, debt outstanding, comment due" on every replica, so a tick on another replica in that window can mint a second ask, and if its claim also runs before the first write-back lands, GitHub hears the same delta twice. That is the identical window the ingest path has always had for a report folding in mid-round-trip (NextRequest reads the same ledger) — the clock did not introduce it; its width is one GitHub round-trip per settlement, and its consequence is one duplicate comment with the same count. A durable in-flight marker would close it — and would reopen the very defect this clock fixes: a pod that dies inside the round-trip leaves the marker set, the clock then excludes that incident for ever, and a stopped fault is frozen again with no report coming to clear it. The only way to unfreeze it is a lease timeout, which is a watchdog. A rare duplicate comment is the cheaper wrong.

The comment is stamped in fault time, on purpose. LogIncidentFiler.Comment writes LastCommentedAt = LastSeen — the incident's own clock, the one CommentDue reads — so a delayed comment for a fault that stopped two days ago is dated two days ago, and a recurrence tomorrow is two days past the window: it comments at once, as "the fault is back after quiet" always has. That is the same sequence ingest would have produced had it settled the debt on time (comment at LastSeen, quiet, then "it's back"); stamping wall-clock time here would swallow the "it's back". A recurrence five minutes after the last occurrence is still inside the window in fault time and stays silent. Both are pinned.

This is the root-cause fix, not a band-aid, because the defect is the absent trigger: nothing failed, nothing needs retrying, and no state is invented — the clock only asks a question the node already answers.

What it deliberately does NOT do

No retry loop around the 401. The credential recovered by itself within the hour; a tight retry would have hidden the event and left the debt exactly where it was. When a post is refused, RecordFailure still parks the incident at Failed with the reason and rings the platform bell once per refusal episode (Systemorph/MeshWeaver#4022). The clock changes none of that; it only means the debt is re-examined every fifteen minutes instead of never — four token requests an hour per indebted incident during a credential outage, strictly gentler than the ingest path, which re-asks on every report while LastCommentedAt stays null.
No backoff, no budget. A budget would be a slower version of the very defect this fixes — a debt nothing ever comes back for. The cadence is the bound.
No second subscription. The clock reads the snapshot Enqueue caches on every emission of the one incident query; a tick costs one in-memory scan.
No override of an explicit request. RequestedStatus != None declines; an admin's Suppress, an agent's File, a claim in flight — all outrank the clock.
No new way to blind the watch. The clock is merged into the same chain as the live incident watch, so a fault escaping a tick would have terminated both — the #3138 outage shape, reintroduced through a timer. Each tick has its own error arm: a throwing sweep is logged at Error (which the log watcher tickets) and the next tick runs on schedule.

Where it is pinned

CommentDebtReconcileTest — the predicate, the outcome's no-op shape, that the clock ticks on a TestScheduler with no node change anywhere, that a throwing tick is reported and does not end the clock (its negative control — the guard removed — stops the clock at the first throw), and the fault-time rate limit after a clock-settled comment (a recurrence two days on asks at once; five minutes on stays silent).
CommentDebtReconcileMeshTest.AnIncidentWhoseFaultStopped_StillGetsItsOutstandingCommentOnTheClock — the pin: the measured shape (69 / 2 / null / Failed on a 401) on a real mesh, the control plane STARTED with a short period and left alone, the comment landing through the live watch and the ordinary claim, once. Its negative control (the clock built but never subscribed) is recorded on the pull request: the read for LastCommentedAt times out, exactly the pre-fix behaviour.
The three seam facts in the same class drive RunCommentDebtReconcile with no plane started: the ask IS written on an indebted incident (the positive control), and is NOT written when the live node has already settled or the incident is Suppressed.

Operating it

Nothing to configure. The clock starts with the control plane (which is idle without a LogWatch:DefaultRepository or LogWatch:Routes, so it never runs where nothing can be filed). A settled debt shows up as a recurrence comment on the issue and occurrencesAtLastComment catching up to occurrences on the incident; a refused one shows up where refusals already do — the incident's error, the platform bell, and the [IssueRefused] warning.