When the ticketing pipeline tickets itself

Measured 2026-09-19 on memex. MeshWeaver#3138 — one failed Postgres query stopped red logs being ticketed for the life of the pod — was fixed by 2e1d498338 : OpenIncidentWatch now re-opens the watch with backoff, unbounded in count and bounded in rate, instead of ending the sequence with .Catch(… ⇒ Observable.Empty).

Then the fix reported itself as a production incident:

No restart in between. That sequence is itself the proof the fix works — pre-fix, a restart was the only way the watch could ever answer again — and it is also a self-referential noise loop.

The loop, in one sentence per hop

1. The retry arm reported at LogError.
2. A .NET console LogError renders with the prefix fail:.
3. mw-log-watcher ingests fail:/crit: and nothing else (Red-log watching and automatic ticketing).
4. So the pipeline that turns red logs into tickets ticketed its OWN retry — and the fingerprint it landed on was the one already mapped to the issue about the defect that retry had fixed.

This is the rule #4257 landed one layer up, arrived at from the other direction: never assert a severity on a DETECTION when a remedy is about to run. Log the detection at Warning, and put the Error on the outcome.

Why the fix is a DURATION and not a demotion

The retry is unbounded in count by design — the watch's lifetime is the service's, not the first fault's — so onGiveUp may never fire at all. Demoting every arm to Warning would have removed the noise and, with it, any report of a ticketing blackout that lasts all night. That is a worse defect than the one being fixed: the pipeline that reports outages is the one thing whose own outage nothing else reports.

So the levelling is:

Five minutes because every occurrence ever measured on this pipeline was a single Postgres CONNECT timeout the next attempt healed — 18 of them over ten days in #3138, and the 2026-09-19 sample recovered in one second — while the backoff reaches its one-minute ceiling at attempt 7 (63 s cumulative). Five minutes is roughly ten attempts: unambiguously "the store is down", and short enough that an operator hears about a real blackout inside one on-call cycle.

The escalation rides on a fault rather than on a timer, so it is observed within one ceiling (≤ 1 min) of the threshold. That is deliberate: a timer whose only job is to watch a clock would be a second subscription to keep alive, and the down store is already producing a fault a minute.

Two decisions that are easy to get backwards

The reset happens AFTER the handler, not on arrival. "Recovered" has to mean the whole emission path worked. A query that keeps completing into an enqueue that always throws is a ticketing outage too, and resetting on arrival would hide exactly that case behind an endless row of warnings with no error anywhere. AnEmissionWhoseHandlerThrows_IsNotARecovery pins it.

The escalation carries NO exception object, though it names the fault inline. This is the part that actually kills the reopen loop, and demoting the message alone would not have: StructuralLogIncidentIdentity keys a burst on its top application FRAME when there is one and discards the reporter's prose, so an Error here that passed the store's TimeoutException would fingerprint as the Npgsql connect fault — the very identity that made the old retry line reopen #3138. With no stack trace the burst is keyed on the log SITE instead, which is the ticket an operator wants: the log-incident watch is blind, not Postgres timed out, which every other component reports better. Nothing is lost — the warning carries the exception in full, at every attempt including the escalating one, at the same instant.

The control

IncidentWatchSurvivesAFailedQueryTest, on a TestScheduler, no clock and no mesh — the composition stays internal static and scheduler-parameterised so the policy is an assertion rather than a timing observation. The three original facts (re-open, unbounded in count, bounded in rate) are unchanged; four were added:

• AShortBlip_IsWarnedAboutAndNeverEscalated — the production case. Before the levelling this fact's zero was a two.
• AnOutageThatLasts_IsEscalatedExactlyOnce_HoweverLongItRuns — one report per hour-long outage, and its duration is within one backoff ceiling of the threshold.
• ARecovery_ArmsTheEscalationAgain_SoTheNextOutageIsReportedToo — a latched error is a one-shot alarm dressed as a continuous one.
• AnEmissionWhoseHandlerThrows_IsNotARecovery — the reset ordering above.