The portal masks what it hashes

MeshWeaver.Plugins#2218. Two defects in the incident-to-ticket pipeline, found from the same measurement: on 2026-09-21 Systemorph/MeshWeaver had 94 open issues opened by systemorph-com[bot], all minted between 2026-09-19T11:44Z and 2026-09-21T09:00Z, and 42 of them were one log site. None of the 94 carried a severity when it was opened.

1. A fold that could never see the fix

What was wrong

LogIncidentIdentityResolution exists because the identity function used to live in mw-log-watcher, a separately shipped image, so a corrected identity reached production only when somebody rolled it — measured five weeks and four revisions behind. #1796 moved the function into the portal, which rolls continuously.

It moved the function and not the function's input. The identity is computed over report.NormalizedMessage and report.NormalizedDetail, and both are produced by LogLineParser.Normalize inside the watcher. So every masking revision since — the episode stamp and the 8-character activation id (#2165), masking by SLOT rather than by what a token looks like (#2167, #2168), a list's length (MeshWeaver#2157), a nodeType: term (MeshWeaver#3545) — shipped in MeshWeaver.Observability.Contract, was present in every portal, and was dead code on the ingest path. The same defect #1796 closed, one layer down.

The measurement

Admin/_LogIncident/54ecfdaea23fd110 on the control instance stores

[ROUTE] Routing back-pressure [e9b2{n}fb#{n} started {time}]: {n} route dispatches in flight …

The activation id is e9b212fb. It was masked only where it happened to contain digitse9b2{n}fb — so every grain activation normalized to a different string, hashed to a different fingerprint, opened a different incident node and produced a different GitHub issue. Three facts, each reproducible:

The portal's identity function over the watcher's stored text 54ecfdaea23fd110exactly the live node id
The same function over the same bursts' RAW lines, re-normalized here 96c85f04cadba7f7 for 42 of the live fingerprints
Live fingerprints whose own samples agree here / distinct identities they resolve to 70 → 26 (the other 16 still hold samples that disagree — pre-existing masking gaps, unchanged)

So the identity function was live and correct; its input was five weeks stale. The agent triaging those incidents had already written the diagnosis into its own tickets — "the fingerprint appears to split per episode stamp, so every crossing opens a new ticket" — which is what the ticket flood looks like from inside it.

The fix

The report already carries its own raw evidence: LogSample.Line is the whole verbatim burst, header and continuation lines, exactly as BurstAggregator.Aggregate joined it. The portal re-parses that with the LogLineParser it ships and hashes its own normalization.

Nothing in the watcher changes, and nothing needs to be rolled for the fix to take effect — which is the property the whole arrangement exists for.

What happens to the 42

Nothing here closes them; the corpus migration is what carries a re-addressed incident's ticket, counts and history onto its successor, keyed on the reporter's own fingerprint. The next crossing of that site lands on the folded identity and brings one of them with it; the rest are a backlog a human closes as duplicates.

2. No automated bug was ever inside the release gate

What was wrong

The issue taxonomy puts a severity on every bug and makes two of the four a release gate: sev:B and sev:H must both be zero to cut a release, and a bug carrying neither "has not been triaged, and working it ahead of a labelled sev:H is choosing by accident."

The pipeline had never assigned one. All 94 open bot-filed issues were opened with bug and nothing else; the 23 that carry a severity were labelled hours or days later by a person — issue #4824 got bug from the App at 2026-09-19T11:51:05Z and sev:M from a human at 2026-09-20T08:18:03Z. The LogTriage agent's instructions spelled the label set out literally as ["bug"], so there was nothing for the model to get wrong and nothing to fail. 71 tickets sat in a queue no gate reads.

The fix, in two halves

The default is deliberately sev:M and deliberately not a blocking class. A machine that did not judge must not be able to hold the release line — one noisy log site would stop every cut — and it must equally not be sev:L, which parks a real defect in a backlog nobody reads. A draft that proposes several severities keeps all of them: choosing between two labels a human can see is a judgement this code has no basis for, and silently dropping one would hide that triage was unsure.

One legacy incident, fifty-two successors

The corpus migration records the same finding from the other side, and it is the cleanest statement of it. When the portal's identity re-addresses an incident, LogIncidentCorpusMigration carries the legacy node forward and every successor's ticket says which legacy id it inherited. 52 of the 94 open bot-filed issues name the same one — e4a97855ab595beb.

That id is the REPORTER's fingerprint for the whole RoutingGrain back-pressure site: the watcher, with all its masking flaws, still called it ONE fault. The portal then split it 52 ways. So the split is provably not in the reporter's grouping and not in the identity function — both agree the site is one thing — it is in the TEXT the portal hashed, which is the reporter's and stale.

It also explains a sentence those tickets carry that reads like a contradiction: "superseded and will not fold, file or comment again". That is true, and it is about the LEGACY node, which is correctly retired. Fold's alreadyMoved guard means only the FIRST successor inherits the ticket and the counts; every later one gets provenance only — and then files an issue of its own. 52 tickets carrying that sentence is the defect's signature, not a second defect.

The same id, masked eleven different ways

One listing of Admin/_LogIncident is the whole finding, because the node NAME carries the normalized text. These are all the same 8-hex activation id at the same log site, as the stale watcher masked it:

[{n}f5{n}ea#{n}   [b5{n}d3{n}e#{n}   [df5d5f6{n}#{n}   [{n}a3d7#{n}     [{n}b2c7c6c#{n}
[aae3{n}b#{n}     [{n}c7f0ee#{n}    [{n}b0{n}bd1#{n}  [{n}c2{n}#{n}    [{n}d6fde0#{n}
[{n}b6{n}d4{n}f#{n}  [{n}d8{n}e3#{n}   [{n}d#{n}      [df8ca0{n}b#{n}  [e9b2{n}fb#{n}

The bare-number rule ate whichever runs of digits the id happened to contain and left its hex letters standing as literals, so the masked text is a function of the id — which is exactly what masking exists to prevent. EpisodeStamp and MixedHexId (#2165, #2167) mask the whole token; they have been in this assembly since 2026-09-19 and had never once reached the hash.

The flood stopped for the wrong reason

Incident DETECTION on the control instance stopped around 2026-09-20T22:30Z and had not resumed 15 hours later: content.lastSeen:2026-09-21* returns 0, not truncated, over coverage.partitions: ["admin"], while content.lastSeen:2026-09-20T2* is still truncated at 50. POST /api/log-incidents is the ONLY way a burst enters the portal, so with the watcher silent no report arrives, no incident is minted and no File is requested — which is why the last tickets (#5058–#5063) were filed from incidents detected BEFORE the stop and the flood appeared to have ended. Nothing was fixed by that outage. This fix is at ingest, so it applies to the first report after the watcher is alive again — alive, not rolled, which is the entire point of it.

And it came back, still splitting. The same query at 2026-09-21T15:41Z returned ≥10 (truncated), with incidents whose lastSeen is 15:40:49Z — so detection resumed some time between 13:31Z and 15:41Z. The first RoutingGrain nodes minted after it resumed are 0fc7665eaf6aae43 ([cf0fc5ed#{n} …), 18104cb0dbf5d912 ([d4cae9e3#{n} …), 0fda5918d57c6050 ([{n}f7{n}a#{n} …), 3708a213a82ec7b0 ([{n}c9{n}d#{n} …) and 4db1545b366d9574 ([a6{n}b4{n}#{n} …): five activation ids, five incidents, one log site, masked five different ways. The defect is not historical and the flood is resuming as this is written.

🚨 And the watcher's liveness is unobservable from the portal by construction: it has no Hosting/Deployment record (namespace:Deployments scope:descendants returns only build, memex, memex-cloud, pearl), so nothing reports its image, samples it or can roll it — and every signal that it has stopped is emitted BY it, including log-pipeline-behind-* and the silent-window finding whose job is to announce exactly this. A missing self-finding is evidence FOR death, never for health. The filer does NOT share that shape — it is a BackgroundService inside the portal, so it rolls with the portal and is covered by the portal's own record and /health.

What this does NOT fix

A second, independent duplication mechanism is still live, and it accounts for the other 8 of the 94: one incident node, several GitHub issues. Admin/_LogIncident/54ecfdaea23fd110 alone carries issues #5017, #5018, #5019, #5020, #5021, #5022 and #5060 — six of them inside 32 seconds — with identical firstSeen, lastSeen and occurrences in every body, i.e. filed from byte-identical incident state, while the node holds only four triage threads. Two other fingerprints did it inside ONE second: #5031/#5032 (4d74fa633047b387, both 22:23:03Z) and #5034/#5035 (2fe3118d64124621, both 22:28:53Z).

The claim that is supposed to make filing happen at most once (ClaimRequest) decides on the content the update lambda reads from this hub's mirror and then emits an RFC 7396 merge patch. Two merge patches carrying the same fields do not conflict, so two workers reading the same pre-claim mirror both grant themselves the File and both open an issue. It is filed separately rather than fixed here because the remedy is an atomic claim — a different change, in a different layer, with its own controls.

Where it is pinned

Reconnecting…
The server was updated. Reloading the page to pick up the latest version.