One log site, one ticket
Measured 2026-09-21/22 on Systemorph/MeshWeaver. One log site — [ROUTE] Routing back-pressure, category MeshWeaver.Hosting.Orleans.RoutingGrain — held 46 open issues. Forty-two
of them were closed by hand as duplicates of #5118, and six more arrived within the hour while
that was being done. Identical-title pairs (#5325/#5326, #5330/#5331) show the same shape filing
twice in the same minute.
What was wrong
LogIncidentFiler.File validates the configuration, resolves the repository from the category
route, builds the labels, and calls repoClient.CreateIssue(...). Nothing anywhere on that path
looks for an issue that already exists.
The system was not undefended — it was defended one level too low. LogIncidentControlPlane.ClaimRequest
makes filing idempotent per incident:
if (request == LogIncidentRequest.File && current.IssueNumber is not null)
return new LogIncidentClaim(/* granted as a Comment */ …);
That is one incident node, one issue, and it is exactly what stopped ROUTER_TRAFFIC being filed
eight times in seven minutes. It cannot see a second INCIDENT: a split fingerprint is a different
node, with no issue number, and it files.
So the defect is not in the filer's dedup — it is that the unit of ticketing was the fingerprint,
and the fingerprint deliberately splits. The incident identity
discriminates on the fault's masked DETAIL, because thirteen NodeTypes parked at CompileError are
thirteen independently-actionable defects. That split is right for triage and wrong for filing:
masking cannot anticipate every message shape (see
The portal masks what it hashes), so a site whose
messages carry an unmasked token fans out into many fingerprints — and, before this, into as many
tickets.
The key is the site fold
StructuralLogIncidentIdentity.ComputeSiteFold already existed, as the aggregator's bound on a site
that fans out past its per-window budget. It hashes WHERE the fault is and WHAT it is, and drops
the detail — which is precisely the axis the splits happen on. So it is stable across exactly the
divergence that caused this, and it is what the filer folds on.
It is now stamped on every incident as LogIncident.SiteFold, by the INGEST, from the report's own
re-parsed evidence and the same majority vote the node id is resolved by — never recomputed later
from the stored, reporter-masked text, which is the defect #2218 closed. It is healed when absent
and never re-addressed once set: the site record is keyed on it, so moving it would strand the
issue the incident folds into.
Why a node, and not a query
The question "does this log site already have a ticket?" cannot be answered by a read that may be
stale, because the write a negative selects is opening an issue. That is the #2229 failure shape
pointed straight at the duplication being removed: Query/ObserveQuery are eventually consistent,
a stale negative is indistinguishable from a genuine absence, and two filings seconds apart would
both miss.
A GitHub issue search is no better, and worse in one respect: search/* carries its own limit of
roughly thirty calls a minute, and its index lags by seconds to minutes — so it fails in the same
direction, under load, exactly when the site is noisiest.
The answer is the shape LogIncidentIngestService.RecordBurst already uses for the incident itself:
- one deterministic record per site, at
Admin/_LogIncident/site-{siteFold}— the same namespace as the incidents, which is already provisioned and access-controlled, and cannot collide (a fingerprint is sixteen hex characters; a site id startssite-); - created with a create-ONLY write, so the STORE decides who owns the site. Refused as "already exists" ⇒ somebody owns it, and the record says who and what came of it;
- read back by path through
GetMeshNodeStream, which is authoritative and live; - every decision taken INSIDE the owner's serialised update lambda, so two deciders can never both take the claim.
What happens, case by case
| The site record says | What happens |
|---|---|
| nothing (no record) | the create lands ⇒ this incident owns the site, files as ever, then stamps the record |
| an issue, OPEN | the incident adopts it and comments — the existing Comment path, not a second one |
| an issue, CLOSED | the same Comment path, so ReopenOnRecurrence and the close-reason/close-time rules decide (Reopening a closed issue) — no new policy is invented |
| a claim taken seconds ago, no issue yet | wait on the record's own stream for the issue number — the actual condition, never a poll — for at most the claim's REMAINING lifetime (ClaimedAt + SiteClaimTimeout − now), then re-decide in the serialised update either way, so the waiter is appended to the fold ledger like any other fold |
a claim older than LogWatch:SiteClaimTimeout |
taken over: the holder died between claiming and stamping, and one crash must not make a site un-ticketable for good |
| an issue this incident just proved GONE | the record gives up its issue, starts a fresh fold ledger (the old one's comments died with the dead ticket), and this incident re-files for the whole site (A vanished fold-in target) |
The stamp is fenced. Only the incident that CURRENTLY holds the claim may stamp its issue onto the record. A holder that was slow rather than dead completes after its claim was taken over; an unconditional stamp would repoint the site at whichever completion wrote last and orphan the other ticket. A stale completion changes nothing and logs a Warning naming both tickets, so the orphan can be closed as a duplicate.
A ticket that predates site records is adopted. When an incident that already owns a ticket comments on a recurrence, it seeds its site's record with that ticket (create-only: a site that has a record keeps it). Without this, the first NEW fingerprint of every pre-existing site would find no record and open one more duplicate beside the ticket already there. | the incident carries no site fold at all | filed exactly as before — nothing is read or written |
SiteClaimTimeout is a crash bound, not a retry budget. The claim is written immediately before
the GitHub round-trip and stamped immediately after, so the only way it stays unstamped is a portal
that died in between. Raising it makes a dead claim block its site for longer; lowering it below a
round-trip lets two replicas both file.
Folding must not become discarding
A folded incident never gets a ticket of its own, so everything that ticket would have carried has to arrive on the one that exists. The fold comment opens by saying what is being folded and why, and carries the folded incident's own distinguishing evidence: its fingerprint, the log site, the category and severity, the exception, the top frame, the masked detail that SPLIT it, the namespace, the pods, the occurrence count, its window, how many fingerprints are already folded here, the incident node path, and what triage wrote about it. The site record additionally keeps the ledger of every fingerprint folded onto the issue.
The ticket the site's FIRST incident opens now says so too: "It also stands for the whole log site
…" — otherwise the first fold comment reads as an unrelated fault arriving on somebody else's
issue.
Splitting a fold back out stays a person's call, and everything that call needs is on the ticket.
What this does not do
- It does not merge two log sites. A different category, a different top frame or a different exception type is a different site fold and still gets its own ticket — the control that stops "never file twice" from passing by never filing at all.
- It does not retro-fold the issues that exist. Those were consolidated by hand; the incidents behind them acquire a site fold on their next burst, and the first of them to recur (comment) or file for that site becomes the record's issue.
- It does not change what a recurrence of an ALREADY-ticketed incident does. That path — the comment rate limit, the comment-debt clock, the transfer following — is untouched.