The Pod-Hub Claim Must Be Re-Asserted

Read Pod-Hub Delivery — the Transport Swap and its Roll Plan first. This page is the bill that plan's own ledger predicted and did not price.

The one-sentence cause

A pod-hub claim publishes an address→silo mapping into Orleans' grain directory, that directory is re-partitioned on every cluster membership change, and the claim was asserted exactly once.

Everything else on this page follows from those three facts standing together.

How a pod-process hub is reachable at all

A hub that lives in a .NET process rather than in a grain — mesh/{meshId}, cache/{meshId}, portal/nodeops-{meshId}, import/{meshId}, every portal/{circuitId} — is reached by a directed call to IPodHubGrain, whose identity is the address. The owning process creates that grain's activation itself (OrleansRoutingService.AttachPodHub → IPodHubGrain.Attach, under [PreferLocalPlacement]) and pins it, so Orleans' single-activation guarantee turns its own grain directory into the address→silo map — with no directory of ours to write, keep durable, or lose.

That is an elegant trick and it is load-bearing. It also means the map has exactly the durability of the grain directory, and no more.

The two properties that turn a lost entry into a permanent one

1 · [PreferLocalPlacement] places on the CALLER. When the directory holds no entry for an address, the next router to deliver to it does not fail — it creates the activation on its own silo. That silo has no local route, so it answers PodHubNotHereException and deactivates. The next delivery does the same. Every refusal re-creates the condition that produced it, from whichever router happens to call next.

2 · The refusal is reported to the SENDER, never to the owner. RoutingGrain NACKs the message's sender with a transient DeliveryFailure. The one process that could repair the mapping — the owner — is by construction the process the router could not reach. Nothing tells it.

Put those together and a mapping that is lost once is lost for the life of the process.

What was measured

memex-cloud, 2026-09-01, via Loki (31 d retention, so the windows below are fully covered) and kubectl get:

The startup population and the lost-after-landing population are two different triggers of one cause. Neither recovers, for the same reason: there is no second assertion.

The user-visible face of it: /api/content answers 503 (#2901)

The measurement above is stated in routing terms. This is what the same fault looks like from a browser, and it is worth writing down because the shape was mis-attributed twice before the routing cause was found — once to a post-upload settle window, once to the memory PubSubStore and the streams-retirement program (#1729/#1742).

/api/content/{node}/{collection}/{file} resolves the owning node's collection config with one GetDataRequest per request — deliberately uncacheable, because that round trip IS the permission check (ContentFileResolver.Resolve). It is issued from portal/nodeops-{meshId} and bounded by ReadBudget.Default (10 s). When the reply cannot get back, the budget is the only terminal, and the route answers 503 after ~10.3 s, per request, for as long as the process lives.

The chain, measured end to end on memex-cloud, 2026-09-02

Rows 3 and 5 together are the whole issue: a membership change moved the grain directory, one process's entry did not survive it, and every answer addressed to that process was refused from then on.

The two corrections this measurement makes

It is not a settle window. Nothing about a freshly written file is involved. The discriminator is which replica served the request, which is why the same file alternated 200/503 across round-robin attempts and why a single-replica probe cannot see the fault at all.

It is not the memory PubSubStore. The refusal line says in its own words that the message was NOT posted to the Orleans stream — the directed pod-hub call replaced that publish (PodHubDeliveryRollPlan). A discarded stream publish is a different failure with a different fix, and attributing this to the streams-retirement program would have parked a closed bug behind an open programme.

How to re-run it

Two Loki queries, no mutation, and the first one alone settles the family:

sum by (namespace,pod) (count_over_time({namespace=~"memex|memex-cloud"} |= "Content read timed out" [24h]))
sum by (pod)           (count_over_time({namespace="memex-cloud"} |= "portal/nodeops-" |= "was refused" [1h]))

🚨 The second filter is portal/nodeops- — the prefix, deliberately, with no mesh id. Every pod-hub address shares it, so the query runs as written against any deployment. An earlier draft wrote portal/nodeops-<meshId>, which returns zero rows unless the reader first substitutes a value they do not have yet — and a zero-row answer here reads exactly like "the refusals have stopped", which is the one conclusion this page exists to prevent someone drawing by accident. If you want a single mesh, add |= "<meshId>" as a third filter once you have one from the first query's output.

🚨 sum by (pod) is the whole method. An aggregate count reads as a low-grade cluster-wide flake; per pod it is one replica at 33 and six at zero, which names the cause. See Measuring a Live Portal Read-Only.

What closes it

Re-assertion repairs the mapping at the next membership change, so the same measurement after the roll must show the failures spread across replicas or absent, never concentrated on one pod again. A fresh single-pod concentration would mean the re-assertion did not land for that address, and the next thing to look at is its ORDERING against the directory re-partition — the claim fires on the membership notification, which is not the same instant as the re-partition completing.

🚨 A green probe through a load balancer in front of one replica has no power to detect a per-replica fault. The replica count is part of the verification criterion, not context for it.

Why the fix is an event and not a retry

The claim already retried indefinitely on failure. That was never the gap. The gap is that landing was treated as a terminal: the moment Attach first answered true, the claim stopped for the life of the registration, even though the structure it had written into can be re-partitioned underneath it minutes later.

So the claim is now re-asserted on every cluster membership change — IClusterMembershipFeed, fed on the silo by Orleans' own ISiloStatusListener:

ClaimTriggers()            // after Active, then once per membership change
    .Select(_ => ClaimOnce())
    .Switch()              // exactly one claim in flight per address, ever
    .Subscribe(…)

This is not a watchdog and not a poll. It fires on the specific event that can invalidate the assertion, which makes the claim's lifetime DERIVED (the rule from #2426) rather than bounded by a counter or by a single success. Orleans itself already makes exactly this move: ClientDirectory re-publishes its whole client routing table to every silo on every membership change, for precisely this reason.

Switch rather than Concat or Merge matters twice. A membership change makes every placement decision the in-flight round has already taken stale, so the new round must replace it rather than queue behind it; and it bounds the work absolutely — one claim in flight per address no matter how fast membership churns, so a scale event cannot become a claim storm.

Where no feed is registered — an Orleans client, the Monolith, a bare mesh in a test — membership cannot change under the process, so the claim is asserted once, exactly as before.

Two windows in which the trigger was dropped anyway (#3931)

Re-asserting on the right event is necessary and was not sufficient. The trigger could be lost in two places after the design above was in, and in both the outcome is indistinguishable from never having had re-assertion at all: no claim, no retry, no log line, and nothing left to re-make the mapping until the next membership change — the #2938 state this page exists to remove.

🚨 Both are DROPS, not delays. The grain-side attach count never moves, so "the claim was made and its result went unobserved" is excluded by the instrument itself: the count is published from inside IPodHubGrain.Attach.

Window one — StartWith subscribed the feed only after the first claim round had run

ClaimTriggers composed the initial assertion as Changes.StartWith(0L). StartWith is a Concat: the source is subscribed only once the prefix has been fully processed, and processing that prefix runs the entire first claim round — Attach included — on the subscribing thread. IClusterMembershipFeed.Changes is hot and deliberately does not replay, so a change arriving in that window was dropped where it was published: Subject.OnNext with no observer is a no-op.

The window is the first claim of a hub's life — i.e. the pod's boot, which is exactly when membership moves. #3983 measured 34 placement failures in 65 s across two pods, all inside it.

Measured on System.Reactive 6.1.0 (the pinned version) with the real shape (…ObserveOn(Scheduler.Default).SelectMany(_ => triggers).Select(round).Switch()) and a round that parks inside its subscribe, so the window is observable rather than inferred:

StartWith: ROUND-0-enter -> PUSH-enter -> PUSH-leave -> RELEASE -> ROUND-0-leave -> SUBSCRIBE-feed
Merge    : SUBSCRIBE-feed -> ROUND-0-enter -> PUSH-enter -> RELEASE -> ROUND-0-leave -> ROUND-42-enter

Under StartWith the push completes with the feed still unsubscribed and no round for it ever runs. The fix is Changes.Merge(Observable.Return(InitialClaim)): Merge subscribes its sources left-to-right, so the durable source is listening before the one-shot initial trigger can start any work, and the merge gate then serialises the two — a change arriving mid-round queues behind it instead of racing it.

Window two — the claim/dispose handshake answered a transient condition terminally

The handshake protecting Attach/Detach ordering was a single token. A round that found the token at "someone is already starting an attempt" answered Observable.Empty — a terminal answer to a transient condition. A retry round re-subscribes on its backoff timer's thread while a membership round arrives on the feed's, so the overlap is ordinary, and it is most likely exactly during churn, because churn is what makes a claim bounce in the first place.

It is worse than a dropped trigger: Switch has already cancelled the round it overlapped, so the membership change destroyed an in-flight claim and made none of its own.

The handshake is now a ledger, not a mutex. One interlocked word carries "disposal has been requested" in bit 0 and the number of rounds currently inside their synchronous attach window in the rest. The only reason a round may refuse to claim is disposal; whoever observes "disposed and the window is empty" owns the release — Dispose when it finds the window empty, otherwise the last round out — so Detach can never overtake an Attach that is still being invoked, and concurrent rounds are ordinary rather than an error.

What pins it

PodHubClaimReassertionTest carries both directions, and both are deterministic rather than probabilistic: the test grain PARKS inside a nominated Attach call, so the contention is created rather than waited for.

The third is not decoration. A "fix" that merely re-triggered the round would satisfy both positive pins and reintroduce the claim storm Switch exists to bound: every attempt makes the owning silo log a line, which is the log-storm shape #2426/#2546 exist to remove.

The diagnosability that was missing

For twelve hours the production line read:

Directed delivery to pod hub 'X' was refused: no silo in this cluster is currently serving that hub.

That sentence covers two different faults with two different fixes, and nothing in the log separated them:

• the owner's claim genuinely is not held, or
• the grain directory has no entry at all, and prefer-local put a throw-away activation on the router's own silo.

The refusal now names the silo whose activation answered (PodHubNotHereException.RespondingSilo). When that is the silo printing the line, it is the second case. And the claim's own refusal — Attach answering false — now says so in its own words instead of borrowing the wire-level sentence, which is what made eight days of logs ambiguous.

🚨 The owner-side half of this failure is invisible in production by design. PodHubGrain's step-aside and refusal lines are Information, and the successful claim is Debug; the deployed log configuration ships neither. That is the correct cost model — those lines are per-delivery — but it means an investigation must reach for the router-side Warning, and that line therefore has to carry the identifying facts. Do not "temporarily" raise a level to investigate; add the fact to the line that already ships.

What this does not fix

Re-assertion repairs a lost mapping at the next membership change. On a cluster that is completely static it would not fire — the mapping would also not be at risk there, since the directory is only re-partitioned when membership moves, but that is an argument from the mechanism rather than a measurement, and it is stated here rather than glossed. A stronger form (the owner detecting unreachability directly) has no seam today: the refusal is raised on a silo that does not know who the owner is.

The claim's placement during silo startup is now ordered on the same OrleansStreamingReadiness signal as the stream subscription (lifecycle stage Active). Before #3983/#3984, RegisterStream issued the pod-hub claim immediately, unordered. The eagerly-registered mesh/{meshId} and cache/{meshId} hubs therefore called IPodHubGrain.Attach before this silo — and sometimes before any silo — advertised the grain type. Orleans rejected the invalid placement with Known nodes with grain type: none; its per-attempt logger produced #3983 while the exhausted Polly call produced #3984. The claim now waits for Active and then moves off the lifecycle thread before touching the grain factory. The membership-change re-assertion remains the repair for a mapping lost after startup.

Related

• Pod-Hub Delivery — the Transport Swap and its Roll Plan — the transport, and the "what the swap traded" ledger this page settles.
• Orleans Stream Pub-Sub Durability — why the transport exists.
• Measuring a Live Portal Read-Only — the method every number above was produced with.
• Error Propagation & Wedges — a silence is a wedge.