The Routing Entry Point

The question this page answers: on an Orleans mesh, every message that is not for a locally registered stream is routed through IRoutingGrain, a cluster grain. Placement of that grain is a dependency on the cluster being healthy — and it is the step that drops live traffic during a roll. Where the caller and the router are in the SAME PROCESS, that dependency buys nothing.

What the entry point is today

Scope. This is the ORLEANS path only, and only its fallback. OrleansRoutingService.DeliverMessage first checks the streams registered in this process (portals, in-process clients) and hands a hit to that callback directly — no grain is involved. A Monolith mesh (UseMonolithMesh, which registers MonolithRoutingService) never touches IRoutingGrain at all. So an incident from a Monolith, or a delivery that hit a local stream, is never evidence about the placement described here.

On a miss, OrleansRoutingService.DeliverMessage ends in one call:

var grain = GrainWhileRunning<IRoutingGrain>("default");
return Observable.Defer(() => grain.RouteMessage(delivery).ToObservable())

RoutingGrain is [StatelessWorker(1)] and non-reentrant: one routing turn per silo. Its turn already does O(1) work and returns — path resolution, the stream post, the per-node grain hand-off and the NACK all run OFF the turn through IIoPool (issue #1028, after a RouteMessage turn was measured executing for 06:00:22 behind NonReentrancyQueueSize=541). So the turn is a dispatcher, not a worker, and it is not the bottleneck.

The bottleneck is reaching it at all.

🚨 Placement fails in two measured ways, and both drop live traffic

A [StatelessWorker] is placed through StatelessWorkerDirector → PlacementService.GetCompatibleSilos, which intersects with the ACTIVE silo set.

1. The silo is leaving — "no active nodes"

The instant a silo begins graceful shutdown it leaves Active, so every placement throws OrleansException: No active nodes are compatible with grain routing. The silo is still running and still processing; it simply may no longer take new activations.

Measured on memex, 2026-08-10: on all three pod shutdowns the first such exception landed within half a second of the host logging "Application is shutting down…" — 11:44:31.341 → 11:44:31.750 — and then repeated 52, 838 and 944 times until the process exited. Every one was an Orleans.Messaging[100071] error AND a terminal DeliveryFailure to the sender. The traffic was ordinary live routing — activity heartbeats, node streams. Nothing was wrong with it except that the silo underneath was going away.

This half is already mitigated, and the shape of the mitigation is the point: IHostApplicationLifetime.ApplicationStopping fires strictly BEFORE the silo hosted service stops, so the router never attempts the placement it knows cannot succeed and answers the sender immediately with the transient ShuttingDown verdict. The fix was to stop asking.

2. The cluster is busy — "placement operation timed out"

Measured 2026-09-20 22:25:14Z, one pod:

fail: Orleans.Messaging[100071]
  Failed to address message Request
    [sys.client/hosted-10.244.9.229:11111@148947182] -> [routing/default]
    IRoutingGrain.RouteMessage(IMessageDelivery) #25BED5B67E2EBFF8
  System.TimeoutException: Grain placement operation timed out for grain routing/default.
   ---> Polly.Timeout.TimeoutRejectedException: … timeout of '00:00:30'
     at PlacementService.PlacementWorker.ExecutePlacementAsync(…)

331 real deliveries to that one grain, inside a ~50 ms burst, all dropped. They are genuine messages — RouteMessage(IMessageDelivery), with incrementing ids, so 331 distinct deliveries and not one message retried. They never reached a turn: "failed to address" is upstream of the grain.

🚨 Why a stateless worker's placement could wait 30 s at all: it asked the grain directory

The obvious reading of failure mode 2 — and the one issue #5037 carried for days — is that a [StatelessWorker] placement is purely local ("no directory lookup and no remote hop"), so a timeout there can only mean the local silo was not running work. The Orleans 10.3.1 source says otherwise. PlacementService.PlacementWorker.ExecutePlacementAsync begins EVERY placement with GrainLocator.Lookup(grainId), stateless worker or not. On the portal's default directory that is LocalGrainDirectory.LookupAsync, which forwards to the silo that owns the grain id's hash on the directory ring. Only after that answer does StatelessWorkerDirector pick the calling silo.

So reaching routing/default waited on some other silo's directory partition, for an answer that can only ever be "not registered": Orleans itself declares StatelessWorkerPlacement.IsUsingGrainDirectory = false and never registers a stateless-worker activation. The address cache normally hides this, but it is dropped on every membership change for grains whose partition owner changed — i.e. on every roll, exactly when an owner is most likely to be a silo that is booting, starved or gone. The lookup is retried as transient until the 30 s Orleans.Placement budget expires, and every message queued on that one placement work item is rejected together — which is why the failure arrives as a burst of hundreds inside milliseconds.

Measured on memex-cloud, 2026-09-23 (image 9a61fed964, before #5571): the 18:11:19Z burst (26 placements timing out at the same millisecond on 7d7f4d84f9-wl8mv, more on -mvzb5) follows twelve minutes in which a THIRD silo, -n7g6b, logged LocalSiloHealthMonitor warnings every 10–13 s (17:58:43 → 18:10:27Z) and then went silent. The pods that dropped traffic were not the pod that was stalled — they were waiting on its directory partition.

Fixed for stateless workers (#5037): ConfigureMeshWeaverServer registers StatelessWorkerGrainDirectoryResolver, which resolves every type whose manifest declares placement-strategy = StatelessWorkerPlacement to an in-process directory that answers "not registered" itself. A resolver rather than a [GrainDirectory] attribute because the attribute travels in the cluster manifest, and a silo still on the previous image would then look for a keyed directory it never registered; the resolver is consulted only by the silo that registered it, so a mixed-version roll is safe. StatelessWorkerPlacementSkipsTheDirectoryTest reproduces the failure on a real TestCluster (a directory stand-in that cannot answer for marked grain ids; negative control: without the registration the routing grain cannot be placed).

What it does not fix: a single-activation grain (messagehub/*, podhub/*, pubsubrendezvous/*) genuinely needs the directory, so its placement still depends on the owning partition answering. Those timeouts are downstream of the stalled silo, not of placement — see #5388 for the thread-pool stalls. Also note the in-process TestCluster runs Orleans' distributed directory (TestingHost adds ConfigureDistributedGrainDirectory), whereas the portal runs the DHT LocalGrainDirectory; the test therefore stands in for "the owner cannot answer" rather than reproducing the DHT's own timing.

🚨 The observation this page exists for

Read the sender in that line. It is sys.client/hosted-10.244.9.229:11111 — the Orleans hosted client inside the same process as the silo S10.244.9.229:11111. And StatelessWorkerDirector prefers the local silo.

So that message went from a process, out through cluster placement, to a grain that was going to be placed back in that same process — and was dropped by the round trip. For the co-hosted case the placement step is pure cost, and it is the step that fails.

The proposal: a local routing service where a silo is co-hosted

Resolve the router from DI and call it in-process when this process hosts a silo; keep the grain for callers that have no local silo. The dispatch BEHIND the entry point does not change — the same IIoPool, the same OrderedRouteDispatcher, the same pod-hub leg.

What it removes: the placement dependency, its two failure modes, the 30-second Polly timeout that holds a route slot while it expires, and the serialization of an envelope merely to cross into the same process.

🚨 Three things such a change must carry deliberately

  1. Ordering is not free any more. The non-reentrant turn is the LAST point at which the mesh's send order is authoritative, which is why the per-channel FIFO is claimed there (Ordered Route Channels). The data-sync protocol is a DELTA protocol with a receive-side monotonicity guard: a frame below the mirror's current version is discarded as stale and never re-sent. Lose the ordering and two frames of one stream swap — the later lands, the earlier is dropped forever, and the subscriber sits on "Building layout…" until its own timeout with nothing logged above Debug. A service must provide an explicit single serialization point; it must not inherit one by accident.
  2. A pure client has no local silo. An Orleans client process cannot host a grain at all — the router already reasons about exactly this when it decides whether a pod hub is reachable. Those callers still need a remote entry point, so the grain path stays and the choice is made once, by declaration, not per call.
  3. The size bound moves, it does not disappear. MessageSizeGuard exists because the grain call SERIALIZES its argument. A local call serializes nothing — a saving — but the bound belongs on the legs that genuinely cross the wire, where RefuseOversizedGrainDispatch already applies it.

A fourth thing it happens to fix: RouteMessage is not idempotent, so a response timeout is ambiguous — the grain may have accepted the delivery and not yet answered, and re-sending queues a second copy (issue #1172, which is why the retry predicate is IsResendableDeliveryFailure and not IsTransientFailure). A local call has no RPC and therefore no such ambiguity.

What this does NOT fix, and what to do first

It does not fix #2299 — RoutingGrain turning a transient pod-hub condition into a terminal DeliveryFailure instead of re-resolving the route. That one is sev:H, has 107 occurrences, and its own evidence is damning: the Orleans exception carries will retry after <n>ms, i.e. Orleans judged the condition transient and was going to retry — and the router surfaced a hard failure anyway.

Sequence accordingly: #2299 first. It is smaller, it stops deliveries being lost, and it removes the 30-second holds that fill the dispatch budget. The entry-point change is the structural follow-up, not the emergency.

What is NOT established

Cross-references