Orleans Test Routing — Dedicated Registered Hubs

When a test sends a message through Orleans, the silo's RoutingGrain must be able to deliver the response back to the originating hub. Production follows a uniform pattern for this; Orleans tests must mirror it exactly — or responses deadlock silently at the routing layer.

Response routing: a hub that never called RegisterStream leaves the silo publishing into a stream nobody reads; client/{id} hubs registered with RegisterStream receive responses via the Orleans memory stream.

The Core Rule

A hub that must receive cross-silo replies has to be at a STREAM-ROUTED address AND registered with RegisterStream — the address type alone is not enough, and the registration alone is not enough.

A hub in the test process is a hosted hub, not a grain on the silo. Routing reaches it only over the cluster-wide Orleans memory stream, and that requires two independent things to line up:

MeshConfiguration.DefaultStreamRoutedAddressTypes is { "portal", "client", "cache", "mesh" }, declared at static-init time so no configurator ordering can make a built-in type "go missing". Modules add their own with MeshBuilder.AddStreamRoutedAddressType(...) — Graph registers import, the gRPC hosting registers its Python and Node types.

🚨 Stream-ROUTED is not the same as CLIENT-HOSTED, and since Durable Streams Are Mesh Nodes the difference decides what a delivery gets. A stream-routed address is first attempted as a directed IPodHubGrain.Deliver call to the pod that claimed it; the memory stream is reached only when the grain answers "not here", and only for an address type separately declared MeshBuilder.AddClientHostedAddressType(...) — i.e. one whose hubs live in an Orleans CLIENT process, which cannot host a grain. Production declares none; the Orleans test rig declares all four built-in types in OrleansTestMeshExtensions.ConfigurePortalMesh, because it genuinely hosts a hub of each on its cluster client. For anything else "not here" is answered with a transient ShuttingDown + TargetUnserved NACK instead of a publish. If you add a test hub on the CLUSTER CLIENT at a new address type, declare it in both places — stream-routed and client-hosted — or its cross-silo deliveries will be NACK'd rather than carried.

The registration step looks like this:

.WithInitialization(hub =>
    hub.RegisterForDisposal(routingService.RegisterStream(hub)))

The canonical production reference is PortalApplication.DefaultPortalConfig (Blazor/SSR). It auto-registers every portal hub on initialization, so the silo can route layout-stream deltas, command responses, and synced-query notifications back to the right circuit.

🚨 Historical note — mesh/{guid} is no longer intrinsically unroutable. This page previously said "never use mesh/{guid} addresses as routable targets in tests", on the basis of a hard-coded address.Type == PortalType || address.Type == "client" check in RoutingGrain. That check is gone: the type list is configuration, and mesh is in the default set. What has not changed is the second half — an address whose hub never called RegisterStream still gets a silently-dropped delivery. Diagnose on the registration, not on the address type.

Why an Unregistered Hub Still Fails

The silo's RoutingGrain.RouteMessage decides between two dispatch paths:

if (meshConfig.StreamRoutedAddressTypes.Contains(address.Type))
{
    var s = streamProvider.GetStream<IMessageDelivery>(addressPath);
    return PostToStream(() => s.OnNextAsync(delivery), …);  // → cluster-wide memory stream
}
// otherwise: path-resolver lookup → grain dispatch

For a stream-routed type the silo publishes onto the memory stream keyed by the address path — and that is all it can do. If no hub in any process subscribed at that key, nothing receives it. For a non-stream-routed type the grain falls through to pathResolver.ResolvePath(...); when that returns null the delivery NACKs as NotFound.

These two failures look different in the log and need different fixes: a missing RegisterStream is silent, a missing node is a [ROUTE] NotFound.

Sharing One Backing Store Between Silo and Client

A single-process test cluster runs the silo and the Orleans client in separate DI containers, so each resolves its own InMemoryStorageAdapter — and a node created on one side is then invisible to the other. The fix is to give both containers adapters that point at the same backing store, exactly as production points every adapter at the same Postgres database.

🚨 The backing store must be an INSTANCE on the fixture, never a static field. test/MeshWeaver.PathResolution.Test/NoStaticCollectionsTest.cs reflects over every MeshWeaver.* assembly in the test output — test assemblies included — and fails the build on any static mutable collection (ConcurrentDictionary, Dictionary, HashSet, MemoryCache, …). A public static readonly ConcurrentDictionary "shared backing dict" on a fixture class is a guard violation and leaks state across every test in the process. Hold the dictionary as an instance field on the ICollectionFixture<> and close over that instance in both containers' registrations; its lifetime is then the fixture's, and it dies with the cluster.

This mirrors production: multiple IStorageAdapter instances (one per host's DI container) all point at the same backing store — Postgres in production, one fixture-owned dictionary in tests. Multiple fixtures coexist without bleed because each owns its own instance; a GUID partition id per fixture is a useful extra label, not the isolation mechanism.

Putting It Together: Cross-Silo Test Pattern

For a test that creates a node via the client and then operates on it across the silo boundary:

1. Fixture setup — hold the shared backing store as a fixture instance field (above) and configure both silo and client containers to close over it.
2. Test hub setup — use OrleansMeshTestBase.GetClient(clientId?, userId). It is synchronous (there is no GetClientAsync) and it does the registration for you: routingService.RegisterStream(client.Address, client.DeliverMessage) at client/{clientId}. Don't target Fixture.ClientMesh.Address, whose hub is not the one registered for your test.
3. Test message flow — target every request at the registered hub's address. Responses route back through the memory-stream subscription that the registration created.
4. Cross-silo operations such as workspace.GetMeshNodeStream(remotePath).Update(…) work because the silo can resolve the remote path via the shared backing store, and the reply reaches the process-unique cache hub described above.

Failure Mode Reference

Production Analogue

The Cache Hub — the Shipped Design

MeshNodeStreamCache used to open its upstream subscription off the parent mesh hub, so the SubscribeRequest it posted carried the mesh hub's own address as Sender and replies had nowhere to land. That is fixed. The cache now owns a dedicated stream-routed hub, and the shape is worth knowing because it is the pattern any process-local cache with cross-silo reads should copy:

var routingService = meshHub.ServiceProvider.GetRequiredService<IRoutingService>();
var cacheAddress = new Address("cache", meshHub.Address.Id);   // ← process-unique, NOT a fixed id
cacheHub = meshHub.GetHostedHub(cacheAddress, config => config /* … + RegisterStream in WithInitialization */);
…
var handle = cacheHub.GetWorkspace().GetMeshNodeStreamBypassCache(p);

Two decisions carry the whole design:

• cache is a declared stream-routed address type, in MeshConfiguration.DefaultStreamRoutedAddressTypes — not a partition with a static node and an IPartitionStorageProvider. The silo dispatches to the memory stream; there is no grain to activate.
• 🚨 The address is keyed by the parent mesh hub's Id, so it is unique per PROCESS. A fixed id such as cache/mesh-node-cache would make the silo's and the client's cache hubs subscribe to the same cluster-wide memory stream, so a silo-side reply to a client-initiated SubscribeRequest can be delivered to the silo's cache hub. That hub has no sync sub-hub for the incoming DataChangedEvent's StreamId, RouteStreamMessage returns request.Ignored(), and the client times out. Do not "simplify" this to a constant.

Superseded design. An earlier revision of this page prescribed making the cache hub a real top-level node at cache/mesh-node-cache via a MeshNodeCacheStaticProvider + StaticNodePartitionStorageProvider, on the premise that RoutingGrain had no address-type check. Neither the provider type nor that address exists; following it would reintroduce the shared-stream cross-delivery bug above. It has been replaced by the description here. Likewise, this page used to cite OrleansUserOwnedModelTest.UserOwnedProvider_RotateKey_ResolverPicksUpNewKey as a skipped repro that would go green after the refactor — that test does not exist; the surviving tests in OrleansUserOwnedModelTest are UserCreatesProvider_ThenResolverFindsKey and UserModelAndProvider_VisibleInSyncedQuery.

🚨 Reachability is not a claim — wait for the transport you are about to depend on

A test that isolates one transport in order to assert another has a precondition the transport itself does not announce, and getting this wrong produces an intermittent failure that looks like a platform race. This was #3298.

RegisterStream establishes two things, on different clocks:

So a freshly registered address is reachable long before it is claimed, and during that window a directed IPodHubGrain call is placed by [PreferLocalPlacement] on the caller's silo — which has no local route for it — and answers PodHubNotHere.

The trap. A "prove the address is live" probe that posts a message and waits for it to arrive proves only reachability, and reachability during that window is satisfied by the stream. If the test then erases the stream subscription and asserts a directed delivery, it has asserted something whose precondition it never checked. Worse, the probe is adversarial to the claim it appears to prove: it posts from the other silo in a loop, and each failed directed call mints a throw-away activation there that the owner's next Attach must bounce off — restarting the backoff.

The rule. Wait on the claim itself. OrleansRoutingService.PodHubClaimSettled(address) is the positive signal — it completes when the claim terminated: it landed, or it hit the one terminal that is impossibility rather than a budget. A claim still retrying never completes it, which is the honest answer.

var routingA = Routing(cluster, 0);
using var registration = routingA.RegisterStream(address, callback);

// Claimed ⇒ a directed cross-silo delivery lands. Reachability would not tell you this.
await (routingA.PodHubClaimSettled(address) ?? Observable.Return(Unit.Default))
    .FirstAsync().Timeout(Budget).Await(ct);

The null-coalesce covers an address with no claim at all — a client-hosted one, where the stream is the permanent transport and there is nothing to wait for.

Do not substitute a poll. "The count stopped changing" measures a pause, and the claim's retry hops the thread-pool scheduler between attempts, so on a loaded shard the poll reads mid-hop. That is why the settled signal exists at all; see its remarks and PodHubClaimReassertion.

Measured. With one bounce forced and the claim's backoff pinned long, the reachability probe goes green in under a second while PodHubClaimSettled has demonstrably not completed — 6 runs, 6 times. That is the whole defect: probe green, claim unsettled, directed transport not yet available.