Read first: Asynchronous Calls. This page is the threading-model substrate that those rules rest on.

The model

Every hub is an actor. Each actor has two components:

• A single-threaded turn loop — MessageService's lock-guarded FIFO queue plus a draining re-entrancy flag, which processes messages strictly one at a time. (This replaced a TPL Dataflow ActionBlock with MaxDegreeOfParallelism = 1; "action block" survives as the name for the same guarantee. See Observable Hub Pipeline.)
• A TaskScheduler (turnScheduler) onto which each drain run is dispatched.

Two hubs can never share a turn loop. Two hubs can share a TaskScheduler — and that sharing is precisely the failure mode this design exists to eliminate. When two hubs share a single-threaded scheduler, they collapse into one effective actor: an await inside one hub's handler queues a continuation that must run on the same thread the other hub's handler is already holding. Tool-call responses can't be delivered while a streaming handler is waiting for them. The result is a deadlock.

The rule is simple:

Hub scheduler topology: the root grain hub stays on the Orleans grain scheduler; every child hub gets its own independent TaskScheduler.Default action block.

Why the root grain hub uses the grain scheduler

Orleans installs a per-grain TaskScheduler so that everything happening "inside the grain" runs on a single, grain-affined thread. This affinity gives Orleans several important guarantees:

• Activity attribution — idle-deactivation timing measures actual grain work, not work that bounced through the thread pool.
• RequestContext flow — state set inside a grain method propagates through await continuations.
• Distributed-tracing scopes — spans are correctly attributed to the grain.
• Lifecycle hooks — OnActivateAsync / OnDeactivateAsync fire on the grain scheduler and see consistent grain state.
• Single-threaded grain semantics — only one continuation runs at a time ([Reentrant] notwithstanding).

If the root grain hub's action block ran on TaskScheduler.Default instead, Orleans would observe only the entry-point grain method call and miss all subsequent work the hub processes after picking up a message.

MessageHubGrain.OnActivateAsync captures TaskScheduler.Current at activation time and passes it via .WithTaskScheduler(grainScheduler) when building the root hub.

Why every other hub uses TaskScheduler.Default

Hosted hubs, per-node hubs, _Exec, and kernel hubs are not the grain. They are sibling actors created from the grain hub's perspective but live independently. If any of them shared the grain's scheduler, every await inside them would serialize through the grain's single thread — exactly the cross-hub deadlock scenario described above.

The default for MessageHubConfiguration.TaskScheduler is null. MessageService resolves that to TaskScheduler.Default (the thread-pool scheduler). The thread pool can run multiple continuations concurrently, so async work inside one hub never blocks async work inside another.

TaskScheduler.Default is the static thread-pool scheduler — fixed at process start and unaffected by the ambient TaskScheduler.Current of whatever code constructed the hub. A hosted hub created from inside a grain method (where TaskScheduler.Current == grainScheduler) still runs on the thread pool. There is no implicit capture.

The configuration knob — WithTaskScheduler

public record MessageHubConfiguration
{
    public TaskScheduler? TaskScheduler { get; init; }

    public MessageHubConfiguration WithTaskScheduler(TaskScheduler scheduler)
        => this with { TaskScheduler = scheduler };
}

MessageService resolves it once and holds it as turnScheduler, then starts every drain run on it:

private TaskScheduler turnScheduler = TaskScheduler.Default;
// …during construction:
turnScheduler = hub.Configuration.TaskScheduler ?? TaskScheduler.Default;

// Each drain run STARTS on turnScheduler, so a handler observes
// TaskScheduler.Current == the hub's configured scheduler:
private void ScheduleDrainOne() =>
    Task.Factory.StartNew(DrainOne, CancellationToken.None,
        TaskCreationOptions.DenyChildAttach, turnScheduler);

The queue + draining flag is what enforces the actor invariant (one turn at a time, FIFO, self-posts to the back) — there is no MaxDegreeOfParallelism knob any more. The scheduler determines where those serial turns run. There is likewise no separate executionBlock: execution requests run on this same loop.

What this fixes — the cross-hub deadlock

The sequence this design rules out:

1. Hub A starts handling a message; its handler awaits a round-trip to hub B.
2. Without .ConfigureAwait(false), the await captures TaskScheduler.Current as the resumption scheduler.
3. If A and B share that scheduler and it is single-threaded, the response from B must be dispatched through it. But the scheduler is busy holding A's handler, which is blocked waiting for the response. The response can't run. Deadlock.

With per-hub schedulers, A and B each have their own thread-pool slots. The await releases A's slot; B's response runs on its own slot; A's continuation resumes when the slot is available. No collision.

What this does NOT fix — the same-hub await problem

Per-hub schedulers resolve cross-hub deadlocks. They do not resolve this pattern:

// Inside hub X's handler
async IMessageDelivery Handle(...) {
    await foreach (var update in someStream) { ... }   // long-running
}

The await foreach holds hub X's turn and prevents it from processing the next message. If the response that advances MoveNextAsync also needs to be processed by hub X, the hub is busy holding its own handler's task — a same-hub deadlock.

The default answer is IIoPool — pool.Invoke / pool.InvokeStream / pool.InvokeBlocking — which offloads and bounds and is joinable at mesh teardown (Controlled I/O Pooling). ThreadExecution's agent-streaming loop is the one place in src/ that uses a bare Task.Run instead, deliberately: the loop re-enters the grain scheduler on every tool-call response, so it must not sit behind a pool permit. It pays for that with explicit compensations at the seam — a CancellationTokenSource registered via RegisterForDisposal (without it, grain deactivation waits up to 120 s on a Task.Run stuck in an AI API call) plus DelayDeactivation/BeginAsyncOperation to keep the grain alive. Treat that as a narrowly-scoped exception with a cost, not the default; the same file's non-streaming leaf carries the comment "I/O pool — NEVER Task.Run, NEVER inline on the hub turn." See Thread Execution Streaming for the worked example.

SubscribeOn(TaskPoolScheduler.Default) inside a grain-hosted service

IMeshNodeStreamCache.GetQuery and MeshQuery.Query / .Query wrap their inner observables with SubscribeOn(System.Reactive.Concurrency.TaskPoolScheduler.Default). When the cache hub lives inside an Orleans grain, this is intentional and correct. Here is what happens step by step:

1. The grain method call (GetQuery(id, queries)) runs on the grain scheduler. Orleans serialises access; the Interlocked.CompareExchange on the cache's _queries dictionary is safe by virtue of single-threaded grain execution. The method returns an IObservable<T> description and the grain releases immediately.

2. Subscribe is called outside the grain method's scope — typically from a downstream consumer such as a layout area, a validator, or another service. The grain holds no lock at this point.

3. SubscribeOn(TaskPoolScheduler.Default) shifts the subscribe-time work — constructing SyncedQueryMeshNodes, opening the upstream IMeshQueryCore.Query subscription, opening database connections and change feeds — onto a thread-pool thread. That is the right place for I/O. Without this offload, those subscriptions would run on whatever thread called Subscribe, which could be the grain scheduler if the caller is mid-handler.

4. Emissions (OnNext to the cache's Replay(1).RefCount() and onward to downstream subscribers) flow on whatever thread the upstream emits from — PostgreSQL change-feed threads, Orleans observer dispatchers, and so on. The cache's internal Replay(1) buffer is thread-safe; downstream subscribers reading from it are also safe.

The risk this does not create — and the risk it does not eliminate

The pattern is safe with respect to the cache's own state. The risk that remains is a consumer-side concern: if a downstream observer's OnNext directly mutates grain state without going through a grain interface call, that is a single-threading violation. The fix lives at the consumer, not at the cache:

// In a consumer that holds grain affinity:
cache.GetQuery(id, queries)
    .ObserveOn(grainContext.Scheduler)   // ← re-enter the grain for callbacks
    .Subscribe(snapshot => /* now safe to touch grain state */);

The SubscribeOn at the cache layer does not widen this risk — upstream change-feed emissions were already arriving from background threads regardless. The offload shifts only the subscribe-time work, which is precisely what you want off the grain.

This SubscribeOn offload is the query-construction half of keeping the grain free. Its leaf-execution counterpart is the Controlled I/O Pooling primitive (IIoPool), which applies the same SubscribeOn(TaskPoolScheduler.Default) move plus a concurrency bound so actual file / blob / HTTP leaf work both runs off the grain and cannot fan out unboundedly.

🚨 The layout-area render pipeline subscribes OFF the hub turn — query-in-render is safe

This is the third — and highest-leverage — place the framework makes the same SubscribeOn move, and it closes the deadlock class that took down multiple production meshes.

A layout area's view generator returns an IObservable<UiControl?>. The framework's render pipeline (LayoutAreaHost.BuildInitialization, and every nested container / dialog / editor sub-area render) subscribes to it to drive content to the client. That subscribe runs on the layout-area's own synchronisation-stream hub action block — a single-threaded actor. When the layout area belongs to a node hosted as an Orleans grain, its owning workspace hub is the root grain hub on the grain scheduler (the table above). So, before the fix, the generator body — and the subscribe to whatever observable it returned — ran on the grain turn.

That is the query-in-render trap:

1. A view generator does, in-render, IMeshService.Query(...) (or hub.Observe, GetRemoteStream, a workspace query — any mesh round-trip).
2. The render pipeline subscribes it on the grain turn.
3. The query must route through Orleans and come back to this grain to resolve. But the grain turn is held inside the subscribe → the response can never be processed → the hub deadlocks.
4. On startup prerender, many such grains block at once → thread-pool starvation → the whole silo wedges (even /healthz). Confirmed offenders: Doc/DataMesh/SocialMedia/Post (List area) and Doc/DataMesh/PythonPandasNode/PandasExplorer.

The fix — one reactive seam. LayoutAreaHost routes every render subscribe through ScheduleRenderSubscribe, the same move made at MeshQuery.Query and IMeshNodeStreamCache.GetQuery (the section just above). The generator, and every observable it subscribes in-render, now runs off the grain turn, which is immediately free to route + answer the round-trip. A pure Rx scheduler operator — no async / await / Task.

// LayoutAreaHost.ScheduleRenderSubscribe — wrapped around every render subscribe
// (BuildInitialization's top-level one AND every nested container/dialog/editor sub-area
// render reached from UpdateArea, which runs ON the hub turn via Stream.Update →
// UpdateStreamRequest):
private IObservable<T> ScheduleRenderSubscribe<T>(IObservable<T> source) =>
    renderSubscribeScheduler is { } scheduler
        ? scheduler.SubscribeThroughPool(source)               // ← preferred: TRACKED + cancellable
        : source.SubscribeOn(TaskPoolScheduler.Default);       // ← fallback: hubs with no I/O pools

🚨 The pooled variant is not a nicety — a bare SubscribeOn(TaskPoolScheduler.Default) is invisible to mesh teardown. During disposal that hop keeps executing on a ThreadPool thread after the hub's Autofac LifetimeScope is disposed (→ ObjectDisposedException from a menu renderer's GetService) and, for a node whose render touches types compiled into a collectible AssemblyLoadContext, after that ALC is unloaded (→ a native use-after-unload crash: the FutuRe.Test exit=139). SubscribeThroughPool makes the subscribe a tracked, cancellable leaf on the mesh's drainable Layout I/O pool, so IoPoolRegistry.DrainAll() cancel+joins it before the scope disposes. The bare fallback is only taken on hubs with no I/O pools registered (bare messaging-only hubs / HubTestBase), which compile no collectible ALCs.

Why it can't reintroduce ordering bugs. The render output (PushRenderResult / UpdateArea) never touches the hub directly — it calls Stream.Update(...), which posts an UpdateStreamRequest to the hub's action block (hub.Post is the actor inbox: safe from any thread, re-serialised in order). So emissions arriving on a pool thread are re-marshalled onto the owning hub's single-threaded turn exactly as before — the offload moves only the subscribe-time work off the hub, never the state writes. Data-before-control ordering is preserved (regression-guarded by the full MeshWeaver.Layout.Test suite, incl. EditorTest / ContainerControlAreaNestingTest).

Why this is the robust fix and not "author around it." The safety lives in the portal/framework binary, so existing deployed nodes — whose cached assemblies never recompile — become safe with no code change. The prior guidance ("move mesh queries into virtual data sources, never IMeshService.Query in a view") was a per-node workaround; this is the framework making the whole class safe at one seam. Regression-pinned by QueryInRenderDeadlockTest (a layout area doing an in-render mesh round-trip on a hub-turn-blocking subscribe: hangs without the offload, renders with it).

Reactive, not blocking, still required. The offload subscribes you off the hub turn; it does not make a blocking subscribe free. A generator that bridges a round-trip back to a blocking Task (.Result / .Wait() / .ToTask() + wait) still burns a pool thread. Compose reactively and return the IObservable<UiControl?> — see Asynchronous Calls.

Offloading a CPU leaf off the action block — Task.Run, identity, and why the gated pool can wedge

A long synchronous leaf (Roslyn Emit, a big reflection/type-load) that runs inline on a hub handler wedges that hub for its whole duration. The canonical trap: CompileAsync(...).ToObservable(). For an in-memory compile CompileAsyncCore has no await before the synchronous Emit, so CompileAsync() runs the entire compile synchronously, and .ToObservable() runs it on whatever thread subscribed — the activity hub's action block — freezing the mesh (≈50 s of trace silence). Three rules came out of fixing it:

1. Never Task.ToObservable() a leaf you want off the action block. The Task's continuation resumes on the current TaskScheduler (the action block when you subscribed mid-handler), so it comes right back onto the blocked turn.

2. The NodeType compile is the one measured exception where Task.Run beat the pool. MeshNodeCompilationService runs the Roslyn compile via Task.Run (its OnThreadPool helper), and its own comment records why: _ioPool.Run "re-entered/parked on the Compile pool's SemaphoreSlim gate (idle 40s wait)" — the activity-driven and request-driven compiles both wanted the same Compile pool slot, and the wait is idle, so dotnet-stack shows no blocked thread (the missed-observation signature). Task.Run schedules on TaskScheduler.Default with no gate and no current-scheduler capture. Do not generalise this into "prefer Task.Run for CPU work" — everywhere else the IIoPool rule stands, precisely because a bare Task.Run is invisible to IoPoolRegistry.DrainAll() at teardown. Whichever you use, never Observable.FromAsync and never Task.ToObservable for a leaf you want off the turn.

   ⚠️ Two things about that exception are worth re-measuring before you copy it: the compile is also bounded by RoslynCompileTimeout (an unbounded leaf parked the type at Compiling for the life of the activation), and a self-serialising pool gate is a design choice — the pool being capped at 1 is the thing to revisit, not the pool.

3. 🚨 Re-establish the login inside the offload. Whichever way you hop threads (Task.Run or the IoPool), the AccessService identity (an AsyncLocal) does not flow across the hop, and the handler's ImpersonateAsSystem scope is long disposed by the time the offloaded work runs. So compile source-reads / write-backs run unauthenticated off-thread. Wrap the offloaded body: using (accessService.ImpersonateAsSystem()) return await work(); — inside the async lambda so the scope spans every await. "Why doesn't the IoPool just work — it's the correct abstraction?" It is the correct abstraction for bounded async I/O; for a CPU leaf with grain-identity needs it adds a gate you don't want and still drops the identity. The pool offloads the work; it does not carry the actor's AccessContext — that is the caller's job at the seam.

The flip side (per the section above): work that touches hub state after the offload must .ObserveOn(grainScheduler) back onto the owning hub's turn — the CPU runs off-grain, the state mutation runs on-grain.

Cross-references

• Asynchronous Calls — the actor-model rules this scheduling model implements.
• Controlled I/O Pooling — bounds the leaf I/O that this scheduling model offloads.
• Thread Execution Streaming — the streaming-loop pattern that depends on this isolation.
• Debugging Message Flow — how to recognise a scheduler-sharing deadlock in trace logs.