Read first: Asynchronous Calls and Orleans Task Scheduler. This page is the I/O-edge counterpart to those two — where the actor model meets real, blocking work.

The problem

Every hub is an actor running on a single-threaded, turn-based scheduler — the Orleans grain scheduler for the root hub, TaskScheduler.Default for every other hub. That single-threading is a guarantee about state, not a claim that the process has one thread: the same process owns the multi-threaded .NET ThreadPool.

Genuine I/O at the leaves — a file read, a blob download, an HTTP call, a Roslyn compile, a Process.Start — must therefore satisfy two requirements:

1. Run off the hub scheduler. A bare await inside a handler captures TaskScheduler.Current and queues its continuation back onto the hub's single turn — blocking the action block, or (across hubs that share a scheduler) deadlocking. The work has to be handed explicitly to the ThreadPool.
2. Be bounded. Without a cap, a mesh of thousands of per-node hubs can each subscribe to the same kind of I/O at once, issuing thousands of concurrent file handles or sockets. This exhausts the resource and — for sync-blocking work — triggers ThreadPool thread-injection that starves the very pool Orleans' grain turns rely on.

Postgres had half of this already: Npgsql's connection pool (MaxPoolSize, sized per role) is a real concurrency governor for DB work. What it never provided is requirement 1 — the old Observable.FromAsync(work, Scheduler.Default) sites did not get the round-trip off the hub scheduler, because FromAsync's scheduler argument schedules notification delivery, not where the function is invoked (see "The hybrid governor" below). File system, blob, HTTP, compile, and process carry no pool of their own and had neither half. IIoPool supplies both, uniformly: off-scheduler by construction, and bounded per resource class.

IIoPool routes all I/O leaves off the hub scheduler onto bounded ThreadPool workers, with per-resource concurrency caps.

🚨🚨🚨 ABSOLUTE: Observable.FromAsync is NEVER tolerated

Observable.FromAsync(...) is FORBIDDEN everywhere in src/ — no exceptions. Not for storage, not for Postgres, not for "it already runs off the scheduler", not for a one-off. There is exactly one place the call may appear in the entire codebase: sealed inside IoPool (the primitive). Anywhere else it is a defect to be removed. Every genuine async/blocking I/O leaf goes through IIoPool.

A bare Observable.FromAsync only schedules notification delivery. It invokes the function's synchronous prologue on the subscribing thread — which is the hub/grain scheduler when the subscribe happens mid-handler — and applies no concurrency bound. That is the entire bug class this primitive exists to kill.

// ❌ FORBIDDEN — runs the prologue on the subscriber (hub) thread, unbounded
=> Observable.FromAsync(ct => httpClient.SendAsync(req, ct));

// ✅ REQUIRED — routed through the resource-class pool: off the hub scheduler, bounded
=> _httpPool.Invoke(ct => httpClient.SendAsync(req, ct));

Pick the method by leaf kind:

There is no "out of scope" residue. If you find yourself typing Observable.FromAsync, stop: the answer is an IIoPool call (or, for an idempotent one-shot, the promise-cache below). The only FromAsync that survives a review is the one inside IoPool itself.

Promise-cache for idempotent one-shots

For work that should run at most once and then be observed by many (schema provisioning, a connect handshake, a container-ready probe), hold the eager pool.Run(...) observable in an instance PromiseCache<TKey, TValue> — or PromiseSlot<TValue> when there is only one — never static:

// PostgreSqlPartitionStorageProvider.EnsurePartitionProvisioned — the canonical example.
// First caller kicks the CREATE SCHEMA off on the per-adapter pool; every later subscriber
// replays the cached completion. No Observable.FromAsync at the call site.
private readonly PromiseCache<string, Unit> _provisioned = new(StringComparer.OrdinalIgnoreCase);

public IObservable<Unit> EnsurePartitionProvisioned(string @namespace) =>
    _provisioned.GetOrAdd(schema, _ =>
        _ioPool.Run(ct => EnsureSchemaAsync(def, ct)).Select(_ => Unit.Default));

// The keyless variant — McpRemoteMeshClient's connect handshake.
private readonly PromiseSlot<McpClient> _connect = new();
private IObservable<McpClient> Connect() => _connect.GetOrCreate(() => _pool.Run(ConnectAsync));

pool.Run is ReplaySubject-backed (see IoPoolExtensions) — eager, single-run, replays to all. That is the "promise pattern": the cache entry is the promise. (pool.RunBlocking is the same for a sync-blocking leaf.)

🚨 Why this is a type and not a ConcurrentDictionary

A ReplaySubject latches terminals, OnError included. So the older recipe — the eager observable in a bare ConcurrentDictionary<key, IObservable<T>> — turned one transient fault into a permanent one: the entry replayed that same exception to every later subscriber for the life of the process, and nothing ever re-attempted. Replay(1).AutoConnect(1) and Replay(1).RefCount() latch identically — one already-terminated subject behind the connectable.

That is not a corner case; it shipped eight times before it was fixed in the recipe (#1369), and its worst instance made a partition permanently un-provisionable after a single connect blip — every later write 42P01-ing until the pod was restarted. PromiseCache exists so the next one-shot someone writes gets the cure for free.

What the type guarantees, and what you must not undo:

Internally the eviction is attached with Do, never a bookkeeping Subscribe: subscribing would be the AutoConnect(1) first subscriber and would connect chains nobody asked for. Invalidate(key) exists for a real domain invalidation (the partition was dropped) — not for test isolation, which a mesh-scoped instance never needs.

Contract pinned by PromiseCacheFaultEvictionTest (test/MeshWeaver.Hosting.Test) and, end-to-end against a real Postgres, PartitionProvisioningFaultRecoveryTests.

The primitive

IIoPool (in MeshWeaver.Mesh.Threading) is the single sealed boundary between the hub schedulers and the I/O. It is hidden inside the leaf adapters — public signatures stay IObservable<T>; callers never see a pool.

public interface IIoPool
{
    // Genuinely-async leaf (blob, HTTP, async file, DB round-trip).
    IObservable<T> Invoke<T>(Func<CancellationToken, Task<T>> io);

    // Sync-blocking / CPU leaf (File.ReadAllBytes, Roslyn compile, Process).
    IObservable<T> InvokeBlocking<T>(Func<CancellationToken, T> work);

    // IAsyncEnumerable leaf (partition objects), bridged to a bounded observable.
    IObservable<T> InvokeStream<T>(Func<CancellationToken, IAsyncEnumerable<T>> source);

    int CurrentInFlight { get; }   // diagnostics / tests only
}

All three return cold observables: the work runs on Subscribe, a pool slot is taken only on Subscribe, and released when the operation completes, errors, or is unsubscribed. This keeps the MeshWeaver.Mesh.RequireSubscribe semantics accurate — a never-subscribed leaf never takes a slot and never runs.

The hybrid governor

The concurrency cap is enforced two ways, chosen per leaf kind:

This design is "compatible with how Orleans wants us to pool": it reuses the ThreadPool the framework already uses and merely puts a governor in front of it — no custom OS threads that Orleans can't see or coordinate with.

Named pools and caps

Pools are keyed by resource class and resolved lazily from IoPoolRegistry (a mesh-scoped singleton, disposed with the mesh — no static state). Caps come from IoPoolOptions, with sensible defaults that a host can override via AddIoPools(o => o with { Blob = 64 }) without any call-site change.

Defaults below are the values on IoPoolOptions — read that record for the reasoning behind each one, which is often not "how much parallelism can this resource take".

Note the prefix-shadowing order in MaxConcurrencyFor: pg-read: is tested before pg: (and sf-read: before sf:), because the read prefix also starts with the write prefix.

Routing is an ISOLATION boundary, not a throttle. RoutingGrain is [StatelessWorker(1)] and non-reentrant, so a silo has exactly ONE routing turn — and Orleans' request timeout applies to callers waiting on a grain, never to the turn itself. Anything the turn does inline is therefore unbounded by construction and blocks every other message the silo needs to route. Prod (2026-08-07) had one RouteMessage turn executing for 06:00:22 with NonReentrancyQueueSize=541; Orleans' own diagnostics showed the work item still Running with Total processed frozen — i.e. RouteMessage had never returned, it was blocked in its own synchronous body. The cure is structural (you cannot time out a synchronously blocked thread): RouteMessage captures its activation-bound handles and hands the composed route to this pool via SubscribeThroughPool, so a leg that never terminates costs one slot and nothing else. See issue #1028.

Hidden inside the interfaces

A leaf resolves its pool from the mesh-scoped registry — for a hub-attached leaf via hub.ServiceProvider.GetService<IoPoolRegistry>(), and for an adapter constructed at a composition root via a required IoPoolRegistry constructor parameter (see the ledgerless-Unbounded rule below — an optional parameter with a ?? IoPool.Unbounded fallback is the shape that let a whole mesh's file I/O escape the teardown drain). Each leaf reads uniformly:

// HTTP leaf (McpRemoteMeshClient) — was Observable.FromAsync(async ct => …).
// Connect() is the promise-cached one-shot handshake (pool.Run, ReplaySubject-backed);
// each call composes off it with SelectMany, so the handshake runs once for all callers.
public IObservable<MeshNode?> Get(string path)
    => Connect().SelectMany(client =>
        _pool.Invoke(async ct => Parse(await client.CallToolAsync("get", …, ct)
            .ConfigureAwait(false))));

// CPU / process leaf — InvokeBlocking on the dedicated limited-concurrency scheduler
=> _compilePool.InvokeBlocking(ct => RunRoslynScript(…));   // KernelExecutor
=> _processPool.InvokeBlocking(ct => RunTestsCore(…));      // MeshPlugin.RunTests

IoPool.Unbounded is a stateless offload onto the ThreadPool with no cap — and, crucially, no ledger: its CurrentInFlight is unconditionally 0, so I/O on it is invisible to every teardown drain and quiescence hold (see "IoPool.Unbounded is LEDGERLESS" below). It is an immutable constant, not a cache. Pools come from the mesh-scoped IoPoolRegistry (registered by MeshBuilder.AddIoPools()), resolved from the owning hub's ServiceProvider or injected via the leaf's constructor — and for adapters constructed at composition roots the registry is a required constructor parameter with a loud failure when the provider lacks one, never an optional parameter with a silent ?? IoPool.Unbounded fallback (that shape is how the FutuRe mesh's entire file I/O escaped the teardown drain, issue #613).

Streaming an agent response into a cell — the precise process

Invariant: the thread hub must never block. A blocked thread turn stops answering GetData / GetPermission / tool-call responses for its own output cell — so the response never renders and the round wedges (GetDataRequest@{thread}/{cell} pending for tens of seconds, GetPermissionRequest timing out). That is the entire "harness doesn't work after submit" symptom. Therefore the streaming round runs in the I/O pool, never on the thread turn. The pool is not an optimisation here — it is the mechanism that keeps the actor's single turn free while the multi-second LLM enumerable drains on a bounded ThreadPool worker.

An LLM round is the archetypal InvokeStream leaf: IChatClient.GetStreamingResponseAsync(...) returns an IAsyncEnumerable<ChatResponseUpdate> — a genuine async I/O source that must run off the thread hub's scheduler and be bounded, exactly like a blob download or an HTTP call. It is never consumed with a bare Task.Run(async () => await foreach …) on (or launched from) the hub turn: that runs the enumerator's continuations under the grain scheduler, and a tool call that needs the same scheduler to answer then deadlocks against the in-flight await foreach. That is the "harness hangs after submit" failure — the thread hub stops answering GetData/GetPermission for its own output cell.

The correct path is exactly three steps, and the output cell is the rendezvous: the pool writes it, the GUI reads it, and neither blocks on the other.

1 — Resolve the output cell and mark it streaming. The round's last entry in MeshThread.Messages is the assistant output cell; its path is {threadPath}/{ActiveMessageId}. Confirm the last cell is the output (assistant) cell, take that as the streaming target, and flip its Status to Streaming so the GUI renders a live cell:

// thread.ActiveMessageId is the canonical handle; the full output path derives from it.
var output = $"{threadPath}/{thread.ActiveMessageId}";   // the last (assistant) cell in Messages
workspace.GetMeshNodeStream(output).Update(node =>
        node with { Content = ((ThreadMessage)node.Content) with { Status = ThreadMessageStatus.Streaming } })
    .Subscribe(_ => { }, ex => logger.LogWarning(ex, "mark-streaming failed for {Path}", output));

2 — Stream in the pool, writing each chunk to the cell's sync stream. Consume the LLM IAsyncEnumerable through IIoPool.InvokeStream (off the hub scheduler, bounded — never Task.Run), and fold every chunk into the output cell via GetMeshNodeStream(output).Update(...). The owning cell hub serialises the writes on its single-threaded action block (no race, no clobber), and the grain scheduler stays free to answer the round's tool-call responses:

var acc = new StringBuilder();
ioPool.InvokeStream(ct => chatClient.GetStreamingResponseAsync(messages, options: null, ct))
    .Sample(StreamingSampleInterval)        // one cell write per sampled tick, NOT per token
    .Subscribe(
        update =>
        {
            acc.Append(update.Text);
            workspace.GetMeshNodeStream(output).Update(node =>
                    node with { Content = ((ThreadMessage)node.Content) with { Text = acc.ToString() } })
                .Subscribe(_ => { }, ex => logger.LogWarning(ex, "stream write failed for {Path}", output));
        },
        ex => SetCellStatus(output, ThreadMessageStatus.Error),
        () => SetCellStatus(output, ThreadMessageStatus.Completed));   // terminal: flip Status once

3 — The GUI subscribes to the same cell stream. The Blazor view databinds the output cell with GetMeshNodeStream(output) (or GetRemoteStream<MeshNode>), rendering Content.Text as it grows and reacting to the terminal Status. It reads the exact node the pool is writing — the cell is the single source of truth, so there is no second channel to reconcile.

Why this is deadlock-free, point by point: the enumerator runs on a pool ThreadPool worker (step 2), never the grain turn, so an in-flight tool call still gets the scheduler. The cell writes go through the owning hub's serialised action block via the stream handle — a non-blocking cross-hub patch, not a synchronous wait. The GUI only reads (step 3). Three actors, one cell, no one blocks another.

🚫 The anti-pattern this replaces. Task.Run(async () => { await foreach (var u in client.GetStreamingResponseAsync(…)) cell.Update(…); }) looks offloaded, but it (a) is unbounded — N concurrent rounds spawn N enumerators with no governor — and (b) bypasses IIoPool, so it is invisible to the pool's diagnostics and cancellation, and any synchronous wait on the output cell from the hub turn (e.g. a sync-handshake read of a cell that isn't reachable yet) still wedges the hub. Route the enumerable through InvokeStream; the offload, the bound, and the cancellation come for free.

🚨 A tool call runs INSIDE the leaf — so its Task must observe the token

The round in the previous section holds one gate permit for its whole duration, and a tool call happens inside that await foreach. So the tool's Task<string> is not a detail of the agent loop — it is the thing standing between Drain() and a real join.

Agent tools are Task-returning by contract (AIFunctionFactory needs a Task-returning delegate), and the usual shape bridges an observable to it with a TaskCompletionSource. That bridge is the one sanctioned Task boundary here — but the token that is bound into the tool's CancellationToken parameter must be able to settle it. It is the round's token: linked to the user's Stop (executionCts) and to the pool token that IoPool.Drain() cancels.

var tcs = new TaskCompletionSource<string>(TaskCreationOptions.RunContinuationsAsynchronously);
// Everything the wait holds — subscriptions AND the cancellation registration — in one bag,
// released by whichever terminal fires first.
var pending = new CompositeDisposable();
void Settle(Func<bool> set) { if (set()) pending.Dispose(); }

pending.Add(cancellationToken.Register(() => Settle(() => tcs.TrySetCanceled(cancellationToken))));
pending.Add(source.Subscribe(r => Settle(() => tcs.TrySetResult(r)), …));
return tcs.Task;

Why a timeout is not a substitute. delegate_to_agent had a 10-minute backstop on one of its two completion paths and none on the other. Ten minutes is 20× the 30 s DrainTimeout, so from teardown's point of view a backstop that generous is indistinguishable from no exit at all: the parked continuation keeps its permit, Drain() sits out its budget, reports a leaked leaf, and the scope is disposed (and collectible node ALCs unloaded) over live code. The user-visible half of the same defect is that Stop does nothing — the round is parked in a Task the Stop cannot reach.

Cancel rather than resolve an error string: ThreadExecution's catch (OperationCanceledException) when (executionCts.IsCancellationRequested || poolCt.IsCancellationRequested) classifies that as the graceful shutdown/stop it is, so the round settles Cancelled instead of writing a false "#147 streaming exceeded the maximum round duration" into the user's response cell.

Pinned by DelegationCancellationTest (unit) and DelegationDrainJoinsParkedToolCallTest (integration — a delegation that never resolves, asserting DrainAll() == 0), the sibling of AiPoolDrainJoinsRoundTest for a round parked on the model call.

Scope — storage and Postgres are pooled too

Earlier guidance carved storage and Postgres out of the pool and left them on plain Observable.FromAsync. That carve-out is rescinded — there is no exemption. FromAsync is never tolerated (see the absolute rule above), so storage / file-system / Postgres leaves go through IIoPool like everything else.

Per-adapter pools: a WRITE gate of 1 and a READ gate of 16, and the split is not optional. A Postgres adapter's writes run on pg:{adapter}, capped at 1 — the IIoPool gate is the write connection ("hook into the pg pool") rather than a redundant bound stacked on top of it. Its reads run on pg-read:{adapter}, capped at IoPoolOptions.PostgresRead (16). The naming + caps live in IoPoolNames.PostgresAdapterPrefix / IoPoolOptions.MaxConcurrencyFor.

🚨 Read the pairing correctly. Only a dedicated single-connection data source makes "the gate and the driver pool are the same size" literally true — PostgreSqlChunkedContentVectorStore is the one place that holds (MaxPoolSize=1 alongside its cap-1 pg:vector pool). The partitioned provider deliberately does the opposite: every per-schema adapter shares ONE NpgsqlDataSource (MaxPoolSize=50 in the portal), because minting a data source per (schema, table) leaked a pool per hub and exhausted the server. There the two caps are a budget, not an identity — 16 reads + 1 write = 17 concurrent connections, comfortably under 50. That budget only holds if both pools are actually wired and each operation is filed on the right one.

Both halves have to be wired, and reads must not be filed on the write pool. This is not a style point — it was issues #1310/#1312/#1313/#1316. The Postgres backend resolved its cap-1 pg:Postgres pool, used it for provisioning, and then never passed ioPool: to the adapters that perform every actual write, so each per-schema adapter fell back to IoPool.Unbounded. Compounding it, eight read-shaped operations (Read, ReadMany, Exists, FindBestPrefixMatch, ResolvePath, GetPartitionObjects, GetPartitionMaxTimestamp, ListPartitionSubPaths) were filed on that write pool rather than the read pool. Net effect: the hottest read path in the portal — per-node-hub activation seeds, URL resolution, write-guard probes, the per-path read fan-out inside StorageAdapterMeshQueryProvider — ran with no bound at all against a 50-connection data source, and memex-cloud duly reported "the connection pool has been exhausted (currently 50)". Keeping reads off the cap-1 pool is also what makes that pool safe: a read issued from inside a write would otherwise be a same-pool re-entry on a cap-1 gate, the one documented way to deadlock an IIoPool. PartitionAdapterIoPoolWiringTests pins both the wiring and the read/write filing.

The cost concern that originally justified the carve-out (a SubscribeOn hop on every hot read under a constrained CI ThreadPool) is real — the answer is to size the per-adapter pools correctly, not to fall back to bare FromAsync. The migration is finished: every query/storage leaf is pooled (see "The sweep is complete" below), and new code (e.g. PostgreSqlPartitionStorageProvider.EnsurePartitionProvisioned) is pooled from day one.

Edge cases — the review checklist

These four properties must hold for every leaf that uses IIoPool:

• Pool the leaf, never the orchestration. IIoPool is injected into leaf adapters only. Orchestration layers (PersistenceService fan-out, version-writing decorator, query providers, MeshQuery) compose adapter observables and hold no slot — a fan-out of N reads acquires N leaf slots and never nests behind a held one. Per-resource pools mean a FileSystem leaf never blocks on a Blob leaf while holding a FileSystem slot. Never let an IIoPool call resolve an observable that itself acquires the same pool — that is the one way to deadlock it.

• Cancellation and release. Every shape releases its slot in a finally. WaitAsync(ct) makes acquisition itself cancellable — a dispose before the slot is granted throws before the in-flight increment, so no slot leaks. The subscription's ct flows into the leaf, so file/blob calls cancel on unsubscribe.

• Disposal. IoPoolRegistry disposes each IoPool, which disposes its SemaphoreSlim. The limited-concurrency scheduler borrows ThreadPool threads — nothing to dispose.

• No sync-context capture. .SubscribeOn(TaskPoolScheduler.Default) + .ConfigureAwait(false) everywhere; InvokeBlocking dispatches via the scheduler-bound TaskFactory, never TaskScheduler.Current. Safe even when Subscribe is called inside a grain handler.

• 🚨 The pool carries NO AccessContext — it is an identity hole, by design. .SubscribeOn(TaskPoolScheduler.Default) re-runs the whole subscribe (gate wait + the leaf's synchronous prologue + every ConfigureAwait(false) continuation) on a bare ThreadPool worker. That worker has no AsyncLocal baton — AccessService.Context reads as null/whatever the pool thread last held, NOT the originating user. The pool is the threading primitive (get IO off the action-block thread, bounded); it is deliberately not an identity primitive. Contrast the mesh write/read primitives (MeshNodeStreamCache.Update / GetMeshNodeStream / GetQuery), which wrap the cold pipeline in CarryAccessContext(accessService) — capturing the caller's identity synchronously on the caller's thread and re-stamping AsyncLocal on each emission. So: never read or write a mesh node from inside an IIoPool leaf and expect RLS to see the user — the leaf runs identity-less. Genuine external IO (subprocess, HTTP, blob) doesn't touch AccessService.Context and is fine. If a pooled leaf genuinely must do an authorized mesh operation, capture the identity before the pool boundary and pass it explicitly, or — far better — do the mesh read/write through GetQuery / GetMeshNodeStream (which carry the baton) and keep only the non-mesh IO in the pool. This is why the Copilot model catalog moved off ioPool.Run(... ListModelsAsync ...) and onto workspace.GetQuery(...): the picker is per-user, and only the GetQuery path stamps the subscriber's identity.

Disposal is reactive — Dispose() fires, the mesh drains

There is no DisposeAsync() and no IAsyncDisposable anywhere — not in the public API, not on any hub or resource. The whole shape is deleted. An await DisposeAsync() inside hub-reachable code (or a shared-scheduler fixture) captures TaskScheduler.Current and queues its continuation back onto the very turn that is tearing down → the turn never drains → deadlock. Disposal is synchronous + reactive instead:

• Dispose() (IDisposable) fires and returns. It does the synchronous unregister (e.g. IRoutingService.RegisterStream returns a synchronous unregister handle) or kicks off reactive teardown — it never blocks on it. IDisposable is the only disposal interface a hub or resource implements.
• Asynchronous teardown is observed through IObservable<Unit> DisposalCompleted (the shape IMessageHub uses), not awaited. Dispose() triggers it; completion is a reactive emission.
• Someone has to subscribe. DisposalCompleted is cold — with no subscriber the completion is never observed. That subscriber is the mesh teardown, never the individual hub or call site: the mesh's disposal merges its hubs'/resources' DisposalCompleted and waits for them all to emit before it completes. Hub-reachable code only exposes DisposalCompleted; it never subscribes to or drains its own.
• Let them all finish. When the teardown owns several reactive completions, drain them together so they all emit before it returns — never an awaited loop that serialises (and can deadlock on) each one.

🚨 The mesh teardown drains THREE things, not one

DisposalCompleted is necessary but NOT sufficient. It drains the hub's action blocks and in-flight message round-trips — but I/O offloaded through IIoPool runs on the ThreadPool, independent of the action block, and DisposalCompleted knows nothing about it. If the teardown disposes the service scope after DisposalCompleted but while an IIoPool operation (or any other async cleanup) is still in flight, that continuation resolves a service from the dead Autofac scope and throws ObjectDisposedException: …LifetimeScope… has already been disposed — unobserved, it surfaces as an xUnit "catastrophic failure" that aborts the whole run.

So the mesh teardown awaits all three, in order, before the scope is disposed:

1. IMessageHub.DisposalCompleted — action blocks + message round-trips. (Resources enqueue their async cleanup onto the AsyncDisposeQueue during this synchronous-Dispose() phase — Dispose() must never block, so async cleanup is queued, not run inline.)
2. IoPoolRegistry.DrainAll() — offloaded ThreadPool I/O. GRACE, then CANCEL + JOIN — never wait alone, never cancel first. The drain first lets every in-flight leaf finish on its own: it re-acquires the gate permits one at a time, each under IoPoolOptions.DrainGrace (8 s), so every completion restarts the clock and a leaf that is going to finish is never cancelled (a write that would have landed in 50 ms lands). Only a leaf that outlives a whole grace with the pool making no further progress is wedged: its call site is captured, and then the pool cancels. 🚨 Do not use the wait-only WhenDrained(timeout) in place of that cancel: a live change-feed leaf never completes on its own, so a polled wait times out and lets the scope dispose while the leaf is still running — its ThreadPool thread then dereferences a collectible node ALC's freed metadata after unload, a native use-after-unload SIGSEGV. DrainAll() cancels so the stream leaves and the wedged leaves stop, then joins, and returns the count it had to leak — and, separately, the leaves it had to cancel (IoPool.LeavesCancelledAfterGrace / CancelledLeafSites), which the teardown logs at Error because that work did not finish (Teardown Layers). 🚨 The cancel runs on its own thread, never on the joining one. CancellationTokenSource.Cancel() executes every registered callback synchronously on the caller, and this token's callbacks are not bookkeeping: SubscribeThroughPool registers one per live pooled subscription that runs that subscription's whole downstream teardown, and every gated leaf links its subscriber token to the pool's, so cancelling also resumes each leaf's gate wait into its observer. Cancelling inline therefore ran arbitrary application teardown on the MESH TEARDOWN thread with no budget over it — the drain timeout covers only the gate join that comes after — so one clean-up leg that would not finish parked teardown silently and forever (#2394: a whole test assembly killed at its 8 min cap with no test named). IoPool now issues the cancel on a dedicated thread and joins it first, ahead of the gate join and under the same budget, because the gate join's meaning depends on the cancel having landed ("once the pool token is cancelled, no NEW leaf can take a permit"); a cancel that does not finish inside the budget is added to the residual. Dispose() issues it the same way and never joins — its wait lives on Disposed.
3. AsyncDisposeQueue.DrainAsync(timeout) — the queued async cleanup. A TPL ActionBlock drains it; DrainAsync Complete()s the block and awaits the remainder (bounded), so it converges even under continuous influx — a version-target wait would not (the queue is a message stream / endless messages). DrainedVersion advances once per item, the test hook.

🚨 A residual with NO site is the CANCEL join — and a subscriber can park it

IoPool.Drain() reports three things under one number: gate permits it could not re-acquire, blocking leaves still running, and a cancel that did not return. The first two always name a site (every leaf registers one on entry); the third is not a leaf and had none. So a trace that read

DISPOSE_IOPOOL_DRAIN_DONE elapsed=30016ms leakedIoLeaves=1 pools=[Query=1]

— Query=1 with nothing in brackets — meant exactly one thing, _poolCts.Cancel() on the Query pool never returned, and nothing on the page said so. It now reads Query=1 [IoPool.Drain: the pool token's cancel did not return within the budget — …].

What parks a cancel. Cancelling the pool token runs, inline on the IoPool-cancel thread, one callback per live SubscribeThroughPool subscription: inner.Dispose(); observer.OnCompleted(); — that subscription's whole downstream teardown. Two facts about the libraries underneath turn a race into a deadlock:

1. CancellationTokenRegistration.Dispose() blocks until a callback executing on another thread has finished (WaitForCallbackIfNecessary; only the callback's own thread is exempt). Unregister() never waits.
2. Rx operators forward from their timers under their gate. Throttle.Propagate runs ForwardOnNext inside lock (_gate), and Throttle.OnCompleted takes the same gate; Take(1) completes and disposes upstream synchronously — still inside that gate.

So a consumer shaped Query(...).Throttle(1 s).Take(1) whose timer fires as the drain cancels holds the operator gate while its upstream disposal waits in Dispose() for the drain callback, and the drain callback waits in Throttle.OnCompleted for the operator gate. Two locks, two threads, no exit. The drain reports the cancel residual after its budget, RSS is flat the whole time (parked, not computing), and the two threads stay deadlocked for the life of the process.

The occurrence. MeshNodeLanguageServiceTest went DIRTY at teardown on 2026-08-28 (#2578, #2616), 08-30 (twice), and 09-03 (Plugins #1260 attempt 1, shard 3) — always the same shape: the test body PASSES in ~1 s (1018 ms / 1044 ms in the two traces that survived), the drain then spends its whole 30 s, and the residual is Query=1 with no site. The consumer is CompletionUsageIndex.EnsureFresh(), whose Throttle(1 s) lands on exactly those ~1 s bodies; a faster machine ends the test before the timer fires, which is why twenty local runs never reproduced it. #2598 fixed a different leaf (the first-in-process script-reference build, on the Compile pool) on the strength of the same anonymous 1; the failure recurred unchanged four hours later.

The rules this leaves behind:

• Never put a raw CancellationTokenRegistration where a subscriber disposes it. The subscription's disposable unregisters (drainReg.Unregister()); the drain callback stays exactly-once through its latch and completes on its own. More generally: never dispose a registration from a thread that may hold a lock the callback needs — in Rx that is every operator gate.
• A request-scoped subscription needs an OWNER. EnsureFresh()'s query used to be fire-and-forget with a 30 s Timeout — a second, unowned 30 s budget coinciding with the drain's. It is held in a SerialDisposable and disposed with the hub (hub.RegisterForDisposal(...), which runs in DisposeImpl — strictly before DisposalCompleted, hence before DrainAll()); the same for the completion memory store's debounced save. Three lifetimes exist for any deferred work — the caller's (WaitAsync(ct)), the owner's (the hub or service that started it), and never (a process-wide memo) — and "no caller cancels this" must not silently become "nothing ever does".
• Read a residual by its site. A bracketed site is a leaf; no site is the cancel join, now labelled. Query=1 and Compile=1 are different bugs, and so are Query=1 [MeshQuery…] and Query=1 [IoPool.Drain: the pool token's cancel did not return…].

Pinned by IoPoolDrainCancelJoinTest (deterministic: a TestScheduler-driven Throttle fired from a thread the test owns, with the two interleavings the deadlock needs made explicit).

🚨 IoPool.Unbounded is LEDGERLESS — I/O on it does not exist to this drain

The three-phase drain only covers what it can see, and IoPool.Unbounded is invisible to it by construction: its CurrentInFlight is unconditionally 0, its Invoke is a bare Observable.FromAsync(io).SubscribeOn(TaskPoolScheduler.Default), and it lives outside the registry — so IoPoolRegistry.DrainAll() never enumerates it, TotalInFlight/WhenDrained never count it, and there is nothing to cancel, dispose, or join. The same blindness applies to every other quiescence hold built on the ledgers (e.g. the silo's routing-quiescence hold). Phase 2 therefore reports a clean drain while the unbounded I/O is still running — and that straggler is exactly the teardown SIGSEGV shape: a leftover Rx emission enters full hub construction after the scope is disposed and faults on a span into the unloaded collectible ALC.

This is not hypothetical. FileSystemStorageAdapter took its registry as an optional parameter with a ?? IoPool.Unbounded fallback, and every production construction site silently dropped it — the storage-adapter factory (the path every config-declared FileSystem data source takes) and both PersistenceExtensions registration sites. Result: every file-system-backed mesh (the FutuRe sample, the one file-system-data-source-backed space) ran all of its file I/O on the unbounded pool, and issue #613's exit=139 teardown crash recurred with zero failing tests in the trx — the drain had nothing to join.

The rule. An adapter or service whose I/O must be drainable takes IoPoolRegistry as a REQUIRED constructor parameter — the compiler then names every dropping site, which an optional parameter never does ("check the call sites, not the declaration"). A DI factory or registration lambda resolves the registry from the provider and fails LOUDLY — a thrown error naming the missing registration and how to add it (AddIoPools(), which MeshBuilder calls by default) — never a silent ?? IoPool.Unbounded. A deliberate IoPool.Unbounded use must be written out explicitly at the call site with a comment saying why bare-ThreadPool is correct there; a pure test convenience is the only acceptable answer, and fixing the test to use a real registry is preferred.

The only sanctioned await is that single three-phase drain at the boundary — the mesh teardown, the same in tests and in prod (the silo's mesh disposal at shutdown). Capture the mesh-scoped teardown services before Dispose() (never resolve DI once disposal has begun), then drain all three, bounded:

// ✅ The one drain, at the mesh-teardown boundary (test mesh OR prod silo shutdown).
//    Either call the canonical helper:
await mesh.TeardownAsync(TimeSpan.FromSeconds(15));   // MeshWeaver.Mesh.MeshTeardownExtensions

//    …or, if you drive Dispose() yourself, do the phases by hand:
var ioPools = mesh.ServiceProvider.GetService<IoPoolRegistry>();        // capture BEFORE Dispose()
var disposeQueue = mesh.ServiceProvider.GetService<AsyncDisposeQueue>();
mesh.Dispose();
// 🚨 ObserveCompletion, never Rx's ToTask bridge (forbidden repo-wide, 2026-08-30) and
//    never a bare `await disposalCompleted` either: both resume this method INLINE on the
//    hub's own disposal thread, which then has to run phases 2 and 3 while the mesh is
//    trying to finish tearing itself down. ObserveCompletion completes with
//    RunContinuationsAsynchronously, so the disposing thread is released immediately.
using var phase1Deadline = new CancellationTokenSource(TimeSpan.FromSeconds(15));
await mesh.DisposalCompleted
    .Catch<Unit, Exception>(_ => Observable.Return(Unit.Default))
    .FirstOrDefaultAsync()
    .ObserveCompletion(
        ex => logger.LogWarning(ex, "disposal faulted AFTER the wait settled"),
        phase1Deadline.Token);                                             // phase 1
var leakedIoLeaves = ioPools?.DrainAll() ?? 0;   // phase 2 — cancel + join, NOT a polled wait
if (disposeQueue is not null)
    await disposeQueue.DrainAsync(TimeSpan.FromSeconds(15));               // phase 3
// ONLY NOW dispose the service scope.

"Only drainage of async pipelines is allowed": the await lives at that one three-phase drain, the work stays reactive. Same principle as IIoPool — the async boundary is pushed to the edge and bounded; it is never an ambient await mid-flow. Full order + failure mode: Mesh Lifecycle. See also Asynchronous Calls.

🚨 Ambient context does not cross the pool. Work handed to IIoPool runs on a pooled thread whose ExecutionContext is not yours: an AsyncLocal you set upstream is not readable inside the leaf, and a value written inside the leaf is not visible to the caller. Capture what the leaf needs into the closure before handing it over — see AsyncLocal Across Scheduler Hops and, for identity specifically, AccessContext Propagation.

🚨 The pool releases its gate on an ADMISSION COUNT, never on "is anyone running?"

Dispose() must not block (a synchronous 30 s join parks a pool thread while the leaves it waits for need pool threads to observe cancellation — a starvation deadlock on a 4-vCPU runner). So it cancels, returns, and lets the last caller out release _gate / _poolCts / the blocking-idle signal. That makes "who is still using them?" the load-bearing question, and the obvious answers are all wrong:

• CurrentInFlight is raised only once WaitAsync has granted a permit. A leaf between the grant and the increment holds a permit and counts as zero, so disposal releases the gate and the resumed leaf calls Release() on it (issue #2146).
• A _disposed flag read at the top of the entry point is read when the cold observable is built, while _poolCts.Token and _gate.WaitAsync are touched on subscribe. Everything the pool returns is cold, so that gap is arbitrarily wide — and a chain built during a render and subscribed as teardown lands threw ObjectDisposedException out of the reactive chain (#2134 / #2135).

IoPool therefore counts admissions, not executions: every path that may touch those primitives — each of the four entry points, Drain(), and Dispose() itself — brackets its whole reach in TryEnterGateRegion() / LeaveGateRegion(), and disposal completes only at zero. Entry is publish-then-recheck (increment, then re-read the disposal flag; Dispose publishes the flag before it reads the count), so of the two check-then-act orders at least one side always observes the other: either disposal defers, or the caller is refused and answers OperationCanceledException. A leaf the pool will not run is a CANCELLATION, never an ObjectDisposedException — that is the contract the region exists to keep, and it is why there is no catch (ObjectDisposedException) anywhere in the file. Adding one would hide a region that was never entered.

Applied to (current scope)

The sweep is complete. Observable.FromAsync no longer appears anywhere in src/, test/, samples/, or memex/ — the only occurrence is sealed inside IoPool. The former "migration debt" query/storage sites (PostgreSqlMeshQuery family, Cosmos, file-system adapters) are now pooled; orchestration that isn't an I/O leaf (layout view generators, message-delivery and routing bridges) was rewritten as pure reactive composition (Observable.Create / Defer + Task.ToObservable()), never FromAsync.

Cross-references

• Asynchronous Calls — rule 9, "the async boundary lives at the real I/O edge".
• Aggregating Providers — "the async boundary lives at the I/O edge", pool-at-the-edge.
• Orleans Task Scheduler — per-hub schedulers and the SubscribeOn offload this complements.