Mesh Lifecycle: Build Up & Tear Down

A mesh is an actor system wired into a DI (Autofac) service scope. Standing it up and taking it down are mirror images, and the tear-down side has ONE rule that is easy to get wrong and produces a brutal, run-aborting failure when you do:

You may not dispose the service scope until every piece of the mesh's work has finished — and "work" is TWO things, not one.

Skip it and a late continuation resolves a service from the already-disposed scope and throws ObjectDisposedException: …LifetimeScope… has already been disposed. Unobserved, xUnit reports it as a "catastrophic failure" that aborts the whole test collection — every later test in the run then times out for reasons unrelated to its own logic.

Build Up

var mesh = new MeshBuilder(/* … */)
    .UseMonolithMesh()                       // or UseOrleansMeshServer() on a silo
    .ConfigureServices(s => s.AddSingleton<MyRepository>())   // mesh-scoped singletons
    .AddGraph()
    .Build();                                // builds the Autofac scope + root hub

Two invariants make tear-down tractable, so honor them at build time:

1. Everything stateful is a mesh-scoped singleton, registered in MeshBuilder (ConfigureServices / WithServices). Its lifetime IS the mesh's — it dies with the scope, so there is nothing to Clear(). No static collections/caches (see NoStaticState.md). The IoPoolRegistry is exactly this: a mesh-scoped singleton owning every IIoPool.
2. Every async/blocking edge goes through IIoPool (see ControlledIoPooling.md). This is what makes the offloaded I/O countable at tear-down — the registry knows how many operations are in flight. Bare Observable.FromAsync / Task.Run work is invisible to tear-down and is exactly the leak that throws ObjectDisposedException.

Tear Down — quiesce activities, drain ALL THREE phases, THEN dispose the scope

IMessageHub.Dispose() is reactive and returns immediately — it only kicks off the disposal state machine. Completion is signalled later through IMessageHub.DisposalCompleted. But DisposalCompleted covers only the first of three kinds of in-flight work; tear-down must drain all three before the service scope is disposed:

The async dispose queue is the key to phase 3: Dispose() is synchronous and must never block, but some cleanup is genuinely async. So in their sync Dispose(), resources Enqueue(ct => …) their async cleanup onto the mesh-scoped AsyncDisposeQueue instead of running or leaking it. A single-consumer ActionBlock drains it in the background; tear-down gives it a bounded quiesce budget to finish.

DrainAsync completes the block, it does not wait for a version target. The queue is a message stream — under continuous influx, "wait until the drained version reaches N" never converges (endless messages). DrainAsync instead Complete()s the block (stops acceptance) and awaits the remainder, so it is bounded even while producers are still posting. The DrainedVersion counter (one per item) is the test hook: enqueue N, drain, assert it advanced by N.

There is also a phase 0 that is easy to miss: an in-flight activity falls through all three phases above — its trigger returned as soon as the activity existed, its command runs off-turn so it holds no grain turn, and the subscribe-window pool permit is long released, so DrainAll() joins nothing. So teardown quiesces the activities first, before anything is disposed — ActivityTracker.Quiesce(ActivityStallBudget): a run that keeps reporting progress (every ctx.Log line) is waited for; one that has made no progress for the stall budget (8 s) is cancelled once, through the same token a user's cancel would trip, and named on the report; one that ignores that is named as abandoned a budget later so the teardown can proceed. Those runs still write their terminal ActivityLog status through hubs that must still be alive. (Measured before any of this existed: a 5 s activity, and teardown returned after 2028 ms with the command still running.) Policy and evidence: Teardown Layers.

The canonical helper does all of it:

// MeshWeaver.Mesh.MeshTeardownExtensions
var report = await mesh.TeardownAsync(timeout);   // activities → Dispose() → DisposalCompleted → DrainAll → AsyncDisposeQueue
// ONLY NOW is it safe to dispose the Autofac scope:
await ((IAsyncDisposable)mesh.ServiceProvider).DisposeAsync();

TeardownAsync:

1. captures the IoPoolRegistry, AsyncDisposeQueue, ActivityTracker and MeshTeardownSignal while the scope is still alive (never resolve DI once disposal has begun — see the note in MessageHub's ShutDown finally),
2. runs ActivityTracker.Quiesce (bounded by the caller's timeout) — phase 0 — and logs every run it had to cancel or abandon at Error,
3. calls mesh.Dispose() (resources enqueue their async cleanup during this reactive disposal),
4. awaits DisposalCompleted (phase 1),
5. calls IoPoolRegistry.DrainAll() (phase 2), then
6. awaits AsyncDisposeQueue.DrainAsync(timeout) (phase 3), and finally
7. fires MeshTeardownSignal with the TeardownReport.

🚨 Phase 2 gives a GRACE, then CANCELS, then joins — it never merely waits. DrainAll() first lets every in-flight leaf finish on its own: it re-acquires the gate permits one at a time under IoPoolOptions.DrainGrace (8 s per completion — a stall bound, so a burst of short writes drains in as many completions), names the leaves that outlived it, and only then cancels. The cancel is still required: a live change-feed leaf never completes on its own, and the wait-only, polled WhenDrained(timeout) — which still exists on IoPoolRegistry and is the wrong primitive here — timed out and let the scope dispose while the leaf still ran; its ThreadPool thread then dereferenced a collectible node ALC's freed metadata after unload → a native use-after-unload SIGSEGV. After the cancel DrainAll() joins, and returns how many leaves did not unwind. A leaf it had to cancel after the grace is a unit of work that did not finish — reported at Error with its call site (TeardownReport.CancelledIoByPool).

TeardownAsync returns a TeardownReport (leaked I/O leaves + whether the async dispose queue drained clean). Surface a dirty report — fail the test class, error-log the shutdown. Proceeding silently over live work is the use-after-unload crash above.

If a caller drives Dispose() itself and keeps its own progress/diagnostic loop around DisposalCompleted (the monolith test base does), it uses the wait half directly — pass the services captured before Dispose():

var ioPools = mesh.ServiceProvider.GetService<IoPoolRegistry>();        // capture first
var disposeQueue = mesh.ServiceProvider.GetService<AsyncDisposeQueue>();
mesh.Dispose();
await WaitWithProgressAsync(...);                                       // phase 1 (DisposalCompleted)
var leakedIoLeaves = ioPools?.DrainAll() ?? 0;                          // phase 2 (cancel + join)
if (disposeQueue is not null)
    await disposeQueue.DrainAsync(timeout);                             // phase 3 (async cleanup)
// THEN dispose the scope

Every phase is bounded by a timeout. A wait that times out means a real bug — a wedged action block (surfaced separately by AnyHubQuiescingTimedOut), a leaked I/O slot (a non-zero leakedIoLeaves / IoPoolRegistry.TotalInFlight), or a wedged async cleanup. The timeout keeps tear-down from hanging; it does not paper over the leak — log it and fix the leak, never just widen the timeout (see the no-band-aids rule).

🚨 Phase 2 was the exception, and it was invisible. IoPool.Drain()'s budget covered only the gate join; the _poolCts.Cancel() ahead of it ran on the CALLING thread — i.e. this very teardown thread — and CancellationTokenSource.Cancel() executes every registered callback synchronously there. Those callbacks tear down whole pooled subscriptions (see Controlled I/O Pooling), so one clean-up leg that would not return parked teardown for as long as anyone waited, writing nothing anywhere: issue #2394, a whole test assembly killed at its 8 min wall-clock cap with no test named and not one line after DISPOSE_INVOKED. The cancel now runs on its own thread and is joined under the same budget, so a stuck leg is reported in leakedIoLeaves instead of hanging.

The order is the whole point

WhenIdle  →  Dispose()  →  DisposalCompleted  →  DrainAll()  →  AsyncDisposeQueue drain  →  dispose scope
(activities)  (enqueue async   (action blocks)    (cancel+join      (async cleanup)
              cleanup here)                        ThreadPool I/O)
                            └────────────── nothing may resolve DI after this ──────────────┘

Dispose the scope one step early — before phase 2 or 3 — and any straggler IIoPool continuation or un-run async cleanup that resolves a service hits a dead scope. That is THE catastrophic ObjectDisposedException.

Where this is wired

See also: TestStateIsolation.md (per-test disposal + static seed), ControlledIoPooling.md (the I/O pool the drain waits on), NoStaticState.md (why everything is mesh-scoped).