An Rx callback may not resolve from a hub that is gone

A callback outlives the hub that armed it. The hub's DI container does not. Between those two facts sits a defect that has cost this repo more CI time than any other single bug: a service resolve inside an Rx onNext body, running after teardown, taking the whole process with it.

The rule is one line: an Rx callback in the AI engine that can reach a hub's container must be gated on that hub's teardown state. Everything below is why that is not obvious, and why four correct fixes did not add up to it.

What it looks like

A suite prints Passed! - Failed: 0 and the job is red. Above the summary, with no stack and no test name:

[xUnit.net 00:00:10.31]     [FATAL ERROR] System.ObjectDisposedException
[xUnit.net 00:00:10.31] Catastrophic failure: System.ObjectDisposedException : Instances cannot be
  resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its
  parent scopes) has already been disposed.
VERDICT src/MeshWeaver.AI.Orleans.Test: THE TEST HOST EXITED 1 WITH NO FAILING TEST

The elapsed time on that line is a background thread's, so it attributes to nothing. There is no failing assertion to find, because nothing failed on a test thread.

Why it is fatal rather than noisy

Rx routes an exception thrown inside Select / SelectMany / Defer to the subscription's onError. It does not do that for the callback delegates themselves. Every Subscribe overload that takes delegates builds an AnonymousSafeObserver, and that observer disposes the subscription and rethrows — out of onNext and out of onError. Adding an onError argument is therefore not a guard: onError receives faults travelling from the SOURCE, and a throw inside it is rethrown exactly like one inside onNext.

The rethrow lands on whatever thread the scheduler used — a System.Threading.Timer callback for a Sample/Throttle tick, or a bare pool thread. That is an unhandled exception. xUnit v3 hooks AppDomain.UnhandledException and not TaskScheduler.UnobservedTaskException, so it reports it as an anonymous Catastrophic failure, lets the run finish, and exits non-zero. In a portal the same throw is a process death: a pod restart, 502s, and a log with no cause in it.

Task-wrapped continuations behave differently and misleadingly: they surface as UNOBSERVED-TASK entries in the straggler artifact and never terminate anything. Both shapes appear side by side in one log, and only the raw ones matter for the exit code.

The two sites Plugins#1390 died on

The straggler artifact (<suite>__teardown-stragglers.log, collected on every red shard) names them outright:

Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
AutofacServiceProvider.GetRequiredService(Type)
MeshWeaver.Data.WorkspaceExtensions.GetWorkspace(IMessageHub)
MeshWeaver.Mesh.MeshNodeStreamExtensions.GetMeshNodeStream(IMessageHub, String)
MeshWeaver.AI.ThreadExecution.…g__PushToResponseMessage|0(…)
MeshWeaver.AI.ThreadExecution.…b__73(StreamingSnapshot)      ← the Sample(100 ms) tick
System.Reactive.AnonymousSafeObserver`1.OnNext(T)            ← disposes and RETHROWS

and

AutofacServiceProvider.GetService(Type)
MeshWeaver.AI.AgentChatClient.ApplyStaleModelFallback()
MeshWeaver.AI.AgentChatClient.CreateAgentsSync()
MeshWeaver.AI.AgentChatClient.ApplyAgents(…)
MeshWeaver.AI.AgentChatClient.…b__0(IReadOnlyList<AgentDisplayInfo>)   ← Initialize's Subscribe
System.Reactive.AnonymousSafeObserver`1.OnNext(T)

Note how far the resolve is from the callback. The first is one hop into a local function, and the word ServiceProvider appears nowhere near it — GetMeshNodeStream → GetWorkspace → GetRequiredService is the path. The second is three hops into an ordinary private method chain. Neither is visible by reading the Subscribe line.

The same resolve, reached three ways

Counting frames in one straggler artifact (run 34191774851, the occurrence that reddened #1431) is what settles where the gate belongs. ThreadExecution.cs:1290 — the parentHub.GetMeshNodeStream(curResponsePath) inside PushToResponseMessage — was reached four times, from two different kinds of caller:

So the loud path and the quiet path share one resolve. Gating only the Rx callbacks would have fixed the crash and left the silent one live — a write that no longer happens, with no exception anyone will ever see. That is a worse outcome than the crash, because the crash at least reported itself.

The gate therefore sits on the resolve — the first statements of PushToResponseMessage and of ApplyStaleModelFallback — where every caller passes, present and future. The subscribe-level gating below is the enforceable rule; this is the root-cause fix.

The general form: when several callers share one resolve, gate the resolve. Gate the callback when the callback is doing more than reaching that one resolve.

Why four fixes did not converge

TeardownSafeCallback — the gate this defect needs — already existed in this assembly, with a type document explaining the mechanism precisely, and it was already applied at four call sites. One of them was the reasoning-heartbeat subscription derived from the same snapshots subject as the crash site, three lines above it in the same file.

That is the whole lesson. Every previous fix (#871, #1443, MeshWeaver#3556 / #1461) repaired a real site and left the class open, because nothing in the build asked "and the others?". A gate that exists but is applied by memory is a gate that is applied unevenly, and the uneven half is invisible until a race finds it.

It also explains why a green run proves nothing here. #1474 failed on this defect and then merged green with no fix for it in the tree — same branch, same defect, one fail and one pass, nothing relevant changed in between. Re-running "to see if it was a flake" produces exactly that green.

The rule, and how to satisfy it

Subscribe through the gate:

stream.SubscribeTeardownSafe(hub, logger, "streaming push", s => PushToResponseMessage(…));

SubscribeTeardownSafe routes every delegate — onNext, onError, onCompleted — through TeardownSafeCallback.Run, which is two halves and nothing more:

1. Skip the work when hub.IsShuttingDown is already true. Everything routed through here has the live mesh as its only consumer — a cosmetic status stamp, a re-subscribe, an identity stamp — so during teardown there is nothing to lose by not doing it. This is a lifecycle decision, not a suppressed error: the destination no longer exists.
2. Tolerate the check-then-act race, and only that. The hub can begin disposing between the probe and the resolve, and no container offers an atomic "resolve-or-tell-me-you-died". So an ObjectDisposedException is absorbed only when it is the container's own and the hub says it is shutting down when the exception surfaces — the filter re-reads, it does not trust the entry probe.

Everything else still propagates. A disposed CancellationTokenSource, a disposed stream, or a disposed container under a hub that claims to be running — that last one meaning something disposed a live hub's scope — all stay fatal. A blanket catch (ObjectDisposedException) would pass the same tests and convert a loud, rare failure into a silent, permanent one.

Where the dependency can simply be captured while the scope is alive — the Blazor-disposal fix's answer (MeshWeaver#3556) — do that instead; it removes the resolve rather than gating it. Gating is for resolves that are correct to repeat, like a node-stream handle whose path changes mid-round.

An explicit if (hub.IsShuttingDown) return; on the same path is equally acceptable and the guard accepts it. What is not acceptable is a bare .Subscribe( from which a container is reachable.

The guard

MeshWeaver.AI.Test.TeardownGatedCallbackGuard scans src/MeshWeaver.AI/ and fails on any subscribe site with a lambda from which ServiceProvider.Get…, GetWorkspace( or GetMeshNodeStream( is reachable — through the callback body and the bodies of same-file methods and local functions it calls, transitively — unless that path is gated.

Four details are load-bearing, and each is pinned by a fixture in the guard itself. Three of them are holes the guard actually shipped with for one review cycle, and each would have made it green over part of its own subject:

• Local functions must be visible. The sibling Blazor guard's signature pattern requires a keyword from a fixed set (public, private, Task, void, …). Under that pattern IObservable<MeshNode> PushToResponseMessage(…) has no body at all — the #1390 crash site would have been scanned, found to reach nothing, and passed.
• No depth limit. The first draft stopped at three hops, the depth the two incident sites happened to need (one and three). seen already prevents cycles, so a limit buys nothing and costs everything: inserting one more helper walks a site out of the resolving set, and the aggregate floors cannot see one site leave.
• The gate must come BEFORE the resolve, in the same body. "A teardown token appears somewhere in the reachable text" is a false negative, and ThreadExecution.cs:792 is the demonstration: its callback resolves at line 799, and an unrelated nested timeout handler's IsDisposing at line 845 — forty-six lines later, on a path that never runs first — made the site read as safe. It was absent from the offender list even though the same change had converted it. That is the worst failure available to a guard: under-reporting its own subject while looking authoritative.
• Both lambdas. A resolve in onError is rethrown exactly like one in onNext, and this repo already carried hand-written gates inside onError bodies for that reason.

It asserts its own denominator: 102 subscribe sites carry a lambda, 27 of them can reach a hub resolve, 0 are ungated. A collapse in either count is red — a scan that has lost its subject reports zero offenders exactly as loudly as a clean tree. The two incident sites are additionally pinned by their gate label, so a rename that silently drops the gate cannot leave the guard green over the exact code it was written for.

MeshWeaver.AI.Test.TeardownGatedSubscribeTest pins the mechanism deterministically — no mesh, no timing, no re-run — against a real disposed Autofac scope rather than a hand-written exception. That matters more than it looks: the gate classifies by the exception's TEXT, and until that test existed nothing had ever compared that text to what Autofac actually says. A string gate whose string was never measured fails open.

What the guard was measured against

A guard that has never been red is a guard nobody has checked. This one was run against origin/main's call sites — the tree that was still crashing — with everything else in place. It named all twenty-two, ThreadExecution.cs:2467 and AgentChatClient.cs:1793 among them:

these Rx callbacks can reach a hub's DI container with no teardown gate:
  AgentChatClient.cs:1793, AgentView.cs:441, :470,
  ChatClientAgentFactory.cs:654, ModelProviderLayoutAreas.cs:313, :355,
  Plugins/DelegationTool.cs:285, ThreadComposerView.cs:238, :396,
  ThreadExecution.cs:376, :599, :792, :1950, :2467, :3297, :3329, :3933,
  ThreadSubmission.cs:352, :640, :740, :1383, :1466

:2467 is the site the straggler artifact blames for the host death on run 34191774851; :1793 is the one in the artifact from 34177049924. That is the evidence this change rests on — not a green CI run, which this defect has already produced on an unfixed tree.

When the guard reds

It is telling you a callback can reach a container it does not own. Do not silence it.

1. Read the reachable path it names. If the dependency can be captured at creation time, capture it — one field, so that "this thing was created" and "the callback has what it needs" are the same fact. A separate bool beside a nullable service can get out of step, and then the work silently no-ops, which is worse than the exception because it leaks without being findable.
2. Otherwise subscribe through SubscribeTeardownSafe, with a short label naming the work.
3. Never try/catch the body, never add an onError argument and call it fixed, and never raise a timeout or add a retry. Each of those hides this defect rather than removing it.