CI disposal audit — 2026-09-08

CI evidence and its limits

The audit read the latest 30 failed Catalog runs and 40 earlier failed runs through GitHub REST, inspected failed-job inventories on nine representative runs, and downloaded three portal-host job logs. The working tree was 962f8702, with CI platform source e1039813e4c080f46d6e54b7a84b09c92326856e.

Evidence	Finding	Attribution
Run 34210183539, shard 1 and run 34222981863, shard 1	FutuRe test host exits 139 at about 1m46s.	Native crashes; these logs do not prove a disposal cause. The related #1527 investigation records zero-MethodTable GC crashes and a separate managed query leak.
Run 34235028593, shard 2	First compile assertion fails in 424ms with NullReferenceException; the process later fails many compile cases and times out.	The compiler canary reports failures even with fresh image-backed references, while flat source still emits. This is the existing Roslyn/CLR investigation, not evidence that every subsequent test independently leaked.
#1390	AI Orleans host reports a disposed Autofac scope while all 51 tests pass.	The managed lifetime family is real; the tracked AI fix had already landed before this audit.
#1513	Teardown NACK assertion times out at six seconds.	Current main already replaces the timing inference with observation of owner shutdown and verdict consumption.

Several sampled reds are policy, validation or bundle-build failures, not intermittent tests. The query-connection leak documented in #1527 is in core's MeshNodeStreamCache, and the pinned platform used for this audit already contains its connection ownership fix. Do not copy that fix into this repository or describe a later green FutuRe run as proof that the native crash is cured.

Instance-sync cleanup gaps found in the source audit

InstanceSyncWorker.DisposeClientLocked discarded the ValueTask returned by the remote client's DisposeAsync. Its comment said the implementation never faults. That says nothing about when it finishes: the MCP client can still be awaiting connection/client teardown after the worker returns. Neither the hub completion signal nor the I/O pool drain counts a discarded cleanup task.

The worker now captures the mesh-scoped AsyncDisposeQueue at construction and queues each detached client's cleanup. Mesh teardown already drains that queue before releasing the service scope. The cached reference is cleared before enqueueing so replacement and repeated disposal do not dispose the same client twice. The client-cache gate also protects the terminal flag: a callback that reaches GetClient after disposal cannot allocate a new transport nobody will release.

InstanceSyncCoordinator retained the change-feed subscription but discarded the startup query's subscription. Stopping before its initial result left discovery alive for up to its 30-second timeout, with a callback capable of starting workers after the stop. Each start now owns both subscriptions in a CompositeDisposable, released before its workers. A fresh bag per start preserves the existing same-instance stop/start contract. A locally captured bag also disposes a subscription whose registration returns after that particular run has stopped.

PR review identified the remaining dispatched-callback race: unsubscription cannot retract an already captured feed callback. A lifecycle gate now serializes worker creation with the stop snapshot, and each callback must still belong to the current start. Stop invalidates that start and detaches its workers under the gate, then disposes subscriptions and workers outside it so unsubscription can safely join a callback. A callback from a stopped run cannot populate even a later restart. Two fenced delivery cases cover both stopped and restarted coordinators.

These are concrete lifecycle defects, but the downloaded CI logs do not establish that either caused the FutuRe native crashes or the Roslyn failure above. The change does not adjust timeouts, disable parallelism, retry CI until green or suppress exceptions.

Validation

InstanceSyncDisposalTest drives the actual client cache and disposal path with a remote client whose cleanup completion is explicitly held open. The drain must stay pending until the test releases cleanup. Separate cases cover client replacement, repeated disposal, refusal to reopen a disposed worker, and cancellation/restart of boot discovery. These assertions test ordering without relying on a lucky scheduler interleaving.

The complete instance-sync suite must also run under DOTNET_PROCESSOR_COUNT=4; its persisted manifest restart case catches a terminal subscription bag accidentally reused after StopAsync.

Measured on .NET 10.0.11, against the platform pin above:

Release build with warnings as errors: zero warnings, zero errors.
The same four regression tests against untouched 962f8702: 4 failed, 0 passed. The failures name the early queue drain, untracked replacement cleanup, post-disposal client allocation and still-subscribed startup query.
Fixed full suite with four logical processors: 59 passed, 0 failed, 0 skipped.
Three additional full-suite passes with four logical processors and four CPU load processes: 59/59, 59/59, 59/59, in 28s, 28s and 27s. The load processes were stopped after the run.
Node-repository validation: 259 nodes across 60 modules passed.
Catalog compile gate against a flat set of freshly built assemblies from the same platform pin: 88 clean, 0 known debt, 0 new breaks, 0 unverifiable.
Release rebuild with CIRun=true and warnings as errors, followed by the full affected suite: 59 passed, 0 failed, 0 skipped (27s).
After the review correction, both dispatched-callback cases fail against the original PR coordinator because a worker survives stop. With the lifecycle gate and start ownership, the full suite passes 61 passed, 0 failed, 0 skipped (27s), with the same CI build settings.

The regression tests establish the defects deterministically. Three loaded passes are interaction coverage, not a statistically established cure for a low-frequency native crash. No matching before/after CI failure rate has been measured for the separate FutuRe/Roslyn families.