Debugging native crashes (core dumps)

A test host that dies on a signal does not fail a test — it fails the shard. There is no assertion, no stack in the trx, often no trx at all. CI reds it via the exit-marker gate:

[CI] MeshWeaver.FutuRe.Test exit=139
##[error]Shard 3: a test host exited non-zero (failure, crash, or timeout kill):

The crashing project name is meaningful; the crashing TEST name usually is not. The signal lands wherever the process happened to be, which is frequently not where the defect is.

The dump is already being collected

dotnet-test.yml sets, for every shard:

DOTNET_DbgEnableMiniDump: 1
DOTNET_DbgMiniDumpType: 2          # heap dump — required for `verifyheap` / object inspection
DOTNET_DbgMiniDumpName: /tmp/coredumps/%e-%p.dmp

and the collect + upload steps are if: always(), so a dump survives even when the shard is killed. It arrives inside the testResults-shard<N> artifact under collected-logs/dotnet-<pid>.dmp, alongside _meshweaver-test-trace.log and _meshweaver-memory-delta.log. Retention is 15 days — pull it before it expires.

🚨 Ask the trace log whether it is complete before you reason from it

_meshweaver-test-trace.log carries [FAULT] records — exception type, message and stack for everything logged at Warning or above with an exception. Its fault records are rate-bounded (100 per 10 s per process), so a storming process has records missing, and reasoning from an absence in a truncated log is how you conclude the wrong thing. The log always says when that happened:

grep FAULT-BUDGET collected-logs/_meshweaver-test-trace.log

Nothing back ⇒ every fault this process logged is in the file. Otherwise each hit names the running count of suppressed records, and a resuming fault records after suppressing N line states exactly how many were lost in that gap. A budget never silences a later fault — the window refills, so the fault next to a wedge is written even if an earlier storm saturated the allowance (issue #982).

# find the shard artifact (the crashing shard's is the big one — the others are ~0MB)
gh api repos/Systemorph/MeshWeaver/actions/runs/<RUN_ID>/artifacts \
  --jq '.artifacts[] | select(.name|test("shard")) | "\(.id)  \(.name)  \(.size_in_bytes/1048576|floor)MB  expired=\(.expired)"'

mkdir -p "$HOME/segv-dump" && cd "$HOME/segv-dump"
gh api repos/Systemorph/MeshWeaver/actions/artifacts/<ARTIFACT_ID>/zip > shard.zip
unzip -q shard.zip -d shard && find shard -name '*.dmp'

🚀 Start here on macOS: name the faulting frame with no container at all

Before reaching for Docker, answer "where did it fault and on what address" in about ten minutes, entirely on the Mac. A createdump core is a plain ELF64 file; every fact needed to name the faulting function is inside it plus one 138 MB download from Microsoft's public symbol server. This is the route that produced the 2026-08-09 confirmation below, and it needs no DAC, no emulation and no dotnet-dump — all of which are what make the container route slow and failure-prone.

Four steps, each a few lines of pure Python over the core (no debugger, no elfutils):

1. NT_SIGINFO → the kernel's verdict. Walk the PT_NOTE program headers; note type 0x53494749 carries si_signo / si_code / si_addr. si_code == 1 is SEGV_MAPERR, and si_addr is the dereferenced address — 0x0 versus a plausible-but-unmapped pointer is already the difference between a null read and a use-after-unload.

2. NT_FILE → the module load bases. Note type 0x46494c45 maps every file-backed range; the minimum start for libcoreclr.so is the load base you subtract to get an RVA.

3. The faulting ucontext — not NT_PRSTATUS, which createdump records from inside its own signal handler (its rip is waitpid in libc). Scan the PT_LOAD segments on 8-byte alignment for a gregs[23] block whose TRAPNO (index 20) is 14 (page fault) and whose RIP (index 16) lands inside libcoreclr; ERR (19) and CR2 (22) then decode the access — ERR == 0x4 is a user-mode read of a non-present page. Read the bytes at RIP straight out of the core through the same PT_LOAD table: that is the faulting instruction, and with the register values it names the exact dereference.

4. RVA → function name, via the public symbol server. The shipped libcoreclr.so is stripped to nine exported STT_FUNC symbols, so resolving against it fails — and that failure looks like the technique not working rather than the file being stripped. Fetch the separate debug file, keyed by build-id:

   curl -sL -o rt.tar.gz \
     "https://builds.dotnet.microsoft.com/dotnet/Runtime/<VER>/dotnet-runtime-<VER>-linux-x64.tar.gz"
   tar xzf rt.tar.gz shared/Microsoft.NETCore.App/<VER>/libcoreclr.so
   # <BUILD_ID> comes from that libcoreclr.so — see below
   curl -sfL -o coreclr.debug \
     "https://msdl.microsoft.com/download/symbols/_.debug/elf-buildid-sym-<BUILD_ID>/_.debug"
   

   Reading <BUILD_ID> is the same Python you already have — walk the section headers (e_shoff/e_shentsize/e_shnum at 0x28/0x3A, names via the e_shstrndx section), find .note.gnu.build-id, and read its note: namesz/descsz/type as three <I, then skip 12 + ((namesz+3) & ~3) bytes and take descsz bytes — that hex string is the id.

   Then walk coreclr.debug's .symtab for the STT_FUNC entry (st_info & 0xf == 2) whose [st_value, st_value + st_size) contains the RVA; symbol names come from the section its sh_link points at. It carries ~31 800 symbols, so the hit is exact — e.g. RVA 0x5cb1e1 → _ZN3WKS7gc_heap16background_sweepEv (WKS::gc_heap::background_sweep(), +0xa61).

Take the runtime version from collected-logs/symbols-<Project>/_runtimes.txt in the same artifact — CI's patch is regularly not the one you have locally, and a mismatched libcoreclr.so yields a different build-id and therefore no symbols at all.

You do need a container for the managed side

The managed commands — clrthreads / clrstack / verifyheap, i.e. anything that goes through the DAC — need the target's own libmscordaccore.so, a Linux ELF library. dotnet-dump analyze on macOS cannot load it, so those commands require a linux/amd64 container (CI runners are x64; Apple-silicon Docker is arm64, so --platform linux/amd64 and emulation are required — it is slow but it works).

🚨 This restriction is about the DAC, not about the file. A createdump core is a plain ELF64 that any tool can read; the native analysis in the previous section runs entirely on the Mac. Do not read "you need a container" as "a Linux core cannot be opened on macOS" — believing that has cost an investigation a slow multi-hundred-MB download and an emulated container for facts that were ten minutes of local Python away.

Stage the dump under $HOME, not /private/tmp — Colima does not mount /private/tmp, so a dump left in the scratch directory is invisible inside the container.

docker run --rm --platform linux/amd64 -v "$HOME/segv-dump/shard/collected-logs:/dumps" \
  mcr.microsoft.com/dotnet/sdk:10.0 bash -c '
    curl -sL https://aka.ms/dotnet-dump/linux-x64 -o /tmp/dotnet-dump && chmod +x /tmp/dotnet-dump
    /tmp/dotnet-dump analyze /dumps/<name>.dmp -c "<command>" -c "exit"
  '

Use the curl single-file download above, not dotnet tool install -g dotnet-dump — the tool installer fails under qemu emulation with There was an error reflecting type '…DotNetCliTool', and the resulting dotnet-dump: command not found looks like a PATH problem rather than what it is.

Free triage before you start a container

strings on the raw core answers two questions in seconds, with no DAC and no emulation:

strings -a dump.dmp | grep -oE "AccessViolation|FailFast|SIGSEGV" | sort | uniq -c
strings -a dump.dmp | grep -oE "/usr/share/dotnet/shared/Microsoft.NETCore.App/[0-9.]+/lib[a-z]+\.so" | sort -u

• AccessViolation + FailFast ⇒ the runtime tripped over bad memory; this is not a managed exception that someone forgot to catch.
• The second line reveals which runtime patch CI actually ran. It is regularly not the one you have locally (2026-08-03: CI on 10.0.10, local on 10.0.9) — on its own a candidate explanation for "only fails on CI", and worth eliminating before blaming load or shard composition.

The command sequence that actually answers the question

Run these in order; each one kills off a class of hypothesis.

1. clrmodules — confirm you have the right process first. Grep for the test assembly (MeshWeaver.<X>.Test.dll). Some dumps are produced deliberately (a DAC/unload probe test), so check you are not analysing an intentional crash. Also count copies of a given assembly: more than one copy means duplicate statics across AssemblyLoadContexts, which silently defeats any process-wide lock.

2. clrthreads — find the faulting thread. It is usually the one in Cooperative GC mode and/or flagged (GC); the Exception column is typically empty for a signal death.

3. clrstack -all — every managed stack in one pass. Grep it for the suspect frames, and to test concurrency hypotheses: if only one thread is inside the library you suspect of being torn by concurrent access, a locking fix is not the answer.

4. setthread <DBG-id> + clrstack -f — the faulting thread interleaved with native frames and module names. This is the definitive stack.

5. verifyheap — clean output means no GC heap corruption, which distinguishes "this code is the culprit" from "this code is an innocent victim of corruption elsewhere". Requires DbgMiniDumpType: 2 (already set).

   ⚠️ It may not run at all. On a large dump dotnet-dump can die inside its own scan:

   Scanning heap: 6 MB / 266 MB (2%)...
   Unhandled exception: System.NullReferenceException
      at Microsoft.Diagnostics.DebugServices.Implementation.Utilities.Invoke(…)
   

   That is the TOOL failing, not a verdict — it says nothing about the heap either way. Do not read it as "clean" and do not retry it hoping for a different answer (2026-08-06, a 1.2 GB FutuRe dump). When it happens, step 5 is simply unavailable: say so rather than implying culprit-vs-victim was established, and get the confidence elsewhere — a deterministic repro is worth more than verifyheap anyway, because it pins the cause instead of describing the wreckage.

6. eeheap -gc — heap size, to rule in/out memory pressure. Cross-check against _meshweaver-memory-delta.log in the same artifact for the deltas around the crash.

When clrstack on the faulting thread prints NOTHING

An empty managed stack on the crashing thread is not a dead end — it is the answer to a different question: the thread is runtime-internal (a GC / finalizer / EE worker), so there are no managed frames to show and SOS has nothing more to give. Switch to native symbols.

Two traps make this look impossible when it is not:

• lldb cannot map modules out of a createdump minidump. image list shows only the executable, so bt prints bare addresses and every frame looks unsymbolisable. You symbolise by hand instead — the addresses are perfectly good.
• A backtrace address is a RETURN address. The frame you want is the target of the call immediately before it, not the function the address lands in. Disassembling the few bytes ahead of the return address is what names the real callee.
• 🚨 The spectacular stack belongs to a DIFFERENT thread. When the faulting thread prints nothing, clrstack -all still prints everyone else — and in this workload that is two ~130/~196 frame application stacks. Reading one of those as "the crash" is the single most expensive mistake available here: it names a culprit that was merely running. Identify the faulting thread FIRST (below), and only then read a stack.

The recipe, end to end (each step is seconds). This is the elfutils spelling of the same four facts the macOS-native section above extracts with plain Python — eu-readelf and eu-addr2line are elfutils binaries that a stock macOS does not ship (and Homebrew does not carry a working eu-readelf either), so run the whole block inside the container, not on the Mac. If you only need the faulting frame, prefer the pure-Python route above and skip the container entirely.

# 1. Which thread, and what did the kernel say? si_code 1 = SEGV_MAPERR; si_addr is the deref'd pointer.
eu-readelf -n dump.dmp | grep -E "SIGINFO|si_signo:|fault address:|  pid: "

# 2. Load base of libcoreclr in the crashed process (NT_FILE mappings inside the core).
eu-readelf -n dump.dmp | grep libcoreclr.so | head -1     # e.g. 7f17d1000000-...

# 3. Native backtrace (bare addresses are expected).
lldb --core dump.dmp -o "thread list" -o "bt all" -o quit

# 4. RVA = frame address − load base. Fetch the MATCHING symbols by build-id and resolve.
BID=$(eu-readelf -n /usr/share/dotnet/shared/Microsoft.NETCore.App/<ver>/libcoreclr.so \
      | awk '/Build ID/{print $3}')
curl -sfL -o coreclr.dbg \
  "https://msdl.microsoft.com/download/symbols/_.debug/elf-buildid-sym-$BID/_.debug"
eu-addr2line -f -C -e coreclr.dbg 0x<RVA>

Run the block in a linux/amd64 container whose runtime patch matches _runtimes.txt from the artifact (the shard already records it) — then libcoreclr.so is byte-identical and the build-id lookup succeeds. The .debug carries a symbol table but no DWARF lines, so you get function names, not line numbers; that is enough to name the phase.

Which thread faulted, and its native backtrace — no lldb, no container

Two plain-Python steps over the ELF core answer both, on macOS, in seconds. They are what turns "something segfaulted in the GC" into "the background GC thread segfaulted, and no application frame is on it".

1. Which thread. Find the faulting ucontext (below), read its pre-signal RSP, then match that against each thread's NT_PRSTATUS rsp: the signal frame sits just ABOVE the crashing thread's recorded stack pointer, because createdump sampled that thread while it was still inside the handler. Cross-check against threads in dotnet-dump — the OS ids line up, and createdump's own "Crashing thread NNNN" line in the shard log is the same number in hex.

2. Its backtrace. Walk that thread's stack upward from the pre-signal RSP and keep every 8-byte value that lands inside libcoreclr's mapping; resolve each against the build-id .debug (nm -C --defined-only -S). Saved return addresses come out in order, and runtime threads have short, unmistakable stacks:

   CorUnix::CPalThread::ThreadEntry
    → CreateSuspendableThread::$_0::__invoke
      → WKS::gc_heap::bgc_thread_stub → WKS::gc_heap::bgc_thread_function+0xdc
        → WKS::gc_heap::gc1+0xf6 → WKS::gc_heap::background_sweep+0xa61   ← FAULT
   

   That is a dedicated background GC thread, and it settles culprit-vs-victim for the whole class: no managed frame is on it, so no application code can be executing the fault. Scan reads whatever is on the stack, so expect a few data pointers among the return addresses — the ordered chain of function symbols is the signal.

Recovering the FAULTING registers (PRSTATUS is not them)

createdump records the crashing thread's PRSTATUS from inside its own signal handler, so the rip you read there is waitpid in libc — not the fault. The real faulting context is the ucontext_t the kernel pushed onto the alternate signal stack; find it by scanning that stack for the mcontext signature (no debugger needed — plain Python over the ELF core):

# gregs[] at ucontext+40; indices: RIP=16, ERR=19, TRAPNO=20, CR2=22
# match TRAPNO==14 (page fault) and a RIP inside libcoreclr's mapping

CR2 is the dereferenced address and ERR decodes the access (0x4 = user-mode read of a non-present page). Then disassemble a window around the faulting RVA — that names the exact dereference, which is what turns "it segfaulted in the GC" into a specific claim.

2026-08-06: MeshWeaver.FutuRe.Test exit=139 (run 31083356138)

Resolved this way in minutes after the 2026-08-04 attempt stalled for want of symbols. Crash was mid-run (75 s in, 44 ms into a fresh fixture's PreWarmNodeTypeHubs), not at process exit. The faulting thread carried no managed frames and symbolised to:

CorUnix::CPalThread::ThreadEntry → CreateSuspendableThread
  → WKS::gc_heap::bgc_thread_function() → WKS::gc_heap::gc1()
    → WKS::gc_heap::background_sweep()      ← faulting frame

and the recovered ucontext pinned the instruction (TRAPNO=14, ERR=0x4, CR2=0x0):

mov rax, QWORD PTR [r15]     ; rax = object->MethodTable      (r15 = the object being swept)
and rax, 0xfffffffffffffff8  ; strip the GC low bits
mov ecx, DWORD PTR [rax]     ; ← FAULT: read MT->m_dwFlags, rax == 0

The swept object's MethodTable pointer is exactly NULL. That single fact is worth more than the whole stack, because of what it rules out:

• It is not the collectible-ALC use-after-unload shape. A dangling pointer into a freed LoaderAllocator is a non-null address that happens to be unmapped. Zero is not that. Several earlier fixes (and the alc-unload-probe workflow) were aimed at that hypothesis; this dump does not support it. Do not keep paying it forward.
• It is not MeshWeaver corrupting the heap. There is no unsafe, no GCHandle, no pinning, no stackalloc of object memory and no custom GC configuration anywhere in src/ — pure managed code has no way to zero a MethodTable. ⚠️ 2026-08-12: that argument is weaker than it reads. It is sometimes restated as "the crashing process ships no native libraries", and that restatement is false. NT_FILE in dotnet-3592.dmp lists 19 file-backed .so mappings — libcoreclr, libclrjit, libSystem.Native, libSystem.Security.Cryptography.Native.OpenSsl, libcrypto/libssl, libicu*, libstdc++, libhostpolicy, libhostfxr — beside the 132 managed .dlls. What holds is the narrow claim (the app's own code is pure managed); what does not hold is "there is no native code in the process, therefore nothing could have written the zero".

A zero MT inside the swept range means the sweep walked memory the GC believed held an object but that is in fact zeroed. ⚠️ 2026-08-17: the two guesses that used to follow here — "a gap that was never filled with a free object, or a walk that ran past the true allocated end" — are both now measured FALSE, for this dump and for the 2026-08-17 one. Here the zeroed block is a well-formed 32-byte object whose neighbours walk cleanly onto it and whose successor's header sits exactly 32 bytes later — so the walk is neither desynchronised nor past the allocated end; only the header word is gone. See the 2026-08-17 section for the second shape. That is runtime-internal bookkeeping either way, so treat it as a CoreCLR GC issue and report it upstream with this dump rather than "fixing" it here. What MeshWeaver contributes is the workload that provokes it: per-[Fact] mesh build + teardown with collectible NodeType assemblies loading and unloading, and 48 gen2 GCs in 75 s.

🚨 Do not "fix" this by turning off concurrent GC in the test host. That hides the fault without changing anything about why it happens, and the same workload runs in prod. This was tried anyway (#1274, <ConcurrentGarbageCollection>false</ConcurrentGarbageCollection> on MeshWeaver.FutuRe.Test) and it did not even hide it — see the 2026-08-12 section, which measures why and removes it again.

2026-08-09: reproduced twice more — the verdict is not a one-off, and +0x5cb1e1 is its fingerprint

Both exit=139s of 2026-08-09 resolve to the identical fault, days and many merges after the original analysis. On runtime 10.0.10 the whole comparison collapses to one number: RVA 0x5cb1e1, WKS::gc_heap::background_sweep()+0xa61. Compare that first on any recurrence.

The mutator phase varies while the faulting instruction does not — and that is the point. One lands mid-teardown, two land inside a fresh test. A background-GC sweep runs concurrently with whatever the mutator is doing, so an identical fault at three different phases is what this diagnosis predicts, and it is the opposite of what a teardown-ordering defect would produce (those reproduce at one phase, by construction). Do not read the phase of any single sighting as evidence about the cause; read the RVA.

⚠️ "Read the RVA" is now known to be half a rule — see the 2026-08-12 section. An RVA is only meaningful against the runtime build it was taken on, and the runtime moves under you. Two dumps on 10.0.11 fault at 0x5cf24c, which is not background_sweep. The portable fingerprint is the instruction plus register state (8b 08 = mov ecx,[rax] with RAX=0x0, si_addr=0x0), which is identical across all five sightings; the RVA and the function name are not.

Two further things these sightings settle, both re-litigated at length in #613:

• The teardown work neither caused nor cured it. The 11:05Z tree contains #967 (node ALCs unload at the end of teardown) and #996 (pending Rx timers no longer root disposed hubs) — it crashed 87 minutes after the latter merged. Across both processes' _meshweaver-test-trace.log, all 22 and all 44 DISPOSE_DONE records read teardown clean — all pooled I/O joined, async dispose queue drained, with zero DISPOSE_QUIESCE_LEAK and zero DISPOSE_DIRTY_TEARDOWN.
• It is not the quiescing-leak family (#981). Over 18 failing shard-job logs across those two days, the signal death and the left Observe subscriptions pending past the Quiescing budget failure each occur twice and never co-occur — not in a run, a shard, or a job. In one job FutuRe.Test ran 69 s to exit=0 while the quiescing leak fired in Hosting.Monolith.Test.

Check the trace log's completeness before reasoning from it (see the FAULT-BUDGET note above): in the 11:05Z file all four suppression windows belonged to a later project's pid, so the crashing process's record was whole — which is what makes "zero quiesce leaks" evidence rather than an absence.

2026-08-12: on runtime 10.0.11 it is plan_phase, not background_sweep — the frame moved, the fault did not

Three dumps from the post-#1274 window, resolved against 10.0.11. All fault at the same instruction with the same registers as the three 10.0.10 sightings — and in a different GC function.

This is not RVA drift. Resolved against the same 10.0.11 coreclr.debug, the old fingerprint 0x5cb1e1 still lands in background_sweep (+0xad1, versus +0xa61 on 10.0.10 — the function moved 0x70 bytes between builds). The two functions are disjoint (background_sweep at 0x5ca710+0x140b; plan_phase at 0x5ccd50+0x4b56), and the crash is 0x4000 bytes away from the old one. The frame genuinely changed.

The faulting instruction is MethodTable::GetComponentSize inlined into the heap walk — mov ecx, [rax]; test ecx,ecx; js … reads the MT flags dword and branches on enum_flag_HasComponentSize. RAX is the MethodTable pointer, and it is zero. Same dereference as before.

What this settles, and what it does not

It settles the question #613 was reopened on. The reopen offered two possibilities and could not choose: (1) the System.GC.Concurrent:false setting is not reaching the crashing process, or (2) the fault is not in background_sweep. It is (2), measured directly. plan_phase is the stop-the-world plan/relocate step of a blocking collection; it is on the path of every blocking GC, including one that runs with concurrent GC disabled. So the premise #1274 was built on — "only background GC can reach background_sweep, therefore disabling it removes the fault" — does not cover the crash that is actually happening, and the unchanged CI rate (4.2 % → 3.8 %) needs no appeal to hypothesis (1) to explain it.

Supporting, though weaker than the frame: System.GC.Concurrent is present as a host config-property key in the crashing process's memory (3 hits in dotnet-3592.dmp, including the UTF-16 AppContext copy), which is consistent with the runtimeconfig having carried it.

It does not settle what zeroes the MethodTable. plan_phase also runs as the foreground collection during a background GC, so this frame alone does not prove concurrent GC was off at the moment of the fault. What five dumps do establish is the invariant: the GC walks the heap and finds an object header holding exactly zero. Which phase discovers it is incidental — and "background-GC bug" named the incidental part. Anything built on that name (a mitigation, an upstream report, a closure as external) needs re-deriving from the invariant instead.

Two hypotheses these dumps falsify — check them off, do not re-open them

Both were proposed again while this was being analysed, and both are decidable from the dump alone:

• Use-after-unload of a collectible ALC (the MessageHubGrain.OnDeactivateAsync "KNOWN GAP": loadContext.Unload() orders only on the hub's DisposalCompleted, and pooled I/O leaves are mesh-shared). Falsified twice over. RIP is inside a file-backed mapping — /usr/share/dotnet/shared/Microsoft.NETCore.App/10.0.11/libcoreclr.so, resolving to a named runtime symbol — not in unloaded or collected JIT code. And a freed LoaderAllocator yields a non-null unmapped pointer; si_addr here is exactly 0x0. This is the same distinction the exit=134 analysis already drew, in the other direction. The gap may be real and worth closing on its own merits; it does not produce this fault.
• "Bounded disposal gives up and unloads on top of live work." Falsified for all three. dotnet-3433's trace has 29 DISPOSE_DONE … teardown clean, zero DISPOSE_QUIESCE_LEAK, zero DISPOSE_DIRTY_TEARDOWN, and no disposal-timeout record of any kind — the only timeout-shaped lines are MEM_WATCHDOG memory samples. Its last teardown finished cleanly in 2009 ms and the next fixture had already started (CTOR / INIT_START / INIT_BASE_DONE at 14:06:27.943) when the fault landed ~1 s later. dotnet-3592 is the same shape: 40 clean DISPOSE_DONE, zero leaks.

Consequences recorded here so they are not re-litigated

• <ConcurrentGarbageCollection>false</ConcurrentGarbageCollection> was removed from MeshWeaver.FutuRe.Test.csproj. It is a mitigation whose mechanism is not the one firing, it measurably changed nothing, and left in place it reads to the next reader as "GC was already ruled out" — the most expensive kind of comment. The instruction above ("do not fix this by turning off concurrent GC") and the tree now agree again.
• Scope it as ALC-heavy assemblies, not FutuRe.Test. FutuRe.Test is where it recurs, but a MeshWeaver.Hosting.Orleans.Test exit=139 has been observed once. Both build and tear down a mesh with collectible NodeType assemblies per [Fact]; that workload, not the project, is the sample.
• The crashing process's phase, again, is not evidence. dotnet-3592's last trace record is FutuReAnalysisTest DISPOSE_INVOKED at 11:21:52.614; 40 DISPOSE_DONE records in that process read teardown clean, with zero DISPOSE_QUIESCE_LEAK and zero DISPOSE_DIRTY_TEARDOWN.

The dump IS uploaded — and here is how to fetch 200 MB in ten minutes

Two beliefs have repeatedly stalled this investigation. Both are wrong:

• "The .dmp is never uploaded; CI ships symbols with nothing to symbolicate." It is uploaded. .github/workflows/dotnet-test.yml copies /tmp/coredumps/*.dmp into collected-logs/ before the testResults-shard<N> upload, and both dumps above came straight out of that artifact.
• "The download is infeasible." Through gh api it is — that measures ~11 KB/s, which is where two investigations gave up. The artifact's blob URL supports HTTP range requests, so fetch it in parallel chunks instead. Budget ten minutes; 2026-08-17 measured 91 seconds for 199 MB with 24 × 8 MB chunks, six at a time (a xargs -P spelling dies with "command line cannot be assembled, too long" — the SAS URL is enormous, so drive the loop from a script that holds it in a variable):

# 1. mint the redirect (do NOT let gh follow it)
TOK=$(gh auth token)
URL=$(curl -s -o /dev/null -D - -H "Authorization: Bearer $TOK" \
  "https://api.github.com/repos/Systemorph/MeshWeaver/actions/artifacts/<ARTIFACT_ID>/zip" \
  | grep -i '^location' | sed 's/^location: //I' | tr -d '\r')
# 2. fetch chunk i of N with a plain range request
curl -sf -r "$START-$END" -o "part-$i" "$URL"

🚨 Two traps, both of which silently corrupt the result:

• The SAS expires in ~10 minutes. Re-mint URL before each wave and fetch only the chunks that are still missing — a resumable loop, not one long run.
• Use curl -sf, never a bare curl with --retry. Without -f, an expired-SAS 403 writes its 544-byte AuthenticationFailed XML body into the part file and curl reports success; with --retry it will also do that on top of a chunk that had already completed. The first attempt here assembled an 8 MB "208 MB" file that way. Verify every chunk's exact byte length before concatenating.

2026-08-17: this sighting's zeroed block is a GC free-list item — and the shape is not constant

dotnet-4418.dmp (run 32033793544, commit 8f163c43a, shard 2, runtime 10.0.11) faults at a third RVA — 0x5d5ab2 = WKS::gc_heap::find_first_object+0x132 — with the same si_addr=0x0 / TRAPNO=14 / ERR=0x4 / mov r14d,[r10], R10=0 state as the six before it. All three fingerprints resolved against the same 10.0.11 symbols land in disjoint functions (background_sweep 0x5ca710+0x140b, plan_phase 0x5ccd50+0x4b56, find_first_object 0x5d5980+0x387), so the frame keeps genuinely moving while the fault does not.

Which is why the frame was never the interesting part. Reading the cursor block as a CoreCLR free object identifies it immediately, and that is the finding:

+0x00 MethodTable    = g_pFreeObjectMethodTable   (SOS: dumpmt <that MT> -> "Free MethodTable")
+0x08 numComponents  = the free block's length
+0x10 free_list_slot = next item on the free list
+0x18 free_list_undo

The faulting block had length and both links correct and reciprocal — its free_list_slot neighbour's free_list_undo pointed back at it, 24 + length landed exactly on the next valid object header, and following the list forward walked a dozen more well-formed free items. Only the header word was zero. So:

A free-list item lost its g_pFreeObjectMethodTable header — one 8-byte zero store on a word only the GC writes — and the next heap walk to reach it faulted reading MT->m_dwFlags at 0.

On the next dump, do this before anything else: read the cursor as a free object. If length + links are consistent you have the free-list shape and the discovering frame does not matter.

🚨 But the shape is NOT the same in every sighting — do not generalise this one. Re-running the same test against dotnet-2969.dmp (2026-08-06, 10.0.10, background_sweep) falsifies the free-list reading there: its zeroed block is a 32-byte, live-shaped object — +0x08 holds a genuine heap reference (implausible as a free-object length), its neighbours walk cleanly onto it, and the next valid header sits exactly 32 bytes later. dotnet-3433 (2026-08-12) has the same live-shaped payload. So there are two shapes on record:

What survives across both — and is the honest invariant to hand upstream — is narrower than either: a single 8-byte header word reads as exactly zero while the rest of its block stays coherent, and the next GC heap walk to reach it faults reading MT->m_dwFlags at address 0. Whether the block was a live object or free space is not constant, so an upstream report must not be written around the free-list detail alone.

Three further things that recurrence established:

• verifyheap is not the way to get the culprit-vs-victim verdict here — SOS segfaulted on it (and on several dumpobj/dumpmt calls) on this 1.29 GB dump. Walk the regions by hand instead: take each region's begin/allocated from eeheap -gc and step object by object. That run covered 1,564,794 objects and 20,492 free items across all 43 regions with zero stalls and found exactly one zero-header block — a stronger statement than verifyheap would have made.
• Scan for references to the cursor. Every PT_LOAD searched for the cursor's 8-byte value yielded only the two free-list links, one stack slot on the faulting thread, and the saved RAX in the signal ucontext. No managed object points at free space — which is what closes the "could our code have written it" question, without relying on the weaker "no native code in the process" claim corrected above.
• The collectible-ALC hypothesis is dead a third time. For this sighting on the general ground: a free-list item belongs to no assembly, so it has no MethodTable to dangle, no ALC, no LoaderAllocator, and the word that broke is in the GC heap, not a loader heap. For the live-shaped sightings the earlier grounds still carry it — RIP inside file-backed libcoreclr, and si_addr exactly 0x0 where a freed LoaderAllocator yields a non-null unmapped pointer. (11 collectible NodeType ALCs were live at the 2026-08-17 fault and the block's stale contents are their dead objects — that is the workload that grows gen2 free lists, not the cause.)

2026-08-18: sighting #8 — the faulting thread is the BGC thread, and the two re-entrant hub constructions are bystanders

dotnet-4411.dmp (MeshWeaver.FutuRe.Test, exit=139, runtime 10.0.11, libcoreclr build-id 989b56df…) is the same fault a fourth time on the same runtime, at the first of the three known RVAs:

si_signo=11  si_code=1 (SEGV_MAPERR)  si_addr=0x0   TRAPNO=14  ERR=0x4  CR2=0x0
RIP = libcoreclr + 0x5cb171 = WKS::gc_heap::background_sweep() + 0xa61
bytes@RIP: 49 8b 07  mov rax,[r15] / 48 83 e0 f8  and rax,~7 / 8b 08  mov ecx,[rax]   ← RAX = 0
cursor R15 = 0x7f00266428e0 — a 24-byte block on a genuine object boundary (a walk from R8 reaches
it after 4 well-formed objects), header word exactly 0, next valid header 24 bytes later

What is new is not the fault — it is which thread took it. This dump was pulled because clrstack -all shows two threads ~130 and ~196 frames deep in re-entrant hub construction (CreateHub → Build → … → CreateHub → Build), one via KernelContainer.StartActivityControlPlane and one via DataExtensions.<GetDefaultConfiguration> → GetWorkspace(), both bottoming out in SynchronizationStream..ctor → GetHostedHub(…, Always). It reads like a cause. It is not:

The crashing thread is the dedicated background GC thread: no managed frames, no application code, nothing that could be constructing anything. The re-entrant constructions are concurrent mutators — the workload, exactly like the 11 collectible ALCs in the 2026-08-17 sighting. And they are not even rare: a green local run of the same project logs 1,350 of them (see the re-entrancy bullet under Reading the result honestly).

Two facts from the same dump that keep the "our code corrupted the heap" branch closed: the process maps 19 native modules and all 19 are the runtime's or the OS's (libcoreclr, libclrjit, libSystem.Native, OpenSSL, ICU, libc/libstdc++ …) — no third-party native code — and of the 129 managed assemblies it loads, none comes from a project that sets AllowUnsafeBlocks: the only one in the repo is memex/Memex.Client, which this process never loads. So unsafe is not merely absent from the sources on the stack — it could not have compiled into anything in the process.

Re-established fleet-wide on 2026-09-04, from the sources rather than from one dump, because the same branch has to stay closed for #890 and its hosts live in the other repository now: git grep AllowUnsafeBlocks over every tracked *.csproj/*.props returns zero in Systemorph/MeshWeaver and zero in Systemorph/MeshWeaver.Plugins — so no assembly either repo builds can contain unsafe code at all. The control: the same pattern does match where the setting exists (memex/Memex.Client and MeshWeaver.BusinessRules.Generator in older checkouts), so the zero is a measurement and not a broken grep. The only span-level memory operations in either src/ tree are one read-only MemoryMarshal.AsBytes (PatchStringSplice.cs) and one fixed-size stackalloc byte[64] (OciDigest.cs) — neither can write out of bounds. MeshWeaver code cannot scribble on the heap directly; if the heap is being corrupted, the writer is the runtime or a native module the runtime loads.

2026-09-03: sighting #9 — a FOURTH frame (background_mark_simple1), and the CI gate named the wrong cause

MeshWeaver.Futu-49849.dmp (MeshWeaver.Plugins run 33778900486, Portal hosts (shard 1), runtime 10.0.11, libcoreclr build-id 989b56df… — the same binary as sighting #8). The register allocation differs and nothing else does:

si_signo=11  si_code=1 (SEGV_MAPERR)  si_addr=0x0   TRAPNO=14  ERR=0x4  CR2=0x0
RIP = libcoreclr + 0x5d89e7 = WKS::gc_heap::background_mark_simple1(unsigned char*) + 0x827
bytes@RIP: 49 8b 08  mov rcx,[r8] / 48 83 e1 f8  and rcx,~7 / 44 8b 09  mov r9d,[rcx]   ← RCX = 0

Sightings #4–#8 read 8b 08 = mov ecx,[rax] with RAX=0; this one is the same three-instruction sequence — load the MethodTable, strip the GC bits, read MT->m_dwFlags — through r8/rcx instead. The instruction plus register state is the portable fingerprint, exactly as the 2026-08-12 note says; the frame is not. Four disjoint gc_heap functions are now on record against the same 10.0.11 symbols:

background_mark_simple1 is the background-GC mark phase, so the family now spans mark, plan and sweep plus a heap walk — which is what "the GC finds a zeroed object header wherever it next looks" predicts, and which no single-phase teardown hypothesis does.

Disposal was clean, and the record is provably complete. In _meshweaver-test-trace.log both FAULT-BUDGET suppression lines belong to a different pid (2722), so the crashing process's (pid=49849) record has no gaps — the precondition the section above insists on. Every DISPOSE_DONE in it reads teardown clean — all pooled I/O joined, async dispose queue drained, with leakedIoLeaves=0, zero DISPOSE_QUIESCE_LEAK and zero DISPOSE_DIRTY_TEARDOWN. The last record is DISPOSE_INVOKED for test 43 of 43, alc=1, gc2=33 in ~62 s.

🚨 The CI gate asserted a cause it could not know, and a fix was written against it

This sighting is recorded as much for what the reporting did as for the dump. xUnit v3 reports a child process killed by a signal through the same Catastrophic failure: banner it uses for an exception that escaped onto a non-test thread, and dotnet test flattens both to exit 1 — so the child's real code (139) survives only in the banner's text. MeshWeaver.Plugins' classify-test-run.py was not reading it, and printed its static HOST_NONZERO_NO_FAILING_TEST remedy: "an exception escaped on a non-test thread … typically a DI resolve inside a single-argument Subscribe(onNext) body … the stack is in the teardown-straggler capture below … fix it by GATING the callback on the hub's teardown state." All four clauses are false here, and the shard's very next line was (no teardown-straggler capture file found for this project).

A core PR was then authored against that invented cause, quoting the gate's own remedy as its evidence, before anyone opened the dump sitting in the same job's artifact. Read the dump before believing a gate's prose — and see MeshWeaver.Plugins#1289, which makes a signal death its own classification, points it at this page, and stops the Rx/DI remedy from printing on a crash.

Base rate, measured

Plugin Catalog CI, 1,197 runs over 2026-08-29 → 2026-09-03, every failed job's annotations read (7,362 rows) plus all 92 cancelled main runs: 5 occurrences, all MeshWeaver.FutuRe.Test, one of them on main (run 33575831719). That is 0.74 % of non-cancelled runs overall and 2.5 % since onset; no other suite shows the signature. Two of the five ran against core commits containing #3072 (87 and 156 ahead of 17ee6d9fa), so the #3026 fix reduced the rate without closing it — consistent with this page's standing verdict that MeshWeaver supplies the workload, not the defect.

2026-09-06: sighting #10 — the fault is OUTSIDE gc_heap, and a symbolization slip named an impossible frame

MeshWeaver.Futu-50119.dmp (MeshWeaver.Plugins run 33923006420, job 101185319665, Portal hosts (shard 1), runtime 10.0.11, libcoreclr build-id 989b56df… — the same binary as sightings #8 and #9). Filed as MeshWeaver.Plugins#1347.

This is the first sighting whose faulting frame is not in gc_heap, and it is worth reading carefully, because the first pass at it produced a frame that cannot exist and a cause (collectible-ALC teardown) that this page has falsified three times.

🚨 The trap: an ELF file offset is not a virtual RVA, and here they differ by 0x1000

The first analysis reported HostCodeHeap::AllocMemory_NoThrow+0xe7 at RVA 0x372887 and concluded "the JIT allocating for a collectible ALC that is already torn down" — which pointed straight back at #2136 and at AlcLeaseRegistry. Every part of that is wrong, and one cheap check falsifies it:

$ llvm-objdump -d --start-address=0x372860 --stop-address=0x3728a0 libcoreclr.so
  372881: 49 8b 4e 18      movq  0x18(%r14), %rcx
  372885: 49 03 4e 28      addq  0x28(%r14), %rcx     ← 0x372887 is INSIDE this 4-byte instruction
  372889: 48 39 c8         cmpq  %rcx, %rax

RIP always points at an instruction boundary, so 0x372887 is not a possible faulting address at all — and addq 0x28(%r14),%rcx could not produce CR2 = 0x4 in any case. The slip is that in this binary a PT_LOAD maps file offset 0x371887 at vaddr 0x372887: file offset = vaddr − 0x1000. The symbol table is indexed by vaddr; the bytes are found at a file offset. Resolve a symbol with one and read bytes with the other and you get a confidently-wrong frame that still "matches bytes".

Three independent measurements agree on the real address:

1. NT_FILE in the core gives libcoreclr's load base as 0x7f4434600000 (the mapping whose page offset is 0, not the executable segment). RIP = 0x00007f4434973887 ⇒ RVA 0x373887.
2. The faulting bytes the first analysis itself quoted — 49 8b 07 / 8b 70 04 / 83 c6 f8 — occur at exactly one place in the 7 MB binary, vaddr 0x373884.
3. 0x373887 is an instruction boundary; 0x372887 is not.

RVA 0x373887 -> LCGMethodResolver::GetCodeInfo(unsigned*, unsigned*, CorInfoOptions*, unsigned*) + 0x1f7

The signal and the faulting registers, re-derived

NT_SIGINFO: signo=11 code=1 (SEGV_MAPERR) addr=0x4. The rt_sigframe's sigcontext (found by scanning for TRAPNO=14, ERR=0x4, CR2=0x4 and a RIP inside libcoreclr's mapping — exactly one match in the whole core):

RIP = 0x00007f4434973887   (RVA 0x373887)   CR2 = 0x4   TRAPNO = 14   ERR = 0x4
RAX = 0x0000000000000000       R15 = 0x00007f3fd1e37188
RBX = 0x00000000ba41bf48       R13 = RDI = 0x00007f3e78d4c010

What the instruction actually does — and why "the JIT allocating code" is the wrong reading

LCGMethodResolver::GetCodeInfo does not allocate code. It fetches a dynamic method's IL from a managed byte[] (src/coreclr/vm/dynamicmethod.cpp):

U1ARRAYREF dataArray = (U1ARRAYREF) getCodeInfo.Call_RetOBJECTREF(args);
DWORD codeSize = dataArray->GetNumComponents();   // movl 0x8(%r15), %ebx      -> RBX
NewArrayHolder<BYTE> code(new BYTE[codeSize]);    // callq _Znam@plt           -> R13
memcpy(code, dataArray->GetDataPtr(), codeSize);  // movq (%r15),%rax          -> RAX = MethodTable
                                                  // movl 0x4(%rax),%esi  ← FAULT (MT->m_BaseSize)

So R15 is the managed byte[], RAX is its MethodTable word, and the fault is a read of MT->m_BaseSize at offset +4 off a null MethodTable — hence CR2 = 0x4 rather than 0x0. That is this page's invariant exactly, reached from the application side instead of from inside the collector: an object's MethodTable word reads as exactly zero.

The object is not merely header-less, it is not a byte[] at all any more:

0x7f3fd1e37188: 0x0000000000000000   <- R15+0, read as MethodTable  => RAX = 0
0x7f3fd1e37190: 0x00007f43ba41bf48   <- R15+8, read as Length       => RBX = 0xba41bf48

RBX is 3,124,870,984 — the low half of a pointer, read as an IL length. No method has 3 GB of IL, and R15 lies in an anonymous mapping (no NT_FILE entry), i.e. heap rather than file-backed runtime code. The block is a recycled/free-list-shaped run of pointers, and its contents are identical at fault time and at dump time ([R15+8]'s low dword still equals RBX), so this is not a stale post-hoc read.

What the neighbouring memory names — the workload, precisely

24 bytes past the dead array sits a live managed System.String (MethodTable, then length 33, then UTF-16):

0x7f3fd1e371d0: 0x00007f43b668dd58     <- String MethodTable
0x7f3fd1e371d8: 0x21 (=33)             <- length
0x7f3fd1e371dc: "LastReleaseRequestHandledAtSetter"

LastReleaseRequestHandledAt is a property of NodeTypeDefinition (src/MeshWeaver.Graph.Contract/NodeTypeDefinition.cs), and core src/ contains no System.Reflection.Emit or DynamicMethod of its own — so the dynamic method being JIT-compiled is a property-setter stub minted by System.Text.Json's reflection-emit member accessor while serializing a mesh node type.

That gives the family a much sharper description than "ALC-heavy assemblies": the provoking workload is LCG / collectible dynamic methods created per serialized type, and the fault is the JIT fetching IL for one whose backing array has already been recycled.

🚨 Stated as a hypothesis, not a result: System.Text.Json's ReflectionEmitCachingMemberAccessor caches these stubs behind a sliding expiration and evicts them on a timer. An eviction that makes a DynamicMethod collectible while a JIT compilation of it is still in flight would produce exactly this. That is the next discriminator, and it is not settled here.

Consequences — two records on this page need correcting

• +0x1f7 is not new. The 2026-08-24 crash (MeshWeaver.Hosting.Orleans.Test, exit=139) that #2136 was written against is recorded as LCGMethodResolver::GetCodeInfo+0x1f7 with the same instruction — and we now know that instruction reads a managed array's MethodTable word. Its rax held UTF-16 text, which under the correct reading means the array's MethodTable slot held recycled string data, the same corruption with a different garbage value. It is not "JIT-compiling a dynamic method whose collectible-ALC allocator had already been unloaded", because the instruction does no allocation and touches no LoaderAllocator. Whether #2136 fixed anything real is a separate question this dump cannot answer; what it settles is that the frame was misread.
• The scope line "FutuRe.Test only" is stale. MeshWeaver.GitSync.Test took the same exit=139 on Plugins#1191, run 34013024540, 2026-09-06 — a second suite, and one that also serializes node types heavily. Re-measure the base rate against the LCG workload, not the project. ⚠️ Superseded in part by sightings #11/#12 below: the fault returned to background_sweep on a dedicated BGC thread the next day, so "the LCG workload" describes this sighting, not the family's boundary. The base rate was re-measured there.

Reproducing this read

Runtime binary: https://builds.dotnet.microsoft.com/dotnet/Runtime/10.0.11/dotnet-runtime-10.0.11-linux-x64.tar.gz. Symbols by build-id, no dotnet-symbol needed: https://msdl.microsoft.com/download/symbols/_.debug/elf-buildid-sym-<build-id>/_.debug. The whole read is struct.unpack over the ELF core plus one llvm-objdump — no container, no DAC. 🚨 Index symbols by vaddr and bytes by file offset, and convert between them with the PT_LOAD table; do not assume they are equal.

2026-09-07 / 2026-09-08: sightings #11 and #12 — two dumps, two frames, two si_addrs, one fingerprint

Two occurrences from the same sweep were read end to end. Together they are the cleanest demonstration yet that the frame carries no information and the zeroed MethodTable word is the whole fingerprint — because they disagree on every incidental and agree on the invariant.

Both faults are the same read of a MethodTable word that is zero, differing only in which field the caller went on to want:

#11  0x373884: 49 8b 07     mov  (%r15),%rax      ; MethodTable word -> 0
     0x373887: 8b 70 04     mov  0x4(%rax),%esi   ; <-- FAULT at 0x4   (MT->m_BaseSize)

#12  0x5cb16a: 49 8b 07     mov  (%r15),%rax      ; MethodTable word -> 0
     0x5cb16d: 48 83 e0 f8  and  $-8,%rax         ; strip GC mark bits
     0x5cb171: 8b 08        mov  (%rax),%ecx      ; <-- FAULT at 0x0   (MT->m_dwFlags)

In both, the null is confirmed at its source rather than inferred from RAX: the bytes at R15 in each core read zero.

Why this pair settles two things a single dump cannot

• si_addr is definitively not part of the fingerprint. 0x4 and 0x0 here are the offsets of m_BaseSize and m_dwFlags — the same corruption seen through two different field reads, 33 hours apart, against the same runtime binary. Sighting #10 already corrected this; #11/#12 make it a measurement rather than an argument.
• The frame does not progress, it revisits. 0x5cb171 is not a new RVA: the table in sighting #9 already lists it as background_sweep()+0xa61, first seen 2026-08-18. So after plan_phase, find_first_object, background_mark_simple1 and GetCodeInfo, the fault has returned to a frame already on the list. Reading #10's move out of gc_heap as the family migrating toward the LCG/serialization workload would have been over-reading a sample of one. The sharpened scope in #10 describes that sighting's workload, not the family's boundary.
• Culprit vs victim, from two directions at once. #12's faulting thread is a dedicated BGC thread with no managed frame on it (CPalThread::ThreadEntry → CreateSuspendableThread::$_0::__invoke → bgc_thread_stub → bgc_thread_function+0xdc → gc1+0xf6 → background_sweep+0xa61). #11's is a mutator inside the JIT (MethodDesc::JitCompileCode+0x262 → JitCompileCodeLockedEventWrapper+0x3eb → JitCompileCodeLocked+0xfa → UnsafeJitFunction+0x180 → CEECodeGenInfo::CEECodeGenInfo+0x11b → CEEInfo::getMethodInfoWorker+0x133 → GetCodeInfo+0x1f7). A defect in either path would not produce the other; a heap whose object headers can read as zero produces both, wherever the next reader is.

Disposal was clean in both, and both records are provably complete

_meshweaver-test-trace.log in each artifact has its FAULT-BUDGET suppression lines on a different pid (2828 for #11's 37600, 2791 for #12's 50607), so neither crashing process's record has gaps — the precondition this page insists on.

alc=1 in both is worth stating: only the default AssemblyLoadContext was live, so there was no collectible context to be unloaded out from under anything. Neither job log contains an Unwind: exception type line, so the managed-exception route through createdump is ruled out rather than assumed away.

Base rate, re-measured 2026-09-06 → 2026-09-08

Plugin Catalog CI, 600 runs listed over 2026-09-06T06:08Z … 2026-09-08T09:52Z. Ten runs are excluded from both numerator and denominator because their job listing could not be read. The denominator is 343 — runs in which at least one Portal hosts (shard N) job reached a terminal conclusion, i.e. the shard actually executed suites. The other 257 had none reach a verdict: overwhelmingly runs cancelled before the shard finished (311 of the 600 are cancelled — the merge-cadence pattern), plus 12 still in progress and 2 startup_failure. All 59 failed Portal hosts jobs in the window (at sweep time; 61 by the time the truncation control below ran) had their check-run annotations read, and none of those reads failed.

🚨 The annotation read was checked for truncation, because a paginated read that silently drops page 2 would undercount the numerator and look identical to a clean sweep. Re-reading every failed Portal hosts job with per_page=100: the largest annotation set on any one job is 15 — below even the endpoint's 30-item default, so nothing was cut off, on any job, in either pass.

4 occurrences, all on Portal hosts (shard 1) — 1.17 %:

All three occurrences before this read were previously unrecorded anywhere. The two undissected ones are listed as occurrences, not as sightings — their dumps were still in retention when this was written, so a later reader can add them if a third data point is wanted.

🚨 The two rates on this page are not directly comparable, and the difference is not a trend. The 2026-08-29 → 09-03 measurement (0.74 %) used non-cancelled runs as its denominator; this one uses runs whose portal-host shard reached a verdict. What they agree on is the shape: a low-single-digit-percent, Portal hosts (shard 1)-only rate that has not gone away, with main among the affected branches in both windows.

Controls run on this read

The symbolization slip that produced #10's impossible frame was checked for explicitly, on #12:

• RVA vs file offset. libcoreclr.so's PT_LOAD table maps vaddr 0x1c9680+ at file offset 0x1c8680+, so RVA 0x5cb171 lives at file offset 0x5ca171 — the 0x1000 difference, exactly the trap. Symbols were indexed by vaddr, bytes read by file offset.
• Instruction boundary. The sequence starts at 0x5cb16a, so 0x5cb171 is where mov (%rax),%ecx begins. RIP is a boundary — 0x372887 in #10's first read was not.
• Bytes agree across two independent sources: the core at RIP, and the stock 10.0.11 libcoreclr.so at the corresponding file offset.
• The symbols are the crashed binary's. The build-id read out of libcoreclr as mapped in each crashed process is 989b56dfb2782aa230f822ee4c520e2ccfea71b7 in both, equal to the build-id of the .debug the RVAs were resolved against.
• Each ucontext is unique. Scanning every PT_LOAD for a gregs[] block with TRAPNO=14, a RIP inside libcoreclr's mapping and CR2 == si_addr yields exactly one match per core.

Scope note for the next reader

MeshWeaver.FutuRe.Test has no ProjectReference to MeshWeaver.AI, directly or through MeshWeaver.Hosting.Monolith.TestBase — so teardown or disposed-scope work in the AI engine cannot reach this suite, and a fix there will not move this rate. Neither will the platform-pin staleness that reds Plugins PRs independently: the only failure annotations on Portal hosts (shard 1) in run 34210183539 are the SIGSEGV verdict and the generic exit code 1.

2026-09-08: sightings #13 and #14 — two CONSECUTIVE main runs, a sixth frame, and the disposal chain walked end to end

Two Portal hosts (shard 1) jobs on main, started one minute apart, both killed MeshWeaver.FutuRe.Test with exit 139. Both dumps were read the same way as #11/#12 (struct.unpack over the ELF core, the rbp chain unwound by hand, RVAs resolved against the .debug for the build-id read out of the crashed process's own libcoreclr mapping). The maintainer's standing hypothesis was a use-after-dispose in the teardown chain, so this read also carries the audit of that chain — what it found, and why it is not what killed these two hosts. Filed as MeshWeaver.Plugins#1527.

Same fingerprint, third register (rax in #4–#8, r8/rcx in #9, r15→rax in #12/#13, r13 here), sixth frame. revisit_written_page is the background mark phase re-walking pages the write-watch flagged as dirty — it reads the header of whatever object sits at R15, and that header is zero.

What the trace log says — and #14 died INSIDE a teardown

Two facts about the fixture that matter for reading the rest: FutuReAnalysisTest declares ShareMeshAcrossTests => true, but no DISPOSE_SHARED_SKIP was ever written — the cluster kill-switch had sharing off, so every [Fact] built and disposed its own mesh (30 full teardowns in 47 s in #13). And alc=1 at every checkpoint — each INIT_MEM/DISPOSE_MEM line is written after a forced full GC — means no collectible AssemblyLoadContext survived any teardown in either process. 🚨 Corrected 2026-09-11 (see the entry “the MANAGED view of sightings #11–#16”): both halves of that sentence are wrong. The count is AssemblyLoadContext.All, which drops a context the moment Unload() is called, and CI never sets the MESHWEAVER_TEST_FORCE_GC that gates the forced collection — #13 died holding 4 contexts mid-unload and #14 held 7. It does NOT mean none existed: asm moves 127 → 129 → 127 → 128 → 130 → 131 → 128 → 130 → 129 … → 133 across #13's checkpoints, so contexts (the per-node DynamicNode_* / node-config-script:* ones) were being created inside tests and fully reclaimed by the next checkpoint. Unloads therefore DO happen in this process, and what excludes an unload race is the fingerprint, not the count: a freed LoaderAllocator yields a non-null unmapped pointer (AccessViolation/SIGABRT, #613) or a #GP on a non-canonical one — never a zero word at a page-aligned, still-mapped address read by the collector itself — and in #14 the death preceded the MeshTeardownSignal every teardown-time unload waits on.

The teardown chain, phase by phase — where a later phase can touch what an earlier one released

Walked on the code the crashed runs ran (Plugins 669b09a9/d9e23446 on core pin 73d94b55; the disposal-relevant files are unchanged to main at the time of writing). The non-shared path of MonolithMeshTestBase.DisposeAsync (Plugins, src/MeshWeaver.Hosting.Monolith.TestBase/MonolithMeshTestBase.cs:1364):

1. Clients — DisposeTestClientsAsync (:1285): each client hub DisposeAndJoinAsync, joined sequentially on DisposalCompleted. ✔ waits on the signal.
2. Hosted services — reverse start order, StopAsync under a DisposeTimeout CTS (:1460). ✔
3. Mesh.Dispose() (:1488) → MessageHub.Dispose (src/MeshWeaver.Messaging.Hub/MessageHub.cs:1878): hostedHubs.CloseCreation() + SignalShuttingDown() FIRST, then Post(ShutdownRequest(Quiescing)). The state machine is Quiescing → DisposeHostedHubs → ShutDown → Dead, each phase entered only on the previous phase's own signal (OnQuiesceComplete; hostedHubs.DisposalCompleted, :2214); every hosted hub's DisposeImpl (:2251) fires its RegisterForDisposal callbacks and reactive dispose actions, and SignalDisposalCompleted (:1833) completes the ReplaySubject. ✔ reactive end to end. This is where MeshNodeStreamCache.Dispose() runs — registered on the cache hub (src/MeshWeaver.Hosting/MeshNodeStreamCache.cs:616), so it executes in the mesh's DisposeHostedHubs phase, strictly before the mesh signals.
4. Wait — WaitWithProgressAsync (:1693): Mesh.DisposalCompleted.ObserveCompletion(…), no .ToTask(), error arm attached, ConfigureAwait(false). ✔ a subscription, not a timer race.
5. IoPoolRegistry.DrainAll() (:1518; src/MeshWeaver.Mesh.Contract/Threading/IoPoolRegistry.cs:169): cancel + join of every IIoPool leaf. ✔ leakedIoLeaves=0 in all 45 teardowns across both runs.
6. AsyncDisposeQueue.DrainAsync (:1525). ✔ clean=True throughout.
7. MeshTeardownSignal.SignalCompleted(report) (:1533) — and only NOW do the hosted hubs' Autofac scopes close: HostedHubsCollection.CloseScopeWhenDisposed (:239) subscribes each child's DisposalCompleted.Take(1) and hands the close to TeardownOrderedScopeDisposal.CloseWhenDrained (src/MeshWeaver.Mesh.Contract/Threading/TeardownOrderedScopeDisposal.cs:41), which on a disposing mesh defers it to signal.Completed. ✔ pinned by HubScopeClosesAfterTeardownDrainsTest.
8. base.DisposeAsync() (FileOutput), then the root ServiceProvider.Dispose() (:1610, DisposeServiceProviderOnTeardown defaults to true), then DISPOSE_MEM with a forced GC.

Every phase waits on a completion signal, none on a timer, nothing blocks in a Dispose(), and no Cancel() is issued from the teardown thread except the bounded StopAsync budget in step 2. The collectible-ALC unload (MeshDataSource.cs:995, UnloadNodeAssemblyContexts) and the lease release (:1017, ReleaseNodeTypeLease) are both gated on teardownSignal — which in #14 had not fired when the host died (no DISPOSE_IOPOOL_DRAIN_START, let alone DISPOSE_DONE), and in #13 had fired 5.5 s earlier with a forced GC after it that left alc=1.

The one place a later phase touches what an earlier phase released — and it is in core, not in the fixture. MeshNodeStreamCache.Dispose() (:784) detached the per-path read streams (step 1), the update queues (2), the probe cache (3), the in-flight writes (4), the storm-breaker windows (5) and the pending self-writes (6) — and never touched _queries. Each synced query is Defer(…).SubscribeOn(TaskPoolScheduler.Default).Replay(1).AutoConnect(1) (GetQueryRaw), and AutoConnect keeps the handle its Connect() returns to itself: the cache had no way to release a chain even had it tried. A first subscriber's Connect() only queues the upstream subscribe on the pool, and nothing in steps 3–8 joins a pool-queued Rx subscribe (DisposalCompleted covers the action blocks, DrainAll covers IIoPool leaves, the queue covers enqueued cleanup). So the item ran whenever the pool reached it — in #13, 4 ms after step 8 — and resolved cacheHub.GetWorkspace() from a scope step 7 had closed. All 11 disposed-scope stragglers captured in run 34222933802's shard-1 artifact are that one Defer — FutuRe 2, Blazor.Views 5 (one of them the inner SyncedQueryMeshNodes.BuildReadStreamCore), ContentCollections.Indexing.Graph 4 — on .NET TP Worker threads, every one caught by the Defer and forwarded to OnError. Fixed in the same change as this entry: the connection is registered with the cache (AutoConnect(1, onConnect)), released in Dispose() (a still-queued connect is cancelled before the pool dequeues it; a late registration is disposed as it is added), and a query opened after teardown terminates with ObjectDisposedException instead of parking on a Replay(1) nothing will feed. Pinned by test/MeshWeaver.Hosting.Test/QueryConnectionsReleasedOnTeardownTest.cs, whose control arm proves the registry sees the live connection before asserting it was released.

The two [FAULT] records in #14's teardown window are the error arms of RegisterForDisposal'd own-stream subscriptions (MeshDataSource.cs:1034, NodeTypeCompilationHelpers.cs:1036) reporting the own stream ending in error as the hub shuts down — the documented "incoming streams error" contract of HubDisposalModel, handled by the arm that logged them, not a resolve from a disposed scope.

Why the audit finding is real and is still not the killer

• It produces a managed ObjectDisposedException, caught by Observable.Defer and forwarded to OnError. Nothing on that path writes to an object header, and a managed exception cannot zero one. Its escalation shape, when a subscriber has no error arm, is Rx's Stubs.Throw → AppDomain.UnhandledException → xUnit's Catastrophic failure … Failed: 0, exit 1 (Plugins#870, #1390) — exit 1, not 139.
• #14 crashed with no straggler at all, and #13's stragglers were 5.5 s and one whole clean teardown before the death.
• Both faulting threads are the background-GC thread with no managed frame; both si_addr are the MethodTable field offset, both source words read zero at their source.

#1507 does not reach this suite

Plugins #1507 (fix/1390-teardown-resolve-guard, merged 13:13Z) converts 22 Rx call sites in src/MeshWeaver.AI/* to TeardownSafeCallback. MeshWeaver.FutuRe.Test references neither MeshWeaver.AI nor anything that does (checked: its own .csproj and MeshWeaver.Hosting.Monolith.TestBase.csproj), and neither crashed sha contains #1507. Portal hosts (shard 1) passing on the 13:13 push run 34230683846 is one green sample at a low-single-digit-percent rate — the reading the base-rate section above already warns against — not the fix working.

2026-09-10: sighting #15 — a NEW RUNTIME BUILD (10.0.12), and the corrupt block is a LIVE, REFERENCED object (corrected 2026-09-11: it is garbage — reachable from no root)

MeshWeaver.Futu-51647.dmp (MeshWeaver.Plugins run 34476948303, job 102870602506, Portal hosts (shard 1), on PR #1603 — whose diff is two Python files under scripts/ and cannot reach a .NET test host). Filed as MeshWeaver.Plugins#1605. Read the same way as #11–#14: struct.unpack over the ELF core, the rbp chain unwound by hand, RVAs resolved against the .debug for the build-id read out of the crashed process's own libcoreclr mapping.

Same fingerprint, a fourth register pair (rax in #4–#8, r8/rcx in #9, r15→rax in #12/#13, r13 in #14, r10 here), and the frame REVISITS again: find_first_object+0x132 is byte-for-byte the same offset within the same function as sighting #7's 10.0.11 RVA 0x5d5ab2, two runtime builds apart.

The two things this sighting adds

1. The family is not a property of one runtime build. Every sighting from #4 on ran on 10.0.11, libcoreclr build-id 989b56df…; #1–#3 on 10.0.10. This one ran on 10.0.12, a binary whose build-id (79945f51…) differs — verified as mapped inside the crashed process, not merely as shipped. So "wait for the next runtime patch" is not a plan, and any upstream report should be written against three patch releases rather than one.

🚨 Corrected 2026-09-11 — the paragraph below does not hold. The 96-byte referrer is an Autofac ServiceRegistrationInfo, and a BFS from every GC root (374,060 objects) never reaches it: referrer and cursor are both the garbage of a disposed hub's registry. "Well-formed" was read as "live". See the entry “the MANAGED view of sightings #11–#16”.

2. The corrupt block is a live object that something POINTS AT. The 2026-08-17 entry established the free-list shape by finding that no managed object points at the cursor. Run the same scan here and the answer inverts. Exactly three 8-byte-aligned occurrences of the cursor value 0x7f5dd6e057a0 exist in the whole 849 MiB core:

• 0x7f5dacd91b50 — the saved RAX in the signal ucontext (i.e. the fault itself);
• 0x7f5d86ff4d08 — one scratch slot on the faulting thread's stack;
• 0x7f5dd6e05640 — a field at +0x20 inside a live, well-formed 96-byte heap object at 0x7f5dd6e05620 whose own MethodTable (0x7f61caeb75d0) is valid.

So this is not GC bookkeeping losing a g_pFreeObjectMethodTable: a published, referenced managed object lost its type slot while every other word of it survived. That is the strongest form the invariant has taken, and it removes the last reading in which the zeroing could be dismissed as free-space housekeeping.

The walk is synchronised — measured, not asserted

find_first_object(start, first_object) walks objects forward from first_object looking for the one containing start. Here RDI (start) is 0x7f5dd6e05900 — a card-marked address. 🚨 first_object itself is only partly recoverable, and saying otherwise would be inventing a derivation: by the fault RSI has been overwritten with first_object >> 12 (0x5d5d18: shrq $0xc,%rsi), so it pins the page 0x7f5dd6dde000 and nothing finer; the byte offset 0x818 is taken from R13, whose assignment is outside the loop and was not traced. What validates the start is the walk, not the register. Replaying it from 0x7f5dd6dde818 over the core:

… 0x7f5dd6e056a0  MT=0x7f61caf24c78  size=48
  0x7f5dd6e056d0  MT=0x7f61cb8fecc0  size=112
  0x7f5dd6e05740  MT=0x7f61cbc334e0  size=24
  0x7f5dd6e05758  MT=0x7f61caf1d6e8  size=32
  0x7f5dd6e05778  MT=0x7f61caf17220  size=40     <- ends EXACTLY on the cursor
  0x7f5dd6e057a0  MT=0x0000000000000000          <- FAULT

172 consecutive well-formed objects were walked before it — every one a valid MethodTable and a size that lands on the next header — and the chain arrives exactly on the cursor, the predecessor's 40 bytes ending on it to the byte. A wrong starting offset does not produce that: it desynchronises within a few objects and reads garbage headers. So the start is confirmed by its consequence, and the cursor sits on a genuine object boundary. The cursor's own successor at +0x20 (0x7f5dd6e057c0, MT 0x7f61caf24bb8, 56 bytes) is a valid object — which the cursor's surviving field at +0x08 also points at. The contiguous zero run is 8 bytes: the header word alone. The walk is neither desynchronised nor past the allocated end; one word is gone out of a coherent heap.

The phase: a blocking GC on a MUTATOR, reached through the card scan

The rbp chain, resolved end to end:

Array_CreateInstance+0x3f9
 → AllocateSzArray(MethodTable*, int, GC_ALLOC_FLAGS)+0x268
   → Alloc(ee_alloc_context*, size_t, GC_ALLOC_FLAGS)+0x1b3
     → WKS::GCHeap::Alloc(gc_alloc_context*, size_t, uint32_t)+0xf0
       → WKS::gc_heap::try_allocate_more_space(...)+0x24b
         → WKS::gc_heap::trigger_gc_for_alloc(...)+0x3a
           → WKS::GCHeap::GarbageCollectGeneration(unsigned, gc_reason)+0x3db
             → WKS::gc_heap::garbage_collect(int)+0x6cb
               → WKS::gc_heap::gc1()+0xff
                 → WKS::gc_heap::mark_phase(int)+0x802
                   → WKS::gc_heap::mark_through_cards_for_segments(...)+0x7dd
                     → WKS::gc_heap::find_first_object(...)+0x132        ← FAULT

A managed thread allocated an array, the allocation triggered a blocking collection, and the mark phase's card-table scan read the header of the object the card pointed into. The family now spans dedicated BGC threads, mutators inside the JIT, and mutators inside a foreground GC — which is what "the GC finds a zeroed object header wherever it next looks" predicts, and what no single-phase hypothesis does.

The trace log — complete, clean, and dead in CONSTRUCTION

_meshweaver-test-trace.log's two FAULT-BUDGET suppression lines belong to pid 3114, not to the crashing pid 51647, so this process's record has no gaps — the precondition this page insists on.

The last five records are CTOR 12:45:24.607, INIT_START .607, INIT_BASE_DONE .609, INIT_PREWARM_DONE .617, INIT_DEVLOGIN_DONE .617 — and then nothing. The host died building fixture 18, not tearing down fixture 17, which had completed cleanly 18 ms earlier. Read the bottom of the stack, not the top: this is construction, exactly as in sightings #1 and #3, and no teardown guard could have been in the path. alc=1 throughout — no collectible context survived any teardown. 🚨 Corrected 2026-09-11: alc cannot see a context that is unloading; this process died with 5 NodeAssemblyLoadContexts mid-unload and none alive, 12 ms after the previous teardown.

The truncation machinery did its job: the trx carries 18 results — 17 Passed plus MeshWeaver.FutuRe.Test.HOST_CRASHED Failed — so the Passed! … Passed: 17 console line is contradicted in the durable artifact, which is the whole point of core #2495.

Disk pressure, excluded explicitly

The job warned only 33G free after reclaim — if this job dies with no failing step, disk is the first suspect again, and it did die with no failing step. It is still not disk, on four independent grounds:

• The dying process wrote a complete 889,839,616-byte core dump and a 1,153,250-byte trx at the moment of the fault; both parse end to end, so ≥849 MiB was free at crash time.
• Nine further suites ran green after it (12:46:24 → 12:49:19: Auth 160, PathResolution 168, Blazor.Views 192, Snowflake 50, Northwind 8, AccessControl 21, ContentCollections.Indexing.Graph 33, Kernel 11, Serialization 14), each writing a trx and per-test logs; 693 files uploaded.
• No ENOSPC / No space left on device appears anywhere in the job log.
• The mechanism does not fit. A full filesystem produces ENOSPC and IO exceptions; it cannot produce a SEGV_MAPERR at address 0x0 with a recovered TRAPNO=14 / ERR=0x4 page-fault ucontext inside libcoreclr.

Controls run on this read

• Managed exception ruled OUT, not assumed away. The job log's only Unwind hit is the verdict's own prose quoting the instruction to grep for it; strings over the core finds Unwind: exception type zero times. Together with TRAPNO=14/ERR=0x4 this is a native page fault.
• RVA vs file offset — the 0x1000 trap of sighting #10. libcoreclr.so's second PT_LOAD maps vaddr 0x1c99a0+ at file offset 0x1c89a0+, so RVA 0x5d5d82 lives at file offset 0x5d4d82. The bytes there (45 8b 32 45 85 f6 78 c6 …) are byte-identical to the bytes at RIP in the core; the bytes at 0x5d5d82 as a file offset are entirely different.
• Instruction boundary. llvm-objdump puts movl (%r10),%r14d at exactly 0x5d5d82.
• The symbols are the crashed binary's. The build-id read out of libcoreclr as mapped in the crashed process is 79945f51fb2612f13b7667a10a8fd29122664791, equal to the build-id of the stock 10.0.12 libcoreclr.so and of the .debug the RVAs were resolved against.
• The ucontext is unique. Scanning every PT_LOAD for a gregs[] block with TRAPNO=14, a RIP inside libcoreclr's mapping and CR2 == si_addr yields exactly one match in the core.
• The crashing thread is identified by its signal frame, not by guessing. The gregs[] block sits at 0x7f5dacd91ae8, 0x1e8 above tid 51726's recorded (in-handler, alternate-stack) RSP 0x7f5dacd91900; no other thread's recorded RSP is within 16 MB of it. NT_PRSTATUS for that thread describes the handler, and the pre-signal RSP (0x7f5d86ff5b48) is on its normal stack — which is why the two do not match and must not be expected to.

Which lanes turn a signal death into a verdict — and the one that does not

A host killed by a signal exits 139, which dotnet test flattens to exit 1 while its console summary still prints Passed!. Whether that is caught depends entirely on the lane. Measured 2026-09-10:

The canary is the one real hole, and it is a gate that cannot fail on missing input. The suites run as dotnet test … || true, and the verdict is computed by platform-canary-delta.py from the two arms' trx as sets of test names: drift = pin_passed & cand_failed. A host killed mid-arm leaves a truncated trx, so every test that never ran is absent from both sets and cannot appear in drift — the job prints No drift — every test that passes at the pin also passes at core main over an arm that measured a fraction of its suites. The only denominators guarded are built.txt and "the pin arm ran no tests"; the two arms' totals are printed in the Observed: line but never compared, and neither the exit code nor a signal is looked at anywhere. Tracked as MeshWeaver.Plugins#1620.

2026-09-11: sighting #16 — the same fingerprint again on 10.0.12, back in background_sweep+0xa61 on a BGC thread

MeshWeaver.Futu-51566.dmp (MeshWeaver.Plugins run 34594554211, job 103254492283, Portal hosts (shard 1), on main — a repository_dispatch). It was read because it is the second dump taken on the runtime that carries dotnet/runtime#131708 (see the next entry), so the one question worth a download was whether the fingerprint survives the fix twice. It does. Read the same way as #15 — struct.unpack over the ELF core, the rbp chain unwound by hand, RVAs resolved against the .debug for the build-id read out of the crashed process's own libcoreclr mapping.

background_sweep()+0xa61  ← gc1()+0xf6  ← bgc_thread_function()+0xdc
  ← CreateSuspendableThread::$_0::__invoke+0x74  ← CPalThread::ThreadEntry+0x201  ← libc

The trace log is complete and clean. The file has no FAULT-BUDGET line at all. For pid 51566: 45 CTOR / 45 INIT_START, 45 TEST_START / 44 TEST_END — it died inside FutuReAnalysisTest.Group_Diagnostic_DataFlow, 3 ms after that fixture's INIT_MEM; 44 DISPOSE_DONE, all 44 teardown clean; zero DISPOSE_QUIESCE_LEAK, DISPOSE_DIRTY_TEARDOWN or leakedIoLeaves>0; alc=1 at all 89 memory checkpoints; gc0=834 gc1=300 gc2=53 at the last one. Its eight [FAULT] records are the Could not lease the NodeType assembly context / Failed-verdict re-drive: own-stream subscription faulted warnings the #13/#14 entry already explains, at 12:17:04 and 12:17:19 — inside teardowns that then completed clean, 18 s and 3 s before the death. The trx carries 58 results: 57 Passed plus MeshWeaver.FutuRe.Test.HOST_CRASHED Failed.

Controls run on this read: the ucontext is unique (exactly one gregs[] block with TRAPNO=14, RIP inside libcoreclr's mapping and CR2 == si_addr), 0x1e8 above tid 51580's recorded in-handler RSP — the same offset as #15; the 0x1000 trap of sighting #10 was checked — the stock 10.0.12 binary's bytes at file offset 0x5ca441 (8b 08 85 c9 78 09 31 c9 …) are byte-identical to the core's bytes at RIP, and the bytes at 0x5cb441 read as a file offset are different; the .debug used is the one for that build-id.

What it adds is nothing new about the fault — which is the point. On the runtime that carries the upstream GC-hole fix, the family has now reproduced twice, and the second time on its most common frame, on a thread that runs no application code. The dump expires with its artifact on 2026-09-18.

2026-09-11: the runtime question — the upstream GC-hole fix ships in 10.0.12, and sightings #15 and #16 crashed ON 10.0.12

This entry records no new dump. It answers the question every reader of sightings #10/#11 eventually asks — is this a known CoreCLR bug that a newer runtime fixes? — and it records the sixteen sightings grouped in one table, so the next reader does not have to re-derive them from the sections above. Tracked on MeshWeaver.Plugins#1605.

The upstream candidate, and what was confirmed about it

dotnet/runtime#131267 (filed 2026-07-23 against 10.0.9, closed 2026-08-05) is an external team's report of the same symptom — "a SIGSEGV reported as test host process crashed while every test passes" — whose faulting frame is LCGMethodResolver::GetCodeInfo, which is exactly sightings #10 and #11. The root cause given by the runtime team on that issue is more general than its title: on x64/arm64 a thread suspended for GC by return-address hijacking, whose return address lies inside CallDescrWorkerInternal (hand-written assembly with no GC info), holds the returned object reference in rax/x0 where no GC reports or relocates it; a compacting GC then leaves the native caller holding a stale pointer. Every MethodDescCallSite::Call_RetOBJECTREF site is exposed — 33 of them in release/10.0 — and GetCodeInfo is one of them. The fix is an early-out in Thread::HijackThread (main: #129714; release/10.0 backport: dotnet/runtime#131708, "Fix GC hole when method return is hijacked for GC suspension").

That the backport ships in 10.0.12 is confirmed three independent ways, all read on 2026-09-11:

The early-out is #ifndef TARGET_X86, so it applies to the linux-x64 runners. v10.0.12 was published 2026-09-08T22:11Z; no later v10.0.x tag exists, and the two release/10.0 pull requests already milestoned 10.0.13 (#133073, debugger sequence map; #133161, JIT visit budget) are not GC fixes.

And the confirmation stops there, because sightings #15 and #16 are counter-examples. #15 ran on 10.0.12 — its job installed dotnet-install: Installed version is 10.0.12 (SDK 10.0.401), and the libcoreclr mapped inside the crashed process has build-id 79945f51…, the stock 10.0.12 binary. So the fix was present at the moment of the fault — and #16 (the entry above) is a second, on the same build-id, on a dedicated BGC thread. The honest reading:

• The upstream GC hole fits sightings #10 and #11 precisely. Their R15 is the byte[] returned by Call_RetOBJECTREF, and #10 found that "array" to be a recycled run of pointers in anonymous memory — the victim-side read of a stale reference, which is exactly what the hole produces.
• It does not account for #15 or #16: on a runtime that carries the fix, a published, referenced heap object lost its type slot (#15, found by a mutator's blocking GC) and another header read zero under the background sweep (#16) — neither is a stale native read.
• Whether the thirteen gc_heap sightings are downstream damage of the same hole (a stale reference written through corrupts whatever now occupies the old address) is not decidable from a core dump, and is exactly what the rate measurement below is for.

The measurement — the crash rate before and after 10.0.12, every run's runtime read from its own log

Method. Plugin Catalog CI runs created 2026-09-06T00:00Z → 2026-09-11T12:50Z: 1,223 runs over 792 head commits. For every commit, the check runs named Portal hosts (shard 0…3) with filter=all, so re-run attempts are included — 3,168 queries, none truncated, none failed.

• Denominator: a run in which at least one portal-hosts shard reached success, failure or timed_out — the definition the 09-06 → 09-08 measurement used — 719 runs.
• Numerator: a run in which any portal-hosts job's annotations carry THE TEST HOST WAS KILLED BY SIG…. Annotations were read for all 181 failed portal-hosts jobs, none missing.
• Runtime: read per run from the head of a portal-hosts job's own log (dotnet-install: Installed version is 10.0.N, or … version '10.0.N' is already installed) — the crashed job for a crashed run, otherwise shard 1. All 719 were read, zero failed reads, each naming exactly one 10.0.x runtime.

The switch is sharp — the last 10.0.11 job started 20:36:38Z and the first 10.0.12 job 20:44:19Z on 2026-09-08 — and it is 1 h 27 m before GitHub's v10.0.12 release object (22:11Z). That is why a run's runtime has to be read from the run: a cut at the release date would have mislabelled every run in between.

Every crash is Portal hosts (shard 1):

What this establishes, and what it does not:

• 10.0.12 does not eliminate the crash, and that is ESTABLISHED by counter-example, not by statistics. Two runs crashed on the fixed runtime and both dumps were read (#15, #16), on build-id 79945f51…. No N can turn that into a zero any more.
• Whether it REDUCED the rate is NOT established. The rate ratio is 0.40, exact 95 % CI 0.04 – 2.02, and the conditional exact test (given the ten crashes, is the post-fix share small?) gives p = 0.20, one-sided. That is consistent with a reduction and consistent with no change.
• What N would settle it. A zero would have needed ≥ 165 post-fix runs at the measured 1.80 % prior (≥ 149 at 2 %, ≥ 255 at 1.17 %, ≥ 404 at 0.74 %) — moot now. Detecting a halving (1.80 % → 0.90 %) with 80 % power at one-sided α = 0.05 needs ≈ 2,030 verdict runs per arm; at the measured cadence of ~134 verdict runs a day the post-fix arm gets there around 2026-09-24. A smaller effect needs proportionally more. Nothing shorter can tell "the fix helped" from "the fix did nothing".
• For the next reader: re-take this measurement at ≈ 2,030 post-fix runs. The sweep's shape is per-commit check runs by name, annotations of failed jobs only, and a ranged read of each run's log head for the runtime — about 4,000 REST calls for five days; per-run jobs listings cost ~4.5 pages each and do not fit the hourly quota. File the upstream report against 10.0.12 with #15/#16: dotnet/runtime#131267 is closed, and its fix is now shown not to cover this family.

The sixteen sightings, grouped

• 16 of 16 share the fingerprint: SEGV_MAPERR, TRAPNO=14/ERR=0x4, RIP in file-backed libcoreclr, and a MethodTable word that reads exactly zero (si_addr is only the field offset — 0x0 for m_dwFlags in 14, 0x4 for m_BaseSize in 2).
• No frame is shared by a majority. By frame: background_sweep 7, plan_phase 3, find_first_object 2, GetCodeInfo 2, background_mark_simple1 1, revisit_written_page 1 — six functions, and the frame revisits rather than progresses.
• By thread: a dedicated background-GC thread with no managed frame in 9 (#1–#3, #8, #9, #12–#14, #16); a mutator in 3 (#10, #11 in the JIT; #15 in a blocking GC); unrecorded in 4 (#4–#7).
• By runtime: 10.0.10 ×3, 10.0.11 ×11, 10.0.12 ×2 — three distinct libcoreclr builds.
• By suite: MeshWeaver.FutuRe.Test 14, MeshWeaver.GitSync.Test 1, unrecorded 1 (plus the undissected 2026-08-24 MeshWeaver.Hosting.Orleans.Test crash at GetCodeInfo+0x1f7).

What the sixteen have already eliminated — do not re-open these

1. ⚠️ Unsound as written (2026-09-11). Use-after-unload of a collectible ALC — falsified three ways (RIP in file-backed runtime code; a freed LoaderAllocator yields a non-null unmapped pointer, never a zero word at a mapped address; a free-list item has no ALC), and alc=1 at every checkpoint of #11–#15. The three arguments exclude only a freed collectible MethodTable being dereferenced — the zeroed word is a default-context object's header, so they are silent on an unload IN PROGRESS — and alc=1 cannot see a context that is unloading: every readable FutuRe dump held 3–7 of them. See the entry “the MANAGED view of sightings #11–#16”.
2. ⚠️ Unsound as written (2026-09-11). A teardown-ordering race — every DISPOSE_DONE in every complete record reads teardown clean, zero DISPOSE_QUIESCE_LEAK / DISPOSE_DIRTY_TEARDOWN; and the phase at death varies (construction in #1, #3, #15; inside a test in #11–#13; inside a teardown in #2, #14). A clean DISPOSE_DONE proves every teardown FINISHED; it says nothing about whether one instance's unload OVERLAPPED the next instance's start, because DISPOSE_DONE is written when Unload() has been requested, before any context is freed. A clean log is exactly what the overlap looks like.
3. Concurrent GC as the mechanism — #1274 disabled it; the rate did not move (4.2 % → 3.8 %) and plan_phase runs in blocking GCs; removed again.
4. An unhandled managed exception routed through createdump — no Unwind: exception type in any.
5. The ClrMD DAC pthread_key teardown — no libmscordaccore.so mapped (#15).
6. MeshWeaver code writing the heap — zero AllowUnsafeBlocks in either repository; every mapped native module is the runtime's or the OS's. (2026-09-11: AllowUnsafeBlocks alone does not prove it — Unsafe.*, MemoryMarshal, GCHandle and Marshal.Write* need no unsafe block. A direct grep of both src/ trees finds none of them writing; the one MemoryMarshal.AsBytes is a read-only hash input. The conclusion stands on that grep.)
7. Disk pressure — #15 wrote a complete 849 MiB core and nine suites ran after it.
8. Re-entrant hub construction — 1,350 per green run; on non-faulting threads in #8.
9. The quiescing-leak family (#981) — never co-occurs with the signal death.
10. The AI engine's teardown callbacks (#1507) — FutuRe.Test does not reference MeshWeaver.AI.
11. The MeshNodeStreamCache query straggler — real and fixed (#13/#14 entry), but a managed ObjectDisposedException (exit 1), not a header write (exit 139).
12. "A single bad runtime build" — three builds.
13. 10.0.12's hijack GC-hole fix as the whole answer — #15 and #16 crashed on it (this entry).

Cause or victim

MeshWeaver.FutuRe.Test is the sampler, not the cause. Nine of the sixteen faulting threads run no application code at all, the faulting instruction is always inside the runtime, no assembly either repository builds can contain unsafe code (AllowUnsafeBlocks is set nowhere in either — framework and third-party assemblies can and do contain it, which is where any managed writer would have to live), and a second suite (GitSync.Test) and a third (Hosting.Orleans.Test) have taken the same fault. What the suite contributes is the densest workload in the fleet for the two things the fingerprint needs — garbage collections (gc0=493 gc1=184 gc2=29 in ~40 s in #15) and LCG DynamicMethod emit (System.Text.Json's reflection-emit member accessors, one per serialized type, per [Fact]-built mesh) — so it is where a process-wide heap corruption is most often discovered. Moving, skipping or shrinking it would move the discovery, not the defect.

The truncation is per suite, not per shard. The Portal hosts lane runs each suite as its own dotnet test process, so a crash ends only that suite's remaining tests; #15's shard ran nine further suites green after it.

Does the lane turn the crash into a verdict? — yes

Portal hosts (shard N) runs classify-test-run.py --record-crash-into … --crash-recorder <platform>/.github/scripts/record-host-crash.py (MeshWeaver.Plugins .github/workflows/ci.yml, the test step, no matrix.shard condition), and #15's trx carries MeshWeaver.FutuRe.Test.HOST_CRASHED Failed beside the 17 passes. The evidence-preserving machinery core #2495 introduced is in place on this lane; nothing needs landing there.

Do the deployed portals run 10.0.12? — no, and the SDK that builds them does not decide it

Both portal images are framework-dependent layers on one hand-built base: core main-cd.yml and MeshWeaver.Plugins portal-ai-image.yml publish with --no-self-contained and -p:ContainerBaseImage=meshweaver.azurecr.io/memex-portal-ai-base:latest, so the runtime inside a portal pod is whatever mcr.microsoft.com/dotnet/aspnet:10.0 resolved to when that base was last built — not the SDK the app was compiled with. The image memex.systemorph.com runs (45306a33, main-cd run 34543984567) was compiled on SDK 10.0.401 with runtime 10.0.12 installed, and its log says Building image 'memex-portal-ai' … on top of base image 'meshweaver.azurecr.io/memex-portal-ai-base:latest'.

That base is built by exactly one lane, base-image-acr.yml, which is workflow_dispatch-only; its last successful run is 2026-08-12T14:23Z (31606495380). v10.0.11 was published 2026-08-11T21:42Z and v10.0.12 on 2026-09-08T22:11Z, so the portals run 10.0.11 by timing — inferred, because that run's log has aged out (HTTP 410) and no portal endpoint reports its runtime — and cannot run 10.0.12. So production does not yet carry dotnet/runtime#131708. Rolling it there is gh workflow run base-image-acr.yml --ref main followed by the next main-cd roll: an operator decision, not taken in this entry. What 10.0.12 is worth is what the measurement above says.

2026-09-11: the MANAGED view of sightings #11–#16 — the zeroed header is in a DISPOSED hub's garbage, and collectible contexts were mid-unload at every FutuRe crash

Every entry above read these dumps for the faulting native frame. This one reads them for what the fault registers cannot say: what the zeroed object was, who could still reach it, what every managed thread was doing, and which collectible contexts existed at the moment of death. The maintainer's reading (2026-09-11) was "not due to dotnet but due to disposal being in progress while new instance starting"; this read tests that against six dumps, and it corrects three claims made on this page.

How it was read — no native SOS. dotnet-dump analyze (SOS 10.0.745401) under an amd64 container on an arm64 host segfaults in dumpobj, in clrstack -all at the first native frame, and ClrMD's own heap walk AVs on the zeroed header. What worked, all read-only against the .dmp:

• setthread N + clrstack one process per thread, so an analyzer crash costs one thread, not all later ones; lno/gcwhere (ClrMD-backed) for the neighbourhood.
• a ~250-line ClrMD 3.1 probe (DataTarget.LoadDump; CreateRuntime(dac, ignoreMismatch: true) for the 10.0.11 dumps, whose DAC version the core does not carry): FindPreviousObjectOnSegment for neighbours; a byte scan of every committed heap segment for the cursor value; a BFS from heap.EnumerateRoots() over EnumerateReferenceAddresses(carefully: true) — never building a ClrObject for an MT-zero child — for reachability; runtime.EnumerateHandles() for the ALC census, reading each AssemblyLoadContext._state (0 Alive, 1 Unloading).
• the native stacks of the non-managed threads (finalizer, tiered-compilation worker) by scanning each NT_PRSTATUS thread's stack for libcoreclr return addresses, symbolized against the build-id .debug.

The fault cursor of each dump comes from the kernel ucontext exactly as in the entries above.

What the six dumps show

• The victim is never a collectible type and never a native wrapper. Every object around every cursor is an ordinary default-context type (Autofac, System.Text.Json, CoreLib collections, MeshWeaver.Hosting / Messaging / Layout). No SkiaSharp, SQLite, libgit2 or other native-handle wrapper is anywhere near one.
• It is what a HUB BUILDS: its Autofac child-scope registry, its JsonSerializerOptions polymorphic metadata, its TypeRegistry, its persistence closures. Where reachability was measured (#15, #16) the object is garbage — reachable from no root — i.e. the leftovers of a hub that has already been disposed. A GC walks dead objects linearly (sweep, plan, card scan, write-watch revisit), which is why it is the collector that trips over the zeroed word, on whatever thread happens to be walking.
• At every FutuRe crash, several NodeAssemblyLoadContexts were mid-unload — Unload() called, _state = 1, held by the runtime's strong handle, their LoaderAllocator objects still present — while the next instance was being built (#15: 5 Unloading and 0 Alive, 12 ms after the previous DISPOSE_DONE, the crashing thread inside the new mesh's HostedHubsCollection.CreateHub → MessageHubConfiguration.Build → an Autofac resolve) or running (#16: 3 Unloading beside the live test's 1 Alive).
• Nothing was executing disposal code at the instant of death. No managed thread is in a Dispose, in AssemblyLoadContext.Unload or in teardown in any of the six; the finalizer thread sits in FinalizerThread::WaitForFinalizerEvent in #15 and #16 (so no LoaderAllocator was being destroyed at that instant) and thread 6 is the tiered-compilation worker. The unloads were pending in the GC: a collectible context is only freed over the following collections, on the finalizer thread, after Unload() returns.
• #11 is a different branch. GitSync ran no dynamic NodeType at all, and its "object" is an interior address in a live array — the stale-reference shape of dotnet/runtime#131267's hijack GC hole, which 10.0.12 fixes. It is not evidence for or against the unload overlap.

What this corrects on this page

1. alc=1 never measured what it was read as. MonolithMeshTestBase.TestMemTrace counts AssemblyLoadContext.All, and the runtime removes a context from that set inside InitiateUnload — the moment Unload() is called (AllContexts.Remove(_id), AssemblyLoadContext.cs, release/10.0). A context that is still unloading, types and LoaderAllocator intact, is therefore invisible to it. And the checkpoints are not after a forced GC on CI: the forced collection is gated on MESHWEAVER_TEST_FORCE_GC, which no workflow sets. So "alc=1 at every checkpoint — no collectible context survived any teardown" (entries #13/#14, #15, #16) is unsupported: the same processes held 3–7 contexts mid-unload when they died.
2. #15's "live, REFERENCED object" is wrong. The 96-byte object whose +0x20 field points at the cursor is an Autofac ServiceRegistrationInfo; it is itself reachable from no GC root. Both are the garbage of a disposed hub's registry. "A published, referenced managed object lost its type slot" — and the conclusion drawn from it that the zeroing cannot be free-space housekeeping — does not follow.
3. Eliminations 1 and 2 of "What the sixteen have already eliminated" are unsound as written. The three arguments of #1 only exclude a freed collectible MethodTable being dereferenced — the zeroed word here is a default-context object's header, so they say nothing about whether an unload in progress is involved — and its alc=1 support is item 1 above. #2 rests on every DISPOSE_DONE being clean, but DISPOSE_DONE is written when Unload() has been requested, before any context has been freed; a clean teardown log is exactly what the overlap looks like.

What it does NOT show

No dump names the writer of the zero. The write happened before the collection that found it, so the thread that made it is long gone from the stack; nothing in either repository writes the heap through Unsafe, MemoryMarshal, GCHandle or Marshal (the one MemoryMarshal.AsBytes is a read-only hash input). What the dumps establish is the state the maintainer described — a disposed instance whose collectible contexts are still being unloaded while the next instance builds and runs — at 5 of 5 FutuRe crashes that could be read. In this workload that state is also the steady state after every fixture, so its presence at the crash is necessary for the hypothesis and not by itself sufficient; the repro below is what separates the two.

The forced overlap, and what it did not reproduce

A crash needs a deterministic repro before a fix, so the overlap was forced first, in two workloads.

• No MeshWeaver code at all. Six workers loop: load a fresh collectible context, build six Autofac child scopes over its types (constructor lambdas are LCG), serialise and deserialise through System.Text.Json (reflection-emit accessors), dispose, Unload() — and start the next iteration immediately, while a hammer thread interleaves background gen2, gen0/gen1 and finalizer passes with a 2 MiB gen0 budget. It ran 16,112 overlapped iterations on macOS arm64 (10.0.11) and 11,289 on linux-arm64 (10.0.12), 180 s each, exit 0. The runtime, Autofac and STJ do not corrupt the heap on this overlap alone, at this scale, on arm64.
• The real suite. MeshWeaver.FutuRe.Test, built against core main, run 20 times back to back through its native xUnit host with a 4 MiB gen0 budget: 1,180 fixtures, 20 of 20 green, no signal death, on macOS arm64.

At the measured CI rate — about 1 % of x64 runs, i.e. about one crash per 4,500 fixtures — neither could be expected to crash, so these are not evidence of absence, and they are recorded so the next reader does not re-run them expecting otherwise. What they do settle is that the state is not sufficient on arm64 at this scale: whatever writes the zero needs more than a context unloading while the next instance builds.

The fix — teardown finishes when the unload has finished

The maintainer's design (2026-09-11): "we need to wait (reactively, i.e. observable.Subscribe()) until all is really finished disposing". "Really finished" for a collectible context is not Unload() returning and not DISPOSE_DONE: it is the runtime releasing the context after destroying its LoaderAllocator, over later collections, on the finalizer thread.

• CollectibleContextUnloads (MeshWeaver.Mesh.Contract, a mesh-scoped singleton next to MeshTeardownSignal) records every context the mesh retires — by its signal, never by reference, so it cannot root what it waits for. AllCollected completes when every context retired before the subscription has been collected, errors when an unload was abandoned (the drain faulted, or Unload() threw — an Unloading handler raising), and emits synchronously when nothing is pending.
• How "collected" is observed. Unload() calls GC.SuppressFinalize on the context, so a finalizer on NodeAssemblyLoadContext would never run. RetireInto instead gives the context a finalizable sentinel that only the context references; the context stays reachable through the runtime's strong handle until the LoaderAllocator is destroyed, so the sentinel's finalizer runs only after that. It stops the entry counting as pending synchronously and releases subscribers on the thread pool — never on the finalizer thread.
• CompilationCacheService retires every context it disposes (UnloadContext and Dispose), once per context even when one generation is aliased under two keys.
• The test bases (MonolithMeshTestBase, HubTestBase, and MeshWeaver.Plugins' MonolithMeshTestBase) end teardown with CollectibleUnloadDrain: drive full collections — an idle test host allocates nothing, so nothing would ever collect — then observe AllCollected. xUnit does not construct the next fixture until DisposeAsync returns. A faulted unload fails the class (DISPOSE_UNLOAD_FAULTED); a context still rooted after collections stop freeing anything is reported (DISPOSE_ALC_RETAINED, naming it), never waited on; the ordinary case writes DISPOSE_UNLOADS_COLLECTED with the round count.

Nothing is cancelled and nothing unloads earlier or later than before — teardown lets the work finish, and simply stops claiming to be finished before it has. In-process recompiles on a live portal (a superseded generation evicted while other hubs keep running) are not sequenced by this; that belongs with the retention work of #4017/#4029.

Pinned by test/MeshWeaver.Compiler.Pipeline.Test/RetiredContextCollectedSignalTest.cs: a start sequenced on the signal does not run while the context is only Unload()-requested, and runs after it is collected; the context is collected (so the fix cannot pass by retaining); a faulted unload releases its waiter with the fault; nothing retired means no delay.

What held the contexts — System.Text.Json's static accessor cache, measured with gcroot

The first version of the fix waited for every retired context to be collected, and the real suite showed that this was not enough. Across three MeshWeaver.FutuRe.Test runs, 84 teardowns ended DISPOSE_ALC_RETAINED, against 81 DISPOSE_UNLOADS_COLLECTED. In half the fixtures, three rounds of full collections freed nothing, and the next mesh still started over contexts that were only unloading.

How the holder was found. A heap dump was taken (dotnet-dump collect --type Heap, macOS-native, so SOS runs natively) the instant a teardown wrote DISPOSE_ALC_RETAINED for BusinessUnit and LocalAnalysis. gcroot was then run on each LoaderAllocator — not on the AssemblyLoadContext. An unloading context is always strongly held by the runtime's own handle, so its gcroot answers nothing. What decides whether an unload can finish is what keeps its LoaderAllocator alive.

What held both. Both have exactly one strong root, and it is the same one:

HandleTable (strong handle)
  -> System.Text.Json.Serialization.Metadata.ReflectionEmitCachingMemberAccessor      (static)
  -> ReflectionEmitCachingMemberAccessor+Cache<(string, Type, MemberInfo)>
  -> ConcurrentDictionary<…> -> …CacheEntry
  -> System.Action<object, string>                  (an emitted property setter)
  -> System.Reflection.Emit.DynamicMethod -> DynamicResolver -> DynamicScope -> List<object>
  -> System.RuntimeTypeHandle -> System.RuntimeType (the collectible node type)
  -> System.Reflection.LoaderAllocator

Every other root gcroot lists is handle type 10, which is HNDTYPE_WEAK_INTERIOR_POINTER in gcinterface.h. It is weak and keeps nothing alive; SOS prints it without a name.

Why the cache has this effect. On CoreCLR, System.Text.Json emits property accessors and constructors as DynamicMethods and shares them through a process-static cache with a 1 s sliding expiry, evicted by a 200 ms timer. A dynamic method's scope holds the handle of every type its IL touches. So serialising a NodeType-compiled instance even once roots that type's LoaderAllocator from a static field until STJ's timer drops the entry.

The unload therefore finished whenever that timer fired: typically a second after teardown, while the next mesh was already being built. That is the maintainer's "disposal in progress while new instance starting", and a timer decided it.

This is the brief's prime lead, confirmed. A process-wide library cache holds delegates over types from a collectible context, with no eviction on Unloading. It has the same shape as the Autofac cache that ReflectionCacheEviction already purges.

The fix. NodeAssemblyLoadContext now also registers JsonMemberAccessorCacheEviction on Unloading. It calls STJ's own published hook: the MetadataUpdateHandler declared on the assembly, whose static ClearCache(Type[]?) is what the hot-reload agent calls. That hook clears the member-accessor cache; the per-options caches it also clears exist only under hot reload. Live options keep the accessors they already hold; the cost is a re-emit when a new options instance resolves a type.

Pinned by ATypeSerializedThroughSystemTextJson_IsCollectedAtTeardown_NotWhenStjsTimerFires, which also asserts that the hook still exists, so a future STJ that drops it fails a test instead of silently resuming the retention.

Measured after this fix. STJ was one holder among several. Over three FutuRe runs with the eviction (macOS arm64), 93 teardowns ended DISPOSE_UNLOADS_COLLECTED and 45 ended DISPOSE_ALC_RETAINED, against 81 and 84 before it:

• 48 teardowns had nothing to wait for;
• 43 collected everything in 2 rounds;
• 2 collected everything in 3 rounds.

The contexts still retained are LocalAnalysis, BusinessUnit and GroupAnalysis.

What still holds them: nothing strong. A second heap dump was taken at a DISPOSE_ALC_RETAINED teardown, this time with the eviction active (pid 61593: LocalAnalysis, BusinessUnit and GroupAnalysis Unloading). gcroot on their three LoaderAllocators finds no strong, pinned or dependent root at all.

Every chain starts at a handle of type 10, HNDTYPE_WEAK_INTERIOR_POINTER. It runs to the collectible assembly's own statics array, then to a static delegate in it (for example the compiler's method-group cache <4>__ProfitByLoB, typed Func<LayoutAreaHost, RenderingContext, UiControl>), then to its RuntimeMethodInfo and the LoaderAllocator. That is the context referring to itself through a weak handle.

But they were not slow — they were held, and the holder was the teardown itself. A controlled run let the drain go on for up to 20 rounds with no progress, 40 full collections with a finalizer pass after each. It still ended 30 teardowns DISPOSE_ALC_RETAINED, each in about 250 ms, and nothing was collected between round 4 and round 20. So something live held these contexts during the drain and let go the moment DisposeAsync returned. That is why a dump taken afterwards shows no strong root.

A third dump was taken inside the drain, at the instant it concluded "retained" (a local diagnostic that ran dotnet-dump collect on its own process). It names the holder. For every Unloading context, the only non-weak root is a stack slot of MonolithMeshTestBase.<DisposeAsync>d__87.MoveNext(), the very frame running the drain:

MonolithMeshTestBase.<DisposeAsync>d__87.MoveNext()  Fp+118
  -> MeshWeaver.Messaging.MessageHub                (the mesh teardown had just disposed)
  -> its properties dictionary -> RecycleAnnouncement -> Action -> MeshWeaver.Data.Workspace
  -> SynchronizationStream<MeshNode> -> ReduceManager<MeshNode> -> … -> MeshWeaver.Graph.MeshDataSource
  -> MeshNodeTypeSource -> MeshWeaver.Mesh.Services.MeshContentTypeRegistry
  -> ConcurrentDictionary<string, DiscriminatorClaim> -> DiscriminatorClaim
  -> System.RuntimeType (the collectible node type) -> System.Reflection.LoaderAllocator

The teardown was waiting to see an unload while holding, in its own frame, the mesh that pins it. The fixture's ServiceProvider field is a second route to the same graph: MonolithMeshTestBase disposes the provider but, unlike ServiceSetup.Dispose, never clears the field.

The fix — the waiter lets go before it waits.

• The test bases clear ServiceProvider before the drain.
• CollectibleUnloadDrain yields before its first collection, so the calling frame and its temporaries are gone by the time anything is collected.

Measured after this fix. Over three FutuRe runs (macOS arm64, all green, 138 teardowns), 126 ended DISPOSE_UNLOADS_COLLECTED (48 with nothing to wait for, 78 collected in 2 rounds) and 12 ended DISPOSE_ALC_RETAINED. None faulted.

The progression, each step measured the same way:

The 12 that remain are reported and named, never waited on.

Correction, the same day: the last row is two changes, and only one of them is shown to matter. That step shipped the frame yield and the teardown's release of its own service provider together, and the two were never measured apart. TeardownWaitsForCollectibleUnloadsTest (core MeshWeaver.Graph.Test) then tried to pin the release on its own, and could not. With the provider disposed cleanly, keeping it roots nothing a harness can build:

• Autofac clears the singletons it built when it is disposed. A container-built holder of a collectible instance was collected with the release reverted, whether it was resolved from the root provider or from the mesh hub.
• Anything registered — a provided instance, a factory's captures — is also held by the fixture's own Services collection, so it stays rooted whether the provider is kept or not.

The release stays: the first in-drain gcroot put the provider on the chain, and it costs nothing. But the 126 / 12 row is not evidence for it. What that test does pin, each by revert: deleting the per-test drain turns 3 of its 4 cases red, deleting the shared-mesh drain turns the shared case red, and swallowing a faulted unload turns the fault case red.

The residual has no managed root, and the cutoff is measured, not guessed.

• A second heap dump taken inside the drain, with both fixes active, shows no non-weak root on either still-unloading LoaderAllocator. None is a stack root, a strong or pinned handle, a dependent handle, or a finalizer-queue root.
• Letting the drain run 20 no-progress rounds instead of 3 then collected 88 teardowns: 32 with nothing to wait for, 56 in 2 rounds. It left 4 retained after 20 rounds. No teardown was collected at any round from 3 to 20.
• So three rounds is right: raising the number frees nothing.

What holds the last 4–9 % is outside the managed heap. The most likely candidates are a native LoaderAllocator-to-LoaderAllocator reference between NodeType contexts (one context's types used by another's) or a runtime-internal reference, and gcroot cannot see either. It is left to the retention work of #4017/#4029, and the teardown reports each such case by name.

For the retention work (#4017/#4029). On a long-lived portal, MeshContentTypeRegistry's discriminator claims hold RuntimeTypes of superseded NodeType generations for as long as the mesh lives. In a test that registry dies with its mesh, so the fix above is enough; on a portal it is a retention candidate of exactly the kind #4017 measures.

Reading the result honestly

The trap in this class of bug is confirmation: the stack shows a plausible culprit and it is tempting to fix that and declare victory. Discipline:

• A stack tells you where the process died, not why. Use steps 5–6 to establish culprit vs victim before proposing a fix.
• Check which phase you are in. Read the bottom of the stack, not the top. MeshBuilder.BuildHub means construction/fixture-init; a dispose cascade means teardown. Getting this wrong sends fixes at the wrong guard — the FutuRe SIGSEGV was framed as a teardown race for weeks while the dump showed it crashing during hub construction.
• A method's name is not evidence of the phase. SubscribeToOwnDeletion is a .WithInitialization(...) hook: it names what the subscription later watches for, not when it runs.
• A frame appearing TWICE in one stack is re-entrancy — a finding only if it is on the FAULTING thread and is not the workload's steady state. CreateHub → Build → … → CreateHub → Build is eye-catching and it is normal here: SynchronizationStream's constructor always calls GetHostedHub(…, Always), and DataExtensions registers h.GetWorkspace() as a synchronous buildup action, so every data-enabled hub builds a sync/{clientId} sub-hub inside its own Build. Measured 2026-08-18 on MeshWeaver.FutuRe.Test: 1,350 nested Builds in one green, exit=0, 23 s run ([ThreadStatic] depth counter in MessageHubConfiguration.Build). A pattern that fires 1,350 times per green run cannot discriminate a ~4 % crash, so finding it in a dump proves nothing on its own — and the 2026-08-03 note that "the registry builder is not re-entrant, so the process died" is withdrawn for the same reason (nested container builds are what those 1,350 are). #774 moved the own-node subscription off the synchronous overload anyway, which is right on its own merits: work that resolves streams belongs after Build returns. So the rule is: repeats are worth scanning for, but qualify them — (a) is this the faulting thread? (b) does it also happen when nothing crashes?
• If a hypothesis survives, write down what would disprove it, then go and check that.

Why the dump may have no precursor in the logs

If a bare catch {} wraps the failing region, nothing is logged before the crash. That is not an accident of the dump — it is a defect in the code. Error paths must log (see ErrorPropagationAndWedges), and the logging itself must be unable to escalate: resolving a logger can throw ObjectDisposedException on a disposing container, so a diagnostic emitted from inside a catch or an Rx onError can convert a handled fault into an unhandled one. Route such emissions through a guarded helper whose single empty catch swallows only the logging failure.

🚨 The managed twin — #890 — produces NO dump, and no amount of asking will change that

#890 is the same "one 8-byte word reads as zero" shape surfacing in managed code: Roslyn's Emit throws NullReferenceException from Cci.MetadataWriter, reading a ContainingType that the guard immediately above it had just read as non-null. Its canary's verdict told triage to "capture a core dump and re-run with tiering disabled" from 2026-08-28 to 2026-09-04, and not one ever arrived — because the advice was unfollowable by construction:

DOTNET_DbgEnableMiniDump fires on a signal. The #890 process never crashes; it throws a managed exception, logs it, and keeps running until CI's 8-minute cap kills it — exit=124, which is timeout --signal=TERM, which produces no dump. Every occurrence in the 9-event 2026-08-22→08-29 sweep ends exit=124 or TESTFAIL. The dump route belongs to this page's family (#613), not to #890.

🚨 The verdict no longer asks for one. EmitPipeline.Verdict's BELOW-ROSLYN branch now names the half of that advice that IS followable — a split-arm DOTNET_TieredPGO=0 / DOTNET_TieredCompilation=0 re-run — and hands over a third canary leg that takes in-process the measurement the dump was being asked for: dissect=, which performs the failing read (ContainingType, then the Cci.ITypeDefinitionMember.ContainingTypeDefinition frame the stack dies in) directly on symbols bound after the fault. READS-HEALTHY there is positive evidence against a corrupted object graph and for code that is wrong only at MetadataWriter's call site; REPRODUCED-OUTSIDE-EMIT would collapse #890 to a one-call repro. See NodeTypeCompilation → Leg 3. The general lesson is this page's own: a remedy nobody can execute is not a remedy — it is a gate that cannot fail, wearing prose. Check that the artifact you are telling the next reader to fetch is one this failure mode can actually produce.

What #890 can contribute that a dump cannot is a measurement #613 has no way to take, because a crashed process has no "after": the fault is total and permanent. On run 33322993649 shard 1, after the first throw at 16:42:42.375, 7 of 7 compiles that reached the metadata writer failed identically over 6 m 15 s and none succeeded — while every compile that needed only diagnostics kept returning correct CS#### codes. Replicated in the other repository on MeshWeaver.Plugins run 33760859754 shard 3 (2026-09-03, a push to main): 39 of 39 compile faults over 13 m 21 s were the same NRE, zero were a CompilationException. Two separate facts pin the other half there: correct CS0103/CS1591 diagnostics were still being produced at +7 m 32 s (13:41:55, on type/RedriveRecovers…'s deliberately broken source), and BrokenNodeTypeAccessTest — which asserts that a non-compiling NodeType answers a terminal error rather than silence — passed at +13 min. Two repos, two harnesses, two shard layouts, same split: parse and bind healthy, emit 100 % dead, no recovery.

A one-off zeroed word cannot produce a 100 % failure rate over freshly allocated symbols; a tier-1 / dynamic-PGO miscompilation of one method can, and also explains the fixed frame and the load dependence.

🚨 Do not lean on "onset ~80 s into a compile-heavy assembly" — it is not a stable threshold. Measured onsets: 33322993649 (core) 130 s in, after 199 TEST STARTs; 33760859754 (Plugins) 33 s in, after 12, on the first compile of that class, with alc=1, asm=138, gc2=4, RSS 548 MiB — i.e. an unremarkable, barely-warm process. A warm-up threshold is what a tier-1 promotion argument predicts; a 4× spread in wall time and a 16× spread in test count do not refute it (promotion is call-count driven and the call counts are unmeasured) but they are not evidence for it either. The tiering hypothesis is still UNTESTED, and the cheap next measurement for BOTH issues is the half of the canary's advice that is followable: re-run with DOTNET_TieredCompilation=0 (or DOTNET_TieredPGO=0, the sharper probe). Design it split-arm — at the measured ~1 % per run a single-arm clean result means nothing.

See NodeTypeCompilation → When the PROCESS cannot emit.

Related

• DebuggingMessageFlow — for hangs and lost messages (a timeout, not a signal).
• DebuggingDisposalAndLeaks — teardown stragglers and retention.
• WritingTests — why a flake is a real race and re-running hides it.