NodeType Compilation & Releases

Compatibility follows the APIs used, not a fixed platform version (Module Adoption Policy, maintainer clarification 2026-09-09). Reuse a compiled artifact only when its cache contract permits it; otherwise compile the source against the current platform and installed dependencies. A different build identity alone is a cache miss, not proof of an incompatible feature and not a reason to pin the running platform. Diagnose concrete missing types, changed member contracts and compiler/load failures. The explicit Modules:RequirePrebuilt policy remains a separate operational choice.

A dynamic NodeType carries its behaviour as C# source (Source/*.cs) plus a configuration lambda — and that source is compiled at runtime, on demand. You never redeploy the portal to add or change a NodeType. This page is the canonical reference for the runtime side of that story: what triggers a compile, how to watch or cancel it, where releases are stored, how to pin an instance to a specific build, and the rules that decide when a NodeType must be recompiled.

For authoring a NodeType (namespace layout, content record, layout areas, CSV data) see Creating Node Types. For the design rationale behind the release model see the NodeType Release Redesign postmortem. For compiling the same source outside the portal — in CI, so the bytes can be shipped rather than recomputed — see Plugin Packaging.

Where the code lives (since #1707): everything that shapes a compile's generated input — the skeleton generator (DynamicMeshNodeAttributeGenerator), source-query resolution (CodeQueryResolver), @@-include shaping, source aggregation/filter/join order, the reference set, parse/compilation options, source-generator execution, and the emit itself — lives in the dedicated MeshWeaver.Compiler assembly, whose full MVID pins the framework build identity. The mesh-actor half — source discovery against the live mesh, access impersonation, scheduling, compile-status write-backs — stays in MeshWeaver.Graph (MeshNodeCompilationService, NodeTypeCompilationHelpers) and orchestrates that toolchain. One pipeline serves every path: the portal's on-demand compile, the batch bake, and the CI bake host all call the same MeshWeaver.Compiler code.

NodeType compilation lifecycle: source changes trigger the Compile Activity, which mints an immutable Release; instance hubs load the DLL from the active release.

🚨 In CI, what gets compiled and what gets selected are decided by the same tree. The module build resolves every selected entry in one Roslyn workspace; that project graph is also what tells select which bundles a src/ change reaches (no more "src/ → everything"), and it is why a superseded main run can be cancelled without comparing affected sets — the next run diffs against the sealed publication, not the previous push. The rule, its two guards (torn seals, starvation) and the maintainer directives behind it are in Module Build Architecture → Superseded runs on main.

The model in one picture

NodeType MeshNode  ──(compile)──►  Release MeshNode            ──►  compiled DLL
{ns}/{Type}                        {ns}/{Type}/Release/{ver}        (on disk / blob)
  Content: NodeTypeDefinition        Content: NodeTypeRelease
    Configuration   (lambda src)       Code, HubConfiguration
    CompilationStatus                  FrameworkVersion
    LatestReleasePath  ────────────►   AssemblyPath
    RequestedReleasePath (pin)         Status (Succeeded/Failed)
    CompiledFrameworkVersion           CompilationActivityPath ──► Activity MeshNode
    CompiledSources {path→version}                                  {ns}/{Type}/_Activity/compile-{id}

Three kinds of MeshNode collaborate here:

Triggering a compile

All compile paths converge on NodeTypeCompilationHelpers.RunCompile.

Automatic — the per-NodeType hub kickoff

When a per-NodeType hub activates, NodeTypeCompilationHelpers.InstallCompileWatcher registers two subscriptions on the hub's own MeshNode stream:

• Kickoff — on first sight of the NodeTypeDefinition, if the NodeType does not already have a usable build (see When is a NodeType recompiled? below) it flips CompilationStatus = Pending on its own MeshNode.
• Watcher — whenever CompilationStatus becomes Pending, it creates a compile-activity MeshNode and posts a RunCompileRequest to that activity's hub.

This is what makes a NodeType "just work" the first time an instance is created, after a Source/*.cs edit, or after a framework redeploy — no operator action required.

🚨 ONE driver runs Roslyn — a resolve request joins the queue, it never starts a second compile

GetCompilationPathRequest (NodeTypeContractHandler) is how an activating instance hub, a cross-process probe or a bake consumer asks "where are this type's bytes?". It must never run Roslyn beside the watcher's compile. Two concurrent compiles on one NodeType produce two assemblies under two IAssemblyStore keys, each terminal write-back names its own, and whichever lands second decides what the record says — the loser's bytes are addressed by nothing, activation resolves a store key with no content, and the instance silently falls back to the default configuration (issue #1368).

The handler therefore does not compile. It dispatches — EnsureCompileDispatched makes one status-guarded flip to Pending, which the watcher turns into the one activity compile — and then waits for that compile to settle before hydrating the answer. The status field is the single-flight lock.

🚨 There is exactly ONE door to Pending, and it records what the dispatch is for

A trigger arriving while a compile is in flight is absorbed when that compile will produce byte-for-byte what the trigger asks for, and parked otherwise. The discriminator is NodeTypeDefinition.DispatchedBuildInputs — a fw=…;mod=…;src=… token (NodeTypeCompilationHelpers.BuildInputsToken) written in the same update that flips Pending, recording the framework identity, the installed-module fingerprint and the source snapshot that dispatch was made against. IsSatisfiedByInFlightCompile absorbs only on an exact match.

Two rules keep the field meaningful, and both are mechanical rather than conventions to remember:

1. Non-null means in flight, and nothing else. Every terminal write — Ok, Error and Unavailable alike — clears it. A stamp that outlives its compile makes non-null mean "in flight or finished at some point", and the absorb read branches on exactly that field.
2. Every flip to Pending goes through NodeTypeCompilationHelpers.DispatchPending. Twelve production sites dispatch a compile — the first-build kickoff, the persisted-Compiling recovery, the framework-stale re-drive, the adoption refusal, the sources watcher's parked auto-retry, the release watcher, the failed-verdict re-drive, EnsureCompileDispatched, two enrichment self-heals, CreateReleaseRequest, and the pre-warmer's missing-bytes rebuild — and DispatchPending is the only place any of them may write the status.

Rule 2 is what makes rule 1 hold. Before #3390 the stamp was a call each door had to remember: eleven of the twelve did not, so a compile started by any of them was unstamped, every request arriving during it parked, and the parked trigger re-fired on that compile's own terminal write-back — one logical event, two compiles, each invalidating the type's instance hubs and raising a newer build available adornment (#2544 measured pairs 65 ms apart and seven compiles for one merge). A write site that simply does not mention a field is invisible on review, which is why DispatchedBuildInputsInvariantGuard now asserts both halves over src/ — one Pending write, all terminal writes clearing — classifying the assigned expression rather than matching a spelling, so a status written as a ternary cannot slip past it.

🚨 Stamping accurately is not the same as absorbing eagerly, and the difference is the safety property. The token describes the inputs the dispatch was made against; it promises nothing about the future. If the sources, the framework or the module set move afterwards, the next request's token differs and it parks — which is also the right answer when one boot resolves two different module sets seconds apart (#3395): an accurate mod= makes that mismatch visible to the absorb read, where an absent stamp merely hid it behind a park that happened to be correct. Force (RequestedReleaseForce) is never absorbed.

The store address is a TRIPLE, and it moves together or not at all

LatestAssemblyCollection, LatestAssemblyPath and LastCompiledVersion are not three independent facts — together they address one blob, and NodeTypeBatchBake relies on that in as many words: "LastCompiledVersion always names a store key that has bytes."

ApplyCompileSuccess takes each of them from the compile result, falling back to the value already on the definition when the result carries none. That fallback is the contract: a compile whose upload did not happen must leave the previous address exactly as it found it. The upload is allowed not to happen — a NullAssemblyStore, unreadable bytes, or a fault or timeout, which is deliberately non-terminal because an upload failure has never failed a compile — and result.Version is set in exactly one place, the projection over a successful PutWithLocation. So a null there does not mean "the version is missing", it means nothing was stored.

🚨 Until #3333, LastCompiledVersion alone broke that symmetry: it fell back to the node's version at stamp time rather than to the previous stamp. A skipped upload then produced old collection + old path + new version — a key the store never held, so TryGetAssemblyPath misses and activation silently falls back to the default configuration, the #1368 outcome by a different road. Because the stamp takes its value from the same projection that performs the Put, that fallback was the only way the recorded version could disagree with the stored one.

Null is the honest value when nothing was ever uploaded, and it costs nothing: every reader already applies its own ?? node.Version (MeshOperations, MeshDataSource, NodeTypeDataModelAreas, NodeTypeEnrichmentHelpers), so the guess belongs at the read — where it is a heuristic — rather than frozen into the node as a claim about the store.

🚨 The wait must be ORDERED AGAINST THE DISPATCH. CompilationStatus == null reads as settled, and correctly so: for a static-only type, or one that already has a usable build, "never compiled" is a final answer and EnsureCompileDispatched deliberately leaves those alone. But a subscriber that attaches to the own-node stream after the dispatch is not guaranteed to be replayed the dispatch's commit — the reduced own-node pipeline seeds a new subscriber from a source that can lag a write the same hub has already applied. So the settle question could be answered by the pre-dispatch snapshot, whose status is still null, and the handler would compile inline anyway (issue #3265).

The remedy is the same one MeshNodeStreamHandle.UpdateOwn uses for its own echo detection, and for the same reason: accept only a state of the node at-or-past the version this caller established — NodeTypeContractHandler.SettledAtOrAfter, gated on the version EnsureCompileDispatched reached. Never make the predicate stricter instead: refusing a null status would hold every static-only and already-built type for the whole 60 s settle budget.

Measured while this was broken (monolith, DOTNET_PROCESSOR_COUNT=2, 8 runs of 8): the dispatch committed v3 = Pending, the settle wait was handed v1 with a null status, two Roslyn runs went off, and two terminal success writes landed 16 ms apart — the handler's first (store key v1, no release path), publishing CompilationStatus = Ok; RunCompile's second (store key v4, with the release), re-stamping CompiledFrameworkVersion, LastCompiledVersion and the assembly coordinates.

That second write is why Status = Ok alone is not a completion signal for anything that intends to write to the node. With one driver there is exactly one terminal write and Ok is the last word; with two, the first Ok opens a window in which a concurrent writer's field is overwritten by the tail. Readers are safe either way (both writes describe a real build); writers must not treat the first Ok as "the pipeline is finished".

The bake ships from a SETTLED record, never from the first one it sees

The record and the store are two states, and a reader that takes them at different moments can name bytes that are gone. The package installer requests a release for every type it installs (the idempotence re-install too), and ObserveNodeTypeRelease completes when the RequestedReleaseAt trigger has been written, not when the compile it starts has finished — so a gate's report, and the bake behind it, can run while that compile is still in flight. The compile's upload then lands under a newer store version, the file-system store evicts superseded versions at write (it keeps the newest three), and the record the bake already read names a version the store no longer holds: "claims a usable build at v7 but the run's assembly store has NO bytes for it" (#3370, #3333 — measured on a tree that already carried the triple fix above).

BakeOutput.CollectOne therefore waits for the record to settle before it names the bytes — CompilationStatus == Ok, DispatchedBuildInputs cleared (a terminal status clears it, #3390), and no release request the watcher has not handled (the watcher's own pending test: RequestedReleaseAt newer than LastReleaseRequestHandledAt) — and, if it never settles inside the read budget, fails naming what was still in flight. The claim and the payload are read as one state; no bound was raised. BakeOutput.NotYetSettled is the predicate, pinned by BakeReadsASettledRecordTest.

Explicit — Create Release

The one entry point is hub.RequestNodeTypeRelease(nodeTypePath, …) (MeshWeaver.Graph/NodeTypeReleaseExtensions.cs) — GUI, agents, and tests all call it. It writes the trigger onto the NodeType node via stream.Update: RequestedReleaseAt (a timestamp, so repeated requests are distinct), plus RequestedReleaseForce to bypass the "sources match the last compile" short-circuit and RequestedReleaseBy to attribute the release to the caller. The per-NodeType release watcher dispatches only while RequestedReleaseAt > LastReleaseRequestHandledAt — an idempotent CAS — and lands on the same RunCompile.

🚨 Do not post CreateReleaseRequest from new code. The legacy request/handler pair (MeshDataSource.HandleCreateRelease) still exists for already-migrated callers, but the canonical surface is the stream.Update trigger above. See RequestViaStreamUpdate.

Use this to capture a named release with author-written ReleaseNotes.

Re-triggering after a source edit

RunCompile records a CompiledSources snapshot — {sourceNodePath → version} for every Code/Test node that fed the compile. When you edit a Source/*.cs node, its version bumps. The mismatch against the snapshot marks the NodeType dirty and triggers a recompile automatically — you never invalidate a cache by hand.

A disk-cache candidate must also match the generated input for the captured source/test and configuration snapshot. Its DLL completion time alone cannot cover an edit made while an earlier compile was running. See Compile Cache Input Freshness for the executed stale-byte/current-stamp regression and the unchanged-input cache control.

Source and test queries — and naming them

Which Code nodes feed a compile is declared on NodeTypeDefinition.Sources / Tests as mesh queries (defaults: namespace:Source scope:subtree / namespace:Test scope:subtree — the conventional sibling namespaces). Each entry may carry an optional name= prefix, e.g. "shared=@SocialMedia/Post/Source/Platform". The name is display-only: the NodeType side menu groups the resolved files under it (unnamed entries land in the default src / test group), while the compiler strips the prefix and behaves identically. CodeQueryResolver is the single expansion/grouping implementation, so the files shown in the GUI are exactly the files that compile.

Watching compile progress

Every compile runs on its Activity hub at {nodeTypePath}/_Activity/compile-{id}, created by NodeTypeCompilationActivity.Start. Subscribe to it as a normal MeshNode stream:

workspace.GetMeshNodeStream(activityPath)
    .Select(n => n.Content as ActivityLog)
    .Where(log => log is not null)
    .Subscribe(log =>
    {
        // log.Status   : Running → Succeeded / Failed / Cancelled
        // log.Messages : streamed Roslyn diagnostics + progress lines
    });

The activity path is surfaced in two places:

• NodeTypeDefinition.LastCompilationActivityPath — on the NodeType itself.
• NodeTypeRelease.CompilationActivityPath — on each release, succeeded or failed.

This means you can always drill from a release back into its full Roslyn output, regardless of whether it compiled successfully.

The NodeType's own NodeTypeDefinition.CompilationStatus reflects the terminal outcome: Compiling while in flight, then Ok or Error (with CompilationError carrying the formatted diagnostics).

Every stage is bounded — a compile can never park at Compiling

CompilationStatus is also the single-flight lock: the compile watcher fires only on a transition into Pending, and the Pending → Compiling flip inside the owning hub's serialized Update elects exactly one dispatcher. Two concurrent triggers therefore produce one run — but the corollary is that a trigger arriving while the type is Compiling is absorbed by design. So a compile stage that never answers does not merely run late: it strands the NodeType at Compiling for the life of the activation, with nothing able to restart it.

Every stage consequently has a wall clock, and the compile subscription is guaranteed to produce exactly one terminal status — an empty completion is caught by a totality guard, and no completion by these bounds. All four are tunable on CompilationCacheOptions:

A tripped bound is never something to raise: it means a stage genuinely stopped answering, and the message names which one. Fix that stage, then retry with Create Release / the Compile button — the terminal status is settled, so the next trigger dispatches normally.

🚨 A compile NEVER runs against an unestablished source set

The source snapshot is the one stage whose failure is not a compile verdict, and it is the only one that settles Unavailable instead of Error.

Discovery races two legs — a direct IMeshService read and the cached synced query — and each leg issues one query per declared Sources/Tests entry. Those queries can fail independently: a shared=@Other/Partition/Source entry crossing into a busy peer silo answers with a SubscribeRequest timeout while the type's own Source query answers fine. The snapshot therefore carries an establishment verdict (SourceSnapshot.IsEstablished), not just a list:

A failed leg used to be swallowed (.Catch(_ => empty)), so the surviving legs' partial set won the race and reached Roslyn — which then emitted a completely genuine-looking CS0246: The type or namespace name 'ScopeLibrary' could not be found about code that was fine. The bake readiness gate cannot tell that from a real image regression, so a rollout stalled on healthy content (issue #1218; memex-cloud 2026-08-11, 14 of 56 sampled types).

An unestablished report never wins the race either — it is held until both legs have answered, so one dead query cannot veto a healthy cached set. It only settles the snapshot when nothing else could.

Downstream, the Unavailable stamp already means the right thing everywhere: the instance overlay drops "please correct the code" for the retry copy (#641), EnsureCompileDispatched treats it as "never determined" and re-dispatches on the next request (never a timer), and DynamicTypePreWarmer.WarmOne reports PreWarmStatus.TimedOut — which NodeTypeBakeGateState files under not evaluated rather than as a regression. A real Roslyn error on an established set is untouched: Error → CompileError → it still gates.

Activations behind an in-flight compile show LIVE progress — never a silent park

An instance hub activating while its NodeType is Pending/Compiling waits for the compile to settle — but only briefly in silence. After a short grace (NodeTypeEnrichmentHelpers.InFlightOverlayGrace, 5 s), the activation stops holding every delivery and settles onto the compilation-in-progress overlay (WithCompilationInProgressOverlay):

• Every area of the instance renders the type's live progress page (NodeTypeLayoutAreas.CompileProgressView): current status, the streaming compile activity log, and — when more types are queued (the framework-bump warm-up recompiles every dynamic type) — the whole sweep as an "N of M types compiled" progress bar with the type currently compiling and the queued count. On Ok it redirects back to the area the caller asked for, so a deep link survives the wait.

  The overlay registers Overview by name plus a catch-all guarded by LayoutDefinition.HasNamedRenderer — the type's own areas (KeyMetrics, …) do not exist on the overlay hub, and covering only Overview did not remove the silent park, it relocated it: every other area answered "**Area not found** — No renderer is registered for area KeyMetrics", a terminal-looking verdict for a state that resolves itself in seconds (issue #1411). The guard is what keeps the catch-all off areas the default node configuration already owns — two renderers for one area are last-wins-destructive.

• Typed requests fail fast with ErrorType.CompilationInProgress naming the NodeType (UnhandledMessageNack), instead of parking until the caller's own 60 s request timeout. Area clients handle that NACK by swapping to the type's Progress area (AreaErrorClassifier.TryGetCompilationInProgressNodeType).

• The standard overlay self-heal watches the type: the compile's terminal write advances the node version, the watcher recycles the instance, and the next access enriches against the settled build.

A compile that settles inside the grace never surfaces any of this — short compiles activate the real hub directly. The grace is a visibility bound, not a compile bound: the compile keeps running however long it needs (see the stage bounds above).

A fault card must not outlive its cause — the overlay RE-EVALUATES

The overlay is a degraded binding with a lifetime, never a verdict. Enrichment binds a per-instance hub's configuration exactly once — the re-enrichment short-circuit (node.HubConfiguration != null) is what makes activation cheap — so a card applied during a bad ten seconds is served for the grain's whole lifetime unless something revokes it. That revocation is ArmOverlaySelfHeal, and its correctness rests on one rule:

🚨 The heal signal must not share a failure mode with the fault. Both original heal routes — the version-advance and the grace re-read — subscribe meshHub.GetWorkspace().GetMeshNodeStream(nodeType): the same stream whose silence made the enrichment slow path time out. When that stream stops emitting, the fault and its cure vanish together and the card is permanent.

That is not hypothetical. On 2026-08-17 (issue #1814) a deploy left the first pod compiling 269 types; page requests inside that window latched the card onto ~12 plugin roots (Training, Video, RolePlay, Edu, Chess, Collaboration, …). Store/Plugin then compiled successfully on both pods — and 1 h 24 m later an anonymous browser still got the card, while neither pod had logged a single overlay or "did not settle" event in the preceding 30–40 minutes. Nothing was retrying. Recycling the twelve roots by hand was the only remedy, and the card's own copy ("the page recovers automatically … this instance recycles itself") was straightforwardly false.

So the watcher has a third route that owes the stream nothing:

• AuthoritativeTypeRead — a one-shot path:{nodeType} query through IMeshQueryCore, as System. It reads the mesh's query providers (storage), not a cached stream, so a mirror that can never learn it is stale cannot suppress it.
• A widening ladder, not a poll — 45 s → 90 s → 3 min → 6 min, then 10 min for ever (ReEvaluationLadder / ReEvaluationCeiling). One read per rung, each capped by ReEvaluationReadTimeout and serialised with Concat, so a slow read can never overlap the next rung. At the 2026-08-17 blast radius that steady state is ~72 single-node reads an hour across the whole mesh. The ladder never stops — a re-evaluation budget that ran out would re-create precisely the defect it exists to remove.
• A faulted or empty read is not a verdict — it is logged and the ladder asks again. Giving up quietly is how the card latched in the first place.
• Loud on non-convergence — a re-read that still finds no usable build logs the type's status/assembly/framework; past the last rung it does so at Warning, next to the existing admin notification at StuckReportDelay.

…and the recycle it orders is SPACED, because it destroys its own watcher

The heal disposes the instance hub — taking the watcher with it. The replacement hub arms a fresh watcher whose ladder starts at the first rung, so a pair whose re-enrichment keeps faulting (a type that reports a usable build the instance still cannot bind — #1814's deterministic cross-hub Conflict) would recycle every 45 s for ever. No state inside the watcher can bound that, because the bound has to outlive the thing being bounded.

OverlayHealBudget (mesh-scoped singleton, registered in AddGraph) is that memory, keyed by (instance, NodeType). The first heal is never delayed; each further heal inside a 30-minute window waits out a widening spacing (45 s → 90 s → 3 min → 6 min → 10 min). It defers a recycle, never cancels one, and a pair that heals once and stays healthy is forgotten.

Pinned by OverlaySelfHealWatcherTest (silent stream → heals unaided; still-broken type → keeps its card on single-digit reads per hour; faulted/empty read → ladder continues; non-converging loop → bounded recycles, with the un-budgeted control in the same test) and OverlayHealBudgetTest.

Cancelling a compile

Compilation is an Activity, so it cancels through the Activity Control Plane (see ActivityControlPlane) — patch the activity's RequestedStatus, never post a bespoke cancel message:

hub.CancelActivity(activityPath);

The activity hub's control-plane watcher sees the patch and tears the compile down; the activity and the NodeType settle to Cancelled / the previous status.

Where releases live

Releases are MeshNodes at {nodeTypePath}/Release/{version}, with content type NodeTypeRelease:

NodeTypeDefinition.LatestReleasePath always points at the most recent release; the full release history is the set of Release/* children.

🚨 A publication is durable and verified, or it is REFUSED — a full volume publishes nothing

Measured on memex.systemorph.com, 2026-09-08. /data — the ReadWriteMany Azure Files share holding /data/assembly-cache, the module generations and the prebuilt bundles — sat at 3 MiB free of 16 GiB. Every recompile of Hosting/InstanceAction then went through unchanged: Roslyn emitted into the pod-local .mesh-cache, the emit was digest-verified there, the bytes were copied into the store, the terminal write minted Release/20260908133359-… naming the copy, and the same pod's first load of that copy failed with BadImageFormatException: Bad IL format. The reader deleted the fragment "for regeneration", the next activation's store miss compiled again, and the cycle produced three Releases (v851, v855, v859) in two minutes for bytes that were never there — while the InstanceAction node behind a live deployment restart stayed an untyped JsonElement on both replicas.

Nothing in the pipeline had lied; it had simply never asked. The store copy was File.WriteAllBytes + rename. On a Linux CIFS mount write(2) lands in the page cache and returns success; the server's ENOSPC surfaces only on writeback — at fsync or close — and .NET's SafeFileHandle discards close(2)'s return value. So the rename published a short file under a complete-looking, content-hashed name, and PublishBytes's first-publish-wins rule then guaranteed nobody could ever replace it: an existing name is "identical bytes by construction".

The rule now. A store publication (AtomicFileWrite.PublishBytes, which FileSystemAssemblyStore.PutWithLocation writes through) is:

1. written to the staging name, flushed to disk (FileStream.Flush(flushToDisk: true) — the fsync that surfaces a writeback error where the kernel reports one), closed;
2. read back by length and compared with the bytes handed in — which catches a volume that dropped bytes without reporting anything at all;
3. renamed into the discovery namespace only when both agree. Otherwise the staging file is removed, the target never appears, and a ShortWriteException carries the path, the bytes written, the bytes the volume kept, and the volume's free/total space.

A short write is deliberately not read as "the concurrent writer won": on a full volume the other replica's file is short too, so PublishBytes throws rather than answering false there.

The refusal is the compile's verdict. UploadToStoreIfNeeded treats an IOException out of the store as TERMINAL (AssemblyPublicationException, carrying the compile's transcript): the terminal handler writes CompilationStatus.Error with the refusal and the disk numbers as CompilationError, skips TryCreateReleaseNode, stamps no LastCompiledVersion and no assembly coordinates, and leaves the previous build's coordinates in place. The activity reads "Assembly NOT PUBLISHED — Roslyn produced it, but the assembly store did not keep the bytes", never "Roslyn failed" — the reader must be sent to the disk, not to the source. Because it is an infra fault, the park registry classifies it as non-deterministic: retried within its bound (a volume freed in time self-heals), then parked with the reason on the record. The "settles Ok with a warning" contract stays for what it was written for — a blob endpoint that TIMED OUT while the local emit still served this silo — and for nothing that reports "accepted" over bytes that are not there.

The reader names what it found. NodeAssemblyLoadContext.LoadNodeAssembly still deletes a file that fails to load (a fragment under a content-hashed name blocks every republication of those bytes), but records why on LastLoadFailure — the exception, the file's length, the volume's capacity — and CompileResultFromAssembly puts that on the verdict instead of "corrupt cached .dll or a missing dependency". A 4 KiB file on a share with 3 MiB free is a full volume; the record now says so.

And /health says it first. storage_capacity (Memex.Portal.ServiceDefaults, over the pure StorageCapacityHealth.Evaluate) reports the volume behind the FileSystemAssemblyStore root as Degraded — never Unhealthy: a replica on a full share still serves every page whose bytes are loaded, and pulling it would turn "cannot compile" into "cannot serve" — when free space is below AssemblyCache:MinimumFreeMiB (default 256 MiB), naming the path and the numbers.

Recovery is operational, not a redeploy: free space on the volume (the prebuilt-bundle identity directories are the usual growth; see the retention work tracked separately), then press Compile on the parked type — or wait for the park registry's bounded re-drive if the space came back within it. What this change does NOT do: it does not verify an existing content-hashed file at put time (a fragment from before this change is caught at load, above), and it does not reach AtomicFileWrite.PublishAsync, the stream-to-path download path, which has no expected length to compare against.

Pinned by ShortWriteIsNotAPublicationTest (the writer, the store, the reader, the health verdict — each arm with a production-writer control) and FullShareRefusesTheCompileTest (the record a real mesh writes: Error with the numbers, no Release, no pointer, "NOT PUBLISHED" on the activity).

Pinning an instance to a fixed release

By default, every instance hub of a NodeType binds to LatestReleasePath — a new release automatically moves them forward. To freeze a NodeType (and all its instances) on a specific historical build, set NodeTypeDefinition.RequestedReleasePath to that Release/{version} path.

• While RequestedReleasePath is set, instance hubs resolve to that release, not LatestReleasePath.
• Creating a new release updates LatestReleasePath but does not touch RequestedReleasePath — pinned NodeTypes stay put until you clear or move the pin deliberately.

This is the supported way to develop against a fixed release: pin the NodeType, compile freely, then unpin (or re-point) when you're ready to adopt the new build.

When is a NodeType recompiled? — verify-before-skip

The kickoff does not trust a bare CompilationStatus == Ok. That value is persisted into the NodeType MeshNode's JSON, so a stale Ok can easily outlive the assembly that produced it — and, conversely, a later failed compile can leave Status=Error behind a perfectly usable earlier build. NodeTypeCompilationHelpers.HasUsableBuild is the gate: a compile is skipped only when all of these hold —

1. LatestAssemblyCollection is populated (only a successful compile write-back sets it)
2. LatestAssemblyPath is populated
3. CompiledFrameworkVersion equals the current framework identity (FrameworkBuildIdentity.FrameworkVersion in MeshWeaver.Compiler)
4. the dependency clause holds — record-first (#1707 slice 2): when the definition carries a per-type CompiledDependencies record (referenced assembly name → surface-id, read off the EMITTED assembly's AssemblyRef table + the reserved !toolchain entry), every stamped pair must still resolve identically in this environment — so a module update invalidates only its dependents, and a type that binds no module is valid on any deployment regardless of composition. A null record falls back to the legacy whole-set CompiledModulesHash vs InstalledModulesFingerprint comparison (null stamp or null caller keep the framework-only behavior)

CompilationStatus is deliberately not a condition — the assembly fields self-heal across a stray Status=Error. The check is also metadata-only: no store probe, no File.Exists — the kickoff prefers a redundant compile over a blocking store round-trip, and a store that has lost the bytes is caught at activation (TryGetAssemblyPath misses → TriggerRecompileAndRetry); the bake probe's NodeTypeBakeStatus.Classify has the BytesMissing state for exactly that gap.

🚨 The !toolchain half of clause 4 is a PROXY, and it now demotes to a trigger (#1976). It hashes the toolchain closure's implementation MVIDs — 16 assemblies, 383 commits/30d — so it moves on changes that touch none of a given type's compile input. When a stale-build verdict has already been formed and the store already holds bytes under the live framework tag, the re-evaluation lane regenerates what a compile would be handed and compares it with the build's stamped !input content key: equal ⇒ the record is restamped and no compile is dispatched; different ⇒ compile (a NEW invalidation — that branch used to skip unconditionally); inconclusive ⇒ exactly the behaviour above, and never a restamp. Full reasoning, including the one half that is deliberately still gated: The Toolchain Re-evaluation Lane.

Adopt-before-compile (#1707 slice 3). "If a pre-built lib exists, take it; only if not, generate" holds at every entry point, not just at boot: a package INSTALL and a git-sync PUSH first run their affected types through the deployment's bundle sources (IPrebuiltAssemblyConsumer.SeedForTypes — the image's prebuilt/ plus the CI-published identity root, each assembly validated by the framework-identity and dependency-record gates), and the release-request watcher satisfies a request that arrives while the node already holds a valid build of the current sources — the trigger is consumed without dispatching Roslyn. RequestNodeTypeRelease(force: true) remains the documented escape hatch and always compiles. Anything not adopted compiles exactly as before — the "generate" branch is untouched.

🚨 "Always compiles" has to hold in the compile watcher, not only in the release watcher (#2818). The release watcher bypasses its satisfied-branch on a force and flips the type Pending — but the on-demand adoption step lives in the compile watcher, where every Pending converges, and until #2818 that step asked the bundle sources again regardless of the force. On any mesh whose bundle still resolved, a force re-adopted the very bytes the operator was trying to replace and settled "without a Roslyn pass" with a fresh LastCompileSucceededAt over the same LatestAssemblyMvid; it only worked where the bundle had gone missing. This is what left the stale prebuilt of #2813 unrecoverable on a node whose source was already fixed. Now a Pending that carries RequestedReleaseForce skips adoption and dispatches (or parks, under RequirePrebuilt, with the park's named reason), and every terminal stamp — ApplyCompileSuccess, ApplyCompileFailure, ApplyGateSettle — sets the flag back to false, so a spent force can never suppress adoption for a later, unforced trigger. The regression (NodeTypeOnDemandAdoptionTest.AForcedRelease_NeverConsultsTheBundleSources) asserts the discriminating fact — the forced Pending never consults the consumer — rather than the MVID moving, because a force on a type whose bundle does not resolve compiled correctly even before the fix. Its third phase found the adjacent gap: the on-demand step judged an adoption "landed" by HasUsableBuild alone, which a type's PREVIOUS build already satisfies — so a consumer that reported an adoption it never wrote back stranded such a type at Pending. Landed now also requires CompilationStatus == Ok, which is what a real PrebuiltAssemblySeeder.Seed stamps.

Anything else triggers a recompile. This makes a cold hub start self-healing:

A compile that FAILED is re-driven too — one attempt per set of inputs

HasUsableBuild and its framework-stale twin both key on assembly coordinates, and a failed compile writes none: ApplyCompileFailure stamps neither LatestAssembly{Collection,Path} nor CompiledFrameworkVersion. For a NodeType that never compiled successfully on this deployment those stay null forever, so every automatic path used to skip it — the first-build kickoff needs a null status, the recovery kickoff needs Compiling, the framework-stale kickoff needs the coordinates, the release watcher needs a human, and the park registry's source-change auto-retry is in-memory (a failure that predates the process is not in it). Only a human pressing Compile got such a node out; a redeploy, a framework bump, a module update and a fix to the failing code reached none of them (#1793; the fix written for fifteen types parked on memex-cloud could not reach the nodes it was written for).

So a failure records the one thing it honestly can: the inputs the verdict was formed from — framework identity, installed-module fingerprint, and the source snapshot the compile consumed — folded into NodeTypeDefinition.FailedBuildInputs. The owner-side re-drive fires exactly when the LIVE inputs differ from that stamp:

It is bounded three ways, and the first is the one that does the work:

1. Structural. The flip to Pending writes the live token in the same update, so the trigger the re-drive fires on is false the instant it fires. A reconcile that can re-arm its own trigger is the 257,000-version write-storm shape; the stamp forecloses it.
2. Loud. A process-wide ledger (NodeTypeCompileParkRegistry.RecordFailureRedrive) logs an error naming the path the moment a type is re-driven twice for the same inputs — i.e. the moment (1) provably did not hold. Non-convergence is never quiet.
3. Terminal. Past MaxAutomaticFailureRedrives the kickoff gives up for the hub's lifetime and says so, naming the type, its error and the remedy. An explicit Compile refunds the budget.

The re-drive is owner-driven, never caller-driven. It fires from the type's OWN hub on facts the node already holds; no request, and no requester's identity, is an input. The compile runs as System and its activity row lands in the owning partition attributed to System — exactly as the first-build, recovery and framework-stale kickoffs have always done — and no user's RequestedReleaseAt / RequestedReleaseBy is touched, so nothing is misattributed. An unauthorized caller who merely activates the hub therefore gains no lever: the trigger is a property of the persisted record, and the three inputs that can move it (framework identity, installed modules, the type's own source nodes) are all writable only by principals who already hold that access.

And when the re-drive declines — a type settled at Error/Unavailable whose verdict was formed under exactly the live inputs — the hub logs one warning per activation naming the type, its error and why nothing will retry it. That state is correct and bounded, and before that line it was also completely silent: nothing anywhere named a NodeType that is broken and will not be retried.

🚨 FailedBuildInputs is mesh-owned operational state: exports strip it, imports preserve the live node's value, and ShippedNodeTypeStateTest bans it from committed node files. An authored token that happened to match the importing deployment's live inputs would suppress precisely the retry it exists to grant.

🚨 A CREATE is an import too. The ownership rule (NodeTypeOperationalContent) used to run only inside the owner's update merge, and the installer's bulk path writes to persistence with no merge at all — so a type definition created from a file (a package installed into a fresh mesh, a first import) landed with the file's embedded verdict as its INITIAL live state. Measured 2026-09-06 (#3474): MeshWeaver.Plugins' node files, last written by GitSync before the export strip existed, carried compilationStatus: Ok, a foreign compiledFrameworkVersion and latestAssemblyPath, and for Store a standing requestedReleaseForce: true. Every fresh disposable mesh framework-stale-kicked Store's core types into a FORCED live-source compile at boot (#2824 honours the flag off the node), the boot sweep then adopted the shipped prebuilt over that compile, and every Edu/Exercise instance came up bound to a foreign assembly path and never answered. Now the two repo→mesh importers strip it where a repo file becomes a node — the installer's AsAuthored (every package file, both parse sites) and GitSync's ParseFile (every imported repo file, AND the static-repo snapshot: ParseSnapshot builds the nodes an InMemoryStaticRepoSource serves for the serveFromPartition partitions by mapping every file through ParseFile, so that path is covered only for as long as the two stay joined) — and PackageInstaller.BulkSave and PreserveLiveOperational(incoming, live: null) strip as well: with no live node there is nothing to prefer, and the operational members are ABSENT until this mesh writes them. 🚨 Deliberately NOT in the owner's generic create handlers: an in-process creator (a move, a restore, a test fixture) legitimately carries compile state, and the file-backed persistence reads the mesh's OWN nodes back through the same parser registry — the rule is about files that come from a repo, so it sits at the two seams where they enter.

🚨 A stale-source decline never leaves a DANGLING record

The decline-before-writing branch of PrebuiltAssemblySeeder (#2813) leaves "the live build's coordinates in place" so the build that is serving keeps serving. That reasoning assumed the live build resolves on the process that declined. On a pod that has restarted since that build, the coordinates name a local collection path (FileSystemAssemblyStore — the pod's own /tmp) in a pod that no longer exists: no process can load them, and nothing dispatched a compile (the branch's own comment claimed "the caller compiles" — the sweep reads a decline as compile-instead only for a bundle declined WHOLE). Measured on memex.systemorph.com, 2026-09-08: Crm/Client pointed at Crm_Client/v31756-….dll in the local collection of a replaced pod; both current replicas degraded every read of its content (MeshNodeContentDegradedException) for hours.

The rule now (PrebuiltAssemblySeeder.AfterStaleDecline, pure, pinned in StaleDeclineNeverDanglesTest): after a stale-source decline the seeder probes the store for the build the record claims; when it does not resolve on this process, the coordinates are cleared and a compile of the live source is dispatched through the one door (Pending with its inputs token) — once per decline, never per activation (a record already Pending/Compiling is left to the compile it carries). On a Modules:RequirePrebuilt mesh nothing is cleared and the seeder logs Critical: nothing that process can do will serve the type. The decline's outcome is reported to the sweep (SeedOutcome.DeclinedStaleSources…), which hands the declined paths to the sync reconciler — see Sealed Publication Reads → "The seal triggers the sync".

🚨 The probe is a fact about THIS process, written onto a SHARED record. Measured 2026-09-08: no host overrides the default IAssemblyStore, which is rooted per PROCESS (/tmp/MeshWeaver-AssemblyStore-pid<pid>, PersistenceExtensions.RegisterDefaultAssemblyStore), so a build compiled on one replica is never loadable by another — the coordinates a compile stamps are per-process facts on a node every replica shares (#3395's shape, for NodeType builds). This rule therefore fires once per replica per sweep whenever the bundle is declined: each replica clears the other's unloadable coordinates and compiles its own — bounded by sweeps (boot, install, push), never per activation, and no worse than the activation-time "bytes missing" self-heal that already recompiled on first access. It does not make replicas converge; whether they must — and in particular option (c) of #3417's open policy, a replica that is behind declines to write NodeType compile records — is the maintainer's decision. If (c) is chosen, this dispatch is one of the writes it must suppress; the seam is AfterStaleDecline's canCompileLocally argument.

🚨 An ADOPTED build must say whether it was ever checked against the source

Adoption — taking a prebuilt assembly from a bundle instead of compiling — is what makes installs and restarts cheap. It is also, until #2813, the one path that could make a NodeType assert something nobody established.

PrebuiltAssemblySeeder.Seed writes CROSS-HUB, so it cannot read the owner's live source snapshot (#1834); it asks instead, via RequestedSourceStampAt, and the owner answers by stamping CompiledSources = CurrentSourceVersions. That stamp is what makes IsDirty false, which is what InstallReleaseRequestWatcher's "satisfied by the existing current build" branch requires and what makes an adoption stick.

And it is also how an adopted build lied. The two signals an operator is taught to trust — CompilationStatus.Ok and CompiledSources == CurrentSourceVersions — both read clean whether or not the bytes have anything to do with the live source, because the adoption writes the second one itself. The staleness detector was never broken; it was answering a question the adoption had already answered for it. On 2026-08-30 a GitSync update pulled new source, adopted a prebuilt built from older source, reported Succeeded, and the stale code destroyed four client documents' bodies — one unrecoverable. Only forcing a compile moved the MVID.

The check, and where it can be made

The producer records a content fingerprint of the sources the bytes were built from. It has to be content, not versions: CurrentSourceVersions is {path → LastModified.UtcTicks}, mesh-LOCAL modification times the producer cannot know and does not have (the bake writes zeros), so a fingerprint over ticks would never match and every adoption would be refused.

What exactly is hashed — NodeTypeSourceFingerprint (in MeshWeaver.Compiler). The fingerprint is taken over the compile input: the NodeCompileShaping.CollectCompileSources fold — deduplicated ordinal-ignore-case, executable cells and blank files dropped — reduced to (node path, SHA-256 of the code text), plus the @@-include closure as (@@{resolved path}, SHA-256 of the code text), folded with PartitionSourceFingerprint, which sorts by path so enumeration order cannot reach the result. Both the runtime and the bake call that fold already, so they cannot fork on which files count.

The obvious alternative — hashing the source MeshNodes' serialised content — is unusable across processes, and each of its three failure modes produces a false refusal, which is an outage strictly worse than the staleness bug:

• Run bookkeeping churns it. CodeConfiguration carries LastExecutedAt / LastExecutedBy / LastExecutedCodeHash / LastActivityPath, written when a reader presses Run on a code cell. The live hash would move with no source change at all, and no producer can know those values.
• The two sides serialise differently by design. The consumer has a hub and therefore a TypeRegistry (polymorphic $type discriminators); the compiler-driven bake deliberately has neither — TreeNodeLoader materialises exactly the two content types a compile reads and leaves everything else null, precisely so a half-populated registry cannot degrade content to JsonElement. Two honest readers, two different JSON strings.
• Node metadata is not compile input. A description, an icon, an order: none of them change a byte of the emitted assembly.

It lives in MeshWeaver.Compiler because FrameworkBuildIdentity.FullMvidAssemblies is that assembly's transitive closure, and adoption is already gated on the framework identity matching — so a producer and a consumer that can adopt across each other necessarily run the same implementation of this hash. Anywhere else, two meshes could disagree about the shape while agreeing about the gate.

The comparison happens on the owner, in ApplyAdoptedSourceStamp — one pure function shared by all three writers that can fulfil the request, so turning assert into check fixes all three at once:

🚨 The legacy row keeps the stamp deliberately. Withholding it makes every legacy-bundle type IsDirty on arrival, the !IsDirty absorb branch stops firing, and every install recompiles everything — the 43 activations / 13.5 s of boot the prebuilt lane exists to remove. On a Modules:RequirePrebuilt mesh a local compile is refused by design, so not stamping would park every legacy-bundle type: the outage that refusing unproven bundles was rejected to avoid, arriving through a different door. A bundle with no fingerprint is of unknown provenance, not proven stale, and those deserve different answers. The requirement is VISIBILITY, not refusal.

BuildProvenance is operational (stripped on export, preserved from the live node on import) and is mirrored onto the compile-state satellite, so a control plane can read it through GetMeshNodeStream(path).

Whether the refused bytes keep serving is CONDITIONAL

Seed has already stamped the adopted build's assembly coordinates by the time the owner judges it, so a refusal that changes nothing else leaves proven-stale code executing. Both answers are wrong in one direction, and the fork is decided at the point of refusal rather than by assuming how a flag is set on a mesh nobody can see:

• this mesh can compile → clear the coordinates. The Pending flip has already dispatched a fresh compile, so the type is unserviceable for seconds — and "marked and still serving" is exactly the state that lets an armed control-plane node fire pre-fix code unattended.
• Modules:RequirePrebuilt → keep them, and log Critical naming the type. The local compile that would replace the bytes is refused by design, so clearing leaves the type with no assembly at all, indefinitely — an outage with no recovery path, self-inflicted by a guard. Only a rebake fixes it, and the log says so.

🚨 Do not collapse this to "RequirePrebuilt is unset everywhere". It is measured absent on memex and memex-cloud (#2194 item 3 records the same) — two instances, saying nothing about pearl, atioz, local installs, or any external instance the registry serves.

🚨 A module's content this mesh does not TRACK never compiles here — it errors, named

Every route above ends in the same fallback: when the bundle is refused, or no bundle for this framework identity has landed, compile the live source instead. That fallback is honest on exactly one kind of partition — one whose files TRACK the module's repository (a configured _GitSync), because there "the live source" is the repository at some commit and a local build of it is the same code the bake would have produced. On a partition nothing syncs, "the live source" is whatever an install left behind. Measured on memex, 2026-09-07 (#3583): the Feedback partition held four of the bundle's five files and no _GitSync; every Feedback bundle was refused on fingerprint (the bundle knew about the fifth file); and on every roll the portal compiled the leftover copy — old code, reading as current, with no line anywhere saying so. The maintainer's rule is the one this section implements: a module whose sources this mesh does not sync is never self-baked here — it errors, and the error names the fix.

The decision is taken in the ONE place every compile passes through, the compile watcher's DispatchOrPark (the same gate that parks a Modules:RequirePrebuilt mesh), from two facts that are already in hand there:

1. Is this a MODULE's content? Either a bundle entry on this identity NAMED the type in the adoption pass that just ran — adopted, already current, or declined — or the record carries adoption provenance from an earlier identity (AdoptedSourceFingerprint survives a local compile; BuildProvenance is reset by one, so it only ever adds). The pass reports the first through a witness the consumer exposes (IPrebuiltAssemblyConsumer.SeedForTypes(paths, onOffered), defaulted so an older consumer reads as "nothing offered"). Authored content — never offered, never adopted — is not a module's and compiles exactly as before.
2. Does the partition TRACK a source? IPartitionSourceTracking (MeshWeaver.Graph.Contract), the one-bit seam the sync layer implements — the GitHub provider answers true when at least one _GitSync of the partition names a repository (a config node with an empty repository is the settings tab's "not configured" state, not tracking). It is the SAME object that feeds the Partition Sync administration page, so the page and the gate cannot disagree. A mesh with no implementation registered — a local mesh, CI's disposable meshes, the bake host — has no notion of tracking and compiles as before.

Module content and an untracked partition → the type PARKS with PrebuiltAssemblySeeder.UntrackedPartitionParkReason: deterministic, the attempt counter at zero, the reason naming the partition, whether a bundle was declined or simply absent, and the two fixes — add a sync source for the partition and sync it, or publish/rebake the package for this identity — "then request a release to retry". It lifts through the same doors as every other park: a source change (the sync that was missing) re-drives it, and a release request after the rebake re-runs the adoption pass, which now adopts. RequestedReleaseForce does NOT bypass it: a force means "build the live source", and on such a partition the live source is the problem.

Whether the answer could be read is never a verdict: a fault or a timeout while asking the tracking seam compiles, as the mesh did before the gate existed — a stranded type is worse than a redundant compile, and the refusal is only ever formed from a positive false.

🚨 The path the fingerprint actually travels — and why it was INERT for months

The comparison above shipped complete, and for months it could not fire. Nothing was broken in it; the value it compares simply never arrived, and every part of the system reported success:

1. PrebuiltAssemblySeeder.Seed had a seven-parameter convenience overload that hard-coded sourceFingerprint: null. Both production callers — PluginBundleClient and ShippedPrebuiltBundles — bound to it, so AdoptedSourceFingerprint was never written, the first guard in ApplyAdoptedSourceStamp short-circuited, and every adoption on every mesh returned AdoptedUnverified. That overload is now [Obsolete(error: true)]: passing null is still allowed (a legacy bundle genuinely records none) but must be written at the call site, where a reviewer sees the claim being waived. It is obsoleted rather than deleted because deleting public framework surface is a breaking change to in-mesh code no compiler can see, and the live-mesh sweep AGENTS.md requires could not be completed — search_chunks answers "searched": false on both reachable deployments, which is a FAILED sweep, not a clean one. Both repos' node trees were swept by hand and hold no caller, so the symbol stays (nothing already compiled breaks) while every source call site now fails loudly, at the call, with the reason.
2. No producer wrote one. BundleWriter.AssemblyEntry.SourceFingerprint / BundleReader.AssemblyRef / BundleReader.Payload now carry it end to end, and all three bake producers record it — the compiler-driven TreeBake and CascadeBuild from the tree, the mesh-driven BakeOutput through the same NodeSources.GetSources query the owning hub reads.
3. The live half was computed only when it was already too late to matter. The owner computed CurrentSourceFingerprint only "when there is something to compare it against" — a condition unsatisfiable in the ordering the incident took. The owner publishes its snapshot first; the adoption's patch then lands carrying both the adopted fingerprint and the stamp request, and the sources watcher does not re-run, because its DistinctUntilChanged keys on the source queries, which did not change. The judgement then read an absent live value and took the "inconclusive" branch. It is now computed on every publication (a SHA over text already in hand, on a path where a Roslyn compile is about to cost four orders of magnitude more), the idempotency check includes it so a node persisted before the field existed self-heals on its next activation, and the two writers that cannot compute it — the standalone stamp watcher and the release-request watcher — now wait rather than consume the one-shot request on an absence.

🚨 The lesson is the general one: "the fix merged" is not "the fix runs." A guard whose input is never supplied is indistinguishable from a guard that passes. The regression that pins this one is AdoptedBuildSourceStampTest, which drives a real bundle through BundleWriter → BundleReader → ShippedPrebuiltBundles → Seed → the owner's stamp on a real mesh and asserts all three rows of the table; a unit test over the pure function cannot see any of the four links above.

Producer and consumer must hash identically, or every good bundle is refused. That equality is pinned twice. BakeEquivalenceTest bakes one content set both ways — through a real mesh (whose value is the consumer-side value) and through the mesh-free tree bake — and asserts the fingerprints are equal, non-empty, and different for different types. And the PR lane asserts it on the REAL trees: bake-then-gate.sh stages the bake and the gate from the same tree, so an ADOPTION REFUSED line in that run can only mean the two producers hash the same content differently, and assert-bake-consumption.sh fails on it by name.

🚨 That second check exists because a false refusal is otherwise INVISIBLE in CI, in the same way the original defect was. Seed returns true, the adopted/declared counts balance (the owner refuses afterwards, when it stamps), the type flips to Pending, recompiles locally, and every per-type verdict is green. Only the log says anything — and only because the refusal logs at Error, above the gate's default level.

🚨 The fingerprint covers the @@-include closure — both sides resolve before hashing

An @@ include pulls a Code node that no source query matches. It reaches the emitted assembly (NodeSetCompiler.ResolveInputs substitutes it after the fold) and it used to be absent from the hash — so editing an included-only snippet moved neither AdoptedSourceFingerprint nor CurrentSourceFingerprint, and a prebuilt assembly baked before that edit still adopted as AdoptedVerified.

That is the worst of the three rows to be wrong on. AdoptedUnverified says "nobody established where these bytes came from", and an operator reads it as the warning it is. AdoptedVerified is an assertion that the shipped bytes match the source this mesh holds — standing over source that was never hashed, it is a false verification, which is the #2813 incident one layer in. So both halves now resolve includes first (#2948):

• The producer already substituted them, so it simply keeps what it pulled in. NodeCompileShaping.ResolveCodeIncludes takes an optional collector and records resolved path → code text at the point it consumes each include; ResolveInputs hands that back on CompileInputs.ResolvedIncludes, and TreeBake / CascadeBuild fold it straight into the fingerprint. No second walk, no second read, and no new blocking bridge at a synchronous build step.
• The consumer resolves it through the mesh, in the one place that already holds the live source nodes: the sources watcher (NodeTypeCompilationHelpers.InstallSourcesWatcher). Resolving a closure means mesh READS, which a pure Update lambda cannot make — so the fingerprint moved out of that lambda into an observable step ahead of it — composed with Switch(), so a newer source set supersedes an in-flight resolution rather than racing it.

The cost is bounded by the shape of the walk, not by the size of the mesh: CollectIncludeClosure scans each source's text for @@ and issues no read at all when there is none, which is almost every type. Only a type that actually has includes pays, and it pays the same reads its own compile would.

Why the walk is order-stable and cycle-safe. A fingerprint that moves when nothing changed is phantom staleness — endless recompiles, or a false refusal — so three properties are load-bearing:

1. Order cannot reach the result. The closure is keyed by resolved path and returned sorted (ordinal); PartitionSourceFingerprint then sorts again. The roots are walked in CollectCompileSources order — itself ordinal by node path — and serially, one SelectMany chain and never a Merge, so there is no read interleaving to observe either.
2. Suppression is result-preserving. The per-root visited set skips a path that root already walked. Skipping a re-read of the same anchored path can only re-derive an entry that is already present, so which parent reached a shared snippet first cannot change the closure.
3. A cycle terminates. A → B → A adds each path to the visited set once; the second visit takes the already-added branch, reads nothing and recurses no further. A self-include is the same case with one hop.

And an unreadable include is INCONCLUSIVE, never absent. This is the direction that would hurt. The producer's lookup is in-memory and never stalls; a consumer that quietly treated a stalled read as "the include is gone" would hash a shorter closure, which is indistinguishable from a stale bundle — so a perfectly good adoption is refused, and on a Modules:RequirePrebuilt mesh that is terminal and needs a human to rebake. SourceFingerprintIncludeReader therefore uses GetMeshNodeOutcome and keeps the three states apart: Present contributes, Absent contributes nothing (it contributes nothing to the bytes either — the directive stays verbatim, so both sides agree), and Unavailable / DeleteInProgress / a timeout raise SourceIncludeUnavailableException. The watcher catches exactly that, logs it, and leaves the previously published CurrentSourceFingerprint standing — the judgement then takes the "nothing has been compared" branch (AdoptedUnverified). Same rule as the emit canary (#890): a probe must not answer its scariest branch on its own inability to run.

Two things this deliberately does NOT do

• The include closure is NOT added to CompiledSources / CurrentSourceVersions. So IsDirty still does not notice an included-only edit, and the sources watcher — which re-runs on its source query — does not re-publish until the type's own source set moves or its hub reactivates. That is the recompile-trigger half, and it is a different mechanism: a change feed, not a hash. It is also a producer/consumer contract — the bundle manifest's sourceVersions mirrors the RAW query match on both bakes, which BakeEquivalenceTest pins deliberately (an include target is asserted absent from sourceVersions and present in the emitted surface and in the fingerprint's input). What #2948 closes is the verified claim, which is decided by the two fingerprints and nothing else.
• Refusing to LOAD is the second line of defence. The damage needs two ingredients — stale bytes and something armed to run them. A type that renders read-only pages from unverified bytes is a degraded system; a type that WRITES from them is the incident. The execute-time half is The Execute-Time Build-Provenance Interlock (#2820): a type whose provenance is AdoptionRefused is never given a durable, write-capable per-instance hub and is never joined into a kernel session's cell surface, while its own pages keep rendering so an operator can diagnose it. AdoptedUnverified is deliberately NOT refused there either, for exactly the reason the legacy row above keeps its stamp.

BuildProvenance is reset to Compiled by ApplyCompileSuccess: Roslyn built those bytes here, from the source this mesh holds, so nothing about an earlier adoption survives a successful local compile. Without that reset the field was write-once-per-adoption — a type refused as stale and then recompiled kept reading AdoptionRefused forever, which turns the operator signal into noise and would make the execute-time interlock refuse a type whose source it had just compiled itself. ApplyCompileFailure deliberately does not reset it: after a failed compile the bytes in place are still whatever the refusal left, so the refusal is still the true story.

🚨 Do not use LatestReleasePath vs the NodeType as a staleness signal. It lags routinely — a healthy type can serve perfectly with its release pointer several versions behind, and a check of that shape fires constantly on healthy meshes. The fingerprint comparison above is the signal precisely because it does not have this problem.

🚨 When the PROCESS cannot emit — a failure that is not a verdict at all

Everything above assumes the compile found something out. Sometimes it does not: Roslyn's Emit throws instead of returning diagnostics, and from that moment the process cannot emit any assembly at all. This is #890 — a NullReferenceException inside Cci.MetadataWriter.GetConsolidatedTypeParameters, reading a ContainingType that the guard immediately above it had just read as non-null.

The condition is total and permanent, not intermittent. Measured on run 33322993649 shard 1 (2026-08-30): the first throw landed at 16:42:42.375, and across the remaining 6 m 15 s of that process 7 of 7 compiles that reached the metadata writer failed identically and none succeeded. Compiles that only needed diagnostics kept working perfectly — the deliberately-broken NodeTypes in NodeTypeCompileParkTest still reported their real CS1040/CS0246 codes — so parse and bind were healthy and only the emit was dead. That split is the fastest way to recognise it in a log: correct CS#### for broken source, NullReferenceException for source that should succeed.

Replicated independently in MeshWeaver.Plugins — run 33760859754 shard 3, 2026-09-03, a push to main, MeshWeaver.Hosting.Monolith.Test. Onset 13:34:23.568, 1.3 s into CodeEditRecompileTest.CodeEdit_ExplicitRelease_IsUpToDate_RecompilesOnSourceChange and 33 s into the process; it is the first NullReferenceException in the whole trace, so it is not downstream of an earlier fault. Then 39 of 39 compile faults over the next 13 m 21 s were that same NRE with canary=BELOW-ROSLYN — zero CompilationException — across 12 unrelated node paths and partitions. Parse and bind stayed healthy throughout, on two independent readings: correct CS0103/CS1591 diagnostics were still returned at 13:41:55, +7 m 32 s after onset, and BrokenNodeTypeAccessTest passed at +13 min. Memory was flat (managed 147 MiB / RSS 961 MiB) through the final wedge, and the ten test hosts that ran after it on the same runner were clean, so the condition is process-scoped, not machine-scoped. Two repos, two harnesses, two shard layouts, one shape.

🚨 Onset is not a warm-up threshold. Core's occurrence fired 130 s and 199 TEST STARTs into the host; this one fired at 33 s and 12, with alc=1, asm=138, gc2=4 — no ALC churn, no memory pressure, and the two ALC-heavy suites (NodeTypeRecompileAlcLeakTest, NodeAlcUnloadTeardownOrderingTest) ran after the onset, not before it. Any hypothesis that needs a precursor workload has to explain that.

It is not a MeshWeaver defect and no compile-side change fixes it. EmitPipeline's canary answers that at the first throw, in two legs: re-emit a trivial, freshly parsed, known-good compilation against the same references, then — if that fails too — against an image-backed CoreLib that shares neither the MetadataReference instances nor their file mappings. REFERENCES and BELOW-ROSLYN both mean the control could not emit either.

What that changes about the verdict

A compile that aborts this way has learned nothing about the code it was handed, so recording CompilationStatus.Error states a verdict that was never formed — and it is durable: the same write stamps FailedBuildInputs, the re-drive above reads it back as "formed under exactly the live inputs", and declines. The type is left saying "your code is broken" about code nothing evaluated, retried by nothing until a human presses Compile or an input genuinely moves.

So IsAvailabilityNonVerdict treats it as the third availability fact, beside an unestablished source set and a recycling address: status Unavailable, no verdict recorded, the park budget untouched, and the type re-drivable — which is what lets a later, healthy process compile it. The bake gate filing this as unevaluated rather than as a code regression is the correct reading: a bake whose process cannot emit has evaluated nothing.

🚨 The predicate reads the VERDICT, never the presence of a canary. Every emit-phase throw carries one, and three of the five verdicts deliberately withhold the claim — OK (the control emitted fine against the same references, so the fault is this compilation's inputs — a genuine Error), INCONCLUSIVE (the control could not be built, so leg 2 never ran) and DIVERGENT (both legs failed, in different frames). Widening this to "any infrastructure fault" is the blind spot SourceSnapshotEstablishmentTest.EveryOtherCompileFailure_StillStampsError exists to refuse.

Reading it in CI — one event, many names

One poisoned process reports as up to ten unrelated test failures; across 2026-08-22→08-29 that was 9 events wearing 37 distinct test names, 23 % of all failing test names in the week. Every occurrence has cost a fresh and always-identical misdiagnosis. In run 33322993649 all five failures — three NodeTypeCompileParkTest, one CodeEditRecompileTest, one ReleaseRequestWatcherHighWaterTest — were this one event, and the shard's exit=124 was its consequence, not a second problem.

Two readings that look right and are not:

• "The retry compiled stale source." It did not. The retry runs a real compile; the broken source's CS#### lines in the same test's output belong to the first compile, before the fix was written. The retry's own error is the NullReferenceException. Compare timestamps, not adjacency.
• "That one is just a slow shard." ReleaseRequestWatcherHighWaterTest's compile threw 174 ms after TEST START; the 50 s wait that followed was waiting for an Ok that could never arrive.
• 🚨 "It's a compile suite and it hit the cap, so it's this." The 2026-08-13 sweep found that exit=124 on MeshWeaver.Hosting.Monolith.Test was almost entirely this defect. That has stopped being true. Measured 2026-09-03→04 on MeshWeaver.Plugins: 3 such timeouts in 122 runs, of which 1 carries the signature and 2 carry none — read on the authoritative sink (467 and 560 [FAULT] records each, zero canary=), so the nulls are not the job log's failing-tests-only blind spot. Both nulls wedged inside one compile test that never ended (one unclosed TEST_START for the whole 15-minute cap, methodTimeout never firing) — a hang, not this defect's fast many-failure cascade, and both were in this issue's own blame set (NodeTypeReleaseGateTest, CodeEditRecompileTest). Discriminate on the verdict, never on the suite name plus the exit code; folding a live wedge into a known-upstream defect hides it.

🚨 The core dump the canary used to ask for cannot be captured this way. DOTNET_DbgEnableMiniDump fires on a signal, and this process never crashes — it throws a managed exception and keeps running until CI's per-suite wall-clock cap kills it (exit=124, no dump): 8 minutes on core's shards, 15 minutes on MeshWeaver.Plugins' Portal hosts shards, so the cap is a property of the harness and not a fingerprint to grep for. That is why weeks of "capture a core dump" produced none. Its SIGSEGV twin, #613, does dump — see Debugging Native Crashes for the invariant the two share (a single 8-byte word reading exactly zero while its block stays coherent) and for how to read one. The verdict no longer asks for one — see leg 3 below.

Leg 3 — DISSECT the null, because the dump route is closed

The verdict's own text called the remaining ambiguity a RESIDUAL: "both legs still run on the one CLR, so this does not separate a corrupted heap from a miscompiled Roslyn method", and it resolved that by asking for a core dump — an instruction that cannot be carried out on this defect at all, for the reason in the box above. From 2026-08-28 to 2026-09-04 every occurrence carried it and produced zero dumps. A remedy nobody can execute is the prose form of a gate that cannot fail: it looks like a next step and it is a dead end.

Legs 1 and 2 both ask can this process EMIT? Neither asks the far cheaper question the stack points straight at — can it still perform the READ the emit dies on? Every #890 stack ends in NamedTypeSymbol.Microsoft.Cci.ITypeDefinitionMember.get_ContainingTypeDefinition(), which in a Release Roslyn reduces to return this.ContainingType.GetCciAdapter();, called from MetadataWriter.GetConsolidatedTypeParameters — whose guard (AsNestedTypeDefinitionImpl) read that same ContainingType as non-null microseconds earlier. Leg 3 makes both of those reads directly, on symbols bound after the fault, with no emit, no metadata writer and no PE stream:

🚨 Both polarities — the read that fails is a NULL that must stay null

Leg 3 shipped reading only Leaf<V>.ContainingType and Inner<U>.ContainingType, two reads that must come back NON-null, and the first two readings it ever produced were earned on that half alone. Look at what the metadata writer actually does, though:

// AsNestedTypeDefinitionImpl — the GUARD
if ((object)ContainingType != null && IsDefinition && ContainingModule == …) return this;
// ITypeDefinitionMember.ContainingTypeDefinition — reached ONLY when the guard answered TRUE
return ContainingType.GetCciAdapter();          // NRE iff ContainingType is null

getConsolidatedTypeParameters recurses up the containing chain, calling those two back to back on the same object, so it walks Leaf → Inner → MwEmitCanary — and at the top-level type the guard is supposed to answer FALSE and stop. If that read answers TRUE, the property is then called on a type whose ContainingType is null correctly, and it throws the #890 NRE at the #890 frame. No corrupted symbol, and no pair of disagreeing reads, is required — which is the framing this issue had carried since 2026-08-13 ("two reads of a readonly reference field, no writer in between, different answers"). A probe that only asks "does a non-null read come back non-null" cannot see that shape and reports it as symbol:OK. The top-level read is now made too, and it must come back NULL; the classification is a pure function so the branch a healthy Roslyn can never produce is still covered by a test.

🚨 The Cci leg deliberately stays on the NESTED type. On a perfectly healthy process, get_ContainingTypeDefinition called on a top-level type throws a NullReferenceException at the exact frame every #890 stack names — correctly, because the containing type really is null. Probing it there would manufacture this defect's signature on every healthy run. The frame is not the finding; the metadata writer having REACHED it is.

🚨 UNAVAILABLE is the branch that has to stay impossible on a healthy process. Leg 3 reaches Roslyn's internal symbol model by reflection (PublicModel.Symbol.UnderlyingSymbol, then the interface map for Microsoft.Cci.ITypeDefinitionMember), because there is no public route to the method the stack dies in. A Roslyn rename would turn every future occurrence into a permanent non-answer with nothing going red — the same way a file-backed leg 2 silently stopped being a control. EmitCanaryDissectionTest.OnAHealthyProcess_BothReadsResolveAndAnswerCorrectly is what refuses that: it asserts symbol:OK and cci:OK, and renaming the reflected member turns it red naming the member (measured — cci:UNAVAILABLE(no UnderlyingSymbol… on NonErrorNamedTypeSymbol)).

A local control — our own nested generic plus an explicit interface implementation — was considered and deliberately left out. Its failing branch would be informative; its passing branch is not, because a different jitted method reading correctly says nothing about Roslyn's, and a probe whose green means nothing is the shape this canary keeps having to remove.

What the verdict says instead of "capture a dump": re-run SPLIT-ARM with DOTNET_TieredPGO=0 (sharper) or DOTNET_TieredCompilation=0, and read it together with the dissect= line. At ~1 % per run a single clean arm proves nothing — that caveat is part of the instruction, because an experiment stated without it gets read as a fix.

Leg 4 — vary the NESTING, inside a real emit

Leg 3 asks its question from OUTSIDE the emit, and that is its ceiling. It reaches the symbols by reflection, so it exercises get_ContainingTypeDefinition's own compiled body from a caller that is not MetadataWriter. If the fault is code that is wrong only where the writer reaches it — the hypothesis the readings themselves point at — leg 3 answers READS-HEALTHY by construction, no matter how broken the process is. Four unanimous readings (2026-09-05, 09-06 ×2, 09-07) are exactly what that ceiling looks like from the outside, and no amount of further reading narrows it.

Leg 4 stays inside a real Emit and varies one thing instead: whether the compilation contains a nested type at all. Same references as leg 1, same process, microseconds later — EmitPipeline.FlatCanarySource, a single top-level, non-generic, member-less class.

Why that discriminates. MetadataWriter.GetConsolidatedTypeParameters opens with typeDef.AsNestedTypeDefinition(Context) and returns IMMEDIATELY when that answers null. For a top-level type the recursive overload — and with it the ITypeDefinitionMember.ContainingTypeDefinition call every #890 stack dies in — is never reached. The class carries no members either, so the NamedTypeSymbol overload of that getter has exactly one caller left in that emit: AsNestedTypeDefinitionImpl's guard having answered TRUE for a type whose containing type is null by construction.

It runs on exactly the two verdicts leg 3 runs on (BELOW-ROSLYN, DIVERGENT) and never on REFERENCES — there the pristine leg emitted, so the process demonstrably can emit and the nesting question does not arise.

🚨 This replaces an assumption that has been load-bearing since the canary was written. The source comment asserted "a flat class would emit fine even on a poisoned writer and the canary would answer healthy wrongly". Nothing ever measured it, and it is why every occurrence has been read as PROCESS CANNOT EMIT — a claim about emit as such, on evidence that only ever exercised one shape. Both branches of leg 4 are informative and neither was previously observable.

🚨 The mutation guard is the leg (EmitCanaryFlatLegTest). The discriminator holds only while the flat source really has no nested type, no generic arity and no members — a field added "for realism" is a second ITypeDefinitionMember, i.e. a second caller of a ContainingTypeDefinition getter, and flat=SAME-FRAME silently stops meaning what it says. The shape is asserted off Roslyn's own binding (SourceModule.GlobalNamespace, ContainingType, Arity, members), not by matching the literal, so a rewrite that preserves the shape passes and one that does not cannot.

The first dissect= readings, 2026-09-06 — and what they do and do not settle

Leg 3 landed 2026-09-04 and its first readings arrived immediately. Two occurrences, both in MeshWeaver.Plugins Plugin Catalog CI, both on platform commits that carry leg 3:

Every reading, in both sinks of both occurrences, is dissect=READS-HEALTHY symbol:OK cci:OK — 78/78 and 53/53 for the first, 62/62 and 28/28 for the second; zero SYMBOL-GRAPH-BROKEN, REPRODUCED-OUTSIDE-EMIT, UNAVAILABLE or NOT-RUN. All faults in each occurrence carry one pid (3033, 2800).

🚨 Read that verdict against what leg 3 measured at the time, not against what it names. Those readings were produced by the two NON-null reads only — the top-level, null-polarity read is added by this change, and a process broken in that direction would have answered symbol:OK too. So the two READS-HEALTHY readings exclude a symbol graph whose nested containers read null, and they do not yet exclude a guard read that answers TRUE for a top-level type. The next occurrence, on a platform commit carrying the third read, is the first one whose READS-HEALTHY means what the word says.

Two things the readings do settle, and they are worth having:

• get_ContainingTypeDefinition called directly — the exact frame every #890 stack dies in — answers correctly on symbols bound microseconds after Emit threw on that very shape. The method's own compiled body is not the broken thing.
• Combined with BELOW-ROSLYN (references, mappings and source already excluded), what is left is code that is wrong only where MetadataWriter reaches it — i.e. the inlined copies of the guard and the property inside getConsolidatedTypeParameters. That is the split-arm DOTNET_TieredPGO=0 hypothesis, and it remains UNTESTED.

🚨 Two sweep traps this measurement walked into, both worth inheriting. GET /actions/runs/{id}/jobs defaults to filter=latest, so attempt-1 shard jobs of a re-run run are invisible — the 2026-09-06 occurrence is on attempt 1 of a run whose overall conclusion is success, because attempt 2 re-ran that shard and passed. A sweep over non-success jobs of the latest attempt finds only the first occurrence. And the trace artifact is written only when a suite FAILED (measured: 29 of 609 teardown-stragglers-* artifacts carry _meshweaver-test-trace.log), so both known sinks are failure-gated: an occurrence that reddened no test would be invisible to either.

The rate has not moved — a null here is worth ~half a coin toss

Continuing where the 2026-09-03→04 sweep ended, 2026-09-04T06:34Z → 22:01Z on MeshWeaver.Plugins Plugin Catalog CI: 73 completed runs, 287 Portal hosts (shard N) jobs, all 93 non-success shard logs fetched (0 unfetchable), detector calibrated first against the known occurrence (job 100666643224 → 56 canary=BELOW-ROSLYN, 20 PROCESS CANNOT EMIT, the counts that occurrence's own report published). 0 occurrences.

🚨 That null is not evidence of anything. At the last measured ~1 %/run, P(0 in 73) ≈ 48 % — a coin toss. The positive control holds (the exposure suite MeshWeaver.Hosting.Monolith.Test really did run in the window: runs 33881146362, 33879617931, 33879408750), so the sweep had somewhere to look; it simply did not look at enough runs to say anything. The last confirmed occurrence is 33760859754, 2026-09-03. Read a #890 sweep the way the 2026-08-09 close should have been read: state the expected count beside the observed one, or do not state the null.

2026-09-07 — the fourth dissect= reading, and the precursor search that came with it

MeshWeaver.Plugins run 34110361260, job 101705567486, Portal hosts (shard 3), attempt 1 of PR #1463. 94 canary=BELOW-ROSLYN, 28 PROCESS CANNOT EMIT, 94 dissect=READS-HEALTHY symbol:OK cci:OK — a fourth occurrence agreeing with the first three, on a platform commit that carries the third read, so this one's READS-HEALTHY does mean what the word says. Both legs threw, both in NamedTypeSymbol.Microsoft.Cci.ITypeDefinitionMember.get_ContainingTypeDefinition; the frames are byte-identical to 2026-09-05 and 09-06. 13 failing tests across 4 classes, shard KILLED (exit 124).

#3425's attribution fix works. PROCESS CANNOT EMIT now carries its exception, so it reaches _meshweaver-test-trace.log: 40 records there, against a measured 0 · 0 on the two 09-06 occurrences. The line whose whole job is "attribute the failures that follow to this line" is on the authoritative sink for the first time.

What ran immediately BEFORE the first failure — and what that excludes

Read from _meshweaver-test-trace.log (teardown-stragglers-34110361260-1-shard3), pid 3039, host first line 10:28:38.948. First poisoned compile type/OverlayDemo at 10:29:50.936 — 72.0 s into the host, after 167 TEST_STARTs — 263 ms into MeshNodeLanguageServiceTest.OverlayCompletions_CompleteAgainstProposedText, which then PASSED. 21 distinct paths were poisoned over the following 13 m 11 s, ending at 10:43:01.

Three candidate precursors were checked against the 2026-09-06 21:02Z occurrence (34059122838, pid 3040, onset 21:01:37.955, 55.4 s / 48 TEST_STARTs in). All three are refuted:

The onset is simply the first emit attempted after the poisoning, and the "first poisoned type" is whatever the running test happened to compile: type/OverlayDemo under a language-service test, type/AlcUnloadLiveProbeStory under an ALC-unload test, TestData/CodeEditType under a recompile test. The warm-up spread is now 4× wider than the number two doc pages used to quote — 33 s / 12 starts, 55 s / 48, 72 s / 167, 130 s / 199 — so onset timing constrains nothing.

🚨 The honest conclusion, stated as a limit rather than a lead: the poisoning event is not visible in any sink the platform writes. _meshweaver-test-trace.log takes a record only for an ILogger call carrying an exception at Warning-or-worse, so it can see the poisoning event only if that event is itself a logged fault — and the last fault before onset differs completely between occurrences (a hub-disposal cascade on type/AppendDemo on 09-07; CompileStateMirror satellite-write failures on 09-06). No event on the authoritative sink discriminates a poisoned process from a healthy one. Adding a fifth read-probe would not change that; leg 4 and the split-arm run are the two instruments left, and they measure different things — mechanism and population.

2026-09-10 — the first flat=EMITS in the wild, a local negative, and the control NO leg has ever been

MeshWeaver.Plugins run 34519677252, job 103014057395, Portal hosts (shard 2), PR #1630. Runner ubuntu-24.04, runtime 10.0.12, SDK 10.0.401, linux-x64 (read off the job's own dotnet-install lines, not assumed).

Both legs threw at NamedTypeSymbol.Microsoft.Cci.ITypeDefinitionMember.get_ContainingTypeDefinition, byte-identical to every prior occurrence. flat=EMITS is the reading leg 4 was built for: a top-level, non-generic, member-less class emitted fine in the same process microseconds after the nested source could not, so this process was not emit-dead and the fault needs the recursion. Note that this contradicts the 2026-09-08 occurrence's flat=SAME-FRAME — the flat leg's answer is not constant across occurrences, so neither branch may be quoted as a property of the defect.

🚨 ALC churn is refuted a third time, and this time by a whole-process constant rather than a before/after comparison. All 51 INIT_MEM/DISPOSE_MEM records of pid 3233 — from 19:25:08.743 to the onset at 19:26:44.700 — read alc=1. Not "one at onset": one throughout. asm moved 110 → 121, gc2 2 → 12, RSS 149 → 526 MiB. Onset landed 462 ms into CompileSingleDriverConsistencyTest.ConcurrentRequests_OnFreshNodeType_BothSucceedConsistently, the first test of its class, ~96 s into the host — and per the rule above that says nothing: onset is the first emit attempted after the fault, and the test that owns it is whichever one was running. A concurrency test being first is not evidence that concurrency is the trigger.

🚨 One caveat that applies to every occurrence report on this thread: a compile that returns real diagnostics does not prove the metadata writer was healthy. Emit returns before the writer runs when the compilation has errors, so the deliberately-broken NodeTypes (here PreWarmTestBatchDown/Broken at 19:25:24, a correct CompilationException) establish that parse and bind work — never that the writer did.

exit 124 is arithmetic, not a second fault

Measured end to end on this job. Onset 19:26:44.700; last PROCESS CANNOT EMIT 19:38:38.462; the final test — CellSurfaceScriptingSeamTest.CellSurfacePackType_IsCallableByBareName_FromAKernelCell — failed at 19:39:37.597 with "Expected the observable to emit a value matching the predicate within 60s" and its own [WATCHDOG-SOFT] … ran 60.0s; the mesh then disposed clean in 14 ms ("all pooled I/O joined, async dispose queue drained"); the harness killed the host at 19:40:03.

So after the fault every compile-dependent test burns its full reactive budget instead of finishing in ~1 s, and thirteen minutes of that exhausts the wall-clock cap. The 124 is the accumulated cost of the poisoning, not an independent hang — which is exactly why raising the cap is refused: it would buy more 60-second waits and end the same way, one verdict later.

A local negative worth having: the emit loop alone does not reach the state

Run in Docker on the CI runtime — mcr.microsoft.com/dotnet/sdk:10.0 resolving SDK 10.0.401 / runtime 10.0.12, the same pair CI installs — driving EmitPipeline's own shapes: the canonical parse options (DocumentationMode.Diagnose), the canonical compilation options (DynamicallyLinkedLibrary + OptimizationLevel.Debug + Platform.AnyCpu), Emit to memory with a portable PDB and an XML doc stream, 175 CreateFromFile references off TRUSTED_PLATFORM_ASSEMBLIES, six rotating nested-generic source shapes, 4 threads, and a collectible AssemblyLoadContext loaded from the emitted bytes and unloaded every 7th emit.

Run on both architectures, because a JIT-codegen hypothesis is architecture-specific and an arm64-only null would say nothing about a linux/x64 runner:

800,000 emits, zero events, against ~20–40 expected at the measured CI rate (roughly one occurrence per 20,000–40,000 compiles). So the null is worth stating: whatever precedes the fault is not in the emit path. Roslyn, the reference set, collectible-ALC load/unload, GC pressure and concurrent emits — on the exact runtime, on CI's architecture — do not reach it.

🚨 Two traps for whoever repeats this. First, --platform linux/amd64 under colima's default qemu-x86_64 binfmt is not a usable arm: dotnet restore dies with Segmentation fault (core dumped) and a 4-thread emit loop never reaches its first progress checkpoint. Rosetta (a separate colima start <profile> --vm-type vz --vz-rosetta profile, so the existing VM is untouched) runs the same image at ~1,450 emits/s and emulates x86-64 TSO, so the JIT emits and executes the same x64 code CI does. Second, build the project on the native arch and run the published IL under the emulated one (FROM --platform=$BUILDPLATFORM … AS build): MSBuild does not survive the emulation, the output is portable, and nothing about the build platform reaches what is executed.

The residual is now the WORKLOAD, not the architecture: the harness is an emit loop, and the CI host is a whole mesh test suite. Note that the CI host runs Workstation GC — #1605's crash frame is WKS::gc_heap::find_first_object — which is what the harness ran too, so that is not the difference either.

🚨 The control no leg has ever been: the COMPILER itself

Leg 1 varies the MetadataReference instances. Leg 2 varies the instances and their file mappings. Leg 4 varies the source shape. Leg 3 leaves the emit altogether. All four execute the same Microsoft.CodeAnalysis.dll and Microsoft.CodeAnalysis.CSharp.dll — the same loaded Assembly objects, the same mapped image, the same JIT-compiled native code, the same statics. Those two files contain every symbol the verdict names — and the fault spans both of them, measured by scanning the 5.9.0 assemblies rather than assumed:

So BELOW-ROSLYN's closing sentence — "the broken state is below Roslyn (CLR heap / JIT / GC), so no reference-set change can fix it" — reads "not the references" as "not Roslyn", and there is a third possibility sitting between them that no leg has ever varied: Roslyn's own image, mapping or native code, in this process. That would produce every observation on this thread — total, permanent, process-scoped, on freshly parsed source, with parse and bind healthy and direct symbol reads correct (reflection dispatches to whatever native code the method currently has, which need not be the copy the writer's call site reaches).

This is the third time the same reasoning error has been found on this issue, and the pattern is now explicit enough to name: a control is only a control for what it does not share. Leg 2 found it in the CoreLib file mapping; leg 3's polarity note found it in the reads the dissection made; this finds it in the compiler binary that every leg shares by construction.

Leg 5 — a pristine COMPILER control — is the experiment that settles it, and it is the exact analogue of what leg 2 did for CoreLib:

1. Create a collectible AssemblyLoadContext and load both Roslyn assemblies into it from freshly read bytes (LoadFromStream(new MemoryStream(File.ReadAllBytes(location)))) — fresh managed bytes, no shared mmap, no shared page-cache pages, and a fresh JIT from IL. Per the table above this is not a detail: the caller and the guard live in different assemblies, so loading either one alone is a dead probe that leaves half the fault on the shared copy.
2. Drive the same EmitCanarySource through the private copy by reflection and record the outcome.

Two residuals, stated because the leg is only a control for what it does not share: the BCL, System.Collections.Immutable and System.Reflection.Metadata still resolve to the Default context, and the private copy starts cold at tier 0 with no profile — so a PRIVATE-COPY-EMITS on a single emit does not by itself separate "fresh mapping" from "fresh native code". The follow-up is one more step, not another design: repeat the private emit until it tiers up. If it then fails, the fault is in what tiering produces; if it never does, it is in the shared image or its mapping.

🚨 This also supersedes the split-arm DOTNET_TieredPGO=0 run as the cheapest next measurement. That experiment needs ~5 events in the control arm for a one-sided Fisher p ≈ 2⁻ᵃ to mean anything — ~500 control runs at ~1 %/run, ~1000 runs total. Leg 5 asks the same question of the next single occurrence, for one ~10 MB read on a path that is already dead.

Is this the same root cause as the #613/#1605 SIGSEGV? Consistent — not established

Both are on runtime 10.0.12, linux-x64, and both are "one value is wrong while everything around it reads correct". But #1605 SIGNALS (a SIGSEGV, a createdump core, a zeroed MethodTable word on a live object in WKS::gc_heap::find_first_object — see Debugging Native Crashes) and #890 never signals; and #890's READS-HEALTHY + flat=EMITS say the object graph the writer is walking is intact, which is the opposite of heap corruption. Leg 5 is also the discriminator here: PRIVATE-COPY-EMITS puts #890 in the compiler's own image and separates the two; PRIVATE-COPY-THREW leaves them joinable.

Framework-version freezing

A compiled NodeType DLL references the MeshWeaver framework assemblies present at compile time. When MeshWeaver is redeployed at a new version, those assemblies change and the cached DLL may be ABI-incompatible — so a release is only usable while the framework version matches.

RunCompile stamps NodeTypeDefinition.CompiledFrameworkVersion with NodeTypeCompilationHelpers.FrameworkVersion on every success. That value is resolved once per process (FrameworkBuildIdentity, #1660 WS3):

• Hosts with a surface manifest (the portals and the CI bake host, which build with MeshWeaverSurfaceManifest=true) — the API-surface identity s<hash>: per compile reference, the SHA-256 of its reference assembly (the compiler's own definition of the API surface — byte-stable under body-only and private-member edits, changed by any surface change), hashed over the canonical content-surface set, with the generated-input-shaping exceptions contributing their full implementation MVID: the toolchain roots MeshWeaver.Compiler (THE compile toolchain since #1707 — skeleton generation, source-query resolution, include shaping, aggregation, options, generator execution, emit) and MeshWeaver.NuGet (the #r "nuget:" parser/resolver), plus their computed MeshWeaver dependency closure — the toolchain calls into what it links, so a body-only change in a closure member (Mesh.Contract, ContentCollections, …) must recompile content too. "Rebuild only when we need to": an internal-only framework release keeps the identity, so every cached and CI-published build stays valid; a breaking surface change mints a new one, and the CI bake for the new surface seeds at boot instead of recompiling per pod.
• Manifest-less CI processes (test hosts) — the commit identity g<sha> stamped as AssemblyMetadata("MeshWeaverFrameworkIdentity") by Directory.Build.props (CI compile inputs are commit-deterministic — no run number or timestamp reaches any compiled attribute). Kept everywhere as logged PROVENANCE.
• Manifest-less local builds — the identity anchor's MVID (MeshWeaver.Compiler.dll; a content hash of the compiled module, no stamp present). Content-exact for a dirty working tree — stable across rebuilds that don't change the toolchain's bytes, changed whenever they do. Note that every host compiling against a persistent assembly store (the portals, the CI bake host, mw-plugin-test) ships a surface manifest and resolves the surface identity even locally — the MVID fallback governs test hosts and ad-hoc tools.

On a framework-version mismatch the NodeType recompiles and mints a new release for the new framework. The old release is left intact as history so instances still loaded on it keep running until they cycle.

🚨 ONE POD BAKES — and cluster membership, not a clock, decides when another may take over

Rule 3 makes every pod on a new image discover the same framework-stale cache at once. The cache is shared but the decision to rebuild is per-process, so with maxSurge during a rollout — or any replicas > 1 — every replica independently starts the same sweep over the same NodeTypes into the same volume. That is not merely duplicated work: it is concurrent cold Roslyn compiles of the SAME type, which is precisely the storm the sequential, dependency-ordered sweep exists to prevent (four of them on memex, 2026-07-28 04:05, dropped six plugin roots to the "did not settle" overlay and needed a scale-to-zero).

Coordination is the build protocol: candidates register a claim on the Admin/Build node and its own hub grants exactly one, while everyone else waits on the per-fingerprint GO. There is no lease file any more — BuildCoordination is the mechanism, and its "Who becomes the build master" section carries the takeover rule: cluster membership decides, not a clock — gone → take over immediately, alive → never take over however old the heartbeat looks, unknown → the ClaimStaleAfter fallback for hosts with no cluster.

What the claim does NOT fix. NodeTypeBatchBake.WriteStamp is a read-modify-write at NextVersion, and the monotonic write guard bounces the loser (compile-state stamp … was REFUSED), so the bytes land on the share while the record does not name them. The claim removes one of the two writers that can race there — a second baker — but the other is the type's OWN per-node hub, which stamps compile state from the activation path, the sources watcher and the release watcher, and is legitimately live while a batch bake runs. The loser's outcome is already correct: the bytes are durable and content-addressed, the record stays pending, and the level-triggered probe (which asks the STORE, not the record) re-bakes and re-stamps on the next pass.

🚨 That means one whole GENERATION of assemblies per deploy — and the store never removed one

FileSystemAssemblyStore keys every file v{version}-{frameworkTag}-{contentHash}.dll, where the tag is the first 8 chars of the framework version. That tag is what stops a new image loading the previous image's bytes (BadImageFormatException → failed grain activations → a portal-wide wedge, prod 2026-06-20) — so a fresh set of files for the whole fleet on every published build is correct, and making the tag stable is not the fix.

What was missing is anything that ever removes an old set. Measured on memex 2026-08-12:

The share is not private to the compiler. /data also holds the DataProtection key ring (/data/dataprotection-keys), the NuGet package cache and the Graph storage base path, so filling it is not a contained failure: key persistence and the compile cache fail together, and a full SMB share reports write failures far from their cause.

What proves a generation is unreferenced

Not its age, and not a count. A pod's generation is fixed for its whole life (it is its image's MVID), and a fully warm pod can go days without touching the share — so file age says nothing at all about whether a generation is in use, and a count is blind to a pod that has not rolled inside the window.

The proof is a live claim. Every process that owns a filesystem assembly cache re-asserts {root}/.generations/{tag}/{holder} on an interval, and a generation any live claim names is never collected. The claim's instant is written into the file's content, not its last-write metadata: SMB metadata caching can report a timestamp that is stale while the holder is alive, and a falsely-stale claim is the one misreading that deletes a live generation.

Every rule is a KEEP rule and they are ORed — a generation survives if any holds:

Anything the sweep cannot attribute to a generation — the claim files, an untagged pre-2026-06 DLL, any foreign file — is counted and never deleted, and any error reading the tree or the claims aborts the sweep with nothing collected. Note the polarity: the coordination path around the bake (the build-protocol claim) fails open, because being wrong there costs duplicated work; this one fails closed, because being wrong here deletes an assembly a live pod is about to load.

Deletion is off by default (AssemblyCache:Retention:Delete). Until it is armed the sweep measures the cache and logs exactly what it would remove — which is the evidence a deployment should arm it against.

Code: AssemblyCacheGenerations (src/MeshWeaver.Graph/Configuration/AssemblyCacheRetention.cs), wired by AddAssemblyCacheRetention (src/MeshWeaver.Hosting/AssemblyCacheRetentionHostedService.cs). BlobAssemblyStore is unaffected: it keys v{version} with no framework tag, so a new image overwrites rather than accrues.

🚨 The cache grows on a SECOND axis, and generation retention is blind to it

Generations are one axis. The other is per-type version accumulation inside one generation — one dll/pdb pair per recompile, forever, every file carrying the same tag. Measured on memex-cloud 2026-08-22, when the 16 GiB /data PVC hit 100% and every NodeType recompile failed with No space left on device (surfacing four steps away as compilationStatus: Error, while the migration pod crash-looped 66 times):

Keeping three generations of that shape still keeps ~12.5k files, so no setting of KeepGenerations could ever have been the answer. The collector for this axis is therefore in the writer: after FileSystemAssemblyStore.PutWithLocation publishes a new version it prunes that type's directory to the newest KeepVersionsPerType versions (default 3, override AssemblyCache:Retention:KeepVersionsPerType). The pass that made the directory grow is the one that trims it, so nothing has to walk the tree to discover the growth.

Why this one may delete on defaults while the generation sweep may not. Different worst cases:

Eviction never crosses the tag boundary, only removes names AssemblyCacheFileName.Parse attributes to this store (so .tmp-* leftovers, bake leases, claim files and pre-tag legacy DLLs are untouchable), never removes the file it just wrote, and treats a delete that throws as "leave it alone" — a file that will not unlink is one something is holding. Both collectors share that one parser deliberately: two collectors disagreeing about which names this store wrote is how one of them would remove a file the other treats as foreign.

🚨 In-MEMORY generations accumulate the same way — a superseded build stays ROOTED while any instance serves it

The store section above is about disk; the same generation arithmetic plays out inside every process that hosts instance hubs, and there the unit is an AssemblyLoadContext. Every publish of a usable build mints a new collectible ALC; collectible only means "may unload once nothing roots it" — and a serving instance hub roots the build it bound. The stale-build banner ("a newer build of this type is available — Recycle") deliberately leaves every instance on its old build until a human clicks, so in a sync-heavy window each publication wave stacks one more full assembly generation — types × Roslyn artifacts — onto the silo hosting the type hubs. The eviction fix on the compile path (#605) cannot free any of it: eviction drops the store's reference, not the instance's.

Measured, 2026-08-25 (issue #2194): two memex-cloud silos flat at 2.7–4.5 GB for five hours, then a hard inflection the moment the first scheduled bake tick after a framework-pin bump published a full-catalog rebake, followed by five content merges — six publication waves in four hours. The two type-hosting silos climbed ~2.5 GB/h to 17–20 GB and four cores of GC; every other replica stayed ≤3.6 GB. Nothing intervened: no OOMKill (the pod limit was never reached), no probe failure (after startup, readiness and liveness both watch /alive, a process-up check a thrashing pod still answers), no alert — the pods served hung requests as Ready for 3½ hours until a human read kubectl top. The same stranded instances also produced the visible half: old and new assemblies serving side by side ($type registration mismatches), pages wedged until the type and instance hubs were recycled by hand.

The missing piece is convergence, and it is policy, not plumbing: Modules:AutoRecycleOnStaleBuild (#2192, default off) turns the banner's offer into an automatic self-recycle — when a NodeType publishes a usable build whose assembly differs from the one an instance bound, the instance posts its own DisposeRequest, re-activates on the new build, and the superseded ALC unroots and collects. Anywhere the catalog updates itself — every self-updating portal — leaving the key off means choosing the accumulation above; #2194 tracks the prod enablement. The structural end state is stronger still: DLL-only module adoption ("never ship uncompiled state") removes in-portal recompilation altogether, so a publication wave costs an assembly load instead of a Roslyn generation.

🚨 Convergence is the missing piece for the mechanism above, and it is not the whole story — measured 2026-09-04. The spiral recurred on 09-03 with Modules:AutoRecycleOnStaleBuild live on both production namespaces. Counted over one replica's entire 28-hour life (186 137 lines, with info: confirmed shipping): zero Stale-build offer:, zero Stale-build convergence:, zero self-heal recompile, zero BatchBake: — no build was ever superseded, so convergence had nothing to converge, while the managed heap still grew from 6 856 to 10 065 MB across 267 gen2 collections. Two things the paragraph above does not cover were doing it instead: no hub or grain deactivates at all (deactivating: reason= is 0 across five replicas and 514 687 lines), so the lifetime ALC lease an instance hub takes is never released; and replicas boot having adopted as few as 8 of 377 dynamic NodeTypes, so the other 369 each pay a Roslyn compile on first access and pin it — the same accumulation as a publication wave, arriving as a drip and driven by bundle coverage rather than supersession. Note also that "no OOMKill (the pod limit was never reached)" above understates why: the GC's own hard limit is 75 % of the container limit, so the runtime defends memory with a sustained 0.66 GC pause share rather than letting the kubelet restart the pod. Why a GC-bound pod stays in rotation has the full derivation.

Triage fingerprint — intermittent hangs while most requests succeed, on a portal that recently synced or baked: suspect a degraded-but-Ready replica, not a global wedge. One or two pods far above their siblings in BOTH memory and CPU in kubectl top pods is this incident (a break-glass read — the per-replica sample is not on the Hosting API yet, OperatingFromThePortal); replace them with a Restart Hosting/InstanceAction (the rolling restart grace-drains each pod and the Deployment replaces it), and read #2194.

🚨 A THIRD root holds a generation, and neither eviction reaches it — the kernel's reference memo

CompilationCacheService evicts superseded NodeAssemblyLoadContexts, and Modules:AutoRecycleOnStaleBuild converges the instances that root them. Both act on the ALC. Neither touches the native metadata mapping a Roslyn PortableExecutableReference owns — and KernelScriptReferences.Materialized, the process-global memo that makes ~350 script references cost one materialization instead of one per kernel session (#2480 / #2578), was keyed by absolute path with no eviction and no bound.

For framework assemblies that is exactly right: their paths are stable, so the key space is the assembly set. For NodeType assemblies it is not a bound at all. EmitPipeline.EmitToDiskWithRetry publishes every recompile into a brand-new {nodeName}_{ticks}_{guid}/ directory that is never reused, and the cell-surface seam (KernelExecutor.EnsureCellSurfaceReferences, #1649) feeds exactly those paths in through GetOrCreateFromFile. MetadataReference.CreateFromFile memory-maps the PE, and the entry survives both the ALC's Unload() and the file's deletion — so one recompile of a cellSurface: true NodeType that any session had referenced left one more mapping alive for the life of the process, in the same class of memory the eviction above exists to reclaim.

The class doc's own NoStaticState compliance claim was half right and half wrong, and the half that was wrong is the one the allowlist rested on. "Holds no Types and no AssemblyLoadContexts — it can pin neither meshes nor collectible NodeType contexts" is true of the object graph: no managed reference into the collectible context is retained, which is why nothing here ever showed up as a pinned ALC. "Bounded by the set of assemblies on disk" is true only if that set is bounded, and for per-recompile release directories it is not. The fix makes the claim true rather than restating it: a file that belongs to a COLLECTIBLE load context is materialized as an ordinary, UNMEMOIZED reference whose lifetime is the caller's — for the cell-surface seam that is the kernel SESSION, which already holds the generation's CellSurfaceAssembly.Lease and drops both when the session dies. The memo keeps exactly the sharing it was written for, over stable paths only, and KernelScriptReferences.IsMemoized(path) makes the bound assertable from a test — per PATH rather than as an entry count, because the count moves with whatever else the shard loaded.

The same file carried a second generation-ambiguity: TryResolveByIdentity matched the live AppDomain on simple name only, without the IsCollectible filter its sibling at MaterializeCurrentAssemblies carries for a documented reason. Superseded generations linger in GetAssemblies() until they are collected, so which one a script compilation got was whichever the enumeration reached first — arbitrary, possibly stale, and indistinguishable from the current one at the call site (the CS0433-between-two-generations shape). It now declines collectible assemblies and answers null, which sends the caller to the sibling probe and then to Roslyn's own resolver. A cell-surface set a session legitimately sees is declared per session and added to its ScriptOptions explicitly, so it never needed this path.

🚨 A READ must never re-order generations — only a PUBLISH supersedes

EvictSupersededContexts is what bounds _loadContexts to the current generation per NodeType, and its trigger used to be "a path-keyed context was newly created". That is a proxy for "a recompile published", and it is wrong for every reader: an assembly-hydration scan of a package's SHIPPED build (GetConfigurationsFromExistingAssembly) and a kernel session resolving a cell-surface pack (CellSurfaceAssemblyProvider) both create a path-keyed context, and neither is evidence about which build is current. Under the old trigger either would supersede the generation a concurrent rebuild had just published.

That alone would be a correctness wart. What made it an outage-shaped defect is that PinForScan re-resolves when a pin is refused — the recovery loop that exists so a transient supersession cannot park a NodeType (#1151). The re-resolve created a context, and creating a context evicted the peers. So two scans of two generations of ONE NodeType each destroyed the context the other had just created:

CompileResultFromAssembly catches that throw, records CompilationStatus.Error, and the watcher PARKS the NodeType — a millisecond-wide race turned into compile:Failed for a package whose source is fine. Measured as #4013: PluginGateRunnerTest.SelfTypedRootWithStaleCompileStamp (the rebuild publishes while the shipped stale stamp is read — exactly two generations, exactly two scans) failed 1 of 362 in a merge-queue group build whose PR run was 7 049 tests / 0 failed on the same commit, dequeuing a documentation-only PR.

The fix names the trigger honestly. ICompilationCacheService now has two doors: GetOrCreateLoadContextForPath is a READ and supersedes nothing, and PublishLoadContextForPath supersedes — reached only through PinForScan(..., publishesTheBuild: true) from the post-emit scan, the one moment a new generation actually exists. A retry never supersedes, whoever asked: a recovery that destroys its peers is what turned one supersession into a ping-pong. The eviction now fires where a generation is born rather than wherever somebody happened to ask for an unfamiliar path.

🚨 What the narrowed trigger costs — stated, not claimed away. The key space that grows without bound is the per-emit {nodeName}_{ticks}_{guid}/ directory set, and every emit publishes, so that half is evicted exactly as before. What is no longer evicted the moment it appears is a generation a process only hydrated: an IAssemblyStore path is keyed v{version}-{frameworkTag}-{hash}.dll and is first-write-wins per version, so a silo reading a version another silo compiled keeps the previous version's context until the NodeType hub disposes (UnloadNodeContexts, which Modules:AutoRecycleOnStaleBuild drives on a stale build). That residue is bounded by the number of VERSIONS one hub outlives, not by recompiles — and it is the deliberate price of the correctness clause. A read that superseded would evict the CURRENT generation's context and then re-create one under the same path, putting two live ALCs behind one file: the two-generations split the section below is about, manufactured by the reclaim itself.

🚨 What was NOT changed, deliberately. The issue's suggested shape was "treat an unloading context as a cache miss in Pin()". PinForScan already does exactly that — twice — and the residue was the exhaustion, not the refusal. Making Pin() itself succeed on an unloading context would be the wrong repair in the other direction: Pin()'s two-state test and IsClosedToLoads()'s three-state test answer different questions (may I START a scan? vs may I LOAD inside a scan I already hold?), and both are right. With a pin outstanding, Dispose is provably still waiting, which is why a LOAD is safe; with none, the drain may already be over, so a new scan would be the TypeLoadException '…format is invalid' tear the pin exists to prevent. The refusal is the contract; the recovery was the defect.

🚨 One build at two paths is ONE generation — a read reuses, it never duplicates

The "narrowed trigger" section above priced its residue at generations a process only hydrated, and pictured a foreign silo's build. It missed the commonest hydration of all: the process's own publish. The post-emit scan publishes from the compile's {nodeName}_{ticks}_{guid}/ directory; UploadToStoreIfNeeded then copies those bytes into the assembly store as v{version}-{frameworkTag}-{hash}.dll; and every instance activation resolves the store path (MeshDataSource / NodeTypeEnrichmentHelpers → IAssemblyStore.TryGetAssemblyPath → GetConfigurationsFromExistingAssembly). Two paths, one generation. While reads superseded, that read quietly evicted the publish's context. Once they stopped, every locally compiled generation got a second collectible context over identical bytes — and the instance hub's lifetime lease (LeaseNodeContexts, which leases every context of the NodeType) pinned both for the hub's whole life.

Measured by NodeTypeRecompileAlcLeakTest.RecompilingANodeType_WithALiveInstance_StillReleasesSupersededContexts (MeshWeaver.Plugins, Portal hosts (shard 3)), which prints the file each live context was loaded from:

Flat, not per-recompile — so not the unbounded curve the assertion's message describes — but a doubled pin on every generation a live instance runs, in every monolith and on every replica that compiles. The assertion (≤ 2: the current build plus the one generation the instance runs) was right; the core change was wrong. Across the 42 shard-3 runs from 05:10Z to 11:32Z on 2026-09-11 it failed 6 of 6 on core sets 8345/8350/8352 (all carrying #4017, b128b804d) and 0 of 36 on sets 8323–8340; the same Plugins commit 401fcadb passed at 09:56Z on 8340 and failed at 10:09Z on 8345.

The repair keeps #4013's rule — a read never supersedes — and closes the duplicate at its source. ResolveLoadContextForPath, on a READ of a path it has not seen, asks whether a live (not retired) context of the NodeType already serves the same build — same MVID, read from the loaded assembly or, failing that, from the file's metadata via ServedBuildIdentity.OfFile (a header read, nothing loaded) — and if so answers that context, aliased under the new path. It is a REUSE: nothing is evicted, so the ping-pong above cannot come back through it. Three details make the alias safe:

• Eviction is by CONTEXT, not by key. A read may alias the store path to the very context a publish is keeping; a key-based evictor would dispose the build it had just published.
• A retired context is never aliased. A publish can retire the context between the search and the alias (its evictor enumerated before the alias key existed); the resolver then drops its own alias and creates a fresh context, exactly as a first read of that path always did.
• One lease per context. LeaseNodeContexts leases each context once however many keys name it, so releasing the hub's one lease performs the deferred unload.

No identity ⇒ no alias: an unreadable file, or an in-memory context, resolves a fresh context as before. Pinned in core by ScanPinSupersessionTest.AReadOfTheStoreCopyOfThePublishedBuildIsThatBuildsContext, …AReadOfASupersededBuildsStoreCopyGetsItsOwnContextAndDoomsNothing and …AnAliasedGenerationIsLeasedOnceAndReclaimedByTheNextPublish.

🚨 What this does NOT close. A replica that never compiles a type still reads each new VERSION another replica published (a different build, a different MVID), and those reader-created contexts are still reclaimed only when the NodeType hub disposes — the residue the section above states. The alias removes the same-build duplicate; it does not decide which of two different builds is current, and must not (#4013).

🚨 Reading a node's content ACROSS two generations already works — #3911's headline, re-measured

The 2026-09-10 control-instance incident (#3911) reported two collectible builds of Hosting/InstanceAction in one pod and attributed the dead control plane to node.ContentAs<InstanceActionContent>(…) returning null across them — "null → Observable.Empty → silence". That link does not survive measurement. ObjectAsExtensions.As<T> recovers a same-SHORT-NAMED foreign type by a JSON round-trip, and it does so on the collectible path too, in both directions — asserted against two genuinely collectible generations in TwoCollectibleGenerationsContentReadTest, which is the case ContentAsForeignAssemblyContentTest (two static types in one non-collectible assembly) does not reach. The collectible branch is the one worth checking, because PolymorphicTypeInfoResolver deliberately refuses to auto-register a collectible type and formats the discriminator without registering it; the round-trip survives that.

So the cross-generation READ was never the silence, and a change aimed at ContentAs would have moved nothing. The part of #3911 that HAS been answered is the recompile LOOP that produced two generations in the first place: a process running a module generation other than the one its activation record names (Module Generation Substitution — Assembly.LoadFrom silently returns an already-loaded byte-identical copy from another path, which is why compiledDependencies disagreed with the runtime and the stale-build kick fired after every activation restart).

#3911 is therefore not closed by either page. Its third ask — the NodeType compiler must reference the LOADED module generation, not the image's /app copy — is still open, and that page is explicit that it does not assert the MVID/generation pairing #3911 reported. What the two pages together do settle is which hypotheses are dead: not the read, and not ContentAs.

🚨 A LEAVING pod never touches shared NodeType state — the adoption sweep observes host shutdown

The NodeType node is one record for the whole deployment. Every generation of pods reads its assembly coordinates, and every generation's adoption sweep writes them — which is fine while every writer is a pod that will go on serving the type, and a clobber the moment one of them is not.

What was measured (issue #3129, memex.systemorph.com, 2026-09-02). A rollout left the old pod terminating for 27 minutes (terminationGracePeriodSeconds=1800, circuits still held, 11.8 GB, 3–5 s GC stalls every ~4 s). While draining it kept running the prebuilt-bundle adoption sweep against the shared NodeType nodes, and Loki shows the loop verbatim:

ADOPTION REFUSED for Underwriting/Workbench (#2813): bundle fingerprint aa0e63c8… vs live 572495ee…
→ coordinates cleared, Roslyn compile dispatched
→ LATE_NACK_TERMINAL code=Unknown TimeoutException (+10 s)   [UpdateQueue] FAILED path=Underwriting/Workbench elapsedMs=12344
→ ShippedPrebuiltBundles: seeding … did not complete — the sweep compiles it instead
→ next access repeats (Workbench/Guideline every 30–90 s; Crm/Client 388× in 30 min)

All 1424 ADOPTION REFUSED and 350 [UpdateQueue] FAILED lines of the window were on the dying pod; the two new pods logged 0 of either. The cross-generation effect IS the reported navigation hang: because the old pod kept clearing Underwriting/Workbench's coordinates, the NEW pods logged Overlay self-heal: instance 'Underwriting/Desk' still stuck on NodeType 'Underwriting/Workbench' after 120s → self-heal compile → Release, ~2 minutes later. A ≥120 s hang per Underwriting type, per roll — three rolls that day.

The clobber, step by step. PrebuiltAssemblySeeder.Seed stamps the bundle's assembly coordinates onto the node first and asks the owner to judge the adoption second (RequestedSourceStampAt, #1834 — the seeder's snapshot may predate the owner's source publication, so only the owner can compare fingerprints). When the bundle is stale the owner refuses (#2813) and, because the coordinates on the node at that moment are the rejected bundle's, it clears them so proven-stale bytes stop executing — that clear is required by the #2813 design and stays. The build that was actually serving on the healthy pods is already gone at that point: the stamp replaced it one write earlier. Each access on the draining pod re-ran the sequence (the Pending flip's on-demand adoption), so the shared record was wiped every 30–90 s for as long as the pod lived.

Why the grace period is the amplifier, not the cause. Thirty minutes is how long the wrong writer stayed alive; the defect is that it wrote at all. Cutting the grace period would shorten the window and leave the same clobber in it — a band-aid.

The rule. A hub that is leaving does not START a sweep pass, does not STAMP, does not CLEAR coordinates, and does not DISPATCH a compile through the refusal path. The predicate is hub.IsLeaving() (HubLeavingExtensions, MeshWeaver.Mesh.Contract), and it reads two signals:

• IMessageHub.IsShuttingDown — this hub's own teardown or an ancestor's cascade; the shape #3109 gave BuildupActions and SubscribeHubWatcher gives emissions.
• IHostApplicationLifetime.ApplicationStopping — the process is leaving. This is the signal that was live for the whole window above and the reason the hub signal alone would have changed nothing: the mesh is disposed by MeshTeardownHostedService.StoppedAsync, i.e. at the very END of host shutdown, after Kestrel has finished with the circuits the pod still holds. Every one of the 1424 refusals was issued by a watcher SubscribeHubWatcher would have dropped had IsShuttingDown been true — so on a draining pod it was false the entire time. It is the same token OrleansRoutingService already consults for its shutdown routing decisions.

It is asked at four places, and every route into adoption converges on them: the start of a ShippedPrebuiltBundles pass (no pass starts — boot, on-demand from the compile watcher, a push's recompile, an install); PrebuiltAssemblySeeder.Seed at subscribe time (the seeder runs one Seed per assembly under Concat, so a pass in flight stops at its next node boundary) and again inside the write lambda (an unchanged node is a no-op Update); and ApplyAdoptedSourceStampAndReport on the owner, reached from the stamp watcher, the sources watcher's fold and the release watcher's fold — a leaving owner judges nothing and leaves the request standing for the owner activation on a pod that stays. The answer everywhere is the caller's ordinary "not adopted / nothing to do" signal, so nothing is ever parked on it.

Decline before you write. The seeder also now refuses to start the stamp-then-refuse sequence when the refusal is already decidable from the owner's snapshot in hand: a bundle whose SourceFingerprint disagrees with a published live CurrentSourceFingerprint is declined before any write, exactly like a framework or dependency mismatch, and the live build's coordinates are never replaced. The owner's three-way check stays for the pre-publication window the snapshot cannot decide (no live fingerprint yet) and as the last line of defence. A decline is always safe — a compile follows; a write a refusal must undo is not.

What a leaving pod still does, deliberately. A user still held on the draining pod who opens a type with no usable build gets a compile dispatched by the ordinary first-access kickoff — that result is a new version, clobbers nothing, and #3115 made its result write land under the pod's own flush bound. Only the refusal-driven clear-and-compile, whose sole effect on a shared record is destructive, is withheld.

Triage fingerprint — Overlay self-heal: … still stuck on NodeType … after 120s on the new pods right after a roll, while a pod of the previous ReplicaSet is still Terminating: read that pod's log for ADOPTION REFUSED. Pinned by LeavingHubAdoptionSweepTest (MeshWeaver.Compiler.Pipeline.Test): the same seed on a shutting-down hub leaves the shared record byte-for-byte untouched, and on a live hub adopts.

🚨 The batch bake compiles the definition the mesh holds WHEN IT COMPILES (2026-09-11)

A pod's bake sweep enumerates every NodeType definition once, at its start (DynamicTypePreWarmer.WarmDynamicTypes → DynamicTypesOf), the batched discovery pass (NodeTypeBatchBake.ResolveSources) resolves source sets from those enumerated definitions later, and the sequential compile reaches each type minutes after that. BakeOne used to hand the compiler the enumerated node together with the later source set. A module update landing inside that window moves a definition and its files together, and the pair Roslyn received was neither the old content nor the new one.

Measured on memex.systemorph.com, new pod on 3.0.0-ci.8372 started 18:50Z, while the Hosting module moved 1.15 → 1.16 (Plugins cff9fb34cb, which added FleetWatch and its shared=@Hosting/InstanceAction/Source/ObservationQueries entry together):

Nothing read anything partially and no resolver dropped the shared= entry: v458 genuinely declared no sources, and the pod's prebuilt DECLINE of the 1.16 bundle (d251868c… against the live ca01bf82…) was correct when it was taken. A fresh pod on the same image baked the type cleanly. It is not a regression of the core range the image moved across either: the repro below fails identically on 45306a33e and on 74d4c8527.

The rule now: NodeTypeBatchBake.CurrentCompileInput re-reads the type's row immediately before the compile, through the same IStorageAdapter.Read the compile stamp already uses (never a routed point read, which on a type a sync has just pruned opens the storm-breaker), and compiles THAT row. The batch's pre-resolved set is reused only when both witnesses say it is still the set: the row declares the very source queries the batch resolved it with, AND the row's own CurrentSourceVersions record is absent or names exactly the versions the batch holds. Otherwise the sources are resolved again from the current queries through the batch's own bounded, truncation-checked RunQuery (same DiscoveryUnestablished invariant), and the row is read once more afterwards; a definition that moved AGAIN meanwhile is not compiled. One deadline covers the re-read, any re-resolution, the compile and the stamp — the compile gets only the time that is left.

Every "I don't know" is a non-verdict (TimedOut, "not evaluated"), never a compile of a pair nobody can vouch for: a re-read that faults or runs out of budget, a row that no longer reads as a NodeTypeDefinition, a current source set that cannot be established, a definition that moved again. A row absent from storage is confirmed by a listing and reported Removed WITHOUT a compile or a stamp — the stamp's insert-if-absent would otherwise re-create the type its repository just pruned.

What it does NOT establish: an atomic snapshot. Neither a row read nor a query is a transaction, and the version record lags the files by the sources watcher's own latency, so a source edited in the last instant before the compile can still be compiled at its earlier version. That residue heals itself — CompiledSources records what was compiled, the watcher's newer record reads dirty, the type rebuilds — and a verdict it produces is the one #1214's recovery watch exists for (see the residue note below).

Test: ABakeCompilesTheDefinitionItResolvedTest (MeshWeaver.Hosting.Test) runs the sweep's own enumeration, discovery and compile against a real monolith mesh. Its repro moves a consumer's definition between enumeration and compile, adding a single-node shared= entry, and expects the verdict the CURRENT content earns — Compiled for sound content, a gating CompileError naming the genuine defect for broken content. Without the fix both cases fail with the production text (Executed source queries (2) … CS0103 The name 'LibAnswers' does not exist) on 45306a33e and on main; two controls — an unmoved definition compiles from the batch set, and a genuine compile error still gates — pass on all three. Three further cases pin the review's findings (#4051): a source edited under UNCHANGED queries is compiled as it now stands once its version record moved; a moved definition whose current source set cannot be established inside the per-type deadline is TimedOut, never a CompileError; and a type pruned during the sweep is Removed and is NOT re-created by a stamp.

Residue, named rather than closed. A mesh that is itself torn when the compile runs — the file has landed and its definition has not — still earns a CompileError, because that is what the content IS at that instant. Its cure remains #1214's WatchForRecovery, which retracts the regression on a fresh usable build on THIS image. Whether that watch can be starved when the type's own hub is hosted by a previous-generation pod (whose rebuild is stamped for the old framework) is consistent with this incident's ~30-minute hold but was not established here.

🚨 That recompile can FAIL — and nothing upstream can warn you

Rule 3 guarantees a framework upgrade recompiles every dynamic NodeType. That recompile builds source stored in the mesh against the new framework — and nothing in the build pipeline has ever type-checked it.

Where NodeType source actually lives

🚨 The configuration lambda is C# inside a JSON string field, so it is not reached by any .cs-shaped search or build. A grep --include='*.cs' for a deleted symbol comes back clean while three NodeTypes still call it from their JSON. Search the node JSON too.

The failure cascade

A failed type produces no assembly, so everything downstream of it is skipped rather than built (NodeTypeDependencyGraph.FirstBlockedBy), and DynamicTypePreWarmer refuses readiness for any type that regressed against its pre-bake baseline:

framework API deleted in src/  →  in-mesh caller no longer compiles
  → NodeType = CompileError
  → every dependent = UpstreamFailed          (transitive, in topological order)
  → DynamicTypePreWarmer: REFUSING READINESS
  → instance hubs never activate → each SubscribeRequest waits the full 60 s → faults
  → hung pages, failed liveness probes, dropped silo

The shape in the log is a CS1061 / CS0103 against framework surface:

CS1061: 'MessageHubConfiguration' does not contain a definition for 'AddTracking'

A timeout is not a verdict, and it does not cascade like one. The chain above starts with a CompileError — Roslyn's answer that the type is broken. When the sweep instead gets no answer (TimedOut: the per-type budget elapsed, typically a cross-silo SubscribeRequest during a roll), that is not evidence of breakage and must never stall a rollout. So the two cascade differently:

Both no-answer statuses are filed by NodeTypeBakeGateState under unevaluated — they can never set Regressed, but they are named in the /health payload, because non-blocking must not mean invisible. Only a failure on a type that was healthy before this image gates at all: a type already sitting at Error before the deploy is pre-existing damage, and gating on it would let one abandoned NodeType freeze every future rollout.

That distinction is what makes the readiness gate safe to arm. Counting timeouts as regressions stalled memex-cloud on 2026-08-02 with "7 NodeType(s) regressed" and not one CS#### diagnostic; counting only direct timeouts leniently still let one timed-out shared source gate through its dependents, which reproduced the same stall one hop downstream.

A sweep that ERRORED is not a sweep that passed

The leniency above is about individual types the sweep could not evaluate. It does not extend to the sweep itself failing. If the ENUMERATION errors or times out, the pod has verified nothing at all, and that is BakePhase.Faulted — refuses readiness, exactly like a regression:

The last two rows are the whole point, and they are easy to conflate. The health check's documented policy — "fail CLOSED on a regression, fail OPEN on 'not running'" — is scoped to NotStarted, i.e. the gate is disabled ("a configuration mistake must never black-hole a pod"). A sweep that errored was armed and measuring and simply failed to, which is a different thing; the same switch already reports Running — a pod that knows strictly more than a faulted one — as Unhealthy.

WarmDynamicTypes used to swallow the enumeration fault and return Observable.Empty, so both "found nothing" and "could not look" reached the subscriber as the same empty completion and the gate marked itself Complete → Healthy. The retired pre-run bake Job carried the counterpart guard from outside the portal — "FINDING NOTHING IS NOT PASSING … a gate that certifies 'I verified nothing' is worse than no gate", exit 3, with a Bake:AllowEmpty escape — and named that Catch as the reason it had to. Retiring the Job (#1357) removed the guard; the distinction now lives at the source, where it can be made honestly.

Cold start. On a roll this is free: maxSurge:1 / maxUnavailable:0 keeps the old pod serving while the new one refuses. On a first deployment there is no predecessor, so the pod fails its startupProbe and restarts — re-running the sweep each time, which is what clears a transient cause without anybody adding a retry. That is the same trade Regressed already makes, and the likeliest environmental cause of a blind enumeration (an unmigrated database) is refused earlier and more precisely by DbVersionGate. The escape hatch is PreWarm:AllowUnprovenBake — it relaxes the verdict only, never the record: the phase stays Faulted and the payload keeps saying the bake was never proven. It cannot waive a real regression.

🚨 The pre-prod sweep is ACCESS-FILTERED — it agrees with the gate only over what the sweeper can read

search 'nodeType:NodeType content.compilationStatus:Error' reads the same field ClassifyDetailed branches on, so its VERDICT per row matches the gate's. Its denominator does not. The sweep runs as the person typing it and the gate runs as the system, and those see different sets of rows:

IMeshQueryProvider.Query filters through ValidateRead; IMeshQueryCore.Query does not (StorageAdapterMeshQueryProvider, useSecurityFilter). So a NodeType parked at Error inside a partition the sweeper has no grant on is not counted — the sweep returns a smaller number, never an error, and the two instruments part company exactly where it matters.

🚨 "Rows the caller may read" is per NODE, not per partition — a grant can sit on a single node below a partition root, so a sweeper denied Helvetia may still be shown one NodeType inside it. Which is why the denominator has to be counted, never inferred from the list of partitions you can open. (autocomplete joined this filtered set in #3890 and is not a compilation instrument: it returns suggestion projections — path, name, node type, icon — and never a compilationStatus.)

🚨 And get/get_diagnostics answer Not found for a node they may not read. Denied and absent are the same string. That is not a hypothetical reading of the code:

#1391 recorded BinaryClickerV2/BinaryToggle at CompileError on every boot of the memex namespace, and noted in the same thread that "BinaryClickerV2 is not visible to an admin MCP read" — it is a private partition. It was then closed on get_diagnostics @BinaryClickerV2/BinaryToggle → {"status":"Unknown","message":"Not found: …"}, read as the type is gone. It was not gone. Four weeks later the same CS1929 on the same two source nodes re-surfaced, unchanged, as #3883 — while search 'nodeType:NodeType content.compilationStatus:Error' on that portal returned 0.

Positive control for "denied, not deleted".

🚨 This control used to have a second step that no longer exists, and its removal is the point. autocomplete '@/<Namespace>/' ran RunQueryNodes(…, useSecurityFilter: false) with the caller's identity dropped, so it enumerated names, paths and node types the caller could not get — which is what made it a witness here, and is also why it was a disclosure surface. On memex.systemorph.com, 2026-09-10, that drill-down named five Helvetia/* nodes, with their titles, to an identity whose get and search on the very same paths answered nothing. #3890 closed it: the drill-down now resolves the viewer and runs the same ValidateRead chain as get, so a suggestion and a point read agree by construction. There is no caller-run read that separates denied from deleted any more, and there should not be one — a control that works by publishing someone else's document titles is a disclosure wearing an instrument's colours.

🚨 And do not read a WRITE tool's permission refusal as an existence proof. compile answers "Compile requires Compile permission on the target NodeType — it schedules a Roslyn build and records an activity under the node. Ask someone with editor access to the node (or a platform admin) to do it.", which reads exactly like "the node is there, you may just not build it". It is not. Measured on memex.systemorph.com, 2026-09-11, three calls, one answer:

The authorization runs before the node is resolved, so the refusal is emitted for a path that could never exist. recycle is the same, measured the same way and the same day: "Recycle requires Update permission on the target node …" for @BinaryClickerV2/BinaryToggle and for @ThisPartitionDoesNotExist_zzz9999/NoSuchType. Run the invented-path control before quoting any refusal as evidence about a real node — the same discipline the Not found string already demands, on the tools that look like they answer differently.

What is left is one read you can run and one answer that comes from the system:

1. get @Admin/Partition/<Namespace> — the partition record survives its data. Active means the partition was never torn down. This narrows the question to the PARTITION; it never answers about the node. Ask a partition you CAN read the same way as the negative control, otherwise a broken instrument reads like a deleted node.

2. The pod's nodetype_bake payload — the only instrument with the full denominator. The boot sweep (DynamicTypePreWarmer) enumerates nodeType:NodeType mesh-wide under ImpersonateAsSystem, so it sees every NodeType in every partition on the replica, not the ones you may read. And its buckets already draw the distinction this section needs — but they are FOUR different statements, not one, and reading them as one is its own false negative:

   bucket	what it actually says
   Regressions	the type EXISTS and failed to build on this image — an image verdict, and the only gating one
   Unevaluated	NO verdict was reached (the warm timed out, or an upstream was itself unevaluated) — evidence of nothing, in either direction
   ContentBroken	a CONTENT verdict (NoSources / UpstreamContentBroken): the type node is still there, its source queries now match ZERO Code nodes — its SOURCES were deleted out from under it
   Retired / Removed	the repository withdrew the type, or a listing no longer names the node at all (PreWarmStatus.Removed, established by a LISTING that came back — never by a point read)

   So Regressions is "denied, not deleted" answered from the system side; Retired/Removed is "deleted" answered the same way; ContentBroken is the half-way case worth naming out loud (the type survived, its sources did not); and Unevaluated must never be read as either. 🚨 Read the arming caveat below before trusting its silence.

🚨 And re-read what the sweep is actually FOR. Its question is not "does this node exist" but "is anything broken", and step 2 answers that one over the whole mesh regardless of who may read what. Denied-versus-deleted was only ever load-bearing because the instrument you were allowed to run had a hole in its denominator; name the denominator and the distinction stops deciding anything. Where you genuinely must know about one node in a partition you cannot read, the answer is its OWNER's — ask them, or elevate (break-glass, their decision, never a sweep step). 🚨 Until you have one of those, a Not found from outside your readable denominator has told you nothing and cannot close an issue. That is precisely how #1391 was closed on a false negative and re-filed unchanged, four weeks later, as #3883.

So a zero from this sweep is not a green mesh; it is a green readable mesh. State the denominator with the result — "0 of N NodeTypes over M readable partitions" — and when the deploy being gated spans partitions the sweeper has no grant on, the honest instruments are the ones that run as the system: the pod's own nodetype_bake health payload, which names every non-Ok type it enumerated, and the boot line's compileErrors= / previouslybroken= counters. Elevation to read someone else's partition is break-glass and is the owner's decision, never a sweep step.

🚨 nodetype_bake is CONDITIONAL, and an absent check reads exactly like a passing one. It is not in this repo — the state lives here (src/MeshWeaver.Hosting/NodeTypeBakeGate.cs) but the IHealthCheck that surfaces it belongs to the host, Memex.Portal.Distributed in MeshWeaver.Plugins (Program.cs, registered as nodetype_bake), so grepping core's src/ for the name finds nothing and reads as "no such instrument". Two ways its silence means nothing:

• It is registered only if (gateBake). With readiness gating off the check is not present at all, and /health names no NodeType because none was asked for — not because none failed.
• GateReadiness=true with DynamicTypes=false is registered, permanently green, and protects nothing (the gate reads bake state that only the sweep writes, and the sweep never runs). The two PreWarm keys are one setting; the host's own comment says so.

So before trusting a green nodetype_bake, confirm the check is REGISTERED and ARMED — the payload must name a positive count of types it actually enumerated. A verdict with no denominator is the skip-trapdoor this whole page argues against, wearing a health check's colours.

🚨 Third way its silence means nothing: the check exists in main and not in the RUNNING IMAGE. /health is served by the image the pod booted, never by the branch you are reading, and the gap between the two is routinely a day's worth of merges. Measured 2026-09-11 on memex.systemorph.com, a portal whose /health was already Degraded — so plainly reachable and reporting — and which published exactly five checks: content-types, pending_module_activation, required_modules, bundle_adoption and the roll-up. No bake-report, no source-discovery, no nodetype_bake.

The instrument had landed in core nine hours after that image was built. Nothing was misconfigured and nothing was refusing — the code simply was not in the binary. So check the running commit before concluding a check is unarmed, and run the ls-tree with a control: an empty ls-tree exits 0 whether the path is absent or the commit is wrong, which is the same shape of false negative as everything else on this page.

The obligation on framework changes

Removing or renaming any public framework surface — extension methods on MessageHubConfiguration / IMessageHub, Controls.*, host.* helpers, content base types — is a breaking change to code the compiler cannot see. Before deleting one:

1. grep -rn "<Symbol>" content samples/*/Data — catches in-repo node source. Grep the node JSON too, without a --include='*.cs': a Code node stores its C# in a content.code string and a NodeType stores its configuration lambda the same way, so a .cs-only sweep reports "no callers" over a page full of them.
2. Grep every OTHER node repository — MeshWeaver.Education, .Plugins, .Reinsurance, .SocialMedia, .Manufacturing, Memex. A symbol on the cell surface is called by bare name from content this repository has never seen.
3. Search the live mesh (search_chunks) — catches nodes that drifted from the repo. 🚨 An answer carrying "searched": false is a failed sweep, not a clean one: that deployment has no embedding provider, nothing was searched, and the envelope deliberately carries no count so an absent field cannot be read as "no callers". Sweep on another deployment or stop.
4. Port or delete the callers in the same change, then sweep.

🚨 A cell-surface symbol has no retirement that ends in a delete

Steps 1–4 can reach zero and still be wrong, because they only cover content somebody can edit.

Installing a course or an app copies its nodes into the installer's own space. The copy is a snapshot: the cell inside it is that person's content, its code is whatever it said on the day they installed, and no repository change and no plugin update rewrites it. So for a type published with cellSurface: true — reachable by bare name from every kernel session in the mesh — there is no state a caller sweep can establish that means "nothing calls this any more".

That is not a hypothetical either. #975 retired TrainingSimResponder on a "no callers left" reading of core and MeshWeaver.Plugins. Thirteen callers were live in MeshWeaver.Education (five of them C# inside a JSON string), and behind those an unknown number of installed copies. Every live prompt cell in the AgenticEngineering course failed with CS0103: The name 'TrainingSimResponder' does not exist in the current context, showing learners a page that promised a prompt box and a ✨ button and rendered neither. Fixing the course repaired the central pages and every future install and nothing already installed (Plugins#1258).

And the copies cannot be enumerated, by design. A learner's copy lives in their partition, so no sweep any agent can run may read it — row-level security is doing exactly its job. Measured on memex: rbuergi/AgenticEngineering/Introduction/Exercise/AskForATable is a 2026-08-13 snapshot that still carries the cell INLINE (a ```csharp --render Chat block in the markdown body plus a codeSubmissions entry) — a shape the central course stopped using on 2026-08-21, when the cell became a separate Source/Chat node. That copy calls TrainingSimResponder.Live, and it is merely the one copy the sweeper happened to own. So "how many callers are left" is not a hard question here; it is an unanswerable one.

So the last step is a forwarder, not a delete. Keep the old name on the same cell surface as a shim that delegates to the successor, [Obsolete] as a warning — never error: true, which would break the very copies it exists to keep compiling, and warnings never reach a cell anyway, since the kernel only surfaces CompilationErrorException. Pin its surface with a test in the NodeType's Test/ area so the shim cannot be narrowed by accident; a parameter quietly dropped from a forwarder is the same invisible break one layer down. The shim may go when the installed copies are gone — which is a fact about a running mesh, not about a repository.

Store Repair is not a substitute for the shim. A learner can pull a page back from the central course — the per-item repair dialog (Store/Installer, with ContentFingerprint telling missing from unmodified from edited) exists precisely so one broken exercise does not mean reinstalling a course. But it is the wrong remedy for a break the platform caused: it asks every affected learner to notice a dead page, go looking for the dialog, and confirm an overwrite that discards their own work on any page they have edited. Repair is for a copy the learner damaged. A forwarder is for a copy we damaged.

ADDING a symbol has the same hazard, in reverse. In-mesh source that references a brand-new framework helper compiles only once the image carrying it has actually shipped — and node content reaches a portal by a completely different route from the image (a GitSync/plugin sync, or an MCP edit, either of which can land first). Merging both halves in one PR does not make them arrive together.

That is not hypothetical: #1386 moved a copy-pasted Article extractor into compiled framework code as MarkdownBody.Of, and the in-mesh callers referencing it went out before the image did. ACME/Article, Cornerstone/Article and Northwind/Article sat at CS0103: The name 'MarkdownBody' does not exist in the current context on memex-cloud until the portal self-updated to the image that contained it, at which point all three returned to Ok on their own.

So when a framework change and its in-mesh callers ship together, the framework half must land first, and the content half is only safe once the target portal reports the image that carries it. The failure is invisible to CI in exactly the same way a deletion is, and it reads identically to a content defect.

The pre-prod sweep

Search('nodeType:NodeType') → LspDiagnosticsForNode('@{path}') per type (reads the cached compilation, no re-emit) → fix roots first, since one red upstream reports as a failure in every dependent → re-check until every type reads Ok. Warnings are in scope: an unregistered $type leaves content as an untyped JsonElement, which renders empty rather than erroring. The full protocol lives in the /code skill.

Quick reference