Build coordination — the Build node protocol
Who builds, what gets built, and when a silo may serve — decided by mesh nodes, not by file leases or in-process sweeps. This page is the protocol contract; the compilation mechanics it orchestrates are unchanged and documented in NodeTypeCompilation and NodeTypeRelease.
Why the current shape has to go
Today the bake is an in-process sweep (DynamicTypePreWarmerHostedService +
NodeTypeBatchBake) arbitrated by a file lease under the shared assembly cache, and
readiness is fed by that same process's private state. Three defect classes follow
directly from that shape, and each has caused a production incident:
- Starved cross-silo discovery. During a rollout the baking pod and the serving pod
share one Orleans cluster, so the baker's per-type source reads land on activations the
outgoing pod already holds. Under churn those round-trips starve, Roslyn compiles
against an empty source set, and a healthy type reads as a regression (CS0246 naming
the type's own classes).
[PreferLocalPlacement]cannot fix this — it only biases where a new activation lands. - Mid-bake content races. The sweep compiles against the moving mesh. A plugin
auto-update landing mid-bake gave Roslyn a source set that existed for two minutes and
never again; the regression latch stuck and the rollout hung. The retraction machinery
in
NodeTypeBakeGateStateexists to paper over exactly this. - Retention. Every compile reads its sources through the mesh, and each touched path
mints synchronization state in the baking process (~22
sync/sub-hubs, ~9 MB retained per trivial recompile), on top of the collectible-ALC accumulation. A long bake grows until the pod hits its memory ceiling.
The protocol below removes all three by construction rather than by compensation.
The shape
Admin/Build ← the build ROOT (durable node, nodeType Build)
Admin/Build/_Claim ← the root's claim LOCK (durable; only atomic writes)
Admin/Build/{chunkName} ← one node per CHUNK (durable, nodeType Build)
Admin/Build/{chunkName}/_Claim ← the chunk's claim LOCK
Admin/Build/{chunkName}/_Activity ← the chunk's execution activity (standard Activity)
{nodeTypePath}/Release/{version} ← unchanged: releases minted per compiled NodeType
- The root carries the build's identity — the framework fingerprint it targets
(the resolved framework build identity,
FrameworkBuildIdentity.FrameworkVersion— the same valueHasUsableBuildcompares against), the commit set of the synced sources, the chunk plan, and the aggregateStatus. - A chunk is a named unit of work defined by an array of mesh queries — as simple as a
list of paths, or a module such as
namespace:MyPlugin scope:subtree nodeType:Code. Chunk names address the nodes:Admin/Build/{chunkName}. - Each chunk node launches one Activity to execute its build (standard
_Activitysatellite: progress, cancel, terminal status — nothing bespoke) and reports back the paths it wrote: the release node paths its compiles minted. Releases stay the system of record for artifacts; the chunk records which ones this build produced. - When every chunk is terminal and none regressed, the root flips to
Ready— the GO signal.
All coordination state is durable node content. There is no other channel: no lease
files, no request/response types, no in-memory gates. Writers go through
GetMeshNodeStream(path).Update(...); the owning hub serialises them.
Who becomes the build master
Nobody is elected. Mastership is a claim written into the root node. Candidates register
under their own holder id in RequestedClaims (per-candidate keys — RFC 7396 merge-safe, so
concurrent registrations compose instead of overwriting each other), and the node's own hub
arbitrates. The claimant then observes its own claim on the stream before doing any work — a
claim you cannot read back is a claim you do not hold. In-memory single-flight flags are
permitted only as coalescers; correctness comes from node state (the
ActivityControlPlane rule).
The same claim shape applies per chunk, which is what makes parallel builders safe later:
each would-be builder claims Admin/Build/{chunkName}; exactly one gets each chunk.
🚨 The grant is taken on the DURABLE ROW — a hub lambda is only exclusive within ONE cluster
The obvious implementation is to decide inside the owning hub's serialised Update lambda:
the action block orders every writer, so the first candidate's lambda sees an unclaimed build
and takes it, and every later lambda re-reads the claimed state and bails. That is correct —
and it is exclusive only within one Orleans cluster, which is not the topology this
protocol runs in.
A second cluster over the same database activates its own hub for Admin/Build, runs its
own arbiter against its own mirror, and grants its own candidate. Three things then line
up to make the collision invisible (#1424):
- both writes are minted at the same next
MeshNode.Version, and the store's monotonic condition applies at equal versions — re-persisting an unchanged node is a legitimate, common shape — so both land, last-write-wins; - neither writer is told it lost: the "store refused" signal is
saved.Version > written.Version, and the versions are equal; - nothing propagates the other cluster's write back. Orleans membership and its memory streams are per-cluster by construction, and there is no cross-process change feed running (see the GO section below), so a mirror can be arbitrarily far behind and never learn.
On 2026-08-13 that is exactly what happened on memex-cloud: the ephemeral bake Job and the
rolling serving pod each claimed Admin/Build and each ran the full 268-type bake.
The claim lives on a LOCK, not on the Build node
Making the grant exclusive is necessary and not sufficient, and the second half is the less
obvious one. Every cluster mirrors Admin/Build in its own workspace, and that mirror is
flushed to storage as a whole node by the ordinary persistence sampler — a plain,
unconditional write from a hub that may never have held the claim. MeshNode.Version is a
per-node counter, not a cross-cluster logical clock, so both mirrors climb it independently and
the later flush simply wins the row. A losing cluster therefore overwrites the winner's
ClaimedBy with its own null, its arbiter sees a free build on the next pass, and the
exclusivity a compare-and-set had just established is undone by a write that knows nothing about
it. (This is observed behaviour, not a hypothesis: the cross-cluster test caught exactly it.)
So the claim moved off the contended row onto a lock node, Admin/Build/_Claim (and
Admin/Build/{chunk}/_Claim). The lock has no hub, no mirror and no sampler. Exactly two things
ever write it, and neither is unconditional:
| Operation | Primitive | Exclusivity |
|---|---|---|
| grant / takeover | IStorageAdapter.WriteIfVersion(node, expectedVersion) |
compare-and-set on the version the arbiter read (0 = "must not exist") |
| heartbeat | WriteIfVersion |
no-op unless this holder still owns the lock |
| release | IStorageAdapter.DeleteIfExists |
rowcount-gated, first delete wins |
BuildState.ClaimedBy on the Build node stays as the observable projection — what the GUI
and ObserveBuildClaim read. A flush clobbering it now costs a stale view in one cluster, never
a second builder, because no decision is taken from it.
The three steps, in this order
BuildNodeType.ArbitrateDurably:
- Read the lock. Not the mirror — the mirror is per-cluster and cannot see a rival's grant.
- Decide with the unchanged
Arbitrate, over the lock's holder state and this cluster's pending registrations (from its mirror, because a candidate that registered milliseconds ago has not reached storage yet — and a cluster can only ever grant one of its own candidates). - Commit with the compare-and-set against the version step 1 read. At most one of N
concurrent arbiters is told
true. Only then is the grant published on the Build node.
Step 3 has to come before the mirror write, not after, because the mirror commits ~200 ms
before the persistence sampler reaches storage and ObserveBuildClaim emits off the mirror.
Adjudicating after the fact would find the loser already baking.
Losing writes nothing and retries nothing: a refused compare-and-set means another cluster's
grant is already durable, so this cluster does not grant, its candidate runs out its
GrantWindow and follows the GO — the path the protocol already has for "held elsewhere".
The arbiter is level-triggered, so the next legitimate trigger re-decides against fresh
durable state. No timer, no election, no backoff loop.
It fails OPEN. With no storage provider owning the path (or no persistence at all — a
monolith, a test, a dev box) the grant is taken on the mirror exactly as before. Being wrong
in that direction costs one duplicated bake: bounded and non-corrupting, because every
downstream write is content-addressed (assembly store keyed by content, release versions
minted from content hashes, GO keyed by fingerprint). Being wrong the other way — refusing to
grant because exclusivity could not be proven — leaves the fingerprint with no GO at all,
which holds every silo's readiness probe down and stalls the rollout. Same asymmetry the bake
gate applies, and the opposite of AssemblyCacheRetention, where the wrong answer deletes
bytes a running pod still needs.
🚨 When may the claim be taken away — cluster MEMBERSHIP decides, not a clock
A builder that dies mid-build must not wedge the fleet, so a claim has to be reclaimable. The
obvious way is a staleness budget on the holder's heartbeat, and that is what this started as
(ClaimStaleAfter, 10 minutes). It is the wrong instrument, and wrong in both directions: a
timestamp answers "when did the holder last manage to write", which conflates dead with busy,
starved and descheduled. So the fleet waits out ten minutes for a pod that is already gone, and
it is licensed to evict a pod that is merely slow — putting two builders on one compile, the exact
storm the claim exists to prevent (#1355).
Where a cluster exists it already answers the real question authoritatively and immediately — it
runs probes, indirect probes and a membership table for precisely this. So the candidate stamps its
IClusterMembership.LocalIdentity into its claim request, the arbiter copies it to
BuildState.ClaimedByIdentity on grant, and takeover is a lookup rather than a guess:
| Membership says about the holder | Result |
|---|---|
| Gone | take over immediately — no budget to wait out once the cluster has positively recorded it as departed |
| Alive | never take over, however old the heartbeat looks |
| Unknown | fall back to the ClaimStaleAfter (10 min) heartbeat clock |
Unknown is the only path that still consults a clock, and it covers exactly the hosts that have no
cluster to ask — a monolith, a test, a dev box, the Orleans client host — plus a claim written
before identities were stamped and an identity membership cannot resolve.
🚨 A holder in ANOTHER cluster is always Unknown, because Orleans membership is per-cluster:
the bake silo's ServiceId is not in the serving cluster's table and vice versa, and absence is
Unknown, never Gone. So a cross-cluster takeover falls back to the ClaimStaleAfter clock —
this is the one place the clock still governs a live fleet, and it is deliberate: the alternative
is treating "I cannot see you" as "you are dead", which is precisely the eviction of a live builder
the membership rule exists to prevent. What #1424 adds is that the takeover WRITE is itself a
compare-and-set, so two clusters that time out on the same dead holder cannot both succeed it.
🚨 Absence from the membership snapshot is Unknown, never Gone. Orleans keeps departed silos
in the table as Dead until the defunct-cleanup window elapses, so a silo that really died IS in the
snapshot and resolves to Gone. One that is missing entirely usually means our own snapshot is not
hydrated yet — and reading that as death on a freshly-started silo would evict a live builder, which
is the one failure this rule exists to make impossible.
🚨 …and the mirror image: PRESENCE in the snapshot is not Alive either (#2076). Alive is read
above as permission-denied-forever — it is the one verdict that skips the clock entirely — so it
must mean "the cluster positively recorded this member as RUNNING", not "a row exists for it". Only
Active, ShuttingDown and Stopping qualify. A silo still Created/Joining is Unknown,
because Orleans only probes ACTIVE silos: a process that dies before finishing its join leaves a
row no failure detector will ever move to Dead. Mapping those to Alive made the ClaimStaleAfter
fallback structurally unreachable for exactly the case it exists to cover, and on 2026-08-22 a pod
deleted MID-BOOT held the claim on memex-cloud while every other pod sat in FollowGo for 25+
minutes — with PreWarm:GateReadiness=true that holds the whole rollout. Note this is NOT an
immediate steal: Unknown hands the decision to the heartbeat clock, so a holder that really is
mid-join and working keeps its claim through the beat it writes.
🚨 The holder's heartbeat starts with the build, not after its planning phase. BakeAsMaster
wraps the WHOLE master path in Observable.Using(StartHeartbeat, …), chunk opening included.
OpenChunk waits up to GrantWindow per chunk for its own grant, so with dozens of chunks the root
claim could otherwise go minutes without a beat — and Observable.Interval does not fire until one
full HeartbeatInterval after subscribe. Across clusters the holder is Unknown by construction
(see above), so the peer's arbiter judges it on ClaimStaleAfter alone: a builder working perfectly
well could age past that budget before its first beat and license a second builder.
The heartbeat needs no defending here: it is a field on the Build node written through
stream.Update, so it is durable mesh state rather than a file's last-write metadata. The
predecessor lease kept its instant in an SMB timestamp, where Azure Files' metadata caching could
report a heartbeat as stale while its holder was alive — that whole failure mode left with the lease.
Code: BuildNodeType.Arbitrate (pure over now and the membership verdict, so every rule above is
tested without a cluster and without wall-clock), IClusterMembership
(src/MeshWeaver.Mesh.Contract/Services/IClusterMembership.cs), and OrleansClusterMembership
registered silo-side by ConfigureMeshWeaverServer.
The bake runs in its own, disposable cluster
The build master is not a portal pod. It is an ephemeral silo with its own Orleans ServiceId (and ClusterId), started per new image version and disposed after GO:
- Same image, different mode. The bake host runs the portal image with a bake
entrypoint. This is non-negotiable: the framework fingerprint is the Graph assembly's
MVID, so only byte-identical binaries produce bakes the portal can use. (A dedicated
bake image was tried in 2025 and retired for exactly this reason — it computed a foreign
fingerprint; see the
memex-bakeretirement, #1347.) - Separate ServiceId = the grain-placement answer. Every grain involved in baking activates in the bake cluster because that is the only cluster it exists in; every regular grain keeps serving in the live cluster. Neither can starve the other, and defect class 1 above is unrepresentable.
- Commit-pinned sources. Chunk compiles read their sources from the on-disk
source replica (
GitModuleReplica) at the chunk's recorded commit (GitHubSyncConfig.LastSyncCommitShaper synced space) — a consistent snapshot, not the moving mesh. Defect class 2 is unrepresentable: the source set cannot change under the compiler, and an unchanged (commit, fingerprint) pair is provably a no-op. - Dispose after GO. The bake process exits once the root is
Ready. Whatever synchronization state and collectible ALCs the compiles minted die with it — defect class 3 is bounded by process lifetime instead of by leak-chasing.
The shared surfaces between the two clusters are exactly three, all already
multi-process-safe: the Postgres store (node state, serialised per node by the owning
hub), the assembly store under /data (content-addressed writes), and the source replica
(commit-addressed, idempotent).
The GO signal and readiness
Every portal silo — old and new — subscribes to the build root and holds its health probe accordingly:
- On startup a silo computes its own framework fingerprint (its resolved framework
identity — a local, mesh-independent computation) and subscribes to
Admin/Build. - The readiness probe reports not ready until the observed root says
Status = Ready && FrameworkVersion == <my fingerprint>. - Old pods keep serving through a rollout untouched: the GO for their fingerprint was
written when their image baked, and a newer build's state transitions never revoke an
older GO — the root's
Readyis per-fingerprint history, not a global boolean.
The broadcast is designed to ride the durable store's change feed (Postgres LISTEN/NOTIFY
via PostgreSqlChangeListener), which is cross-process and therefore cross-ServiceId by
construction. It deliberately does not ride the Orleans stream relay — the bake cluster
is not in the portal's cluster, and intra-cluster relays must not be a readiness
dependency. The subscription itself is a remote-path watch and uses the
SubscribeWithReEstablish fault taxonomy: transient faults re-establish; a poisoned or
deleted root is terminal and loud, never a silent 1 Hz retry loop.
🚨 That feed delivers only to a session that is already listening — it never replays. (It is
running: "registered but never started in either partitioned-PG overload" was #1440, the middle leg
of the #1814 outage, fixed by #1816 and pinned by ChangeListenerWiringTests in the adapter's repo.)
A NOTIFY fired while a follower's LISTEN session was down, or before its mirror activated, is
gone; within a cluster the GO still reaches every silo, because GetMeshNodeStream resolves to the
ONE per-node hub that owns Admin/Build there, but across clusters a follower that came up after
the write observes only what its own hub read at activation. The design page
Durable Streams Are Mesh Nodes records the loss
semantics.
Nothing in the protocol may therefore depend on being notified. The arbiter never did — it READS the durable row on every pass (above). The follower used to, and that is #1440.
The follower has two doors, and it closes them behind it
BuildProtocolDriver.FollowGo used to be a single ObserveBuildGo(fingerprint).Take(1). That is a
projection of this cluster's mirror, so a cross-cluster follower was waiting on an event that
could not be delivered to it — and it had also stopped observing its own claim, so the one event
that could still reach it in-process went unheard. It had no door at all. It now ends on either of
two real events:
- The GO becomes visible —
ReadBuildGoreads it off the durable build root (the same witness the arbiter decides on) andObserveBuildGowatches this cluster's mirror, whichever answers first. The read is what lets a peer cluster's GO mean anything at all here. - The arbiter hands us the claim. Registering as a candidate outlives the grant wait, and the arbiter grants the moment the build falls free — whether the builder finished or died. On a grant the follower re-reads the durable witness: a GO already there means the winner finished and this process stands down without re-baking; no GO means the builder went away mid-build and this process bakes. The grant is the level-trigger; the witness is the verdict.
Standing down is not bookkeeping. RequestBuildClaim's registration survives until a grant
consumes it, so a follower that reached its answer some other way is later handed a build it will
never run — holding the claim at Planning, never heartbeating, and never taken over, because its
process is alive and the takeover rule below defends a live holder by design (a stopped heartbeat on
a live process means busy, not dead). The next image's fingerprint could then never be claimed and never get a GO, holding
every silo's readiness down. WithdrawBuildClaim removes the registration and gives back a claim
that raced it.
🚨 Standing down cannot be made safe from the candidate's side alone (#1193). Both halves of
WithdrawBuildClaim are conditional on state the arbiter is about to change, and an arbitration
pass spans two storage round-trips: it reads the candidate set off the mirror, reads the lock,
commits the compare-and-set, and only then publishes the grant. A follower that stands down inside
that window removes a registration the pass has already captured, and finds no lock of its own to
release — so the pass commits a grant to a process that has finished with the build. Conditional
writes make each half safe on its own; they cannot order two writers. Depending on the
interleaving the debris is either the mirror's claim projection or the durable lock, and the
takeover rule then defends it because the process really is alive. Measured on
BuildCoordinationTest.Follower_StandsDown_SoTheNextBuildCanStillBeClaimed: 2 failures in 15 runs
on an idle machine, the build node left at ClaimedBy=<the follower>, Status=Planning.
So the arbiter re-checks at the point where it can: ApplyGrant refuses to publish a grant whose
winner is neither the holder the mirror already names nor still a candidate, and a refused
publication hands the lock straight back (HandBackAStoodDownGrant) — the arbiter is the lock's
only writer, so undoing its own commit is the only place this can be closed. The guard's first
clause is load-bearing: a holder mid-bake has no registration left either, its own grant having
consumed it.
…and the mirror image of it: a stand-down DECIDED before the grant and APPLIED after it (#3567)
That refusal closes the interleaving in which the stand-down has already happened when the grant is published. It cannot close the one in which the stand-down is decided first and applied last, because there the publication was legitimate at the moment it happened — the candidate was still registered. What arrives late is the stand-down.
WithdrawBuildClaim runs on the candidate's hub, which does not own Admin/Build. So its
stream.Update lambda is evaluated against that hub's mirror and travels to the owner as an RFC
7396 merge patch of the fields it changed (MeshNodeStreamHandle.UpdateQueued →
ComputeMergePatchDiff) — and a field the lambda leaves unchanged is absent from the patch,
which by RFC 7396 leaves the owner's value untouched. Its hand-back half is conditional on exactly
such a field (state.ClaimedBy == holder && state.Status is Planning), so on a mirror the grant has
not reached yet the patch carries the registration removal and no claimedBy at all. The
arbiter, by contrast, writes the node it owns, so its lambda is serialised against fresh state.
That asymmetry is the bug — it is the general shape, not a build-protocol quirk:
Conditional Writes Across Hubs.
| order | outcome |
|---|---|
| stand-down decided and applied before the grant | ApplyGrant refuses, HandBackAStoodDownGrant drops the lock. Closed above. |
| grant applied, then stand-down decided | the hand-back half is true → the holder is cleared. Always worked. |
| stand-down decided before the grant, applied after it | patch carries no claimedBy → the holder survives. #3567. |
A candidate cannot evaluate a condition the owner will apply, so it states a fact the owner acts
on — the RequestedX-plus-owner-watcher shape, and RequestedStatus is the same idea one field over
on this record:
BuildState.StoodDown— holders that withdrew, keyed by holder id and written unconditionally byBuildNodeType.StandDown. Its own key makes it merge-safe against every other candidate, and being unconditional makes it true whatever the owner's state turns out to be. It is the half that is always there to be read.ReleaseStoodDownClaim— the arbiter releases aPlanningclaim whose holder has stood down, and consumes the mark. ABuildingholder is never released: a mark from an earlier life of the same id must not pull the lock out from under a running compile.ApplyGrantalso refuses a marked winner — the same refusal stated off the fact rather than off the absence of a registration, which the in-flight patch may not have delivered yet.DropLockHeldByStoodDowndrops the claim LOCK when the mirror released, so the two records agree; the debris lands on either, so fixing one alone just moves the wedge.- Marks are consumed, never accumulated: on release, on re-registration (
RequestBuildClaimclears its own key) and otherwise aged out onClaimStaleAfter, the claim's own budget.
🚨 A decision the arbiter is never woken to take is not a fix. The mark lands on a node with
RequestedClaims empty — the grant consumed the follower's registration on its way in — so the
own-stream trigger has to ask about more than "is anyone queued?", or the emission carrying the mark
is filtered away and the release waits for the two-minute stale tick. ArbitrationTrigger is that
predicate, named so it can be tested as the expression the arbiter actually subscribes: it wakes a
pass for a pending registration or a stood-down holder, and the mark is part of the change key,
not merely of the filter, because it arrives on a node whose other trigger fields did not move.
(On a host with a durable store the withdraw's flush also publishes on IStorageAdapter.Changes,
which is why this only ever showed up where the mirror is the claim.)
There is deliberately no timer and no bound bolted onto the wait. A bound would end the wait by guessing, and a follower that guesses "the build finished" certifies a share it never probed — a silent wrong answer, strictly worse than a hang that announces itself. Both doors are level-triggered on durable state, exactly like the arbiter (and see #1366 for why the poll clock went).
Starting the listener is a separate decision. It would make the GO propagate promptly across clusters instead of at the arbiter's next pass, but it changes behaviour for every partitioned-PG deployment and wants its own justification and measurement. The follower is correct without it.
The PRE-WARMER has the same two doors — and it did not, until #3404
The follower's two doors only ever opened once a process had already got inside. Everything the
pre-warmer knew about the build arrived through ONE door: BuildProtocolDriver.Run opens with a
claim handshake, and a claim handshake is a SubscribeRequest to Admin/Build. Exhaust its
attempts and the driver threw BuildCoordinationUnreachableException, the sweep faulted, readiness
was refused, and the rollout held the previous image.
🚨 A pod can therefore refuse a build that was already approved. Measured on memex-cloud,
2026-09-06: two pods of one deployment refused at 11:44:32Z and 11:49:37Z for a fingerprint whose GO
had been written on an already-Ready build root at 11:28:56Z — sixteen minutes earlier. Their
own hubs were healthy and idle throughout (RunLevel=Started, empty queue); the silence was between
them and the hub that owns Admin/Build. They could not open a subscription to read a verdict that
was already durable — and the verdict was the only thing they needed.
So Run's subscription-borne composition is now treated as ONE DOOR, and when it cannot be opened
at all the durable witness is asked (WhenTheSubscriptionDoorIsShut). Same read as door 1 of
the follower — ReadBuildGo on the durable row, the record the claim arbiter itself decides on —
at the one point in the protocol that never had it.
Four properties are load-bearing, and each is pinned by a case in
PreWarmerReadsTheDurableGoTest:
Per-fingerprint, never "a GO exists". The
Readymap is keyed by framework version precisely so an old-image pod stays ready while a new image is unproven.ReadBuildGolooks the pod's own fingerprint up by exact key; a GO for any other image reads as no GO and the door stays shut. Granting on a foreign GO would certify a build the process is not running — strictly worse than the refusal it replaces.Fail-closed is unchanged where the witness ANSWERS. A witness that says "no GO for this fingerprint" re-throws the ORIGINAL
BuildCoordinationUnreachableException, so the sweep still faults, readiness is still refused, andDescribesUnreachableCoordinationstill separates "no verdict" from "a bad verdict" in the health payload — and it refuses even when this pod's own share is fully baked, because a pod does not certify from its own volume a build the coordination node never approved (NoDurableGo_StillRefuses_EvenWhenTheShareIsFullyBaked).The transport fault stays fully visible. Nothing here widens a timeout, retries, polls or swallows.
RetryUnreachableCoordinationlogs the unreachability at its own severity with its own diagnostic detail either way, and the grant logs a secondWarningnaming what could not be reached. That fault is the only signal the pod↔Admin/Build-hub path is broken; the durable door changes the readiness VERDICT, never the visibility of the fault, and does not diagnose or repair it.No stand-down on this path. The follower withdraws its claim before probing because it really registered one. This path did not: the registration is written by
RequestBuildClaimthroughGetMeshNodeStream(path).Update(current => …), and anUpdatewhose stream never delivered current state never computed a patch, so there is no candidate entry to hand back — and callingWithdrawBuildClaimwould post a second write into the same unreachable hub. The two paths share the post-GO share probe; they deliberately do not share the stand-down.
🚨 …and the second door was in the FIRST door's failure domain (#3404, second half)
The first cut of that door had two branches — a GO, or no GO — while ReadBuildGo folds
three outcomes into its null: the row carries no GO, there is no durable store, and the read
failed. So a read that never completed was acted on and narrated as a definitive negative
("the durable witness carries no GO for framework X").
That is not a corner case here, because the second door is not in a different failure domain from
the first. In the fleet's portal wiring AddPartitionStorageHubs replaces IStorageAdapter with
RoutingProxyAdapter, whose Read is
hub.Observe<ReadNodeResponse>(new ReadNodeRequest(path, options), o => o.WithTarget(addr)) — the
same hub transport under the same 60 s budget as the SubscribeRequest. Of the five candidates
below, the first, fourth and fifth take both doors down together, and the durable read then
fails with the same TimeoutException. On the very fault this door exists for, the expected
reading is undetermined, not no GO.
So the reading is now a three-state BuildGoReading (Go / NoGo / Undetermined, always with a
Detail and, when there was a fault, its Error), read through ReadBuildGoReading; ReadBuildGo
keeps its fold for the callers whose negative branch is bake, and its contract now says a caller
that REFUSES on the negative must use the three-state read instead. On Undetermined the pod does
not guess in either direction: it measures on the one witness the broken transport cannot touch —
its own IAssemblyStore — and grants only when NodeTypeBakeReport.GateRelevant is empty, which is
stricter than the GO branch (a still-pending type after a GO is non-gating, so a half-baked
share passes with a GO and is refused without one). The full argument, and the general rule, is
Undetermined Is Not No.
🚨 The durable door is a FAIL-SAFE, not a cure — and the cause is still open. The pod's own diagnostic names three candidates for the silence: the request never reached the target (routing), the target received it and is wedged (a per-node hub that stops answering — the #2896 class), or the target answered and the reply was lost. Two more have since been measured, and neither subsumes the other:
Fourth — deferred-queue ordering (#3408). MessageService.OpenGate drained the deferred queue by
appending it to the main queue, so a message that arrived before the gate opened but had not yet
been turned sat ahead of the deferred turns appended behind it — and the parked message ran last.
Appending is always the wrong end, not merely unlucky: deferral happens at TURN time, not arrival,
and the loop is strictly FIFO, so everything deferred is by construction older than everything still
waiting. A SubscribeRequest is exactly the message that triggers a per-node hub's activation, so it
is the one that loses its place — load-sensitive by construction, green in isolation, and capable of
being processed after the requester's 60 s budget has expired. That fits a requester measured healthy
and idle against a target never observed dead better than a wedge does.
Fifth — a root that STOPS EMITTING with a holder still set (MeshWeaver.Plugins#1193). A
controlled load experiment on BuildCoordinationTest.Follower_StandsDown_SoTheNextBuildCanStart —
18 cores, two arms of ten runs, same binary and same sha, only background load differing — passed
10/10 under 12 burners (load ~31–39) and failed 1 of 10 under 64 burners (load 49–91+), at 32.9 s,
with the CI signature. Instrumenting the wait recorded exactly one root emission in 15 seconds
and nothing after it.
🚨 The control is what makes the reading honest. The PASSING state was captured at the same point in the same test (by temporarily making the predicate unsatisfiable — a diagnostic never seen to run reads exactly like one that works). The two states differ in one field:
FAIL: ClaimedBy=<machine-scoped id> Status=Planning RequestedClaims=[] Ready=[fp]
PASS: ClaimedBy=<null> Status=Planning RequestedClaims=[] Ready=[fp]
So Status=Planning and the empty RequestedClaims are not symptoms — the healthy run carries
both, and the build completed in both (Ready holds the fingerprint either way). Any account
resting on those two fields rests equally on the passing run. What is measured is exactly this:
ClaimedBy stays set, one root emission in 15 s, nothing after. A reader waiting on such a root
waits forever.
The mechanism is CLOSED (#3567) — the holder was never told to act, and could not have been. The
stand-down was decided on the candidate's own mirror and applied as a merge patch that carried no
claimedBy, so a grant the arbiter committed inside that window survived it; the takeover rule then
defended the live-but-idle holder by design. That is the third interleaving tabulated under
"a stand-down decided before the grant and applied after it" above, and the fix is there too: the
candidate records StoodDown as an unconditional fact and the arbiter releases on it — the general
shape is Conditional Writes Across Hubs. The permanent diagnostic
that produced the reading above landed as
MeshWeaver.Plugins#1401 (diagnostic only, no production code); the repro recipe is on #1193, which
closes once that repo's MW_PLATFORM_REF carries the fix.
🚨 It reproduced with #3408 already in the core, so #3408 does not remove it and #3131 did not close it: fourth and fifth are independent, and neither subsumes the other.
This shape is the one the durable door most clearly answers — and note the claim is about the shape, not about any mechanism. A root that has stopped emitting is neither dead nor slow, so nothing about the transport is going to recover it, while the durable GO is already published and already readable. The durable door is correct whichever of the five it is, it is the only one that does not depend on diagnosing the transport first, and in this shape it is the only fix in flight that helps at all. None of that makes it a cure: each cause is a separate change, owned separately, and this one must not be read as having addressed any of them.
The probe semantics of NodeTypeBakeGateState are preserved unchanged — fail closed
on a measured regression (the rollout stalls, the old image keeps serving), fail open
when nothing is measuring (a configuration mistake must never black-hole a pod), gate
/health and never /alive. Only the feeder changes: a subscription to the build root
instead of the in-process sweep. The elaborate retraction/cascade machinery loses its
main customers (timeout-blindness and content races are gone by construction) but keeps
guarding the one honest case: a type that really does not compile on this image.
Chunking: plugins are the natural unit
The chunk plan is derived, not invented:
- one chunk per installed plugin — a plugin's footprint is a chunk query
(
namespace:{Plugin} scope:subtree nodeType:Code), and its registry commit is the chunk's commit key; - residual chunks for non-plugin content (samples, user NodeTypes), partitioned along namespace lines;
- a plugin may declare its chunk (override queries, order dependencies) in its manifest — declarative content, shipped with the plugin.
Two consequences fall out for free:
- A module update is a build. "This plugin's commit moved" re-runs exactly
Admin/Build/{pluginName}— same protocol, no separate mechanism. The install/update path compares recorded vs target commit and materialises only what differs. - Parallelism is claim arbitration, not state transfer. When multiple builders run, the content-addressed stores (assembly cache, source replica) do the heavy coordination; the protocol only arbitrates who takes which chunk — the claim shape above, unchanged.
What the coordinator is NOT
- Not a dynamic NodeType. The protocol engine makes dynamic content usable; if it were itself dynamic content, a compile failure in it would leave the portal permanently not-ready with no deterministic way back, and none of it would be CI-tested. The engine ships in the image. Plugins contribute declarative chunk manifests and observation UI, never coordinator code.
- Not a static declared node.
AddMeshNodesentries are process-local and never persisted — and a static claim at the root's path would shadow the durable row (the #1209 class). The root is created if absent by the claimant, durably, and existence is answered by the store. - Not request/response. No
StartBuildRequest. A build is requested by writingRequestedStatus = Runningon the node (RequestViaStreamUpdate); cancel isRequestedStatus = Cancelledvia the standard activity surface.
What this retires
| Retired | Replaced by |
|---|---|
.bake-lease file lease under the assembly cache |
the claim field on Admin/Build, arbitrated by the owning hub |
in-process sweep orchestration (DynamicTypePreWarmerHostedService driving the whole bake) |
the ephemeral bake silo executing chunk activities |
| readiness fed by process-private sweep state | readiness fed by the GO subscription |
| cross-silo discovery timeouts + their leniency rules | bake-cluster-local discovery + commit-pinned sources |
The compilation mechanics themselves — Roslyn invocation, release minting
({nodeTypePath}/Release/{version}), the assembly store, HasUsableBuild — are reused
as-is. This protocol changes who runs them, against which source snapshot, and how the
verdict reaches the probes.