The seventh pin — the platform SOURCE ref, and what it delays

⚠️ Superseded 2026-09-09 (MeshWeaver#3842). MW_PLATFORM_REF is no longer a pin — the env literal, the bump lane (platform-ref-bump.yml) and the staleness arm are deleted; the run's core commit is the newest SEALED set's, resolved by scripts/resolve-platform.py (PlatformResolution). The repo VARIABLE of the same name survives only as the incident freeze. The worked example below is kept for what a stale source ref costs.

Hosting/PlatformBuilds documents six platform pins and proves they are one promoted build. All six describe an image. There is a seventh, it describes the source, and nothing in that page names it:

That asymmetry is the whole subject of this page. A core fix is merged, green and released — and this repo keeps failing on the bug, because the gate that fails is compiled from a commit that predates the fix.

Status, 2026-09-06 evening. Both halves of this page are now closed for THIS repo, and the page is kept because the second half is closed nowhere else.

The worked example is closed: MW_PLATFORM_REF names 1b5350d54, which contains MeshWeaver#3408, so the FIFO reorder is no longer reachable here. Note HOW it moved — the ordinary wave bump (#1415) carried it, not anything watching for the fix. A fix waiting on an unrelated pin move is precisely the failure mode this page describes, so the example is worth reading even though its symptom is gone.

The gap is closed too: #1419 adopted node-repo-platform-ref-bump.yml as .github/workflows/platform-ref-bump.yml, so this repo now has the caller the section below prescribes. The other satellites still do not — .Education, .Reinsurance and .SocialMedia remain on the "nobody" row, which is why that section stays.

Two other things that bump revealed, both content drift rather than platform faults, recorded so the next mover expects them: two MeshWeaver.PluginCatalog.Test fixtures had to declare Format = "package-json" (core #3384 made a catalog node's source format explicit and defaulted it to node-repo), and an Edu/CourseCatalog wait ceiling of 2.5 s was too tight for a loaded bake host (#1421). Expect a pin move to surface a handful of these; that is the bump working, not failing.

The worked example: a fix that was 49 commits out of reach

ActivationBacklogFifoTest.MessageArrivingWhileActivationBacklogDrains_MustNotLeapfrogIt failed on this repo's main twice on 2026-09-06, four hours apart, with the identical permutation:

Processing order: B, C, A
Expected collection {"B", "C", "A"} to equal {"A", "B", "C"}

The same permutation twice is not a random race. A genuinely random interleaving scatters the order; a repeated one is a deterministic reorder under a condition. A — posted first and released first — ran last, both times.

PR #1401 is a diagnostics-only change to build-root wait messages whose diff cannot reach this test. It was collateral, and attributing the red to it would have been wrong twice over: wrong pull request, and wrong repository.

The reorder is not in routing

The test's because string blames "the #1145 kernel CS0103 reorder" — the routing hole where a message took the Mesh.GetHostedHub short-circuit while earlier deliveries were still queued in the per-address ActivationSerializer. That guard is intact and was never the cause here. A test can name the wrong mechanism and still be a true assertion, which is why the observed ORDER, not the because, is the evidence.

The defect was one queue further in, in MessageService.OpenGate (core MeshWeaver.Messaging.Hub). When the last initialization gate opened, it drained the deferred queue by appending:

while (deferredQueue.Count > 0)
    mainQueue.Enqueue(deferredQueue.Dequeue());

under a comment promising the deferred turns run "before any message that arrives after the gate opens". That is only the easy half. A message that arrived before the open and was still sitting un-turned in mainQueue — because the loop was busy — was already ahead of the deferred turns being put behind it. The parked message then ran last.

Appending is always the wrong end, not merely unlucky. Deferral happens at turn time, not at arrival: a message is dequeued, found on-target with a gate closed, and pushed onto deferredQueue. The turn loop is strictly FIFO, so a message that has not been turned yet cannot have arrived first — everything deferred is by construction older than everything still waiting. The fix (MeshWeaver#3408, core c8c7dd327) rebuilds the main queue as deferred-then-waiting, each run keeping its own order, which is total arrival order across the two queues.

This is exactly why the symptom was load-sensitive on CI and green in isolation: it needs the turn loop to be busy across the gate open, which a saturated shard produces and an idle single-test run does not.

The measurement, both ways

Core's DeferredTurnsResumeAheadOfLaterArrivalsTest forces that window structurally — the turn loop is FIFO, and a handler posting to its own hub enqueues behind its own turn — so no delay, poll or sleep is involved.

The five reverted runs all reproduce the field permutation, which is what ties the core defect to this repo's red rather than merely to a plausible story.

🚨 This repo's own test is not a local discriminator. ActivationBacklogFifoTest passes in isolation with or without the fix — #1394 records 5/5 local passes, and a full 805-test single-process run of 803 passed / 0 failed, while main was failing. The discriminating repro is core's, and it had to be, because the condition is a busy turn loop. A test that only fails under load will never falsify anything on a developer's machine; treating its local pass as evidence is the isolated-run trap PortalHostSuites catalogues.

Why the source ref moves ALONE

scripts/check-platform-pins.py states it directly, in the fifth invariant:

There are three core-ish pins here — the uses: …@sha workflow DEFINITION, MW_PLATFORM_REF the tested COMMIT, and MW_IMAGE_DIGEST/MW_PLATFORM_SET the compiled-against IMAGE — all of which may legitimately disagree; #1268 moved two of them together and went red, and its landed fix splits them.

So the "a pin moves by grepping its OLD VALUE" rule in PlatformBuilds is about the image set. It does not reach this one, and applying it here would drag the sealed identity along for no reason. Name what a source-ref move actually reaches:

Reached — portal-hosts ($(MeshWeaverRoot) for src/ and the moved suites); the type-forward gate's canonical surface list; the public template's other half; the gRPC-web protos; and the two node-repo-module-pack calls' platform-ref, which is the pack tool's checkout and one input to the content-addressed build key (so modules rebuild once).

Not reached — the module reference set. node-repo-module-pack.yml restores it from the portal image's /app under platform-refs-<platform-image-digest> and binds with --bind-to-image, so the API surface every module compiles against, and the identity its bundle is sealed under, are the image's. Moving the source ref alone changes no bundle identity and forces no upstream re-seed — which is what makes it a private decision, where an image-set bump explicitly is not.

🚨 Two platform-ref: literals in ci.yml — on the node-repo-gate and node-repo-publish-bake calls — are not copies of MW_PLATFORM_REF. They are copies of the lane's own uses: sha ("the central scripts are fetched at the lane's own sha"), and must stay equal to it. Grepping the old pin value and replacing every hit would silently decouple each lane from its own scripts.

The gap: a bump lane almost nobody calls

Core ships .github/workflows/node-repo-platform-ref-bump.yml, a workflow_call lane that opens a pull request moving MW_PLATFORM_REF on a schedule. Its own rationale describes this incident in advance:

On 2026-08-18 MeshWeaver.SocialMedia's pin sat days behind while a module fix landed on a later platform commit — which meant the fix could be merged, green, and still never reach the shipped bundle, because the bundle is built against the pin.

Measured 2026-09-06, morning: no satellite called it. Not MeshWeaver.Plugins, not .Education, not .Reinsurance, not .SocialMedia. The image set has an automated mover (PlatformPinUpdater, driven by the CD webhook this module hosts); the source ref had a lane that existed and was wired to nothing.

That evening this repo adopted it (#1419, .github/workflows/platform-ref-bump.yml). The other three have not, so the paragraphs below are written for them — and for this repo's own record of why the caller exists.

That is the difference between the two pins in this page's opening table, and it is the reason a fix merged into core at 13:06:47Z was still failing this repo's main at 13:31:03Z: the pin was bbcb22f25 (05:43:18Z), 49 commits before the fix. Nothing was broken, nothing was mis-measured, and no gate was wrong. The fix simply had no route in.

Adopting it is a caller contract, not a fork

name: Bump the platform pin
on:
  schedule: [{cron: "23 5 * * *"}]
  repository_dispatch: {types: [meshweaver-framework-released]}
  workflow_dispatch:
jobs:
  bump:
    permissions: {contents: write, pull-requests: write}
    uses: Systemorph/MeshWeaver/.github/workflows/node-repo-platform-ref-bump.yml@<sha>

It opens a pull request and never pushes to main, which is the point of keeping a pin at all: the bump stays a reviewable commit the full gate suite runs on, so a platform regression arrives as a red pull request rather than as a silently-changed build input. Whether this repo adopts it is a maintainer decision — the cost is one more pull request a day to triage, and the thing it buys is that "the fix cannot reach us" stops being a state nobody is watching.

Checking a bump, both ends

The rule in ci.yml is check both ends, and it bites in opposite directions — too old and this repo's own code stops compiling; too new and an unreconciled core contract change arrives. For a source-ref move that means, at the candidate ref and against this tree:

# 1. the hosts the required context builds, Release + warnings-as-errors
python3 scripts/build-stream.py --msbuild "MeshWeaverRoot=$CORE/" \
  src/Memex.Portal.Gui src/Memex.Portal.Monolith src/Memex.Portal.Distributed

# 2. the suite that was failing
dotnet build src/MeshWeaver.Hosting.Monolith.Test/MeshWeaver.Hosting.Monolith.Test.csproj \
  -c Release -warnaserror -p:MeshWeaverRoot=$CORE/
dotnet test  src/MeshWeaver.Hosting.Monolith.Test/MeshWeaver.Hosting.Monolith.Test.csproj \
  --no-build -c Release -p:MeshWeaverRoot=$CORE/ --filter FullyQualifiedName~ActivationBacklogFifoTest

# 3. both frozen inputs still inside the staleness bounds
python3 scripts/check-platform-pins.py                    # copies agree, every checkout names a ref
python3 scripts/check-platform-pins.py --check-staleness --ref <candidate>

--check-staleness measures both frozen inputs against core main under the same two bounds (24 h, 120 commits). Reading it after a source-ref move is not a formality: on 2026-09-06 it reported the source ref at 2.9 h / 53 commits and, in the same breath, MW_PLATFORM_SET at 10.3 h / 102 commits of a 120 bound — the image set is the input approaching its wall, and Validate node repos is a REQUIRED check, so that arm reds every open pull request in this repo when it crosses. A source-ref bump does not relieve it.

🚨 The staleness gate was GREEN throughout, and that is the lesson

This is the part worth carrying away, because it generalises past pins.

Through the whole incident above, --check-staleness passed. bbcb22f25 was 10.3 h and 102 commits behind core main — inside both bounds, comfortably. Nothing was red. Nothing was mis-configured. No gate was wrong or bypassed or skipped. And the fix still could not reach the repository it had been merged for.

"Roughly current" is not "carries the fix we merged for this repo's red."

A staleness bound answers "is this input drifting?" — a question about distance, asked continuously and cheaply, with no knowledge of content. It cannot answer "does this input contain commit X?", which is a question about membership, and the only question that mattered here. The two are not approximations of each other: a pin one hour and three commits old still fails the membership question if the fix is the fourth commit.

So a green staleness arm must never be read as "the pin is fine". When a specific core change is what unblocks this repo, the check is membership and it is one command:

git merge-base --is-ancestor <fix-sha> <candidate-pin> && echo "pin carries it"

The corollary for triage: when a failure here is fixed in core, the very next question is which pin the failing job compiles against and whether that pin contains the fix — before any theory about the failure being intermittent. On 2026-09-06 that question, asked at 13:31Z, would have answered itself in one command; instead the red read as a recurring flake for another four hours.

🚨 The same commit can be green at 07:58 and red at 09:32 — the gate is a function of the CLOCK

The section above is the gate reading green while the pin is useless. This is the other direction, and it looks like flakiness in a way the first one does not.

Measured on Plugins#1507, 2026-09-08, one unchanged head (99227455):

Nothing was pushed between them. The branch did not move, the pin did not move, main did not move under it. Core's trunk moved, and the commit bound is measured against core's trunk at the moment the job runs. The branch crossed from 119 to 121 commits behind while sitting still.

Two things follow, and both are easy to get wrong under time pressure:

1. "It was green an hour ago on this exact commit" is not evidence of a flake. It is the expected behaviour of a bound whose other operand is somebody else's moving trunk. Re-running will not recover it and will burn a full matrix; the bound only un-trips when the pin moves. Attributing it to flakiness sends the reader hunting a race that does not exist — the same category error the section above describes, arriving from the opposite side.
2. It reds every open pull request at once, and a bump on main clears none of them. The gate reads the pull request's own ci.yml, so each branch has to take the merge before its own copy of the pin is fresh. A repo-wide red with one cause looks, on any single PR, exactly like that PR's problem.

The tell that separates this from a real red is in the message itself: it names core's head and a distance, not anything in the diff. And it arrives with company — every gate that needs: the validate job fails too, saying so explicitly (validate=failure … failing RED instead of skipping, issue #890). Three red contexts, one cause, none of them the change under review.