Image Pair Skew

A portal image is two commits from two repositories, and continuous delivery pairs them at different moments. The core half is the commit the run was created for; the Plugins half is refs/heads/main of MeshWeaver.Plugins at the moment the run's gate job finally ran. When the run queues for hours — it did — the two halves can be hours apart. Each half was green in its own repository, against its own pin. Nobody ever ran the pair.

On 2026-09-03 that pair was core e7f1d699 (08:05Z) with Plugins 12500c9 (13:31Z), tagged memex-portal-ai:3.0.0-rc9.ci.7658, and it answered 503 to every signed-in request on memex.systemorph.com for twenty minutes. This page records the property, the timeline, why every guard missed it, the runtime fix that shipped, and the delivery gap deliberately left open.

🚨 The one sentence

Promote and Verify every image shipped green means the images EXIST; it says nothing about whether the pair inside them can serve a signed-in request. Only booting the pair proves that, and nothing in CD does. That remains true as this is written.

The property

Threading the Plugins sha once (#2622) made a run internally consistent — the portal image and the module bundles come from one Plugins commit. It did not make the run externally consistent: a core commit that queued for five hours is paired with a Plugins main five hours younger than anything that core commit was ever tested with. The Plugins pin moves the other way too — the Plugins repo pins a platform sha, and on the same afternoon its own tests went red in both directions as that pin crossed the same boundary (Plugins #1281 asserts both arms by detecting the platform at runtime).

The class is: a promoted set whose two halves encode different contracts. Sign-in is where it surfaced, because sign-in is the first read every request makes.

🚨 The sharper form: what is TESTED and what is SHIPPED read different refs

Measured in MeshWeaver.Plugins on 2026-09-03, and this is the durable defect rather than the queue delay:

So the Plugins repo tests against a frozen core and ships against a moving one. The two agree only while the pin happens to equal the tip. Every hour the pin lags, the tested pair and the shipped pair are different pairs — and no run anywhere reports the difference, because each lane is internally consistent and green. Core's own CD adds the mirror-image gap: it pins its core half at run creation and resolves the Plugins half when gate runs.

A pin is the right tool for a test lane: it makes a suite reproducible. The defect is that nothing compares the pin to what the image lane actually used, and nothing executes the pair the image lane produced. A stricter pin does not close it; only executing the pair the image lane built does.

The timeline (UTC, 2026-09-03)

The roll cleared more than sign-in. On the sister portal, measured across the same change, the cross-schema fan-out storm went from 1,917 slow 201-schema unions per pod per 30 minutes to 0–1 per pod per 10 minutes — the anchored reads of #3206 removing the unions themselves, not merely the refusals of them.

ci.7693 is up but unsealed: its Plugins: bake + seal job failed on the #3175 one-producer guard (MeshWeaver.Markdown.Collaboration composed as a module and still shipped in /app), as did every promoted build since #7683. NodeTypes binding module types decline at adoption on it. The first sealable set is the first CD run after Plugins #1268 lands.

Why every guard missed it

• Plugins CI was green — its tests ran against the platform pin, and the pin was on one side of the boundary or the other, never both. #1281 fixed the tests to assert both arms; that fixes the tests, not the image.

  🚨 But the class DID announce itself, hours earlier and for free, in a PR check. Two access suites in the plugin repository asserted the pre-#3206 query literals and reddened trunk the moment the pin resolved core's tip. Replacing them with the post-#3206 literals then failed in the OPPOSITE direction, because the pin had meanwhile been set to a commit that predates #3206. That mirror-image red — the same test failing both ways within a day — is the cheapest diagnostic signature this class has, and it appears in a check long before it appears as a 503. Read as "our tests are wrong" it produces a literal-chasing fix; read as "the pin moves both ways" it names the boundary. The shape a test must take to survive a pin that moves both ways is its own subject — see Pin-Boundary Contracts.

• Core CI was green — UnanchoredQueryCensusTest and SignInReadsAreAnchoredTest (#3206) judge core's current callers with the current planner rules. They cannot see an older core paired with a newer planner, because that pair exists only in ACR.

• The grace list could not protect the window by construction. It is regenerated from the census of callers still issuing an unanchored shape and fails the build when a listed shape has no caller. The moment core anchored the sign-in read, the list had to drop it — while every image built from an older core half still issued it.

• Promote and Verify every image shipped were green. They prove existence and provenance, not behaviour. The seal was red, but for an unrelated reason (#3175), and nothing surfaces the seal at the tag.

• The front door lied twice. During the crash loop the old ReplicaSet answered 200; after the cutover the anonymous shell answered 200. The failing leg needed a signed-in request. /api/og and /healthz are negative controls here, exactly as The /api/content 503 already records for its own leg.

The fix that shipped, and the one that did not

Runtime — a refusal is a CI invariant, never a production answer (MeshWeaver.Plugins #1300). PostgreSqlPartitionedMeshQuery gains an UnanchoredQueryPolicy. The default is Refuse (fail-closed: a host that never heard of the property keeps the CI invariant). The production host opts into ServeAndReport in its own committed appsettings.json, baked into the image so no chart or ConfigMap can forget it, and a contract test discovers the Postgres hosts from source so the rule cannot silently cover nothing. Under ServeAndReport an unlisted unanchored shape is served by the fan-out and logged at Error naming the offender, which the red-log ticketing turns into an incident. The trade this reverses: a hard refusal converted a performance hazard (a cross-schema UNION contending on the lock manager) into a total availability outage on the request path. After the change the allow-file governs two different things: in CI the Grace-versus-Refuse verdict, unchanged; in production the log level (listed → Warning, unlisted → Error). It may still only shrink.

So a repeat of THIS fault is caught at the layer where it fired. An image whose two halves disagree about anchoring no longer takes sign-in down; it serves the query and files an incident.

🚨 What is still open: nothing executes the pair before it is promoted

A CD gate that booted the run's own image pair, signed in through DevLogin and asserted a cookie-carrying GET / and GET /Store was written and proven against both controls — it passes the pair that shipped fixed and fails ci.7658 on the signed-in GET / — and was deliberately not adopted: a new required job in front of promote was judged not worth the standing cost and the first-run risk to the release lane. That decision is recorded here rather than argued: it is a real trade, and the residual it leaves is real too.

The residual, stated plainly. promote and Verify every image shipped still attest only that the images EXIST. No step boots the pair, and none signs in. The pair's two halves are still chosen at different moments from different refs, so a different disagreement — one the runtime policy above does not cover, because it is not about unanchored queries — would ship exactly as this one did, and the first thing to notice would again be production. Anyone reopening this should know the proof already exists as a standalone script with a two-control verification; it is the adoption that was declined, not the mechanism.

Reading the signals next time

kubectl logs <pod> -n memex -c memex-portal --since=20m | grep -c 'UNAVAILABLE for'
kubectl logs <pod> -n memex -c memex-portal --since=20m | grep -oE "no partition could be determined from '[^']{0,100}" | sort | uniq -c
az acr manifest list-metadata -r meshweaver -n memex-portal-ai --orderby time_desc --top 30 -o json   # tags read <core>-p<plugins>
git merge-base --is-ancestor <fix sha> <core half>                                                     # is the fix in this image's half?

• 🚨 Filter the 503 count BY CAUSE or you will over-attribute it. UNAVAILABLE for with UnanchoredQueryException is this class. UNAVAILABLE for with DeliveryFailureException is an unrelated transient transport fault — and the older pods, running before this outage began, logged 23 of those. An unfiltered grep "UNAVAILABLE for" therefore hands you pre-incident noise as incident evidence; the cause-filtered query is the one quoted above.
• A pending surge pod on Insufficient cpu with the pool at its maximum: look for a Terminating pod still holding a node. 🚨 It is very likely NOT wedged — the chart's preStop hook blocks for up to 28 minutes draining Blazor circuits, and the Killing event marks the START of preStop, not SIGTERM. Such a pod is out of the Service but running normally by design. Force-deleting it frees the node immediately at the cost of cutting those sessions off, which is a decision to take knowingly rather than a wedge to clear.

Recorded, not fixed

• A "terminating" portal pod at 5,705 millicores — DETERMINED, and it is the chart working as designed. The pod was not past SIGTERM, not deadlocked, and not failing to drain. It had not been signalled at all.

  deploy/helm/templates/memex-portal/deployment.yaml gives the portal a preStop hook that polls /drain and BLOCKS until this pod's last Blazor circuit closes, bounded at drainSeconds − shutdownMarginSeconds (1800 − 120 = 1680 s ≈ 28 min). Kubernetes delivers SIGTERM only when preStop returns. So for up to half an hour a "Terminating" pod is out of the Service but running completely normally — no ApplicationStopping, no dispose, still serving every circuit it already had. That is deliberate: it is what stops a roll cutting people off mid-task, and it is why terminationGracePeriodSeconds is 1800 rather than the default 30.

  🚨 The trap, and the reason this took three attempts to read: the Killing event marks the START of preStop, NOT SIGTERM. Every instinct says otherwise. Read against that, the specimen is unremarkable in every particular — ready=true, ordinary application work in the log, zero shutdown-shaped lines in 25 minutes, and high CPU — because nothing had asked it to stop yet. Its sibling on the same image sat at 238 m simply because it held fewer circuits. The chart's own comment already records an earlier round of exactly this confusion (memex, 2026-08-21: "a terminating pod still executing application code 68 s before the ceiling").

  What the CPU number is actually worth keeping for: 5,705 m and 13 GB is the pre-anchoring fan-out storm, measured on a pod still serving its circuits — the same load the roll to ci.7693 cleared, and the reason the sister portal's slow cross-schema unions fell from 1,917 per pod per 30 minutes to 0–1 per pod per 10 minutes. It is evidence about the query fan-out, not about shutdown.

  Two earlier readings on this page were wrong and are withdrawn. "A dispose deadlock caused by the absence of core 71a41b231" — falsified by the sibling on the same image. "Still serving after SIGTERM, so the shutdown path is not quiescing intake" — wrong in its premise, because SIGTERM had not been sent. Both were built by reasoning from the pod's phase instead of reading what the chart does.

  🚨 A tooling note that outlives the specimen: kubectl debug cannot attach an ephemeral container to a pod that already carries a deletionTimestamp — the API accepts the command, prints its "Targeting container" line, and creates nothing, silently (ephemeralContainerStatuses stayed empty on both attempts). So the published recipe for profiling a live portal pod does not work on a draining one. Attach the probe before the delete is requested, or go in from the node.

• Nothing executes the image pair before it is promoted — the largest open item, and a deliberate decision rather than an oversight. See What is still open above.

• The silos node pool has no headroom for one surge pod when both portals are at full replica count. A rollout that must surge waits for a termination, and a wedged termination waits for the grace period.

• The self-updater cannot cross a schema bump on its own (the migration is a helm-revision Job) — The Self-Update Schema Wall.

See also

• Unanchored Security Reads — why the sign-in fold reads the way it does, and the census this outage was the runtime twin of.
• The Cross-Repo Pair Gate — the same class at the source level: two halves that must land in an order, and the gate that holds the deleting half last.
• The Continuous Delivery Contract — all-or-nothing publication; verify the image, never the tick.
• The Self-Update Schema Wall — why the roll to ci.7658 wedged silently for three and a half hours before it could fail loudly.
• Reading CI Signals — which greens mean what.
• Pin-Boundary Contracts — the same boundary seen from the TEST side: what shape a test must take to survive a pin that moves both ways.
• Dead-Circuit Fan-Out Storm — the refusal verdict that made the #2426 eviction unreachable; the storm this outage's roll also cleared.
• Database Migration Procedure — the schema moves before the image, which is the wall this incident's rollout spent three hours behind.