Candidate Release Protocol

Releases are fast; deploys are gated. A new version ships as soon as it is ready. What is verified — and where — depends on who can answer the question.

Inside the mesh, a module can name its own dependents, so it verifies them: it builds itself, publishes the build as a candidate, and every dependent builds against that candidate. A clean closure promotes it to a release; a broken one publishes a preview carrying the complete list of what it broke.

The framework cannot do that, and should not try. It ships as a container image, it has no enumerable set of dependents, and holding its release until every downstream repo was rebuilt would slow the train while still answering the wrong question. The set that matters is per-instance — the plugins a given portal actually installed — so for the framework the gate sits at the deploy boundary: a portal verifies a candidate image against its own installed plugins, and refuses to adopt one they cannot compile.

This page specifies both: the states, the failure semantics, and where each gate lives.

Why — the gap it closes

The framework deleted an extension method (AddTracking on MessageHubConfiguration; tracked changes now derive from history). Three NodeTypes in a plugin still called it. Every check that exists today was green:

So the break was invisible in both repos at once, and surfaced only when a portal started on the new core: CompileError → dependents UpstreamFailed → REFUSING READINESS → the instance hub never activates → every request burns the full 60 s activation budget → hung pages and failed probes.

The pin is not the bug — it is a deliberate, correct decision (a moving :latest makes two runs of identical code disagree). The bug is that nothing re-runs a dependent's build when the thing it is pinned to moves. That is exactly what this protocol adds.

The protocol

1. Build self. The module compiles its own sources. On failure it reports the diagnostics to the requester and stops — no candidate is published.
2. Publish a candidate. On success it records the built assembly and announces the version as Requested, carrying a reference to the built code. A candidate is a real, addressable build; it is simply not the current release.
3. Dependents build against the candidate. Every direct dependent starts its own run at step 1, resolving this module to the candidate rather than to the last release. Recursively.
4. Promote only on a clean closure. If every transitive dependent reports success, the candidate becomes the release. If any fail, the candidate does not promote, and the result names every failure — the walk continues past the first one, the way a build reports all errors rather than the first.
5. A failed closure still yields a preview. The candidate is retained and published as a preview with the failure list attached, so the author can inspect and iterate without rebuilding from scratch.

State

A version is in exactly one of:

Released and Preview are both terminal and both keep their artifact. Only Released moves the pointer instances follow, so promotion is a single pointer write, not a rebuild.

Failure semantics — what "collect all errors" can and cannot mean

Breadth-complete, depth-stopped. Every sibling at a level is attempted, so one broken dependent never hides another. But a dependent of a module that produced no assembly cannot be compiled at all — reporting speculative errors for it would be fiction. It is reported as blocked, naming its blocker.

This distinction is not academic. In the incident, SocialMedia/Post, Profile and PostsHub each carried the identical .AddTracking() call. Only Post was ever reported; the other two were recorded as UpstreamFailed: blocked by SocialMedia/Post, so two of the three bugs were invisible until the first was fixed. Under this protocol all three surface in one run, because they are siblings in the dependency order — none is downstream of another.

The report distinguishes three outcomes, and a run is only clean when the failed and blocked sets are both empty:

• Compiled — built against the candidate.
• Failed — attempted, with its own diagnostics.
• Blocked — not attempted, naming the upstream that produced no assembly.

Design decisions

Cycles. Store/Coupon, Store/Order and Store/Plugin form a genuine source cycle today (they share sources through cross-type sources entries). A cycle is compiled as one unit — a single compilation containing every member's sources — and promotes or fails atomically. Rejecting cycles is not an option; they already exist and are legitimate.

Concurrent version requests. Two modules each verifying against the other's previous release can both come back clean and still be jointly broken. Candidates are therefore versioned as a set — a release train. A dependent resolves every module in the train to its candidate, and the train promotes atomically. Where trains would overlap, they serialise per dependency closure.

Cross-partition dependents. A module's dependents may live in partitions the publisher cannot read. The closure walk runs as System so it is complete, but the report is filtered to the caller: full diagnostics for paths they may read, aggregate counts for the rest. Completeness of the gate must never leak the contents of a partition.

Cost. A full cold sweep of 232 NodeTypes takes about ten minutes. This protocol rebuilds only the transitive closure of actual dependents, runs independent topological levels in parallel rather than strictly serially, and caches on (source hash, framework version, upstream candidate ids). Anything it bounds — a truncated closure, a skipped level — is logged explicitly; a silent cap reads as "verified everything" when it did not.

Where it is enforced — the gate is on the INSTANCE, not the release

Core releases immediately. It does not wait on its dependents, does not fan out to node repos, and does not need to know who depends on it. Holding a release until every downstream repo has been rebuilt would slow the release train, require cross-repo credentials, and still not answer the only question that matters — because no central set of dependents is the right set.

The right set is per-instance: the plugins THAT portal actually has installed. A NodeType broken on a plugin nobody deployed is not an outage; a NodeType broken on a plugin one portal installed is an outage for exactly that portal. Only the instance knows its own set.

So the gate moves to the deploy boundary:

Before a portal adopts a candidate image, that exact combination — the candidate image × the module versions THAT instance has installed — is built and its tests run. Every module must build and every test must be green for that very combo. If anything fails, the instance does not roll: it keeps serving the image it is on and reports what broke.

This is a deploy gate, not a release gate. A broken candidate stops at whichever instances it would break and rolls everywhere else.

Availability is a CONSISTENT sealed set, not a present file (#3175)

The first gate a candidate meets — before the combo — is ReleaseAvailabilityService: does every installed package have a sealed bake under the candidate's framework identity? Until 2026-09-03 that question was answered by PRESENCE, and memex-cloud rolled to ci.7621 on it: every bundle was there, and the portal then declined SocialMedia at adoption because its NodeTypes had been built against a MeshWeaver.Markdown.Collaboration build the same identity's sealed module set did not carry.

Available now means CONSISTENT: every dependency record inside every installed package's sealed bundle names module builds that ARE the module builds sealed for that identity, and no module is sealed twice at different builds. Anything else is SealedSetInconsistent — a hold naming the bundle, the NodeType, the module and both MVIDs — and "nothing goes" (maintainer, 2026-09-03: "we must have a clear confirmation that all plugins deployed to an instance are available for the correct platform version; if not ⇒ nothing goes"). An unreadable module set is Indeterminate, which also holds; a false "available" rolls a fleet, a false "hold" freezes it, and the two are told apart in the verdict on purpose. The mechanism and its limit — a module an instance installed from the registry outside any publication is not visible to this check — are in ModuleBuildArchitecture → "One producer per (module, framework identity)".

The unit of verification is the COMBO, and it includes TESTS

Two things are easy to get subtly wrong here, and both were wrong in earlier drafts of this page.

It is a combo, not a repo. A node repo's CI verifies that repo at HEAD against a pinned image. Neither factor matches production: an instance runs a specific image and a specific, cross-repo set of module versions, which no single repo can see. PackageManifest already pins exactly what is needed per instance — Id, Version, ModuleVersion, Source/SourceFolder, Requires — so the combo is fully identifiable from the instance's own install records.

It is build AND tests, not compilation. A NodeType that compiles can still be broken: a signature survives while its behaviour changes. "Green" means every module builds, every default area renders, and every module's Tests area passes — for that combo.

That is not a new harness. mw-plugin-test (tools/MeshWeaver.PluginTester) already does precisely this: it boots a fresh in-process mesh, installs each package, waits for every NodeType to reach a terminal CompilationStatus (printing Roslyn diagnostics on error), renders each type's default area, and executes each type's Tests layout area — a red test fails the run. Exit 0 = all green.

The combo pipeline exists, as three composable steps. mw-plugin-test takes a repo root; the gate points it at an instance's set:

1. Read — InstanceComboReader (a mesh singleton) states the instance's combo: every module with its source and pinned ref, folded from both recording shapes. Its JSON is combo.json.
2. Assemble — mw-combo-assemble (tools/MeshWeaver.ComboAssembler) materialises every module at its RECORDED ref into a repo-root layout, plus the manifest combo-assembly.json naming module → resolved ref → content hash.
3. Verify — mw-combo-verify (tools/MeshWeaver.ComboVerifier) runs the whole check as one job: assemble, then execute mw-plugin-test over that root inside the candidate image (docker run … --entrypoint /app/mw-plugin-test, the same contract as the plugins repo's test-repos CI job — never a container: job), read the tester's structured combo-gate-report.json (--report), and fold everything into one verdict. Same executable, different input.

The combo IS recorded — in two shapes, not one

An earlier revision of this page claimed an instance cannot state its combo. That was wrong, and the correction matters because it changes the first task from "add recording" to "read what is there".

Modules reach an instance by two paths, and each records its coordinate differently:

Verified live on memex-cloud — SocialMedia/_GitSync carries repositoryUrl=…/MeshWeaver.SocialMedia, branch=main, subdirectory=SocialMedia, lastSyncCommitSha=d19534d6…. That is a complete, exact combo coordinate.

The reason the mistake was easy: Plugins/* holds only _Policy on that portal — zero install records — and it is tempting to read "no install records" as "no version recorded". But ModuleDiscoveryService documents the true state plainly: "on real instances modules arrive through per-Space {Space}/_GitSync entries, not the plugin catalog: memex carries 37 sync configs and zero install records." The information was never missing; it was in the other shape.

So the first task is a READER, not a writer: one query that returns an instance's full combo — every module with its source and pinned ref — folding both shapes into one list. A reader that handles only Package records would report almost nothing on a real portal and look like a healthy empty set, which is the same false-confidence failure in a new place.

Therefore the surge pod is NOT sufficient on its own. DynamicTypePreWarmer compiles this instance's NodeTypes on the candidate image, which is the right scope — but it is compile-only and it runs after the pod is already up. It is a good last line; it is not the combo check, and it must not be mistaken for one.

Most of this already exists — and it fired too late

DynamicTypePreWarmer already computes exactly the required signal: it captures a WasHealthy baseline before baking, then refuses readiness for any NodeType that regressed on the new image —

REFUSING READINESS — N NodeType(s) regressed on this image.
The rollout will stall with the previous image still serving.

That sentence is the intent. It is also, today, a lie — and that is the whole defect.

The gate is not armed. The health check that consumes the regression state is registered only when PreWarm:GateReadiness is true. On all three portals it is false. So the sweep runs, records every regression into NodeTypeBakeGateState — and nothing reads it. The gate state is registered unconditionally, so the REFUSING READINESS line fires regardless, while the pod goes Ready and takes traffic. Anyone reading the pod log believes they were protected. That single misleading line is why the outage looked inexplicable from the logs.

Why it was switched off, and why that reason expired the next day:

The rollout shape is mostly NOT the problem. It is tempting to blame single-replica deployments; that is wrong for two of the three portals. maxSurge: 1 / maxUnavailable: 0 is surge-first — Kubernetes creates the new pod and keeps the old one serving until the new one passes its probes, and never deletes first. While the startup probe fails, readiness is suspended and the surge pod stays out of the Service. A 1-replica portal is fully protected by readiness refusal — provided the check is registered.

🚨 But the strategy is not uniform, and one portal is genuinely unsafe. Measured live:

With maxUnavailable: 1 at replicas: 1, Kubernetes may delete the only serving pod before the replacement is ready — so on that portal readiness refusal protects nothing even once the gate is armed, and any slow start is a hard outage. Arming the gate without first setting maxUnavailable: 0 there would create false confidence. Fix the strategy and the gate together.

The surge pod is the LAST line, not the gate. Adoption is not the image patch; adoption is when traffic moves, and readiness controls that — so a surge pod that never joins the Service does contain the damage. But it only compiles, it runs no module tests, and it discovers the problem by failing in production rather than before shipping. Treat it as the backstop that catches what the combo check missed, never as the check itself.

The combo check therefore runs before the image is offered to that instance at all, off-cluster, in the candidate image — not in the self-update poller. The poller patches the image and cannot know compatibility without running the candidate's assemblies: a framework-identity change invalidates the whole assembly cache by design, and UpdatePolicyContent carries no declared-compatibility metadata to evaluate instead. A check there would be a guess; the combo run is an answer.

What actually has to change

1. Arm it — PreWarm:GateReadiness = true, and replace the stale rationale with the post-974016bf4 reasoning. The paired startup budget is already correct.

2. progressDeadlineSeconds ≥ the startup budget. It is unset, so Kubernetes defaults to 600 s against a 3 h startup budget: a legitimately-baking pod reports ProgressDeadlineExceeded after ten minutes and a healthy long bake reads as a failed rollout.

3. Make the log honest. When the health check is not registered it must say "gate not armed: this pod WILL take traffic with N regressed types" — never claim a stall it cannot enforce.

4. Report the verdict where an admin looks — WIRED. A blocked upgrade used to be invisible: the admin tab showed "update available" forever, and the only evidence lived in the log and /health of a pod that never becomes Ready — the hardest place to look. The verdict now lands on Admin/UpdatePolicy: UpdatePolicyContent.ComboVerifications carries, per candidate tag, the verified-at time, the image digest it ran against, the manifest reference (module → resolved ref → content hash), and one of three verdicts — Green (every module compiles, renders and tests green), Red with the complete per-module failure list, or NotVerifiable with the caveats naming why the question could not be answered (the three are never conflated: "we could not find out" reads as neither "broken" nor "all clear"). The admin Updates settings tab joins this against the poller's LatestAvailableTag, so a red candidate renders "cannot update to X — these modules do not compile or test against it" instead of an eternal "update available". Writes go through UpdatePolicyNodeType.RecordVerification (stream.Update, upsert by tag, bounded).

5. HONOUR the verdict in the roll decision — WIRED. Recording a verdict that nothing consults would leave the settings tab showing real answers while the instance rolled regardless, which looks even more like a working gate than an empty one. ComboVerificationGate folds the verdict into the self-update decision: Red REFUSES the roll (a hold naming every failing module), Green clears it, and NotVerifiable is neither — it grants no clearance and takes no refusal, and the check verdict says the roll was taken UNVERIFIED. See Combo Gate Wiring for the full decision table, the producer/consumer split, and why "could not find out" must not fail closed here.

Running the gate for one instance × one candidate

# 1. The instance states its combo (as System — e.g. an admin execute_script on that portal):
#    hub.ServiceProvider.GetRequiredService<InstanceComboReader>().Read()  → save as combo.json
# 2. Verify the candidate against it (docker + a GitHub token for the module repos):
GITHUB_TOKEN=… mw-combo-verify combo.json meshweaver.azurecr.io/memex-portal-ai:<candidate-tag> \
    --source plugins=https://github.com/Systemorph/MeshWeaver.Plugins
# exit 0 = GREEN. Anything else: the summary names every failing module and every caveat,
# and the work root is kept for inspection.
#
# 🚨 The run is pinned to --platform, default linux/amd64 — what the fleet runs. Do NOT drop it
# to "use the local architecture": see the note below.  Verifying arm64 instead:
#     … --platform linux/arm64
# 3. Land the verdict where the instance's admins look: combo-verdict.json is a
#    ComboVerification — merge it into Admin/UpdatePolicy → content.comboVerifications
#    (upsert by candidateTag), e.g. via the meshweaver MCP: get → merge → patch.
#    🚨 This is no longer only a report: the instance's next update check CONSULTS it, so a Red
#    landed here REFUSES the roll (Doc/Architecture/ComboGateWiring).

The verify job itself holds no portal credential — reading the combo (step 1) and landing the verdict (step 3) are the operator's/CD's authenticated touches on the instance; the verification in between needs only docker, the candidate image, and read access to the module repos.

🚨 A verdict is about ONE architecture, and it has to say which. A candidate tag is a multi-arch manifest list, and the amd64 and arm64 variants carry genuinely different bytes — they resolve different framework build identities, which is why the CD bake is a matrix with one lane per architecture rather than one job with two --platform flags. Docker reports the SAME manifest list digest for both, so a run that lets docker pick the host's architecture produces a Green naming a digest that covers bytes it never executed. On an operator's arm64 laptop that is a false pass about the amd64 the fleet actually serves, and nothing in the output distinguishes it from a real one. mw-combo-verify therefore pins --platform (default linux/amd64) on every docker call and records it on the verdict as verifiedPlatform; a host that cannot run the requested platform yields NotVerifiable, never a Green about the wrong one. (Measured 2026-08-27 on mw-plugin-test:main: list digest sha256:4a63eda…, amd64 sha256:ab6efc31…, arm64 sha256:e353c397… — #2274.)

Run it where the fleet's architecture is NATIVE. The same combo against the same tag, on an arm64 host, went Green unpinned and NotVerifiable once pinned to linux/amd64 — the emulated amd64 tester died with exit 139 before it could write a report. That is the gate working: an architecture this host cannot execute is a question it could not answer, not a pass. So produce fleet verdicts on an amd64 runner (CI, or an amd64 ops box); use --platform linux/arm64 only to make a deliberate statement ABOUT arm64.

Known boundary: the sweep covers dynamic NodeTypes only. A break in a non-NodeType surface — a standalone script, a layout area — is not swept and this gate will not catch it.

The node repo's own pin stays

A node repo's Compile every NodeType (vs core) remains pinned to a fixed core digest. A plugin PR must not go red because core moved underneath it, and a moving :latest makes two runs of identical code disagree. Pin bumps stay deliberate — the instance gate is what makes an unsafe bump harmless, because a portal that cannot compile its own plugins simply never adopts it.

What this does not do

It verifies that dependents compile. It does not verify they still behave correctly; a signature that survives with changed semantics passes this gate. Behavioural compatibility remains the job of each module's own tests, run in its own repo against the candidate as part of step 3.

It also assumes the dependency graph is discoverable from declared sources. A module reaching another module's types through a path the graph cannot see is invisible to the closure walk — which is another reason sources entries are a contract, not a convenience.

Related

• NodeType Compilation & Releases — the runtime side: triggers, HasUsableBuild, framework-version freezing, where releases live.
• Deployment — how a promoted image reaches the portals.