The instance-action dispatch seam, and how it is measured

Hosting/InstanceAction is the control instance's whole action surface. Two of its kinds do not launch an in-cluster operator Job at all — they dispatch a workflow in the deployment's own config repository and follow the run:

🚨 They are NOT the same shape, and this page is careful about which one it is describing. Both POST a workflow_dispatch, persist the run GitHub named, and then poll that one run until it concludes, folding every line the follower emits onto the request node — that much is shared, and HelmReleaseRunner.Follow literally is the one follower. What is InfraDeploy's alone:

• Only InfraDeploy consumes an approval. A helm release is gated upstream of the phase, by InstanceActionContent.Refusal() requiring the request to name the instance, and there is no ConsumeApproval anywhere in RunHelmRelease. (The workflow's own confirm INPUT is a different thing again: the record's helmRelease, which the runner sends and never guesses.)
• Only InfraDeploy reads the workflow before dispatching it, and reads the estate's namePrefix out of the parameters file that workflow names.

The helm release used to record nothing (Plugins#1975). HelmReleaseRunner.Run emitted bare string lines, so RunHelmRelease had no id to stamp and the node ended with no runId/dispatchedAt/observedUntil. AdoptOrphan routes on Executor == "Actions", RunId or DispatchedAt, so an orphaned release carried none of the three, fell into the operator-Job branch, found no jobName and answered "the outcome cannot be recovered from here — take a Sample of the deployment" — honest, and a run GitHub was still holding, named by nothing but a prose line in the node's log. Both kinds now write the same record through the same helper, and the branch's own wording says workflow run rather than aks-ops run, because three kinds reach it.

The order IS the safety property

The defect that was reviewed out, not measured out. Plugins#1935 originally consumed the approval after the dispatch. A dispatch that succeeded and a stamp that then failed would have left an approval that still looked unspent on a node — and an approval is credential-equivalent: it is what lets a run mint an admin kubeconfig and change a client's estate. A second run could have started from it.

RunInfraDeploy's shipped order is therefore:

1. consume the approval, in its own write, before anything is sent — and therefore before the workflow is even read, so a GitHub-side refusal costs it too. The refusals that cost nothing are the LOCAL ones (no config repository, no infraMode), answered before the approval is touched;
2. dispatch — InfraDeployRunner.Dispatch, which reads the workflow, POSTs, and answers ONCE with the run GitHub named. It does not poll;
3. persist the run — id, URL, dispatch instant and observation bound — in the same write as the dispatch line, so a control process that dies during a two-hour deploy leaves a node its successor re-observes (AdoptOrphan routes on RunId/DispatchedAt);
4. follow to a verdict, and fail the node naming the conclusion and the run.

RunHelmRelease is the same from step 2 on, through the same two helpers.

Consumed-then-failed costs a fresh approval. Dispatched-then-unconsumed can start a second run. Single-use fails closed, and that is a property of the ORDER of two writes — which is exactly what a test calling ConsumeApproval directly cannot see. It passes either way.

Step 3 happens-before step 4, and that is structural

🚨 It did not used to. Both consumers folded each progress line with SelectMany, which merges rather than serialises: the runner's own stream carried straight on from the dispatch line into the follow's first poll while the dispatch line's write was still landing, and an instant follow failure would have disposed a write still in flight — leaving the dispatch line on the node with no runId, which is exactly the state AdoptOrphan cannot re-observe. In practice the local workspace write beat the HTTP call every time anyone measured; "in practice" is not a guarantee (Plugins#1976).

The fix is a SPLIT, not an operator swap — a .Concat() inside the old fold would not have helped, because the source still ran ahead and still disposed on error. Each runner now offers two calls:

and the control plane composes them through ONE named operator:

InfraDeployRunner.Dispatch(hub, repo, mode, subscription, token, send)
    .SelectMany(dispatched => FollowOncePersisted(
        PersistDispatch(workspace, path, run, dispatched, InfraDeployRunner.RunTimeout),
        () => FoldFollow(workspace, path, run,
            InfraDeployRunner.Follow(hub, repo, dispatched.RunId, token, send))));

internal static IObservable<Unit> FollowOncePersisted(
    IObservable<Unit> persist, Func<IObservable<Unit>> follow) =>
    persist.Concat(Observable.Defer(follow));

Concat subscribes to the follow only when the persist has completed, so the run is on the node before the first poll is issued, and there is no in-flight write for a follow failure to dispose. The preparation line travels in Dispatched.Lines with the dispatch line, so the node's record of a dispatch is ONE write rather than a sequence a process death can land in the middle of.

Scope of the claim. It is about the DISPATCH line — the write that makes the run recoverable. The follower's own status lines are still folded with SelectMany, deliberately: each is an independent WithOutput append, the polls are 15 s (helm) / 30 s (infra) apart, and serialising them would queue writes behind one another on a source that does not wait. Nothing downstream of the dispatch line is load-bearing for adoption.

🚨 The operator is named, and shared, so that the property has somewhere to be asserted. The only experiment that tells the ordered shape from the racing one is a persist that has not completed — a state no live case can arrange, because in a live case the write always does complete. That is why #1976 was filed rather than asserted on the live seam, and why the ordering is pinned on the operator (below) while the live cases pin that the seam is built out of it.

What is doubled, and what is refused

Only the GitHub edge: the installation token and the one HTTP send, as parameters that default to the real ones.

public delegate Task<HttpResponseMessage> GitHubSend(HttpRequestMessage request, CancellationToken ct);

public static IObservable<HelmReleaseRunner.Dispatched> Dispatch(
    IMessageHub hub, string repoSlug, string mode, string? subscription,
    Func<IObservable<string>>? token = null, HelmReleaseRunner.GitHubSend? send = null)

public static IObservable<string> Follow(
    IMessageHub hub, string repoSlug, long runId,
    Func<IObservable<string>>? token = null, HelmReleaseRunner.GitHubSend? send = null)

It is a parameter, never a settable static hook: a process-wide hook would be static mutable state that outlives the mesh and bleeds between cases. It sits at the same place the real send sits — inside IIoPool, at the async leaf — so injecting it changes nothing about how the runner runs.

🚨 There is no workspace double. The cases run against the live mesh's own hub, workspace, IO pool and serializer, and stamp a real Hosting/InstanceAction node they create and delete. A workspace double is precisely the thing that would let a broken write order pass: Stamp reads the node's content through ContentAs and silently no-ops when that read degrades to null, so the only honest assertion is a positive one, read back off the node.

Two consequences the cases are written around:

• A Take(1) straight after a write is answered from the PRE-write snapshot (Plugins#1930, #1931). Every positive assertion goes through a Where(reached).Take(1).Timeout(...), and the negatives are asserted on that same emission — by which point the run has terminated, so nothing is left to write.
• The probe node must be inert. The live control-plane watcher is wired to this very NodeType. A probe request it picked up would dispatch a REAL workflow against a real client's repository, so the seeded content names no deployment and sits at Done: ShouldRun() is false for every state a case can leave it in, and AdoptOrphan only fires on Running.

The transitions, and the denominator

The seam has 19 distinct transitions from the phase's entry to its terminal answer.

19 of 19, by thirteen cases in Hosting/InstanceAction/Test/DispatchSeamTests.cs — twelve in the Tests area's LIVE half, and transition 19's second half in the PURE half, for the reason below.

🚨 Two boundaries the table does not cross, stated so nobody reads more into it. Transition 9's first half is pinned as a line the follower emits reaches the node — no case would catch a consumer that invented an unchanged line of its own, because the seam's poll period is a constant and a case may not wait 15 seconds on one. And transition 19 is pinned in two pieces — the dispatch cannot poll (live, on both runners) and the operator does not subscribe the follow until the persist completed (pure, on FollowOncePersisted) — which together mean no poll can precede the write. What no case here would catch is a future call site that composed those two halves with Merge INSTEAD of the operator: the operator's own case would stay green. That is why there is one named operator and both seams go through it, and it is the reason to leave it that way.

Why each case can fail

A case that cannot fail is worse than no case, so each one's instrument is a state in which the broken shape and the shipped shape give DIFFERENT answers — never merely "the run completed":

• Consumed before the POST is measured on a dispatch that FAILS. With the consume before the POST the node ends with the approval spent; with the consume after it — the shape #1935 was reviewed out of — the same failed dispatch leaves the approval intact. The case additionally asserts the POST was attempted, because a consume that happened while nothing was ever dispatched would prove nothing about the order.
• Persisted on the dispatch line is measured on a run that then CONCLUDES failure. The id is on the node anyway, and the node's own log puts the bound-recording line immediately after the dispatch line — a persistence deferred to the end of the follow could not pass either check.
• A failed run fails the node runs the control plane's own composition (ProvisionPlan.Run → InstanceActionControlPlane.Fold) and asserts Failed and not Unobserved: nobody's observation was lost, so the generous direction (#3961) is a failure here.
• A refusal costs nothing reads a negative — no GitHub call, no approval spent — and is only trustworthy because the same fake records every call the other cases make.
• The 401 recovery answers the first poll 401 and asserts the next poll carries a different, newly minted token. A follower that minted once at dispatch would 401 for ever on a deploy longer than an hour, which is the review finding on #1935.
• A helm release persists its run uses the same instrument as the infra case — a run that then concludes failure — and additionally asserts the bound is the HELM follower's 30 minutes, not the infra deploy's 150: a persistence that reached for the wrong runner's constant would pass a "there is a bound" check and be wrong by two hours.
• The dispatch half never polls is a count, not a race: Dispatch emits once and completes, and nothing inside it can poll, so the number of run polls at that moment is stable. A follower moved back inside the dispatch is a non-zero count however the timing fell.
• The follow is not subscribed until the persist completed holds the persist OPEN and asserts the follow has not been subscribed. Under Merge — the shape that shipped — it is subscribed at once and the case reds on its first assertion; under Concat it is subscribed on the persist's completion and not before. The second half asserts the persist's completion is DELIVERED even when the follow fails instantly, which under Merge it is not: the error arrives first and disposes it.

…and each one was WATCHED failing

Every case was run against a seam broken on purpose, and the row it owns went red for its OWN stated reason. The instrument is mesh-test's gate, which renders the table per NodeType and prints it for a RED one; across all seven broken runs Hosting/InstanceAction was the only Tests area that failed and the area's other cases stayed green, so nothing below is collateral.

The first four broken runs scored 255/258, 255/258, 256/258 and 255/260 against a tree that then scored 260/260. The three above each scored 262/263, with Hosting the only failing package; the restored tree scores 263/263 and the whole-repo gate is 98 node types green.

🚨 The third break is the one to read carefully. Under Merge — the shape that shipped — the live cases are all still green: the run still reaches the node, the follow still reaches its verdict, the log order still holds. Only the case that holds the persist open can tell the difference. A reader tempted to replace that case with "a live one that really exercises the seam" would be removing the only instrument there is.

🚨 The breaks are applied by exact string replacement that ABORTS unless the anchor matches exactly once — a break that silently did not land would turn "the case passed" into evidence of nothing, which is the same defect class the cases exist to prevent.

Running them

The live half needs a mesh, so it runs under the mesh gate rather than the compile cascade:

python3 scripts/mesh-test.py Hosting   # compiles; reports the live cases as `needs-mesh`
python3 scripts/mesh-test.py --gate    # boots a mesh and renders every Tests area, these included

See also

• Hosting/ObserverExpiry.md — the three-answer rule (success / failure / unobserved) the follower and the adoption both obey.
• Hosting/AksOperationsViaActions.md — the wider design these two kinds are the config-repo half of.