When an instance action outlives its observer

An Hosting/InstanceAction run reports itself on its own node: InstanceActionControlPlane watches the node, runs the phases, and folds each phase's progress onto it. That reporting chain is an in-process reactive subscription. The operation it reports on is not — it is a Kubernetes Job, and it keeps going when the process that launched it goes away.

The gap, in one sentence. A run whose observer died read exactly like a run in progress: state: Running, phase: Run operator job, message: ⏳ Step 5/8 — Run operator job…, no finishedAt — and there was nothing on the node to compare the elapsed time against.

The measurement (control instance, read-only, 2026-09-10)

Deployments/memex-restart-20260910-1140-activate-ai16 — a Restart of memex, requested at 11:33:56Z, node version 20:

🚨 The rollout it was watching SUCCEEDED. Deployments/memex-sample-20260910-1207-settled (12:05:44Z) reports 2/2 ready, 0 restart(s), running meshweaver.azurecr.io/memex-portal-ai:3.0.0-ci.8238. The operation completed and the record never learned it had.

The action's own plan had predicted a different mechanism, and written the warning into its log:

⚠️ The deployment's rollout budget exceeds the 3000-second observation window of one operator job. The job may stop waiting before the rollout is complete. Check the rollout status afterwards.

That window would have expired around 12:24Z. It never got the chance to matter.

Three facts about the code, and only the third is surprising

1. The observer's lifetime is the process's. Process → ProvisionPlan.Run → Track is one subscription held by the per-node hub. Kill the process and the chain is gone, mid-phase, with no write.

2. Nothing ever revisits a Running node. InstanceActionContent.ShouldRun() deliberately excludes Running — the watcher writes to the node it watches, and that exclusion is what stops it scheduling itself. Correct, and it also means an orphaned run is orphaned for ever.

🚨 3. The operator Job's NAME was never written to the node. run.JobName lived exclusively in the watcher's in-process RunState. The content type declared a JobName property and the page rendered an "Operator job" row from it, and nothing ever populated either. So a run whose process went away was not merely unobserved — it was unobservABLE: the one identifier needed to go and ask Kubernetes what happened died with the observer.

And for one action the loss is by construction

A Restart or Roll of the deployment that hosts the control plane rolls away the pods running the control plane. The control instance can never observe its own restart to completion. That is not a race, a slow day or a flake — it is an identity: the observer is inside the thing being replaced.

What the control plane records and does now

A bound, stamped when the job is launched. HostingOperator.ObservationEnd(settings, launchedAt) — the Job's own activeDeadlineSeconds plus the poll's slack — is written to the node as observedUntil, alongside jobName. One definition, two readers: the same function produces Track's own .Timeout(...). A second, separately-derived number would be free to disagree with the observer it claims to describe, which is this bug one level up.

A third state, Unobserved. Not Running (nobody is watching) and not Failed (nothing is known to have broken). Error stays null and FinishedAt stays unset, both deliberately: the only thing that happened is that observation stopped. The action is cleared, so nothing loops — and a person may re-request it, which is the only route back, on purpose.

Adoption on the first emission. A node this hub finds already Running on its very first emission cannot be this process's run, because this hub has only just come into existence. So the watcher re-observes the job the node names:

A derived verdict where nothing has had the chance to record one. The fleet run log (Runs) comes from a query: most of those nodes' hubs are not up, so nothing has adopted them. OutcomeNotMeasured(now) derives the same verdict from observedUntil alone, and the page collapses both sources into ONE sentence — a reader must never have to tell "a process recorded this" from "I worked it out" apart.

An observation that ends without a verdict is no longer reported as a failure. Track's timeout used to emit a terminal failed event saying the job "did not finish within its deadline". The Job carries activeDeadlineSeconds, so a job still alive at that point is impossible: reaching it means the API could not be read, or the object went away. That is now OperatorEvent.Unobserved, and InstanceActionControlPlane.Fold turns the resulting phase failure into Unobserved rather than Failed.

What this deliberately is NOT

• 🚨 Not a watchdog that re-posts the request. Re-running a Restart whose rollout already succeeded restarts a healthy deployment. Adoption re-observes the job that was already launched; it never re-runs anything, and it never invents a step.
• Not a longer observation window. The operation is not slow. The observer is short-lived, and a bigger number would only move the cliff.
• Not a poller, and not a catch. Adoption is demand-driven — it happens when the node's hub next comes up, which is when someone reads the node, which is when it matters. Every containment arm reports: a verdict that could not be written logs at Error naming the consequence, because at that point the log is the only place the information exists.
• Never generous. A terminal verdict comes only from the Job's own status. Getting this wrong in the forgiving direction would retire a record on a teardown nobody watched.

What is still open

• The operator job's own rollout window. kubectl rollout status … --timeout=3000s inside the Job can still stop waiting while the rollout continues; the plan warns when the record's budget exceeds it. Ending the plan with the rollout observed, not complete and re-observing from the control plane until updated == ready == replicas is the other half of #3782 and is not built here.
• A dead watch informer. The 2026-09-10 log shows kubectl losing its informer three seconds in and never recovering. Whether the operator image should survive that belongs with the operator.
• An unclaimed request — one nobody has picked up, which is byte-identical to one nobody will ever pick up. That is the other defect in the same incident, still open, and its remedy (a claim with a heartbeat behind it) has real write-volume consequences on a shared record. See core's Unclaimed Control-Plane Requests.
• Six sibling control planes have the same shape. OperationRequestReliability already named it: "a fresh hub that finds an in-flight state must re-attach … not stamp Failed, which would tell the approver the deletion stopped while it is still deleting." Essentials/OperationRequest and the five Store/* control planes still stamp nothing at all. This change is that re-attachment, for Hosting/InstanceAction only.

The relationship to the deploy lane (Systemorph/Memex#188)

Memex#188 is the same shape one layer down: helm-release.yml could not report an outcome for a rollout longer than the 10-minute az aks command invoke limit. It is not the same defect — different repository, different runtime, different window, and a GitHub Actions runner is never killed by the rollout it is watching. It is also largely fixed (Memex#192/#193: helm upgrade without --wait, observation across separate invokes, RED naming the re-run at the budget, and an action: observe dispatch that re-observes later).

Its shipped remedy is the template this page follows: an observation window bounds how long an operation may take, which is the wrong constraint. Report honestly at the budget, and re-observe across the boundary.

The control

Both directions, in InstanceAction/Test/ObserverExpiryTests.cs, because a fix that answered "I don't know" to everything would pass a one-sided test and destroy the control plane: