How an instance action finds its record — index first, never a point read

A Hosting/InstanceAction names the record it acts on in its deployment field, and the operator may write it either way:

{ "requestedAction": "Provision", "deployment": "Deployments/build" }   // a PATH — the runbook's form
{ "requestedAction": "Provision", "deployment": "build" }               // a bare ID

Until Plugins#1731 those two spellings took different routes, and only one of them obeyed the platform's rule for reading a node that may not be there.

The incident

Measured 2026-09-12 on the control instance (memex.systemorph.com) while ramping up Deployments/build:

The refusal is a claim about the mesh — there is no such record — derived from a read that failed. Nobody measured it. The record was there the whole time.

The defect: a point read of a node whose existence is the question

InstanceActionControlPlane.ResolveNode branched on the reference:

if (value.Contains('/'))
    return workspace.GetMeshNodeStream(value).Take(1).Select(n => (MeshNode?)n)
        .Catch(Observable.Return((MeshNode?)null));        // ← the path route
// …the bare id asked the index, then streamed the path the index listed

Three defects in those two lines:

1. The exception was discarded. .Catch(Observable.Return(null)) kept no cause and logged nothing, so the caller's null became "no record found".
2. No DefaultIfEmpty(null), which the id route had — a stream completing without emitting terminated the whole resolution silently instead of answering null.
3. It is the forbidden access shape. AGENTS.md, CQRS: "A node that may NOT EXIST YET needs BOTH halves — a scope:children listing for EXISTENCE, then GetMeshNodeStream(target) for CONTENT. A point read of an absent node is a framework defect, not merely slow: the owner answers a routing NotFound that terminates the stream AND opens the storm-breaker on that path." Core says the same thing in its own words, at MeshNodeStreamCache.cs: "The PRIMARY fix is to read optional nodes via GetQuery (empty-on-absent, never NotFound); this breaker is the framework backstop so NO point-access can storm."

Defect 3 is what explains the asymmetry, and it is why "let the path route fall back to the query route" was the wrong fix: a fallback leaves the point read in place, keeps tripping the breaker, and hides the cause behind a retry.

The breaker did not cause the refusals — the arithmetic says the NotFounds were true

MeshNodeStreamCache's constants: StormBaseCooldown 2 s, window 2 s · 2^(n-1) per consecutive failure, StormFailThreshold 5 (which gates only the suppression log). Against the timestamps above, fail 1's window expired at 12:41:08 and the next attempt came 14 s later; fail 2's at 12:41:24, next attempt 34 s later. Every attempt fell outside the previous window, so each one genuinely re-probed the owner and genuinely received NotFound. The refusals were honest about the owner and wrong about the mesh.

And 12:42:00 — the id route's success — falls inside fail 3's 8 s window, on the identical path. A cached error still open would have fast-failed it; it did not, so the negative entry had been cleared, and ResetFailureState says what clears one: a post-commit write on the path. A real write landed on Deployments/build between 12:41:54 and 12:42:00, ~90 s after GitSync reported the import done.

So the asymmetry is about WAITING, not about which path is read

Both routes end in GetMeshNodeStream on the same string. The difference is whether the index was asked first. The index trails the store, so "the index has seen it" implies "the store has it": an index-first resolution can only ever over-wait, while a point-read resolution can only ever under-wait — and it converts "not yet" into an authoritative "not ever".

The fix: one route, two spellings

ResolveRecord resolves both forms identically:

1. Ask the index — every query anchored, projected and bounded; never the unanchored scope:subtree, which unions every partition schema of the mesh (MeshWeaver#3545, and see Unanchored query policy). Two kinds, and both are asked:

   • the exact candidate paths the reference can name — itself when it is path-shaped, {home}/{id} in each home when it is a bare id, since a node's path is {namespace}/{id}: path:{candidate} nodeType:{type} select:path,id limit:1. path: with the default scope:Exact is an anchored point query — an empty frame for an absent path, never a routing NotFound — and it cannot be pushed off a bounded page;
   • the anchored listing of each home, namespace:{ns} nodeType:{type} select:path,id limit:500, which is what lets a bare id match case-insensitively and what the refusal quotes as evidence.

   The exact query is not redundant with the listing. A listing alone matches only inside its page, so a home that filled its limit would answer "no record found" for a record that exists — reintroducing this page's defect at a different scale. (Review finding on Plugins#1896.)

2. Match the row the reference names — by Path for a path-shaped reference, by Id for a bare one. That is the only difference between the two spellings.

3. Stream exactly the path the index listed — one GetMeshNodeStream(hit.Path).

A hit answers on the first complete frame. Only a miss keeps listening to the live index for a short grace (IndexGrace, 10 s), so a record the index has not caught up with is found rather than refused — the bounded, explicit form of the over-wait that index-first buys you.

The homes

A bare id is looked up where records of that type actually live — the same pair FleetWatch.RosterQueries reads, and for the same reason:

What a refusal says now

• The index lists nothing for the reference → NoRecordRefusal: the reference, every query that was asked, and every path they listed. An operator reading it sees the measurement, not a conclusion drawn from a failed read.
• The index listed it but the stream could not deliver it → a throw naming the path and the cause, with the exception attached, so the run's failure log carries it. Never "no record found".
• The index answered no complete frame → a throw naming the queries and the budget.

How it was made to fail first — without the #1794 seam

#1731 sat open for a day because the obvious control could not exist: reproducing the real thing needs a node whose write has committed but whose owner has not materialised it, which is #1794's seam and does not exist. Two candidate controls were tried and are vacuous — recorded here so nobody re-tries them:

• "resolve an absent path, create the node, resolve again and watch the breaker refuse it" — cannot fail: the create is a post-commit write on that path, and ResetFailureState treats it as authoritative proof the negative entry is stale.
• "after the fix a differently-cased reference resolves" — proves the wiring only if the point read is case-sensitive, which is unverified; if it is not, the control is green before and after.

The instrument that does work needs no seam, because the defect leaves a trace of its own: a point read of an absent path leaves a negative entry, and MeshNodeStreamCache.IsStormWindowOpen reads it. RecordResolutionTests.Live_ResolvingAnAbsentReference_OpensNoBreakerOnItsPath resolves a reference to an absent path and asserts the breaker window on that path is closed — i.e. that nothing point-read it.

Two properties make that a control rather than a green light:

• It is internal to MeshWeaver.Hosting, and this source compiles in the portal, outside every assembly it is visible to — so it is reached by reflection, and the case fails when it cannot be reached. A case that cannot measure does not pass.
• It proves the instrument first, on a direct point read of a second absent path: if that does not open a window, the negative reading the case exists for could not be trusted, and the case refuses rather than reporting an undetectable negative.

Measured: RED on the test-only commit (Hosting/InstanceAction compile=Ok render=ok tests=FAILED — "resolving '…' opened the storm breaker on that path — the resolution POINT-READ a node whose existence was the question"), GREEN on the fix.

What is still dark

A record the index lists but whose owner refuses to stream — #1794's held state. That path now refuses naming the path and the cause instead of claiming the record does not exist, which is the observable #1731 asked for, but it is still not reproducible in a test. #1794 remains the seam that would close it.

Rules that follow

• Never point-read a node whose existence is what you are asking. Index first (empty-on-absent), then stream the path the index listed. The breaker is the backstop, not the design.
• Never .Catch(Observable.Return(null)) on a read. A read that failed and a node that is absent are different answers; collapsing them turns a failure into a false claim about the mesh.
• Anchor every resolution query on namespace:, and project it (select:path,id) and bound it (limit:).
• A refusal states what was measured — the queries asked and what they listed — not a conclusion about the world.