Unanchored query policy — refuse in CI, serve and report in production

Read first — the core side of the same mechanism: get Doc/Architecture/PinBoundaryContracts (Pin-Boundary Contracts). This page is the policy — what the planner does when a query arrives unanchored, and why that answer differs between CI and production. That page is the boundary — why the pin moves in BOTH directions, so a test encoding a core contract is correct on only one side of it, and a mixed image pairs two halves that were each green alone. The window this incident opened is an instance of exactly that: the grace list cannot cover it by construction, because the two halves never existed together in any tree a gate could have run against.

Incident: memex.systemorph.com, 2026-09-03 17:02–17:22Z. Image memex-portal-ai:3.0.0-rc9.ci.7658 paired core commit e7f1d699 (built 08:05Z, before MeshWeaver #3206 anchored the platform's sign-in read) with Plugins 12500c9 (13:31Z, after #1263 made the unanchored-query refusal live). The platform's OnboardingMiddleware.LoadUserRoles issued nodeType:AccessAssignment content.accessObject:"…" scope:subtree limit:all, the planner refused it, and the middleware answered 503 "temporary problem on our side" to every signed-in request. The Store page died the same way on nodeType:PluginCatalog; GitHubSyncConfig, InstanceSyncConfig, Notification, Email, ModelCreditCharge and free-text search all faulted. Loki: 138 refusals and 19 identity 503s on two pods in twenty minutes. Even on the fixed image (ci.7693) a free-text query 'MessageHubGrain' was still being refused in a background path.

The maintainer's directive, verbatim intent: "this must not completely break us — we must have some form of higher up try/catch if such stuff happens."

This page is the design that follows from it. Read first: core's Cross-Schema Fan-Out Elimination (why fan-out is a hazard at all) and this repo's NodeType instance locations (the narrowings that shrink it).

The trade a hard refusal makes

An unanchored query — no concrete path:/namespace: first segment, no routing hint, no partitions:all — is answered by a UNION ALL over every searchable partition schema. That is a performance hazard: one such union takes heavyweight locks on hundreds of relations against a backend's sixteen fast-path lock slots, the lock manager serialises, and every other query queues behind it (measured on memex-cloud 2026-09-02: 7 786 fan-outs in thirty minutes, pinned point-reads waiting 2–5 s on LWLock/LockManager).

#1231 made fan-out opt-in by refusing such a query with UnanchoredQueryException. Correct in CI — a refusal is a red test naming the caller. On the request path of a running portal it converts the performance hazard into a total availability outage: the refused read was the one every signed-in request depends on. That trade is never acceptable in production.

Why the grace list cannot cover the window — by construction

unanchored-queries.allow (#1263) names the offender shapes the current census of callers still issues, and UnanchoredQueryAllowFileTest regenerates it from that census and asserts equality — a listed shape nobody issues is stale and red. So the moment core anchors a caller, the shape leaves the list. But a deployment is two halves built at different times: an image whose core half predates that anchoring keeps issuing the old shape, against a Plugins half whose list has already dropped it. That is exactly ci.7658. No list regenerated from the current tree can describe what an older tree still does; the window is structural, not an oversight in the census.

The policy

PostgreSqlPartitionedMeshQuery carries an init property (not a constructor parameter — a published module binds the constructor's signature in its IL, so a new parameter is additive here and breaking there):

public UnanchoredQueryPolicy UnanchoredPolicy { get; init; } = UnanchoredQueryPolicy.Refuse;

Three properties of the ServeAndReport arm are load-bearing:

• It serves by exactly the Grace path. No new code path, no second planner: an unlisted shape is treated as a listed one would be, except for the level it is reported at.
• It reports at Error, once per subscription. Error, so the log-incident control plane opens an incident and Loki shows the debt; per subscription (like the grace warning, in the same Observable.Defer around the fan-out), so the log counts fan-outs that actually ran and a request whose later query is refused never reports its earlier graced one. The UnanchoredQueryException is attached — never thrown — so CI's fault and production's report carry one discriminator.
• The default is Refuse, fail-closed. A planner nobody configured — every test fixture, every satellite rig, every host that never heard of the property — keeps the CI invariant. A fail-open default would silently strip the guard from every fixture, and nothing on a green wall would say so.

Where a host states it

The registration (AddPartitionedPostgreSqlPersistence, both overloads) resolves the policy from the host's IConfiguration through UnanchoredQueryPolicyConfiguration.Resolve:

"Graph": {
  "Storage": {
    "Type": "PostgreSql",
    "UnanchoredQueryPolicy": "ServeAndReport"
  }
}

• The key sits beside Graph:Storage:Type — the host's own convention for "this is the Postgres planner" (core's MemexConfiguration binds that section). The two lines read together.
• Absent or empty ⇒ Refuse. The same fail-closed default as the planner's. A test mesh's configuration (MeshWeaver.Fixture.ServiceSetup: an in-memory collection plus the test project's own appsettings.json, never a host's) has no such key, so every fixture stays strict without a core change.
• A value naming no policy is a fault at the first resolution, naming the key and the valid values. A typo must never quietly select either arm.
• The production host commits the line in its own appsettings.json — src/Memex.Portal.Distributed/appsettings.json, the only host that wires Postgres today (the Monolith runs on FileSystem storage). It is baked into the image, so no Helm chart, ConfigMap or environment variable can forget it, and a deployment that never heard of the key is still safe. Never set Refuse on a production host.

What the allow file governs after this change

Nothing about the list's lifecycle changed: it may only shrink (scripts/check-unanchored-queries.py --base-ref fails any added line), a stale line fails UnanchoredQueryAllowFileTest, and a caller that genuinely reads the whole mesh declares it (partitions:all, core's MeshWideQuery) rather than getting a line. What changed is what a line means on each side:

• In CI: the verdict — a listed shape is Grace (served, Warning), an unlisted one is Refused (fault). Exactly as before.
• In production: the level — a listed shape is a Warning, an unlisted one is an Error that opens an incident. Both are served. Listing a shape is therefore never a way to quiet the log; the only thing that closes the incident is fixing the caller.

The guards, both arms

A single-arm test passes against a policy that stopped being read at all, so every claim above has a test on each side of it (src/MeshWeaver.Hosting.PostgreSql.Test):

Operating it

• Finding the debt. Loki: {app="memex-portal"} |= "[FanOut] UNANCHORED" — one line per subscription, carrying the shape (nodeType:X scope:Y), the query text as the caller wrote it, and UnanchoredQueryException. The incident control plane files one LogIncident per structural fingerprint. [FanOut] GRACE at Warning is the listed debt, the same way.
• Closing it. Always in the caller: anchor the query (path:/namespace: with a concrete first segment), pin its type with a QueryRoutingRule, or declare the fan-out (partitions:all) when the whole mesh genuinely is the answer. Core's own callers are censused by UnanchoredQueryCensusTest there; this repo's by UnanchoredQueryAllowFileTest.
• What not to do. Do not set Refuse on a production host. Do not list a shape to lower an Error to a Warning — the ratchet refuses the added line, and it would only hide the debt. Do not raise a bound or add a retry around the fault: there is nothing to retry, the query is wrong.