What a synthetic probe may assert

The Prod synthetic probe watches the one endpoint that crosses the cross-silo content hop — /api/content/{node}/content/{file}, a collection-config GetDataRequest to the owning node's hub. That is the transport that actually wedges, so the choice of endpoint was right. The choice of target was not, and it cost eight days of red.

The failure: red for a true statement

The probe's matrix paired each portal with an asset of an installed package:

portal target
memex.meshweaver.cloud /api/content/AgenticEngineering/content/og.png
memex.systemorph.com /api/content/AgenticPrimer/content/og.png

AgenticPrimer is not installed on the staff portal and never was. So from 2026-08-31 the probe failed every 15 minutes, for 59 of its last 60 runs, while the portal it was pointed at was completely healthy — /healthz 200, every page rendering, the content route serving other assets fine.

A check that is red in an empty room is worse than no check. It is the same defect as a gate that skips: after a day nobody reads it, and every real failure it would have caught arrives wearing the colour everyone has learned to ignore. The eight days were not the bug. They were the cost of the bug.

Why it was broken by construction, not by accident

A per-deployment install decision is not a property of the platform. The moment the fleet has two portals with different content, an assertion naming one portal's packages is wrong on the other — and this repository cannot even see which is which. The deployment inventory is deliberately private (see chart-drift.yml: "do NOT move the overlays into this public repo to dodge the credential — that would publish the deployment inventory"), so a public workflow that hard-codes what a named portal serves is encoding a private fact it has no way to keep current.

So the target could only ever drift, and when it drifted the probe blamed the portal.

The first instinct — install the content, or repoint the target — is a repair of this instance and nothing else. Installing course content on the staff portal to satisfy a monitor is the tail wagging the dog; repointing puts a fresh guess in the same slot to drift again. What the probe needed was to stop guessing.

The three things a probe may assert

Everything the probe checks is now either shipped by the platform image or read from the deployment at probe time. Nothing is hard-coded about what a named portal contains.

1. The platform floor — a constant that cannot drift

/api/content/Doc/Architecture/content/platform-overview.svg.

Present on every deployment by construction, and each link in that chain is load-bearing: the bytes are an EmbeddedResource in MeshWeaver.Documentation, AddDocumentation gives every Doc/* child a content collection with isStatic: true (which is what publishes a collection on this route at all — the default is not publishable, and the route answers 404), and Doc/_Policy sets PublicRead so an unauthenticated probe may read it. It resolves through ContentFileResolver with the same longest-node-prefix match and the same collection-config request to the owning hub as any package asset, so it crosses the hop being watched while depending on no install decision.

This is the only assertion that can answer "is the route healthy", because it is the only one whose expected value is known without asking the deployment anything.

2. A negative control — the assertion whose absence made the incident unreadable

A path under the same route that cannot exist, which must answer 404.

The old probe sampled one URL three times and reported content route unhealthy — 3/3 samples failed. Three samples of one URL cannot distinguish "this asset is missing" from "the route is down" — they are the same evidence. The issue was filed on exactly that wrong inference, proposing that the content-serving module had failed to load, and the next two readers each had to re-measure from scratch to find that the route was fine.

With a control the two separate cleanly:

platform asset control verdict
200 404 route healthy, target present — green
404 404 the route discriminates, so the asset is genuinely absent — a content red
5xx / timeout any a route red
any not 404 the route answers the same thing for everything — a route red

The route verdict outranks the content verdict, because a wedged route makes every content answer meaningless. Only once the control has proved the route discriminates does a 404 mean what it says.

3. The deployment's own declaration — /sitemap.xml

The probe holds each portal to its own claims rather than to a guess made in this repository.

/sitemap.xml is anonymous and is the only unauthenticated surface that says what a deployment serves: SeoEndpoints enumerates every top-level Store/Plugin, Store/Catalog and Space root that passes AnonymousGate, applied per node and fail-closed. Measured 2026-09-08 it already encodes the difference the matrix used to guess at — 79 roots on memex.systemorph.com (no AgenticPrimer), 101 on memex.meshweaver.cloud (with it).

The probe reads that list, samples roots from it by index (first, middle, last — deterministic so a red is reproducible, spread so the sample is not the same three alphabetically-first names on every portal), and requires 200. A portal is therefore only ever failed for contradicting itself, and content installed later is covered with no edit here.

An empty denominator must be louder than a failure

A loop over an empty list exits 0 having asserted nothing, and GitHub paints that green. So the count is asserted before the loop and printed either way — a zero has to be readable rather than silent. Three distinct reds come out of reading the declaration, and they are deliberately not one message:

🚨 A 200 is not a sitemap

Found by falsifying the step rather than by reviewing it. The portal's SPA fallback answers 200 text/html for any unrouted path, so "the sitemap endpoint is gone" arrives looking exactly like "the sitemap is empty" — and the second reads as a portal that publishes nothing, which is a statement about the portal rather than about the probe's own input. The step now asserts the shape (<urlset present) before counting, so the two get different messages.

This generalises past this probe: on any host with a catch-all route, an HTTP 200 is not evidence that the thing you asked for exists.

🚨 code=$(curl … || echo 000) concatenates

The idiom every step in this workflow used. On a timeout curl prints its own 000 and exits non-zero, so the capture reads 000000 — a six-digit status nobody can look up, produced in exactly the situation where the error message is all the reader has. code=$(curl …) || code=000 assigns once. Worth grepping for wherever a status code is captured.

Which half is wrong: the assertion, or the environment?

Every instance of this class looks the same from the outside — a check is red because the thing it names is not there — and the two possible repairs are opposites. Fixing the wrong half is how a monitor gets quietly weakened.

The discriminator is whether the thing asserted is guaranteed by construction, and by what.

So before repointing a red assertion, name the mechanism that was supposed to make it true. If you can name one, the red is a finding and repointing destroys it. If you cannot, the assertion was a guess and repointing is the fix — but then point it at something a mechanism does guarantee, or you have only moved the guess.

What this file may never contain again

The workflow's matrix names hosts. It may not name a package, a space, or a course. Anything content-shaped appearing there is this defect returning — and it will not be visible as a defect, because it will look exactly like a target that used to work.

See also

Reconnecting…
The server was updated. Reloading the page to pick up the latest version.