Webhook Inbox

External services (Stripe, GitHub, …) notify a MeshWeaver portal by HTTP POST — they cannot authenticate as a mesh user, and their payloads are integration-specific. The webhook inbox is the ONE generic bridge: it stores each delivery verbatim as a mesh node and leaves every integration-specific concern — above all signature verification — to the consuming plugin. No payment (or other integration) code ever lands in the portal.

The endpoint

POST /api/hooks/{target} (anonymous — see WebhookInboxEndpoints in Memex.Portal.Shared) stores the delivery as a WebhookEvent node at {target}/_Inbox/{id}:

• Body — the raw request body, verbatim: the exact bytes an HMAC signature was computed over.
• Headers — the request headers, minus credentials (Authorization, Cookie, … are never persisted). Signature headers (Stripe-Signature, X-Hub-Signature-256, …) survive verbatim.
• ContentType, ReceivedAt.

Fail-closed, three times

1. The allowlist. Only targets listed in configuration accept deliveries; everything else is 404 (no detail leaks about which paths exist):

   { "WebhookInbox": { "Targets": [ "Store/Payments" ] } }
   

2. The owner must exist. A satellite must anchor under a real node — an ownerless satellite NotFound-storms the router — so a delivery to an allowlisted path whose node does not exist is refused too.

3. The signature, when the target declares one — see below. A target that declares no SecretConfigKey keeps the original contract and is stored unverified.

Bodies over 1 MB are refused with 413. The event is written under the System identity (the anonymous caller has no write access anywhere; the allowlist is the authorization to store — never to act).

The consumer's contract

A plugin that receives webhooks:

1. Ships the target node (e.g. Store/Payments) and documents its endpoint URL ({portal}/api/hooks/Store/Payments).
2. Watches its inbox with a live children query over {target}/_Inbox from its hub initialization, processing strictly one event at a time.
3. Verifies authenticity itself over the stored raw Body + Headers — e.g. Stripe's t=…,v1=… HMAC-SHA256 with the endpoint's signing secret. Only a verified event authorizes an action. A consumer whose target declares a SecretConfigKey has already been verified at the endpoint, and still re-verifies: the endpoint's check is what makes a drifted secret VISIBLE to the sender, not a replacement for the consumer's own gate.
4. Deletes processed (and unverifiable) events — a poison event must never loop; unprocessed events replay naturally on the next start, so every action taken from an event must be idempotent.

🚨 What a 2xx proves — and what it used to hide (#3312)

A 2xx from this endpoint used to mean "I stored bytes", never "a consumer accepted them". That was an honest answer for a deliberately dumb inbox, but two callers read it as acceptance, and one of them is the platform's own release path:

• main-cd.yml → notify-platform-update signs the platform-build fact with secrets.PLATFORM_WEBHOOK_SECRET;
• node-repo-publish-bake.yml → register-publication signs the publication record with the caller's webhook-secret.

Both are verified against the control instance's Hosting:PlatformWebhookSecret — but that happened in the Hosting plugin's watcher, after the POST had already answered. So from CI three states existed and only two were distinguishable:

The empty case was found precisely because it failed closed. The mismatch failed open and was byte-identical to success, so it could persist indefinitely while every dependent quietly fell back to its schedule poll — the same consequence #3311 fixed, with nothing anywhere going red.

The fix: a target may declare how it is authenticated

A target entry MAY name the configuration key holding its shared secret. The declaration rides on the allowlist entry itself — a configuration section carries both a value and children — so the record that makes a target reachable is the record that says how it is authenticated, and the two cannot drift:

WebhookInbox__Targets__0: "Hosting/PlatformBuilds"
WebhookInbox__Targets__0__SecretConfigKey: "Hosting:PlatformWebhookSecret"

It names a key, never a secret value: nothing here belongs in a ConfigMap, and a secret pasted in by mistake resolves to no key and refuses everything rather than leaking.

🚨 The declaration is paired with a SLOT, and slot __0 is not the same target on every deployment. The instance that receives platform builds renders webhookInboxTargets: ["Store/Payments", "Hosting/PlatformBuilds"] from its Hosting/Deployment record — so its __0 is Stripe, and the platform target is __1. A blanket default would demand X-Hub-Signature-256 from Stripe, which signs Stripe-Signature, and 401 every payment delivery. The chart therefore renders both slots' declarations empty, and each instance sets its own on the index that actually carries the target — in the same record that sets the slot (extraPortalConfig on the Hosting/Deployment record, which HelmValues projects onto config.memex_portal). Set the target and its declaration in one change, never separately.

When a target declares one, the endpoint verifies X-Hub-Signature-256 over the raw body before anything is stored:

That order is contract: target first (an unlisted path answers 404 without revealing whether it is signed), size next (an oversized body is refused before it is hashed), signature last and always before the node is created. A delivery that fails to verify must leave nothing behind — otherwise the fix has only moved the silent drop from the consumer into the store.

Everything else keeps the dumb contract. Schemes this endpoint does not speak — Stripe's t=…,v1=… — declare no key and are still verified by the consuming plugin over the verbatim stored body, so no integration-specific code lands in the portal.

Why the body carries the verdict too

"verified" and "not-required" are both 200, and they mean very different things to a sender that signed. The second says this instance declares no SecretConfigKey for the target, so the signature was never looked at. Without that distinction, "we verify now" would degrade silently back to "we used to verify" the day a chart value goes missing — the same shape as the bug, one level up.

How a publishing lane judges that answer

Both lanes — main-cd.yml's Did the inbox VERIFY the build fact? and node-repo-publish-bake.yml's Did the inbox VERIFY the publication record? — read the body, not the status, in a step of their own. There are three answers, and only one is a misconfiguration:

🚨 The third row is the one that is easy to get wrong, and getting it wrong is worse than having no check. The first form of these steps tested for the single string verified and reported everything else as not-required, so the absence of a verdict was printed as a verdict — and named the wrong fix. Measured on 2026-09-06 (core CD run 34039122957, the notify job at 14:44:39Z and the satellite bake leg at 15:16:47Z): the control instance answered 200 with an empty body on both lanes, because it still runs the pre-verdict endpoint, and both lanes told the reader to provision a chart key on a pod that could not have read one.

What makes not-required a misconfiguration and an absent verdict not

The expectation is the act of signing — there is no flag, no input and no opt-out, because a "verification expected" knob is a skip-trapdoor by another name. Both lanes ALWAYS sign: main-cd.yml's preflight asserts PLATFORM_WEBHOOK_SECRET, the satellite lane's POST step fails RED without webhook-secret, and both always send X-Hub-Signature-256. So inside these lanes a not-required answer is a broken pair: the sender's secret is doing nothing, and a drifted one would be as invisible as it was before the verdict existed. The fix is one key on the receiving instance's record, and the error message names it.

The legitimate not-required case is a target with an unsigned sender — Stripe on Store/Payments, which signs Stripe-Signature; GitHub on its own target — and neither of these steps ever runs against one. A target cannot be moved into that category silently: doing so means removing the lane's secret, at which point the POST step fails first, naming what to provision.

An absent verdict is not a receiver declining to verify; it is a receiver that cannot answer because it has not rolled the endpoint yet. That half arrives by helm upgrade from the private Systemorph/Memex env folders, never through self-update (which is a set image), so failing on it would red every promoted build until a roll no repository here controls happens.

The escalation therefore arms itself. The moment a receiver answers a verdict at all, the not-required branch becomes reachable and fatal — nothing has to be remembered, switched on, or re-decided. An unrecognised verdict word is fatal too: this step judges the receiver's own word, so a word it does not know is not a pass.

InboxSignatureVerdictGuard lifts each step's script verbatim out of its workflow and RUNS it against synthetic answers, because a substring assertion cannot tell "classifies three states" from "classifies two and guesses" — which is precisely the defect that shipped. That is also why both steps read their response file through RESP="${RESP:-/tmp/resp}".

Register is not persist: the declaration has already been lost once

SecretConfigKey is not a field on DeploymentContent — it can only ride in the free-form extraPortalConfig bag. And a Hosting/Deployment record is git-synced: the control instance's lives in the private Systemorph/Memex repo as mesh/Deployments/memex-cloud.json, so an edit made on the live node that is not committed there is reverted by the next sync. Measured on that record's own version history:

The key appears nowhere in Systemorph/Memex — not in a record, not in an overlay — so it was never persisted at all. It was set by hand 2026-09-05, round-tripped away 1 h 42 min later, and nothing anywhere went red: the shape this page exists to end, arriving one level further up than last time.

Until it is committed the receiver has nothing to verify with, and the escalation above is what will say so — it fires the first time that portal answers a verdict at all. So: set it in the repo (never only on the node), on slot __1, because this instance's __0 is Store/Payments and demanding X-Hub-Signature-256 from Stripe would 401 every payment delivery.

The two shapes that were rejected, so they are not re-derived

1. Read the record back. The lane cannot: the inbox is anonymous to WRITE only, and reading a registration needs a mesh credential in CI plus a surface that does not exist.
2. Wait for the consequence — poll the satellites for the repository_dispatch the broadcast produces. Rejected on two counts: it is a timeout-bounded poll (the bound, not the fact, would decide the verdict), and it makes core's CD verdict depend on other repositories' state, which is the CI-to-CI coupling main-cd.yml has already deleted twice.

Where the code lives

• memex/Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs — the anonymous POST /api/hooks/{target} endpoint: allowlist check → 404, ContentLength > MaxBodyBytes → 413, target-node existence, then the WebhookEvent write.
• src/MeshWeaver.Graph/Configuration/WebhookEventNodeType.cs — the WebhookEvent node type, the allowlist reader (WebhookInbox.ReadTargets → WebhookTarget(Path, SecretConfigKey)), the HMAC check (VerifyHmacSha256) and the constants both ends share: TargetsConfigSection, SecretConfigKeyName, SignatureHeader and MaxBodyBytes = 1024 * 1024.
• test/MeshWeaver.Graph.Test/WebhookInboxTest.cs — the pair that pins the fix: SignedTarget_WithTheRightSecret_IsAccepted against SignedTarget_WithADriftedSecret_IsRefused_AndStoresNothing. Both returned Accepted before #3312; if a change ever makes them agree again, the hole is back.