Action-Block Wedge Prevention

Every portal wedge we have diagnosed — the production _Activity NotFound storms, the DeliveryFailure/ShutdownRequest ping-pong, the composer UpdateStreamRequest AccessContext-null cascade — is the same failure: a per-partition hub runs a single-threaded action block, and a message whose failure produces more messages saturates that one thread until it can no longer drain. The portal serves HTTP 200 but the partition is dead.

A wedge is never one bad message. It is amplification: rejected post → DeliveryFailure → resubscribe → rejected post → … on one thread. Remove the amplification and no input — including a defect we have not seen yet — can saturate the block.

This is prevention by invariant, not by chasing each root cause. The three invariants below are the contract; the Tests section is the acceptance criteria. "Solved for good" means these tests are green and stay green.

Invariant 1 — Rejection is terminal, never amplifying

A post that the never-null AccessContext guard (or any inbound gate) rejects must produce O(1) follow-up: log once, drop. It must NOT emit a DeliveryFailure (or any reply) that triggers a resubscribe/repost that fails again.

• Infrastructure/lifecycle messages ([CanBeIgnored], [SystemMessage], DeliveryFailure, ShutdownRequest, DisposeRequest) are exempt from failure-reporting at every emit site — MessageService.ReportFailure, both HierarchicalRouting sites, both RoutingServiceBase sites. (This is the DeliveryFailure-storm fix; the invariant is that it holds at all emit sites, permanently, enforced by a test — not re-checked by hand.)
• Root-aligned, not a band-aid: it removes the cause of amplification.

🚨 The exemption must be read off the ENVELOPE, never the payload's CLR type

Issue #1485 — the exemption above was written seven times as delivery.Message is DeliveryFailure || delivery.Message.GetType().HasAttribute<CanBeIgnoredAttribute>(), and on the routed path not one of them could ever match. Every mesh delivery reaches a router as delivery.Package(hub.JsonSerializerOptions) (MeshBuilder) — the single call site of IRoutingService.DeliverMessage — and Package replaces the payload with RawJson. So the guards were inspecting RawJson, which is neither a DeliveryFailure nor [CanBeIgnored]. The trace tell is RouteMessage: NotFound for RawJson → ….

The consequence is exactly the amplification this invariant forbids: a departed owner that is heart-beaten every SyncStreamOptions interval was answered with a DeliveryFailure every time, and a DeliveryFailure could be answered with another one. It affected both hosts equally — the comments claiming the two routers "both agree" were true only in that both were dead — and it reached one level above the routers too, in MessageService.ReportFailure, which reports whatever the route handler hands back and therefore also sees RawJson.

The fix is AnswerPolicy (src/MeshWeaver.Messaging.Contract/AnswerPolicy.cs): Package stamps the fact "this payload must not be answered" onto the envelope's properties immediately before the type is erased (the same mechanism IDiagnosticKeyed already uses for the storm breaker), and every guard calls delivery.MayAnswer(), which reads the stamp and falls back to the CLR type for a delivery that never crossed a packaging boundary. Only the suppressed case is stamped, so ordinary traffic costs nothing and an unstamped delivery degrades to the old behaviour — never to "answer something you must not".

So when adding a new emit site: call delivery.MayAnswer(). A hand-written CLR-type check is a silent no-op anywhere downstream of a hub hop. Pinned by OrleansRouterAnswerOnceAfterPackagingTest, MonolithRouterAnswerOnceAfterPackagingTest and RoutingTailAnswerOnceAfterPackagingTest — one per host plus the reporting tail, each of which fails on the pre-fix tree. This is also what the proposed FailureExemptAtEveryEmitSite test below would have caught years earlier, and it is the strongest argument for writing it.

Invariant 2 — A subscription to a missing target dies after N, it does not retry forever

The phantom _Activity storm and the rsalzmann subscribe-wedge were both unbounded resubscribe to a node that does not exist. Bounding must be the default behavior of the external/missing-target subscribe primitive, not a per-call-site patch (AreaStreamRetry, the JsonSynchronizationStream bounded retry) that each new caller re-opens.

• The primitive: .Take(1).Timeout(t).RetryWhen(≤ N on DeliveryFailure/Timeout) → terminal OnError, then stop. No caller can spin to infinity even if it forgets to bound.
• Root-aligned: a missing target is a terminal condition, not a retry loop.

Invariant 3 — No single action block can be driven past its drain rate (the safety net)

Invariants 1–2 remove today's causes. Invariant 3 makes the wedge structurally impossible for any future defect: a per-hub aggregate backpressure breaker.

• The existing MessageStormBreaker trips per-key (one path, one stream). Every wedge we saw was many distinct keys — each phantom path, each failed area is a different key — so no single key crossed the threshold while the aggregate saturated the thread. Per-key is the gap.

  🚨 "Per-key" means (sender, target, message-type, **payload identity**). The fourth component is not optional decoration — without it the claim above is false, because the mesh funnels traffic through dispatchers where one sender talks to one target with one message type about many different things (every sync stream an owner holds to the shared node cache; every CreateOrUpdateNodeRequest a bulk importer sends to the mesh hub). Keyed on the routing tuple alone, a wide legitimate fan-out is arithmetically identical to one thing looping, and the breaker drops it — real writes discarded at ingestion (#1200). The identity comes from IDiagnosticKeyed.DiagnosticKey on the message (a stream id, a node path); once a hub hop has erased the payload type to RawJson, it comes from the envelope property MessageDelivery.Package stamped there, so the breaker never parses a payload on the ingestion path. A message exposing no identity keys on the bare tuple — the fallback is the old, stricter behaviour, never "allow".

• The fix: a per-action-block watermark on inbound depth/rate. When one block's queue exceeds the watermark, shed [CanBeIgnored]/failure-class messages (never user-facing or lifecycle messages) to keep it draining. The breaker is keyed on the hub, aggregated across message keys.

✅ This has landed. MessageStormBreaker now carries DefaultAggregateWatermark = 10_000 (overridable per hub with MessageHubConfiguration.WithAggregateWatermark(...)), exposes AggregateSheds / AggregateShedCount, and the shed shows up on a delivery's fate trail as SHED_AGGREGATE (see Debugging Message Flow).

Tests (the acceptance criteria — "done" = these are green)

All deterministic. Run the saturation tests under DOTNET_PROCESSOR_COUNT=2 (the CI 2-core sim that reproduces the real wedge — a fixed sleep/Task.Delay is forbidden; assert on the condition). Wait on the actual signal via await stream.Should().Within(t).Match(...).

Only one of the five exists under the name below. The other four are still proposed names for guards that have not been written; some of what they describe is covered incidentally by tests with different names (AccessContextNeverNullTest, DeletedAddressNackClassificationTest, DeferredDeliveryNackedOnDisposeTest in test/MeshWeaver.Messaging.Hub.Test). Treat 1, 2, 3 and 5 as a backlog, not as a checklist you can grep for and find green.

1. Rejection_DoesNotAmplify (proposed — does not exist) — post a message to a hub with no AccessContext and no exemption. Assert: exactly one rejection is logged/observed and zero DeliveryFailure re-posts follow (count the DeliveryFailure traffic — it must be 0, not "fewer"). Would pin Invariant 1.

2. FailureExemptAtEveryEmitSite (proposed — does not exist) — reflection/architecture test: enumerate every site that constructs a DeliveryFailure; assert each is guarded by the [CanBeIgnored]/[SystemMessage] check. Fails the build when a new emit site forgets the guard. (Would mirror NoStaticCollectionsTest in test/MeshWeaver.PathResolution.Test, which is the working example of this architecture-guard pattern.) Would pin Invariant 1 permanently.

3. SubscribeToMissingTarget_Terminates (proposed — does not exist) — subscribe to a node path that does not exist under a live prefix (the rsalzmann shape: {partition}/_Thread/does-not-exist). Assert the subscription emits terminal OnError after ≤ N attempts within a bounded time and stops (no further SubscribeRequest/NotFound after termination). Would pin Invariant 2.

4. ✅ ManyDistinctMissingSubscribes_DoNotWedge — exists and is green (test/MeshWeaver.Messaging.Hub.Test/MessageHubTest.cs). It holds the single turn thread, floods the hub with [CanBeIgnored] traffic past a low injected watermark, and asserts a user-facing probe still round-trips. It also proves the per-key breaker is not what saves it — the flood is [CanBeIgnored], so ShouldDrop's TripCount stays 0. Pins Invariant 3.

5. AggregateBreaker_ShedsOnlySheddable (proposed — does not exist as a separate test) — drive a hub past the watermark with [CanBeIgnored] traffic; assert shed messages are dropped but a concurrently-posted user-facing message is still delivered. Test 4 already asserts the "user-facing message survives" half; the "shed count > 0" half is what a dedicated test would add.

Status / ownership

Invariant 3 has landed — MessageStormBreaker carries the per-hub aggregate watermark and test 4 above is the guard. Invariants 1–2 remain point-fixes rather than enforced invariants (the DeliveryFailure-storm exemptions in MessageService.ReportFailure and the routing sites; the AreaStreamRetry / JsonSynchronizationStream bounded retries) — the outstanding work is promoting them via tests 1–3. This all lives in the messaging action-block + sync layer (MessageService, RoutingServiceBase, MessageStormBreaker, SynchronizationStream) and should stay with a single owner of that layer — two parallel editors of the action block is itself a source of regression.

The separate root causes that fed these wedges (System identity dropped on activity/import writes; user credential dropped on the composer stream.Update; the Agent/Model public-read grant) are tracked elsewhere — fixing them removes the load; the invariants here remove the amplification. Both are needed.