AccessContext Propagation — the identity baton

A user's identity (AccessContext) flows from the authentication boundary through every async hop, every message hand-off, every handler, and every downstream post. The framework makes this look like a single-threaded execution under one principal — even when work crosses thread pools, schedulers, hubs, and grain activations.

This document explains the model, the six propagation phases, the sanctioned exceptions, and the anti-patterns to avoid.

🚨 For the OWNER case — read Owner Injection. When a node's own hub acts with no live caller (a watcher tick, a deferred sync write, a cold-start activation), the node owner is the standing identity, injected everywhere and carried forward via CircuitContext (not Context, which is wiped across Rx hops). Genuine infra (doc sync, cache hydration) runs as System; an empty context is rejected, never faked. That page covers the cold-start submit deadlock this prevents.

🚨🚨🚨 THE INVARIANT: AccessContext must ALWAYS be set — never null

Every message a hub posts carries an AccessContext. There is no null, and there is no fourth source — exactly three:

Posting identity is a per-hub CONFIGURATION decision, declared at hub startup

It must be unambiguous which identity a hub posts under — this is a config property, not a per-callsite concern. Set it once when the hub is built:

// DEFAULT — user-facing hub. Posts as the ambient user; UNHAPPY when no user context.
config                                              // PostingIdentity.User (implicit)

// Framework infrastructure (routing, persistence). Posts as system-security.
config.WithPostingIdentity(PostingIdentity.System)

The three modes (PostingIdentity enum, consumed by UserServicePostPipeline):

1. User (default). The hub posts as the user — it wants the identity set on the AsyncLocal AccessService.Context. When that AsyncLocal is not set and the message is not exempt, the post is UNHAPPY: the PostPipeline logs an error (the CI tripwire on the MeshWeaver.AccessContext channel) and fails the delivery immediately — no identity, no delivery. The awaiting hub.Observe(...) gets a clean OnError; the post is never thrown out of Post (Post is fire-and-forget from countless callsites — a synchronous throw would be unobserved or crash an unrelated path) and never silently left null (that masked the empty-agent-registry / prod EventCalendar bug).
2. System. Routing and persistence. The hub's own otherwise-unattributed posts are stamped system-security. Never overwrites a user identity already on the delivery (a forwarded user delivery, or a response inheriting the request's identity via ResponseFor) — System is only the fallback for the hub's OWN posts.
3. Cannot post → fail the delivery. The User-mode fallback above is this: a hub that can resolve no identity does not get to post a non-exempt application message.

Exempt traffic — the only messages that may carry a null context

Genuinely identity-free framework traffic is exempt from the never-null rule and is delivered (not failed) even with no context:

• [SystemMessage] — heartbeats, hub-lifecycle, subscription management, SetCurrentRequest, Save/DeleteMeshNodeRequest.
• [CanBeIgnored] — fire-and-forget control (Shutdown / Dispose / HeartBeat) with no awaiting requester.
• DeliveryFailure — the courier's own error channel (inherits the request's identity via ResponseFor when there is one; failing it would turn a NACK into a NACK-of-a-NACK).

🚨 Cross-cutting: only hubs REGISTERED IN THE MESH may post

This invariant only holds if every posting hub is a real, registered participant in the mesh — exactly how the portal hub is keyed and registered (PortalApplication → hub.GetHostedHub(CreatePortalAddress(circuitId), …), one hub per circuit, addressable + routable). A hub that is not registered in the mesh is not allowed to post — except a hosted hub, which is registered with (and owned by) its parent via GetHostedHub. The rule:

• Posting hub ⇒ registered hub. If something needs to originate messages, give it a proper mesh address and register it (a top-level hub in the mesh catalog, or a hosted sub-hub under a registered parent). Don't post from an ad-hoc, unaddressable object.
• Declare its posting identity at registration. A registered user-facing hub defaults to User and must carry the user's context; a registered infrastructure hub is WithPostingIdentity(System).
• The portal is the worked example. It is per-circuit, keyed on the stable circuit id, carries the circuit user (via ICircuitContextAccessor.UserContext → a per-hub PostPipeline step that stamps the circuit user when a post has no ambient context), and is reachable through the mesh's stream-routed portal/* address type. Mirror that shape for any new posting hub.

This is a cross-cutting concern: it touches hub construction (MessageHubConfiguration.WithPostingIdentity), the post pipeline (UserServicePostPipeline), the circuit/auth boundary (CircuitAccessHandler → CircuitContextAccessor), routing and persistence (declared System), and every hub-registration site. Reference: memory feedback_access_context_always_set; tests AccessContextNeverNullTest, PostPipelineAccessContextTest, SubscribeRequestIdentityRoutingTest.

Mental model — piecewise single-threaded flow

Think of the system as a chain of short, synchronous pieces of work. At any moment, exactly one piece is running on one thread under one principal. That principal is the identity baton.

The baton follows a predictable life cycle:

• Starts at the authentication boundary (Blazor circuit, Minimal API, or ApiTokenAuthenticationHandler).
• Is set on AsyncLocal at the start of every piece via AccessService.Context.
• Is read from AsyncLocal every time the piece posts a message, stamped onto delivery.AccessContext.
• Travels with the delivery across the async boundary — thread pool, Rx scheduler, grain hop, network.
• Is read from delivery.AccessContext by the next piece's dispatch code and restored onto AsyncLocal before the handler body runs.
• Is restored to its previous value when the piece finishes (wrapped in try/finally).

Inside any single piece, application code can ignore identity entirely — it's already correct. The framework owns every hand-off.

The baton is never absent during a piece's execution. Application code can trust AccessService.Context to reflect the originating user — that's the whole point.

The identity baton travels through six phases; Phase 6 loops back to Phase 2 whenever the handler posts further messages or starts reactive chains.

Security guarantees

The baton model exists to enforce a small set of strong security promises. Each is enforced by a specific framework primitive and verified by a specific test class — break one and the corresponding test fails loudly.

The whole model rests on failure modes being loud and tested, not silent. The error log, the violation tests, and the fail-closed default together make a privilege-escalation regression visible the moment it's introduced.

Phase-by-phase contract

Phase 1 — Entry: authentication middleware sets AsyncLocal

Authentication happens at the platform boundary. Exactly one place per platform sets the baton:

After this phase, the entire AsyncLocal-respecting execution path runs under the user.

Phase 2 — Posting: AsyncLocal → delivery.AccessContext

Any call to hub.Post(...), hub.Observe(...), or meshService.CreateNode/Update/Delete/CopyNode(...) reaches the PostPipeline (MessageHubConfiguration.UserServicePostPipeline). The decision order is:

1. If delivery.AccessContext is already set (e.g. via PostOptions.WithAccessContext(...), ImpersonateAsHub, or ImpersonateAsSystem at the post site) → use it; do not overwrite.
2. Else read accessService.Context ?? accessService.CircuitContext; if non-null → stamp on the delivery.
3. Else, if the hub is PostingIdentity.System and the message is not exempt → stamp the well-known system-security identity (the routing/persistence fallback).
4. Else, if the message is not exempt → log an Error on the MeshWeaver.AccessContext channel and FAIL the delivery (d.Failed(reason)), short-circuiting the rest of the pipeline. NotifyAsync sees State == Failed and ReportFailure posts a DeliveryFailure back, so an awaiting hub.Observe(...) gets a clean OnError. It does not throw (Post is fire-and-forget from countless call sites) and does not deliver a null-context message.
5. Exempt messages ([SystemMessage], [CanBeIgnored], DeliveryFailure) keep their null context and are delivered normally — they carry no security-relevant payload.

The baton is now on the delivery, ready for hand-off.

The PostingIdentity used here is read from syncPipeline.Hub.Configuration, the final with-copied configuration — not from the configuration instance the pipeline delegate was captured against, which predates any WithPostingIdentity(...) call.

Phase 3 — Crossing the boundary: delivery carries the baton

delivery.AccessContext is part of the message payload. It survives:

• The action-block scheduler (Dataflow.ActionBlock → ThreadPool dispatch)
• Hub-to-hub routing (in-process via MessageService.RouteMessageAsync, cross-grain via MessageHubGrain.DeliverMessage, or remote via JSON serialisation)
• Rx schedulers (framework primitives capture into a closure and re-set on emission — see Phase 6)
• Thread-pool re-scheduling generally

If you find a place where identity is lost during Phase 3, it's a framework bug — patch the carrier, not the application.

Phase 4 — Receiver: delivery.AccessContext → AsyncLocal (beginning the next piece)

Before the handler body runs, two cooperating pieces of code set AsyncLocal from delivery.AccessContext:

• MessageService.NotifyAsync → MessageHubConfiguration.UserServiceDeliveryPipeline sets AsyncLocal for the delivery pipeline body.
• MessageHub.HandleMessageAsync sets AsyncLocal for the rule-chain / handler dispatch body, and MessageHub.RestoreUserContextOnEmission re-stamps delivery.AccessContext on every emission of the response observable.

Both wrap their inner work in try/finally so the previous value is restored when the piece finishes — the action-block thread that dispatched message N is correctly re-entrant for message N+1.

After this phase, application code runs with AccessService.Context = delivery.AccessContext.

Restoring the caller on the receiving hub

There is a third restore site, and it exists because it runs before the two above. AccessControlPipeline — the [RequiresPermission] gate — is added with AddDeliveryPipeline, and the pipeline list composes outside-in (Aggregate), so the gate is the outermost wrapper and UserServiceDeliveryPipeline the innermost. At the moment the gate evaluates, Phase 4 has not happened yet: AccessService.Context still holds whatever the action-block thread was left with — typically system-security from the hub's own bootstrap ImpersonateAsSystem, or nothing at all.

That matters because the permission fold snapshots the caller on the calling thread (PermissionEvaluator.ResolveFoldServices captures accessService.Context ?? accessService.CircuitContext, because AsyncLocal does not flow through the Rx schedulers cache.GetQuery uses), and it reads exactly two flags off that snapshot:

So the gate restores delivery.AccessContext itself, before subscribing the fold. Two rules:

• The cue is a REAL PRINCIPAL, never a payload shape. The restore is keyed on the delivery carrying a context with a non-empty ObjectId that is not hub-shaped. It is deliberately not keyed on anything the caller happens to have filled in.
• Hub-shaped ids keep the mesh-wide treatment. A sync/…, mesh/…, node/…, activity/… or portal/… id is not a user identity and is never installed as one — the same rule AccessService.SetContext's leak tripwire, UserServiceDeliveryPipeline's shouldStamp and MeshNodeStreamCache's pass-through all encode.

🚨 Why this is written down: the gate's own input used to decide whether the gate ran (#2976). The condition was delivery.AccessContext is { Roles: { Count: > 0 } } — restore only when the caller carries role claims. Its comment justified that in terms of claim-based role resolution, a mechanism that no longer exists (the 2026-08-05 paywall fix took claim roles out of node permissions; #2974 removed their last foothold). AccessContext.Roles is read nowhere in PermissionEvaluator — so the condition outlived its reason, and it was never the right one.

The consequence was silent and permissive. Most identity providers emit no role claims, so ApiToken.Roles is normally empty; such a token reached a per-node hub with Roles = [], the restore was skipped, capturedContext was null, IsApiToken was never seen, and the clamp did not run — the Bearer delivery was evaluated as an interactive session. The exact-read path was never exposed (MeshNodeStreamCache.GetStreamRaw captures the caller itself and wraps the evaluation in SwitchAccessContext), so api: false held for a read that named the page and was skipped for every message-routed check against it.

This is the runtime cousin of the CI rule in AGENTS.md — a gate never tests its own inputs — because "the check did not run" and "the check passed" were indistinguishable from outside: no exception, no log, a served read. When a condition guards whether a security decision is made, it must name the thing the decision needs (here: a principal to evaluate), never a field the decision does not read.

Pinned by RoutedApiTokenClampTest, whose control arm is the same person, same node, same routed message without the Bearer flag — so the pin measures a capability decision rather than a blanket deny.

Phase 5 — Handler runs under user identity

The handler body executes synchronously (or as a single observable chain — no await; see AsynchronousCalls.md). Every read of AccessService.Context returns the originating user. Every check (securityService.HasPermission(...), RlsNodeValidator.Validate(...)) is evaluated under the right principal.

Phase 6 — Handler posts → back to Phase 2 (chain continues)

The handler typically posts further messages or starts reactive chains. Two cases arise:

Direct post within the synchronous body. hub.Post(...) reads AsyncLocal → stamps the new delivery. Identity baton carries forward.

Cold observable + Subscribe (the Rx case). A framework write primitive returns a cold IObservable<T> whose side effect runs on Subscribe — but Subscribe may emit the callback on a different thread (an Rx scheduler) where AsyncLocal is wiped. The framework solves this internally via CarryAccessContext:

// AccessContextCaptureExtensions.CarryAccessContext (cross-cutting wrap)
public static IObservable<T> CarryAccessContext<T>(
    this IObservable<T> source, AccessService? access, bool restoreNullCapture = false)
{
    if (access is null) return source;
    // Capture Context ONLY — never `?? CircuitContext`. PostPipeline already reads
    // `Context ?? CircuitContext` at post time, and synthesising CircuitContext here
    // would leak the Blazor circuit identity into background Subscribe paths.
    var captured = access.Context;
    if (captured is null && !restoreNullCapture) return source;
    // Each OnNext/OnError/OnCompleted runs inside a SwitchAccessContext SCOPE that is
    // disposed as the callback returns — AsyncLocal is touched only for the callback.
    return new CarryAccessContextObservable<T>(source, access, captured);
}

🚨 Never re-implement this as source.Do(_ => access.SetContext(captured)). That was the earlier shape and it was reverted (2026-05-22 / re-fixed 2026-05-28): SetContext without a matching restore mutates AsyncLocal on whatever thread Subscribe ran on — often the caller's — and leaves the captured identity live for every subsequent operation on that logical execution context. That is the McpUpdate user1/user2 cross-contamination bug. The scope-per-callback shape above is the fix; a bare Do(...) re-introduces the leak.

restoreNullCapture is the second half of the contract. The write-result observables (MeshNodeStreamHandle.Update/Overwrite) pass true: a null capture is restored as null, so the framework identity ambient on the emission thread (the stream cache's read path runs ImpersonateAsSystem) cannot leak into the caller's callback and turn a nested write into a post-as-System escalation. Read/query pipelines default to false — a null capture passes through unwrapped, preserving whatever identity is ambient at emission. Flipping that default to clamp is blocked on migrating the ops/MCP call sites that currently depend on the leaked ambient identity.

This is applied by every mesh write primitive — MeshService.CreateNode/Update/Delete/CopyNode, MeshNodeStreamHandle.Update, IMeshNodeStreamCache.Update, and the content-file writes hub.ImportContent(path)…Post() / hub.SyncContentFiles(path)…Post() — so callers keep writing the natural shape:

meshService.CreateNode(node).Subscribe(_ => …);   // identity preserved across the Subscribe seam

Eager capture is half the contract, and the half that is easy to forget. A primitive that only wraps its result with CarryAccessContext still reads the ambient AsyncLocal when its Observable.Defer body runs — i.e. on the subscribing thread. The two content-import builders shipped that way and were the last write primitives without an eager snapshot: an import that built its operations under ImpersonateAsSystem and then drained them through a Concat pump had every node write land (those capture eagerly) and every content write failed closed for a null AccessContext (MeshWeaver.Reinsurance#46 — 412 applied, 409 refused, one call site). Snapshot the context where the caller calls you; pin it on the post with o.WithAccessContext(captured); wrap the result with CarryAccessContext. All three, or the primitive is not identity-safe. Pinned by ContentImportAccessContextTest.

Phase 2 then happens again from the Subscribe callback's thread, under the right identity. The chain continues.

Sanctioned exceptions — fine-grained, exact, controlled

Some components are the actor, not a proxy for a user: cache hydrators, redistributor hubs, framework bootstrapping. These are sanctioned by giving the component a named, dedicated identity — never an accidental hub-address-derived shape — paired with precise access rules that grant only what the component actually needs.

Three rules apply to every sanctioned identity:

1. Named, dedicated identity — e.g. cache/mesh-node-cache, not mesh/xxx. The address reflects the component's role, not the hub it happens to live on. Hub addresses (mesh/xxx, sync/xxx, node/xxx) are accidental — they describe placement, not purpose.
2. Fine-grained permissions — each identity is granted ONLY the specific operations it actually needs. Not "all protocol operations for sync/*"; rather "cache-read for cache/mesh-node-cache on these specific paths".
3. Tested boundary — every sanctioned identity has a test verifying that misuse fails. Posting a write under a read-only identity must yield UnauthorizedAccessException. Without this test, the sanctioning is voodoo.

Example 1 — cache/mesh-node-cache (read-only hydrator)

The MeshNodeStreamCache pre-loads MeshNodes from storage to serve cache hits for every user. It cannot run under a user identity because it services many. It is sanctioned via:

• Identity: cache/mesh-node-cache (single, reserved, internal-only constant).
• Grants: Read on the paths the cache hydrates. NOT Create, NOT Update, NOT Delete.
• Compensating control: every GetStream(path) call re-validates the requesting user's Read permission before returning data — the cache is the hydrator, not the gate.
• Test: post a CreateNodeRequest under cache/mesh-node-cache → expect UnauthorizedAccessException("Access denied: Create permission required …").

If anyone else tries to impersonate cache/mesh-node-cache (the constant is internal to the cache assembly), they fail at AccessControl because the address grants only what the cache needs.

Example 2 — IsPortalIdentity (user-node onboarding)

User onboarding is the canonical example of a hub-as-actor seat configured correctly:

• Identity: any address matching portal/* (the running portal hub's own address).
• Grant: UserNodeType.WithPortalCreate → AddAccessRule([Create, Update], (_, userId) => IsPortalIdentity(userId)).
• Why it's fine: portal hubs are the natural actor for onboarding — the hub identity IS the role here. No narrower seat would add safety because exactly one component class uses this code path and that class IS what portal/* matches.

Leave this pattern in place. The fine-grained refactor (dedicated component address, as in Example 1) applies when the hub address is accidental — the code happens to live on a mesh hub or sync hub but the role has nothing to do with message routing or sync streaming. When the hub IS the role, IsXxxIdentity(hub-prefix) is the right shape.

Example 3 — sync stream protocol vs sync stream user-data

This is the bug to be careful about. Sync streams carry two completely different kinds of traffic:

A SynchronizationStream.OnNext that unconditionally stamps ImpersonateAsHub(Hub.Address) collapses both onto the same hub address. The receiving owner then sees sync/xxx for a user's edit, and CreatedBy on the resulting MeshNode is sync/xxx. That's the user-identity-disappears bug.

Correct shape: protocol messages stamp protocol/sync-stream; user-data messages carry the user via CarryAccessContext (no impersonation at the post site). Two message types, two code paths, two identities — never collapsed.

Example 4 — system-security (true infrastructure)

accessService.ImpersonateAsSystem() switches the baton to "system-security", granted Permission.All unconditionally by PermissionEvaluator. Use only for genuine framework infrastructure with no user context AND no narrower sanctioned identity that fits — schema migration, framework-level bootstrap, internal recursive lookups inside PermissionEvaluator itself.

Every ImpersonateAsSystem callsite is a deliberate choice to bypass all access checks. Grep for them; review each. If a narrower identity would suffice, use that instead.

🚨 An impersonation scope must not ESCAPE the operation it was opened for

accessService.RunAsSystem(work) — not a hand-written Observable.Using(access.ImpersonateAsSystem, _ => work) — at every site, whether the scoped observable is returned to a caller or subscribed on the spot.

The hand-written idiom is correct for the work itself and leaks in two directions at once — forwards onto what the subscriber composes, and BACKWARDS onto the thread that subscribed:

// ❌ the scope escapes: Rx forwards OnNext to the subscriber BEFORE Using disposes its resource,
//    so the write below is CONSTRUCTED while the impersonation is still open — and the write
//    primitives eager-capture AccessService.Context when they are CALLED.
Observable.Using(access.ImpersonateAsSystem, _ => ReadGatedSource())
    .SelectMany(rows => meshService.CreateNodes(Plan(rows)));   // lands as `system-security`

// ✅ the scope covers the read and stops at its boundary
access.RunAsSystem(ReadGatedSource)
    .SelectMany(rows => meshService.CreateNodes(Plan(rows)));   // lands as the CALLER

Because those primitives also re-stamp the captured identity around their own emissions (CarryAccessContext — "MessageHub sets, framework primitive preserves"), a leaked System identity is not merely inherited one hop — it is re-acquired at every hop after it. That is how a package install wrote its home root, five copy batches and its manifest as system-security while _UserActivity — the one write on that path posted directly from the request thread rather than composed on an emission — attributed to the real user (#1444).

It is an authorization concern, not only an audit one. AccessControlPipeline.HandleGetPermission carries the scar: SecurityService's bootstrap-time system scope leaked past its using-block onto the action-block thread, and trusting the ambient there returned Permission.All for every caller, anonymous included. That call site defends itself by resolving the identity explicitly; RunAsSystem fixes the class at the source so the next one does not have to know Rx's disposal order.

…and the other direction: the SUBSCRIBING thread keeps the identity (#1790)

Impersonation is an AsyncLocal store/restore pair, so both halves must land on ONE logical flow. Observable.Using runs its resource factory on the subscribing thread and disposes the resource when the inner observable terminates — for a cross-hub request/response, the owning hub's response thread. Nothing ever disposes it on the subscriber, so:

// ❌ opens on THIS thread, closes on the hub's response thread
Observable.Using(access.ImpersonateAsSystem, _ => hub.Observe(request)).Subscribe(...);
// …everything after this line on this thread runs as system-security

Measured consequences, all silent:

• The auth bootstrap. UserContextMiddleware.ValidateTokenViaHub subscribes on the ASP.NET request thread. The latch left the request running with Permission.All, and InvokeAsync's existing.Email == userContext.Email reuse branch then adopted the System context as the caller's own.
• A script run. ActivityLogLogger's first publish is issued from the script's own thread; the latch made the rest of the script System, and a --render export resolved embedded areas the submitting user may not read.

RunAsSystem / RunAsHub / RunAs open the scope at Subscribe and close it on the way out of that same Subscribe. That still covers the work: everything a cold pipeline does eagerly happens inside Subscribe — the factory runs, the primitives eager-capture the identity, the post is stamped, and any scheduled continuation captures the ExecutionContext as it stands right then. A captured ExecutionContext is an immutable snapshot, so restoring the subscribing flow afterwards cannot un-elevate work already scheduled.

🚨 ContainIdentity does not close this half — it restores around NOTIFICATIONS, never around the Subscribe that opened the scope. Where the capture is genuinely synchronous, a plain using around the call and its Subscribe is equally correct (ActivityLogLogger).

The second half of the leak — the caller's previous identity being written onto the terminating thread, which is an identity injection into a hub action block, not a cleanup — is closed inside AccessService itself: a scope's restore is thread-affine, so a dispose on a flow that never carried its store writes nothing. ImpersonationScopeSiteRatchetGuard + test/ImpersonationScopeSites.allow keep the inventory of sites still written the old way from growing.

🚨 It never invents an identity. What is restored is exactly what was ambient at Subscribe: a real user for a user-driven flow, system-security for a genuinely system-initiated one, and nothing at all when the subscriber had no identity — so a background worker that never had a user is not fail-closed by adopting it. The only behaviour that changes is the defect: a caller who had an identity, silently continuing as System.

Pinned by SystemScopeDoesNotEscapeTest (test/MeshWeaver.Messaging.Hub.Test), whose first case asserts the raw Rx ordering every other case rests on.

Implementation: define + grant + test

For each sanctioned identity, you need all three:

1. Define an internal const string for the identity's name inside the component's assembly:

// src/MeshWeaver.Hosting/MeshNodeCacheIdentity.cs
internal static class MeshNodeCacheIdentity
{
    internal const string Address = "cache/mesh-node-cache";
}

Keeping it internal means only the assembly that defines it can stamp the identity — external code cannot impersonate.

2. Grant precise permissions via per-NodeType access rules:

config.AddAccessRule(
    [NodeOperation.Read],
    (_, userId) => userId == MeshNodeCacheIdentity.Address);

3. Test the boundary:

[Fact]
public async Task MeshNodeCache_Identity_CannotWrite()
{
    using (accessService.SwitchAccessContext(new AccessContext { ObjectId = "cache/mesh-node-cache" }))
    {
        // 🚨 Materialize, not act.Should().ThrowAsync() over an Rx ToTask bridge:
        // ToTask is forbidden repo-wide (2026-08-30), and folding OnError into a
        // value keeps the ORIGINAL exception type assertable. The write is cold, so
        // the assertion's Subscribe IS the write.
        var error = await meshService.CreateNode(someNode).Take(1).Materialize()
            .Should().Match(n => n.Kind == NotificationKind.OnError);
        error.Exception.Should().BeOfType<UnauthorizedAccessException>();
        error.Exception!.Message.Should().Contain("Access denied");
    }
}

The triple — define / grant / test — is the contract. Without all three, the sanctioning is voodoo and the system is one change away from privilege escalation.

Persistence runs as System — the persistence queue

Authorization is enforced once, at the user-facing write — the app handler / stream.Update on the owning hub's action block, where the user's identity is live and RLS gates the write. Once a change is accepted there it is already authorized. The durable DB write happens later, on the dedicated persistence hub (CreatePersistenceAddress()) inside DataSourceWithStorage.Synchronize, which fires on the workspace emission scheduler — a background thread that has wiped the AsyncLocal identity.

So the persistence write must run under System:

protected override void Synchronize(ChangeItem<EntityStore> item) =>
    persistenceHub.InvokeAsync(async ct =>
    {
        using (accessService?.ImpersonateAsSystem())   // already authorized → durably store as System
            await UpdateAsync(item, ct);
    }, ex => { logger.LogWarning(ex, "Updating {DataSource} failed", Id); return Task.CompletedTask; });

If you instead let the persistence write inherit the (null) ambient context, it fails closed and the change is silently dropped — a write that "succeeded" upstream never lands. When the dropped write is an _Activity node, every progress reader then subscribes to a node that does not exist → a [ROUTE] NotFound resubscribe storm that wedges the partition (production outage 2026-06-17, compile + import activities). Persistence is the bottom of the stack: it stores what was already approved; it never re-gates and never fail-closes.

Posting/redirecting to a node: do it inside Create().Subscribe()

A SubscribeRequest — or any post/redirect — to a node that does not exist is the root of the resubscribe-storm/wedge class. Never advertise, subscribe, or redirect to a node path until you have proof it exists, and the only proof is the create's own completion:

// ✅ The node provably exists inside the Subscribe — only THEN advertise / redirect / let readers subscribe.
meshService.CreateNode(activityNode).Subscribe(
    created => { /* now stamp the path, point readers at it, redirect, … */ },
    ex => logger.LogWarning(ex, "create failed — advertise NOTHING; no reader can storm a phantom"));

// ❌ Stamp a `_Activity/compile-*` path BEFORE/without the create landing → readers subscribe
//    to a phantom node → endless [ROUTE] NotFound → wedge.

RunCompile (NodeTypeCompilationHelpers) is the worked example: it provision-orders the create, observes it (bounded), and stamps LastCompilationActivityPath only on the confirmed create — null otherwise, so no reader ever subscribes to a path that was never written. Combined with "persistence runs as System" above (so the create actually lands), there is nothing to storm.

Anti-patterns (the baton drops)

Debug aid: the hub-shape error log

AccessService.SetContext / SetCircuitContext log an ERROR with a stack trace whenever a hub-shaped principal (sync/, mesh/, node/, activity/, portal/) is set on AsyncLocal:

[Error] [MeshWeaver.AccessContext] SetContext: hub-shaped principal sync/xxx set as AccessContext — must never happen. Source stack: …

When this fires:

• Sanctioned init / cache hydration / protocol traffic — expected at startup; can be downgraded once the legitimate sites are audited and tagged as sanctioned.
• Application code under user-driven flows (layout-area renders, click actions, watcher emissions caused by user writes) — this is a bug. Trace the stack to find where the user's identity was lost upstream and apply CarryAccessContext at the seam.

IMeshNodeStreamCache.GetStream is access-checked

Beyond ensuring writes preserve user identity, reads through the process-wide IMeshNodeStreamCache are also gated by the caller's effective Read permission on the path.

🚨 The gate is evaluated LOCALLY — it does NOT post a GetPermissionRequest to the path's hub. MeshNodeStreamCache.ProbeEffectivePermissions calls meshHub.GetEffectivePermissions(path, userId) (→ the static PermissionEvaluator) directly. Only when the result has Permission.Read does the gated observable forward upstream emissions; otherwise it terminates with UnauthorizedAccessException.

Why local, and why the hub round-trip was removed: GetEffectivePermissions walks the path's whole scope hierarchy — root, partition, every ancestor, the node itself — so access defined on the main node already covers every satellite under it ("who can read the main node can read all its satellites"). Asking for the PATH is therefore sufficient. The old probe targeted new Address(path), and a satellite / cell sub-path with no hub of its own — a {thread}/{messageId} the GUI subscribed to that was never persisted, or a brand-new thread — routes to NotFound, so nothing answered and the probe blocked for the full 15 s timeout: the side-panel "thread won't open" spinner. Do not reintroduce a leaf-hub permission probe on the read path.

The caller's captured AccessContext is restored around the synchronous evaluator capture (accessService.SwitchAccessContext(captured)) so claim-based (Bearer-token) roles on AccessContext.Roles resolve — PermissionEvaluator snapshots accessService.Context on the calling thread before any Rx scheduler hop.

Per-(path, userId) validations are cached in-process for 30 s (the AccessTtl constant in MeshNodeStreamCache.cs). Revocation surfaces within at most that window; the cache is not invalidated reactively. The shared upstream is unchanged — only the returned subscriber-side observable is gated.

GetPermissionRequest contract

GetPermissionRequest still exists as a message — it is how a caller asks a specific per-node hub what it grants (the test helper PermissionTestExtensions, and the request-time pipeline). It is simply no longer what the stream cache uses.

public record GetPermissionRequest : IRequest<GetPermissionResponse>;
public record GetPermissionResponse(Permission Permissions);

The request carries no path — the receiving hub answers for its own path. Callers route the request to the per-node hub at the path they care about; routing decides which hub responds. The handler (AccessControlPipeline.HandleGetPermission) resolves the per-hub-scoped PermissionEvaluator and evaluates it against the caller's delivery.AccessContext.

Worked example — Blazor data binding → SyncStream → owner update

Walking through all six phases with a concrete scenario:

1. Phase 1. User logs in. CircuitAccessHandler sets accessService.CircuitContext = user for the circuit's lifetime. Each inbound circuit activity wraps the dispatched callback in using SwitchAccessContext(user) so AsyncLocal is set per-render too.

2. Phase 5 (inside a layout area render). The user edits a text field bound to JsonPointerReference("$.title"). The binding pushes the new value into a SynchronizationStream<MyContent>.

3. Phase 6 (Rx hop). The stream's OnNext fires. The Rx scheduler's thread has wiped AsyncLocal — but the stream's subscribe-time wrap captured user into a closure (via CarryAccessContext on the cold pipeline that fed OnNext). Before posting SetCurrentRequest, AsyncLocal is restored to user.

4. Phase 2. Hub.Post(new SetCurrentRequest(value), o => …) reaches PostPipeline → reads AsyncLocal → stamps delivery.AccessContext = user. No ImpersonateAsHub.

5. Phase 3. Delivery travels to the owner hub.

6. Phase 4. Owner hub's MessageHub.HandleMessageAsync reads delivery.AccessContext → SetContext(user) → runs HandleSetCurrent.

7. Phase 5. HandleSetCurrent writes through the owner-side data layer. AccessControlPipeline validates that user has Update on the path. CreatedBy / LastModifiedBy on the resulting MeshNode rows is user.

If step 3 had stamped ImpersonateAsHub(Hub.Address) instead of carrying user, step 4 would set AsyncLocal to sync/xxx (caught by the error log), step 5 would validate sync/xxx (no grant configured) and either deny or — if a misconfigured rule grants it — attribute the write to the sync hub. That's precisely the bug this model is designed to prevent.

Related docs

• @Doc/Architecture/AsyncLocalAcrossHops
• @Doc/Architecture/AsynchronousCalls
• @Doc/Architecture/CqrsAndContentAccess
• @Doc/Architecture/AccessControl
• @Doc/GUI/DataBinding
• @Doc/Architecture/ActivityControlPlane
• @Doc/Architecture/LogonActions