Debugging Message Flow & Hangs

When a hub handler looks like a deadlock — a test times out, a response never arrives — resist the urge to bisect blindly or rerun the test two or three times to see if it sticks. The framework already emits a structured trace at Trace level. Turn it on, run once, grep, fix.

Healthy request-response trace: Hub A routes the request through the hierarchy to Hub B, which executes the handler and sends the response back.

Log Levels — Edit the Test appsettings, Not the Source

ABSOLUTE: Never flip LogInformation ↔ LogDebug ↔ LogTrace in src/ for a debugging session. Log levels in source code are a production cost contract — every Information line is emitted to stdout and shipped to Loki (via Promtail), where ingest/retention has a cost. Toggling them temporarily silently bleeds budget the next time the branch is deployed.

Two sanctioned paths to raise verbosity:

If a Log* call is genuinely too noisy or too quiet at its current level, fix it permanently with a commit explaining the cost/value trade-off — never sneak it in alongside an unrelated change.

One-Shot Recipe

1. Crank logging to Trace in the test's runtime appsettings:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "MeshWeaver.Messaging": "Trace",
      "MeshWeaver.Data.Serialization": "Trace",
      "MeshWeaver.Mesh": "Debug",
      "MeshWeaver.Hosting.Persistence": "Debug",
      "Microsoft": "Warning",
      "System": "Warning"
    }
  }
}

2. Run the failing test once, capturing all output:

dotnet test test/<Suite> --no-build --filter "FullyQualifiedName~<TestName>" \
    > /tmp/trace.log 2>&1

3. Grep the structured tags:

grep -E "MESSAGE_FLOW|SYNC_STREAM|exception occurred|deliveryId" /tmp/trace.log

If the test hangs inside the runner, per-test log files are captured here: test/<Suite>/bin/Debug/net10.0/test-logs/<TestClass>_<TestMethod>.log

Structured Trace Tag Reference

Every tag below is emitted as a single structured log line. The source column names the file and the emitting method — grep for the tag itself rather than a line number, which goes stale on the next edit to the file.

Reading the Trace

For a request-response hang, the healthy timeline looks like this:

[T0]  MESSAGE_FLOW: Unpacking …                           (request arrives at Hub A)
[T0]  MESSAGE_FLOW: ROUTING_TO_HIERARCHICAL Target=B …
[T1]  MESSAGE_FLOW: Unpacking … Hub=B
[T1]  MESSAGE_FLOW: ROUTING_TO_LOCAL_EXECUTION Hub=B
[T2]  <handler logs at Information level>
[T3]  MESSAGE_FLOW: Unpacking …                           (response arrives back at Hub A)
[T3]  hub.Observe(request) emits, then completes          (the caller's subscription fires)

hub.Observe(...) is the only request/response primitive — there is no RegisterCallback registry and no AwaitResponse on IMessageHub to grep for. Observe is AsyncSubject-backed and pre-registers before posting, so a synchronously-handled response cannot slip past it.

Find the last MESSAGE_FLOW: line that fired and look at what should have happened next:

The handler-side fate trail — read it before reconstructing the table above by hand

The table above is a manual reconstruction from Trace-level MESSAGE_FLOW: lines, which are off by default and invisible in CI. For a request someone is still awaiting, the framework now keeps that reconstruction for you: RequestFateLedger records the stages the delivery actually reached, and the trail is printed whenever a pending callback is reported.

[QUIESCE-TIMEOUT] client/475b…: 1 callback(s) still pending after 0.3s.
  Pending: 0iFBGDmtTUyOnOgbHS3ezQ=SwallowedRequest@host/1(302ms)
  handler-side fate (what happened to the delivery):
    0iFBGDmtTUyOnOgbHS3ezQ=SwallowedRequest:
      POSTED target=host/1@client/475b…(+0ms) → RECEIVED runLevel=Started@client/475b…(+0ms)
      → ENQUEUED@client/475b…(+0ms) → RECEIVED runLevel=Started@mesh/1(+1ms)
      → ROUTED onTarget=False state=Forwarded@mesh/1(+1ms) → RECEIVED runLevel=Started@host/1(+1ms)
      → ROUTED onTarget=True state=Submitted@host/1(+1ms) → HANDLER_ENTER@host/1(+1ms)
      → HANDLER_EXIT state=Processed@host/1(+1ms)

Each trail ends in a verdict (⇒ …) naming which failure shape it is, so you do not have to infer it from the stages. Verdict and stages share one line, deliberately: these records are read out of /tmp/meshweaver-test-trace.log with grep, and a newline before the verdict would let grep <messageId> return the evidence while hiding the conclusion.

The first stage is always AWAITING …, written when the callback is registered. That guarantees a trail is never empty — so "AWAITING and nothing else" is a positive statement (nothing was ever posted under this correlation), not an absence of evidence.

Read it by what is missing:

Reading an ORDER defect: the queue-and-depth stamps

🚨 A hub has TWO queues, and a delivery moves between them at TURN time. mainQueue holds turns in arrival order; a turn that is dequeued and found on-target with an initialization gate closed is pushed onto deferredQueue — so it leaves mainQueue — and OpenGate later drains the deferred run back to the front (MeshWeaver#3408). Any question of the form "why did B run before A?" is therefore a question about which queue each delivery was in and how many turns were ahead of it, and a trail that records only ENQUEUED / DEFERRED cannot answer it: a delivery parked in front of an empty queue and one parked in front of two messages that then ran look identical.

Every transition consequently carries both depths:

🚨 A turn can be in NEITHER queue — dequeued, and not yet at its gate check. That is where the reorder that survived #3408 lived, and no depth stamp can see it because it is in neither count. Read Turn-Loop Arrival Order before attributing an order defect to a queue: it gives the two windows, why the permutation names the message that straddled the gate open rather than the defect, and the invariant that now holds.

🚨 The depth is a SEPARATE QUEUED stage, and that is not cosmetic — a stage token is a matched CONTRACT. Stages render as {stage}@{hub}, and suites wait on that literal substring:

// DisposalRaceNackTest, SubscribeDuringRecycleTest
.Where(trail => trail.Contains($"ENQUEUED@{victimAddress}", StringComparison.Ordinal))

Writing the depth inside the token — ENQUEUED queue=main depth=1 — deletes ENQUEUED@…, those waits never fire, and the suites time out looking like routing stalls that have nothing to do with the change. Measured while adding these stamps: 0/5 with the token altered, 5/5 with it restored. So append new stages; never edit an existing token. MessageTrace lines are free-form, but a fate stage's leading token is API.

Why this exists (Plugins#1394). ActivationBacklogFifoTest has produced four different permutations — A, C, B, B, C, A, C, A, B and B, A, C — for one address and, in the captured case, Distinct probe-hub instances: 1. A structural read of RoutingServiceBase → MonolithRoutingService.RouteImpl → MessageService says every one of them is unreachable: the per-address ActivationSerializer is inbox.Select(RouteOne).Concat(), so a later message's routing begins only after the earlier one's hub.DeliverMessage has run; the defer decision and the deferredQueue.Enqueue are inside the same lock (gateStateLock) that OpenGate also takes; and DeferredTurnsResumeAheadOfLaterArrivalsTest pins the restore ordering deterministically. When the code says a reorder cannot happen and it keeps happening, the missing evidence is where each delivery waited — not more reading.

🚨 A permutation does not name a mechanism. Four distinct ones from one test is itself evidence that this is not a single fixed swap, and triaging from the permutation alone has already sent two investigations to the wrong subsystem.

Three things to know before trusting it:

• It only covers awaited requests. Entries exist for exactly the ids with a live Observe callback and are dropped the moment one resolves — that is what keeps the ledger bounded. A fire-and-forget post leaves no trail by design.
• It is scoped to one hub TREE. The root hub creates the ledger and hosted hubs inherit it, so a request that leaves the tree (another silo, another grain) records POSTED and then goes quiet. That silence means "left this tree", not "vanished".
• It cannot see a reply sent under the WRONG correlation. That case reads as HANDLER_EXIT with no RESPONSE_POSTED — same as a handler that replied to nothing. If you suspect it, look for a spurious RESPONSE_POSTED on a different request's trail.
• A long trail keeps its head and its TAIL, suppressing the middle. The terminal stages decide the verdict, so they are never the part that gets dropped.

When the handler replies from work it detached

HANDLER_ENTER / HANDLER_EXIT bound the rule chain only. The canonical mesh handlers return request.Processed() immediately and owe their reply from a composed observable they subscribed and let run — so for those, HANDLER_EXIT state=Processed tells you nothing about whether a reply is coming. A handler that answers in 3 s, one that faults silently, and one whose chain completes EMPTY all look identical from the pipeline.

Only the handler can split them, so it records its own terminal arms:

hub.NoteRequestStage(request.Id, $"CREATE_CHAIN_EMITTED mode={mode}");   // produced a result
hub.NoteRequestStage(request.Id, $"CREATE_CHAIN_ERROR {ex.GetType().Name}");
hub.NoteRequestStage(request.Id, "CREATE_CHAIN_COMPLETED_EMPTY …");      // ← terminated with nothing
hub.NoteRequestStage(request.Id, $"CREATE_SAVE_DECLINED adapter={…} path={…}");

The two handlers behind every cross-hub mesh-node write record their arms the same way (MeshWeaver#2543 — the bake wedge's trail ended at HANDLER_EXIT state=Processed with nothing after it, on both hops, and could not say which wait it was in):

NoteRequestStage is a no-op unless something is awaiting that id, so it is free to call unconditionally. Add the onCompleted arm: a chain that completes empty posts nothing, and without a stage there it is indistinguishable from one that is still running. HandleCreateNodeRequest is the worked example.

A detached chain must ANSWER on every terminal arm, not just record one

Recording the arms is diagnostics; answering on them is the contract. HandleCreateNodeRequest carries both, and the shape is worth copying:

• Every terminal post goes through ONE local Respond(...) that flips a responded flag. That is what makes the backstop exact rather than approximate.
• The onCompleted arm posts a failure only when nothing emitted AND nothing answered. Both guards are load-bearing: emitted means the reply is owed by the post-success subscription (whose own arms answer), and responded means a branch already answered and posted its own, more specific, rejection.
• CREATE_SAVE_DECLINED names the adapter behind a null from IStorageAdapter.Write — the try-then-claim sentinel meaning "this adapter does not own this path", not "the write succeeded". That null used to be filtered away with .Where(n => n is not null), which is exactly how a create terminated with no reply. It now FAULTS with the same message family the composite PersistenceService.Write already threw, so the answer no longer depends on which adapter the hub resolved.

The general rule: if a handler owes its reply from a detached observable, the observable's onCompleted is a terminal arm like any other. Leaving it unhandled is not "nothing to do" — it is the one path that hangs the caller forever.

Terminated vs still-running: a Timeout catches only ONE of them

This is the distinction that left several #981 captures unexplained, and it is worth internalising because it applies to every bounded reactive chain in the mesh:

.Timeout(...) faults on SILENCE, not on a clean finish. A source that completes without emitting sails straight through Timeout, through any Catch behind it, and through every SelectMany downstream — producing nothing, reporting nothing, bounded by nothing.

So a chain guarded only by Timeout still has an unhandled terminal case. The two failure shapes need different evidence and have different fixes:

Silent Completion treats that second row on its own terms — the shape, its instances outside the request/response path (a filtered decline sentinel, a render generator that never delivers), and how to guard a chain so its empty terminal case fails closed.

EnsurePartitionBootstrap's authorization probe carries both stages for exactly this reason — BOOTSTRAP_PERM_AWAIT before the fold, then one of BOOTSTRAP_PERM_VERDICT / BOOTSTRAP_PERM_FAULTED / BOOTSTRAP_PERM_COMPLETED_EMPTY. A capture that ends at BOOTSTRAP_PERM_AWAIT is a create waiting on a slow-but-healthy permission fold, not a terminated one.

That distinction matters for reading a quiescing-timeout report. The permission fold is bounded at 15 s (a cold-start synced query on a fresh partition legitimately takes seconds), while the teardown quiescing budget that detects the pending callback is 2 s. A create caught mid-probe is therefore reported as a leaked callback long before its own bound would have fired — the report is the detector, not the defect, and raising either number fixes nothing.

Can that fold complete empty in practice? Through the shipped evaluator, no — and the reason is the same reason it can stall. PermissionEvaluator's fold rides SyncedQueryMeshNodes, whose allChanges is upstream.Merge(externalChanges).Merge(feedRemovals); externalChanges is a Subject that is never completed, and Merge completes only when every source does. So that substrate can never complete at all — only stall, which the Timeout does catch. The empty case remains reachable only through the EffectivePermissionsDelegate DI extension point (an evaluator returning Observable.Empty<Permission>() is a legal implementation), which is why the empty arm is a guard rather than a hot path.

The same block is printed by [STALE-CALLBACK] (every 5 s, for callbacks older than MESHWEAVER_STALE_CALLBACK_MS, default 30 s) — that is the one that fires while the mesh is still live, so a repro run that dials the env var down gets the handler side before teardown is involved.

The Cross-Hub Border — JsonSynchronizationStream

workspace.GetRemoteStream<TReduced, TReference>(addr, ref) subscribes via a SubscribeRequest posted to the owning hub. If the owning hub has no handler for SubscribeRequest — or no matching reducer — the subscription receives a DeliveryFailure instead of a SubscribeResponse, and the SynchronizationStream errors out:

[Warning] [MeshWeaver.Data.Serialization.SynchronizationStream] Stream <id>
   received DeliveryFailure: No handler found for message type SubscribeRequest

This is the smoking gun for the "remote read returns nothing" class of hang. The fix lives on the owning side (register a SubscribeRequest handler or add a MeshDataSource so the hub has the reducer), not on the caller side.

Type-Registry Mismatch

The handler is registered, but the message arrives at the receiving hub as JsonElement because the receiver's ITypeRegistry is missing the type. The sender surfaces a clean exception:

DeliveryFailureException: Could not deserialize message in hub <addr> —
    type 'MyRequest' is not registered in this hub's TypeRegistry.

MessageService.DeserializeDelivery catches the JsonElement fallback, calls ReportFailure(delivery.Failed(...)), and posts the DeliveryFailure back via the standard ResponseFor(delivery) path — the sender's hub.Observe(...) surfaces it as OnError.

Fix: add the type to the receiving hub's config:

hub.WithTypes(typeof(MyRequest), typeof(MyResponse));

For Orleans deployments, register on both sides — the silo's hub config and any client/portal hub that posts the request. A ping-pong guard suppresses DeliveryFailure responses when the inbound $type is itself DeliveryFailure, so a misconfigured pair won't spin forever.

FQN vs Short-Name Mismatches

The wire $type discriminator must match the receiver's registered typeName. The polymorphic serializer picks the discriminator from the sender's ITypeRegistry:

• Sender registered WithType(typeof(T), nameof(T)) → wire $type is the short name ("CreateNodeRequest").
• Sender's registry lacks the type → falls back to FullName ("MeshWeaver.Mesh.CreateNodeRequest") at serialize time.

A receiver that registered WithType(typeof(T), nameof(T)) only matches short names, so an FQN on the wire fails the lookup and produces a DeliveryFailure even though both sides technically "have" the type.

Triage with the file trace (MESHWEAVER_MSG_TRACE=1). The file is Path.GetTempPath() + meshweaver-msg-trace.log — %TEMP%\… on Windows, but on macOS that is $TMPDIR (a per-user /var/folders/… directory, not /tmp) and on Linux /tmp. Find it with ls "${TMPDIR:-/tmp}"/meshweaver-*.log; a POSIX shell expands $TEMP to the empty string, so a copied-from-Windows "$TEMP/meshweaver-msg-trace.log" silently reads /meshweaver-msg-trace.log and finds nothing. The NotifyAsync ENTER line stamps msg=... with the JSON $type discriminator from RawJson.Content. If it reads msg=MeshWeaver.Mesh.CreateNodeRequest (FQN) instead of msg=CreateNodeRequest (short), a hub somewhere along the hop didn't register T in its TypeRegistry — register on every hub the message transits, not just the originator and the final target.

For test setup specifically: MessageHubConfiguration.TypeRegistry is mutable per call (WithType returns the same instance), so configuration.TypeRegistry.AddAITypes(); (discarded return) is sufficient — but the call must reach the configuration of every hub that serializes the message, including hosted sub-hubs like {path}/_Exec and any cross-cutting ConfigureDefaultNodeHub chain.

Auto-registration warms only the SERIALISING hub

A type nobody registered explicitly still gets a short-name $type: PolymorphicTypeInfoResolver auto-registers an unregistered, non-collectible type the first time a hub writes one (it logs a Warning naming the hub). That entry lands in that hub's registry only — GetOrAddType writes the local map and never the parent, and a child registry can read its parent but not the reverse.

So which hub does the writing decides where the mesh learns a content type. Move a workload to a different hub and every hub whose registry chained to the old one starts reading that content back as an untyped JsonElement: Content is T goes false, validators stop firing, views render empty. The tell is a pair of lines — Unregistered type … on hub A followed by Received '$type':'…' which is NOT registered in this (receiving) hub on some hub B.

When two hubs must be one serialization identity, share the registry rather than registering twice:

mesh.GetHostedHub(address, config => config
    .WithTypeRegistry(mesh.TypeRegistry)   // FIRST — before AddData()/WithType/WithHandler
    .AddData());

WithTypeRegistry carries anything the configuration already registered over to the shared registry (and never clobbers an entry the shared one already owns), so a call made out of order still cannot lose a registration. It is deliberately rare: the mesh's node-operation execution hub uses it because node CRUD serializes node Content on behalf of the whole mesh.

Common Gotchas

• A handler that throws uncaught — e.g. workspace.GetStream<T>(reference) where the reducer isn't registered throws InvalidOperationException("Failed to create stream") and crashes the delivery pipeline. The exception appears in the An exception occurred … line. The original caller receives no response and times out. Wrap the call or verify upstream that the stream exists.

• Take(1).Timeout(15s) on a never-emitting source — results in a 15-second wait followed by TimeoutException. Always pair Timeout with Catch so the chain emits a sentinel value rather than dying.

• Subscribe callbacks run on arbitrary threads. State updates that need the hub scheduler must happen in the handler body, not inside a Subscribe callback. See AsynchronousCalls.md.

🚨 Reading a request timeout: it describes the hub that GAVE UP, not the one that stayed silent

TimeoutException: No response received in hub X within 00:01:00 for request Y → target Z is the most common banner in this system, and the most commonly misread. It is written by the waiter. The waiter cannot see the target at all — it can only report that nothing arrived, which is consistent with several different worlds.

The message used to end "The request may have been undeliverable or the target hub was not found". Two buckets, asserted as though they were the whole set. They are not, and the missing third is the one to rule out first:

The last one is invisible from the sentence and indistinguishable from the others. A single-threaded actor whose action block is busy, gated, or backed up looks — from inside — exactly like a peer that never replied. So the message now carries the waiter's own RunLevel and queue snapshot, and states the classification explicitly:

• queue non-empty, gates open, or a message executing → this hub was not idle, so it cannot attribute the silence upstream. Investigate here first; the answer may be sitting behind the work in that snapshot.
• idle → the silence really is upstream, and the message says so — then names the three remaining worlds and states that it cannot distinguish them. An explicit unknown, because a diagnostic offering two options when there are four teaches the reader to pick the nearer one.

The production case that motivated it

2026-09-02: a document read failed with the old wording, naming cache/… as the waiter and a node path as the target. Both offered buckets were wrong. The node existed — version 53, edited the previous evening — and its hub resolved fine; exactly one per-node hub was wedged while every sibling answered instantly. The sentence sent the reader to "does this node exist?", the one question never in doubt.

The control arm is what settled it, and it is cheap enough to run every time:

point-read the failing path      -> times out (twice, different ids => deterministic, not transient)
point-read a SIBLING path        -> instant  => partition, store and routing are fine
list the namespace via the index -> healthy  => the node exists and is indexed

Two probes separate "the mesh is broken" from "one hub is wedged", and the second is the answer far more often.

🚨 recycle fixes it and destroys the evidence

recycle posts a DisposeRequest and forces re-initialisation. It cleared the wedge above with no data loss (all 53 versions intact) — and it also destroyed the only state that could have explained it. Capture before recycling when the situation allows: the owner's RunLevel, queue depth, and whether it sits in Starting with an open gate. Recycling first is the right call when someone is blocked on live data; just say so plainly rather than reporting a diagnosis you no longer have the evidence for.

The string coupling nobody can see

🚨 MeshNodeStreamCache.IsTransientOwnerFailure and AreaErrorClassifier.IsTransientHubFailure decide retryability by substring match on this message. "No response received in hub" is what keeps a hub timeout classified RETRYABLE. Change the wording without that phrase and every such read becomes terminal — silently: no compiler error, no exception, just reads that stop being retried and wait out their budgets. MessageService documents the same hazard from the other direction (a deleted-address NACK must NOT contain any of those markers, or a provable absence degrades back into "retry shortly"). Pinned by TimeoutMessageNamesTheCallersOwnStateTest.TheTimeoutMessage_KeepsTheMarkerThatClassifiesItAsTransient.

The Golden Rule

Run once. Grep the trace. Fix the root cause.

Don't rerun "to see if it still sticks" — it will, and you'll waste minutes per cycle. The trace tells you exactly which message went missing and why.

When you find the broken edge, leave the relevant LogTrace / LogDebug lines in place. They cost nothing at higher log levels and are the only way to debug the next analogous failure without re-instrumenting the code from scratch.

"Deadlock" that is really a missed observation — resurrection on init

A whole class of "hangs" are not locks. The signature in the MESHWEAVER_MSG_TRACE file (meshweaver-msg-trace.log under the temp dir — see above) is decisive:

• Real lock-deadlock — one large gap where nothing runs, then the test times out. The action block is wedged on a blocking continuation.
• Missed observation — the hub runs a burst of work (seconds), then goes completely silent for the rest of the timeout. The work finished; the thing waiting on it never saw the terminal state. No gap-during-work, no lock.

To tell them apart, compute the max gap between handler-enters for the stuck node (grep "hub=<path> " trace | grep "HandleMessageAsync ENTER" → diff the timestamps). A big mid-work gap ⇒ lock. Continuous work then silence ⇒ missed observation. The volume asymmetry is another tell: a synced-query subscription re-emitting on every change shows far more GetDataResponse than GetDataRequest, and the count scales with load — slower round-trips ⇒ more iterations.

The root pattern

A long-lived operation (a thread round, a parent waiting on a delegated child, an activity) is driven by an in-memory observer — a Subscribe on a node stream, a TaskCompletionSource resolved by a callback. Two ways that observer silently dies and the operation parks forever:

1. One-shot with a give-up. stream.Take(1).Timeout(15s).Subscribe(...) — if the loaded-state emission is dropped during the subscribe handshake (see the init-gate-drops-patches note) or merely arrives late under load, the Timeout fires onError, the recovery gives up, and never retries. The node stays non-terminal forever.
2. Lost on reactivation. The observer lives only in the agent-loop / grain that set it up. When the grain deactivates and reactivates (Orleans) or the hub re-inits, the subscription is gone and is never rebuilt, so the child's eventual completion is never observed.

The fix — self-healing resurrection on init

Lifecycle recovery (ThreadExecution.InitializeThreadLifecycle, and the analogous activity init) must obey:

• Re-establish, never give up. Wait for the first real state emission however long it takes; if the observation faults before it drives the node to a valid state, re-subscribe (restart the watcher). No Timeout(...)-then-give-up.
• Restart if any observer dies before terminal. An observer that completes or errors while the node is still non-terminal (Cancelled/Done/Failed/settled Idle) must be restarted.
• Re-observe children on init. A parent frozen mid-delegation must NOT blindly re-run its agent loop (that re-delegates / duplicates the child). It must re-observe the existing child, and when the child reaches terminal, write the child's result back so the parent can settle/continue.
• Guarantee terminal. A last-resort watchdog forces a wedged round to a terminal Idle after a generous grace of no progress (Rx Throttle resets on every node emission, so live streaming never trips it; threads legitimately waiting on a child are skipped — that staleness is the heartbeat ticker's job).
• Children always reach terminal. A sub-thread's own init must drive itself to a terminal state so the parent's re-observation is guaranteed to fire.

Don't "fix" this by bumping the test timeout — that hides a missed emission behind a longer wait. Find the observer that died and make it restart.

The subscribe handshake that delivers SubscribeAck but never the initial Full

A specific, high-value instance of the missed-observation class: a consumer GetMeshNodeStream / GetQuery / GetRemoteStream never emits for ~one heartbeat interval, then recovers. The fingerprint in the MESHWEAVER_MSG_TRACE file is unmistakable:

SubscribeRequest → HandleMessageAsync → SubscribeAck routed      (handshake acked)
   … ~45 s of dead silence on the owner node …
HeartBeatEvent HandleMessageAsync ENTER                          (the heartbeat fires)
RawJson HUB.DeliverMessage ENTER                                 (content FINALLY arrives)

The owner acknowledged the subscription but never sent the initial Full — the subscriber sits dark until the next SyncStreamOptions.HeartbeatInterval (45 s) re-emits. ~42 s between "subscribe" and "first data" ≈ one heartbeat is the tell.

Confirm it's idle, not a deadlock — dotnet-stack

Before assuming a lock, dump the managed stacks mid-freeze — it settles deadlock-vs-missed-observation in one shot:

# launch the test, then mid-freeze (the compile/source-read window).
# macOS/Linux — `tasklist` is Windows-only; use pgrep:
pid=$(pgrep -f testhost | head -1)
dotnet-stack report -p "$pid" > /tmp/stacks.txt
grep -vE "System\.|Microsoft\.|xunit|testhost" /tmp/stacks.txt | sort -u   # any APP frame?

dotnet-stack is a separate global tool (dotnet tool install -g dotnet-stack); it is not part of the SDK.

If every thread is parked on LowLevelLifoSemaphore.Wait / Task.Wait / WaitOne and there is no MeshWeaver frame on any thread, nothing is running — it's a missed reactive emission, not a blocking deadlock or CPU starvation. (8 cores + total trace silence + idle stacks ⇒ a dropped emission, never a hot loop.)

Two owner-side drops that cause it

1. ChangedBy collapsed to empty → echo-filter false-positive. The owner forwards changes through reduced.ToDataChanged(c => !reduced.ClientId.Equals(c.ChangedBy)) to suppress echoing the subscriber's own writes back. ChangedBy is the stream-echo-suppression key — the identity of the originating stream, always the StreamId, never empty. It is NOT the access identity: deriving it from CaptureCallerAccessContext()?.ObjectId ?? ClientId collapses to "" when there is no ObjectId and ClientId is empty, and the filter becomes !"".Equals("") == false → the initial Full is dropped. Fix: ChangedBy = StreamId in SynchronizationStream.BuildChangeItem / BuildFullChangeItem. The AccessContext (RLS / LastModifiedBy) flows orthogonally and must never leak into ChangedBy.

2. The echo-filter dropped FULLS. A ChangeType.Full is the owner's complete authoritative snapshot (initial subscribe state, or a re-assert/rollback) and is never the subscriber's echo — subscribers only ever send Patches via DataChangeRequest. So the owner-side filter must always forward Fulls: c => c.ChangeType == ChangeType.Full || !reduced.ClientId.Equals(c.ChangedBy). This is the echo filter (owner side) — it always forwards Fulls because a Full is never a subscriber echo. Do not confuse it with the client-side version guard in UpdateStream, which now drops both stale Patches and stale Fulls (Version < Current.Version). A real Full is never below Current because every owner frame rides one clock (OwnerVersion()); see DataSyncAndCrdt.md §2–3.

The heartbeat must NOT re-broadcast the Full

The 45 s heartbeat is a keepalive (it keeps the remote owner grain alive). It must not be the mechanism that re-delivers content — relying on it both (a) masks the dropped-initial-Full bug above behind a 45 s stall, and (b) would re-ship every stream's entire content every 45 s, which is lethal at scale. Fix the handshake so the initial Full lands on subscribe; keep the heartbeat content-free.