Data Synchronization & CRDT

How MeshWeaver keeps a piece of state consistent across hubs — the version model, the monotonicity rules, conflict resolution, and the minimal-bytes transport. This is the contract every ISynchronizationStream<T> obeys.

1. The actors

A SynchronizationStream<TStream> is one synchronized value (a MeshNode, a collection, an EntityStore, a layout-area control tree, …). It has three hub references that are easy to confuse — and the version model hinges on the difference:

Hub is a child of Host (Host.GetHostedHub(...)). They are different clocks. Host.Version is the host's global message counter — it can sit still for many stream updates, or jump by hundreds between two of them. Hub.Version is this stream's counter — it ticks once per message the stream processes. For per-stream ordering, only Hub.Version is meaningful.

The stream IS the owner ⇔ Owner.Equals(Host.Address).

• Layout areas are owned by their own stream/hub → the stream assigns its own version.
• MeshNodes / domain data are owned by the host node → a write must travel to that host's queue (below).

2. Version: assigned by the owning hub, in the owning hub's queue

The Version is the one reliable ordering signal. DateTime is not — there is no universal clock across hubs. So:

1. The owner assigns the version, inside its execution queue. A normal update is hub.Post(UpdateStreamRequest); the handler runs on the stream's sync-hub action block (serialized — one update at a time), and there the version is stamped from the hub that just ticked. Reading a version outside the lock would race; reading it inside the serialized handler is monotonic by construction.
2. A subscriber never mints a version. It either:
   • adopts delivery.Message.Version verbatim when it applies an owner frame (Full or Patch), or
   • carries the BASE version it last observed (Current.Version) on the change it requests of the owner — so the owner knows what the writer saw.
3. No "other" hub stamps it. Not the sync sub-hub of a subscriber, not Host when Host isn't the owner.

Net effect: every frame on a given stream carries a strictly increasing version assigned by one clock — the owner's. (Pinned by StreamVersionMonotonicityTest.)

One helper, every emission path. SynchronizationStream.OwnerVersion() is the single place that picks the clock — Owner.Equals(Host.Address) ? Hub.Version : (Current?.Version ?? 0L). Every frame an owned stream emits funnels through it: a value Update (BuildChangeItem), a full overwrite (BuildFullChangeItem), and the init/base frame (Initialize — the layout area's "Building layout…" shell, a data source's initial snapshot). The init frame used to read Host.Version directly; because Host.Version (the parent host hub) runs hundreds of ticks ahead of a freshly-created sync/<id> sub-hub, the base frame outranked the render content that followed on Hub.Version, and §3's guard dropped the content. Funnelling the init frame through OwnerVersion() keeps it on the same clock as the renders, so base.Version < content.Version always holds.

3. The monotonicity guard — patches and Fulls, but OWNER frames only

When a subscriber receives an owner frame (UpdateStream):

ANY OWNER FRAME (Patch or Full) : drop it if  Version < Current.Version
                                  …unless a Full is consuming the resubscribe latch (below)

🚨 Version means two different things, and the guard reads only one of them

UpdateStream handles messages travelling in both directions, and the meaning of Version flips with the direction:

A subscriber's write is optimistic by definition: StandardReducers.PatchJsonElement stamps it with stream.Current?.Version — the frame the writer had in hand — so it is below the owner's clock by construction whenever an owner frame is in flight. §4 already names that case as the one the owner must merge. Comparing it against the owner's clock and dropping it therefore discards a legitimate write, silently: no rollback Full, no DeliveryFailure, one Debug line.

That was Systemorph/MeshWeaver#2701. The measured shape is an editor whose re-render takes 100 ms: a burst of UpdatePointer writes lands while that render's frame is still on the wire, every one of them is dropped, and the control stream that should have carried the result emits nothing at all — indistinguishable from a slow machine, which is why it read as a flake for months. It is a data-loss bug, not a timing one: a user's edit typed while the server pushes an unrelated re-render was thrown away. Pinned by StaleBaseSubscriberWriteTest (test/MeshWeaver.Layout.Test).

The same asymmetry was already drawn one block further down in the frame-loss check, which applies "to DataChangedEvent (owner→mirror) only: a PatchDataChangeRequest's chain is stamped by the SENDING mirror and is not comparable to this stream's applied version". The version handling simply never got it.

Having applied a subscriber's write, the owner keeps its own clock. Adopting the writer's base would move Current.Version backwards, and the frame the owner then broadcasts would be stamped below what its other subscribers already hold — so their (correct) guards would drop it, and the same loss reappears one hop out. The owner's version is a floor: applying a subscriber write can only ever move it forward.

A Patch is a delta computed against a specific base version; applying a reordered older patch corrupts the mirror, so it is version-guarded.

A Full is the owner's complete authoritative state — but it is also version-guarded. The guard once let every Full through unconditionally; that let a resubscribe's point-in-time Full, snapshotted before a write the mirror had already applied, overwrite the newer state (the lost-message data loss). So a Full whose Version < Current.Version is a stale snapshot and is dropped too. (SynchronizationStream.UpdateStream.)

This is safe only because every frame the owner emits rides one clock — the owner's Hub.Version (§2). A legitimate re-assertion can never carry a version below Current: a reject→ROLLBACK Full re-asserts the owner's CURRENT state, stamped with its current (higher-or-equal) version, so it still lands (§6); only a genuinely older snapshot can be below Current, and that is exactly what we drop.

The one sanctioned exception: the rebased-resubscribe Full. A grain that idle-recycles resets its Hub.Version to ~0, so the fresh snapshot it sends to a mirror that asked for one carries a frame version below the mirror's cached (pre-recycle) value — and the guard above would drop it, orphaning the mirror (#325 symptom 2, multi-replica only). So a version-gated resubscribe — issued only when the change feed announced a node version higher than the mirror holds — arms a one-shot latch (SynchronizationStream.ExpectResubscribeFull). The next Full consumes the latch, is accepted despite the regression, and the mirror adopts the owner's re-based clock. Only a Full consumes it (a stray reordered patch is still dropped), and because the latch is armed only when the mirror is genuinely behind, it can never let a stale snapshot clobber a newer optimistic write. The corollary is unforgiving: if even one frame is stamped from a different clock — e.g. the init/base frame stamped with Host.Version while the render content rides Hub.Version — the version order breaks and the guard discards real content. That was the layout-area "stuck on Building layout…" non-emission; the fix is OwnerVersion() (§2, §11), which forces every frame onto the owner's stream clock.

4. Where a write goes (ownership routing)

Self-owned (layout area): Owner == Host.Address

The stream is the owner. stream.Update(...) posts an UpdateStreamRequest to its own sync hub; the handler validates, applies, assigns the version, emits. Done — no network hop.

Host-owned (MeshNode / data): Owner != Host.Address

The subscriber CANNOT assign a version. It must transfer the change to the owner:

1. The subscriber's local change is converted to a DataChangeRequest (ToDataChangeRequest) and hub.Post(..., WithTarget(Owner).WithAccessContext(caller)).
2. The request lands on the owner's execution queue.
3. Inside that queue the owner validates (RLS) → accepts or rejects.
4. On accept it applies the change and assigns a fresh version off its own sync-stream clock.
5. The new state streams back to every subscriber (the requester sees its own optimistic change reconciled; others see the merge).

This is the canonical cross-hub write — JsonSynchronizationStream lines ~179–219. The version is born in the owner's queue; the subscriber only ever proposed a change.

5. CRDT — conflict resolution by version + string splice

Because the request carries the base version it was computed from, the owner can resolve concurrency without a universal clock:

(StreamConflictResolution.Resolve.)

String fields merge by splice, not clobber

A string field changed by both sides is reconciled with StringDelta: the writer's splice (Start, RemovedLength, Inserted) is replayed onto the current text. Disjoint edits to the same big string both survive — "The VERY quick brown fox" + "…fox leaps" → "The VERY quick brown fox leaps". (StringDelta, StreamConflictResolutionTest.)

6. Roll-back / undo

When the owner rejects a proposed change (validation/RLS fails), the subscriber holds an optimistic value the owner never accepted. The fix is a roll-back: the owner re-asserts its authoritative state as a FULL. That Full carries the owner's current version (≥ the subscriber's optimistic bump, which was only ever a base the subscriber carried — a subscriber never mints a version, §2), so it passes §3's guard and overwrites the optimistic value. The undo is clean because the rollback Full is current, not because Fulls bypass the guard — they no longer do.

Request a Full when unsure. A subscriber that detects it is out of sync (a patch arrived with no base, a patch failed to apply, a write was rejected) calls RequestFreshSnapshot() — it re-SubscribeRequests the owner, which replies with a fresh Full. Gated by resyncInFlight so a burst of confusing patches triggers exactly one resubscribe, not a storm.

The frame chain, and how to read Frame loss detected

The transport under the fan-out is at-most-once — a frame published before a subscriber's stream subscription attached, or dropped under pressure, simply never arrives and nothing re-sends it. Before this was detectable, that loss was silent: later patches kept applying cleanly (they touch other entities), so the mirror tracked the owner forever at a constant deficit with no error anywhere.

So every frame the owner emits carries BasedOnVersion — the version of the frame this same forwarding subscription sent immediately before it (-1 for the first). A mirror compares an incoming Patch's BasedOnVersion against the version it last applied; a mismatch proves the gap, and the only sound reaction is a fresh authoritative snapshot: RequestFreshSnapshot() (above). Frames the owner skips (value-equal, no updates, an echo-suppressed patch) never enter the chain, so a legitimate version gap cannot false-trigger a resync. (JsonSynchronizationStream.ToDataChanged → SynchronizationStream.UpdateStream; test StreamFrameLossResyncTest.)

🚨 The mirror's side of the same rule (#3520): a frame it RECEIVES and skips still moves its clock. The owner cannot skip a patch whose Updates are non-empty but whose diff is — a JsonElement payload written back as it was is a new element to the owner (a struct with no value equality) and an identical document to the mirror — so it ships [], and chains the next frame onto it. The mirror's value dedup in SetCurrent rightly emits nothing for that frame, but until #3520 it also left Current.Version where it was, so the next frame's BasedOnVersion pointed at a version the mirror "never applied" and the detector fired on a loss that never happened. Measured on an Education gate run: 164 of 164 Frame loss detected warnings were preceded by exactly that skip of the frame the next patch chained onto — a layout area re-rendered with an identical control on every page load. The skip now adopts the frame's version silently (no OnNext, no consumer wakes) — test SameValuePatchKeepsTheChainTest. Corollary for readers of the warning: a run whose loss lines all sit right after Skipping SetCurrent … same value (patch) for the chained-onto version was never losing frames.

⚠️ The trade this makes, invisible afterwards: before #3530 the spurious detection was accidentally a self-heal — had ValuesEqual ever answered equal for values that differ, the mirror would have diverged and the very next frame's broken chain would have pulled a Full that corrected it. After #3530 a false-equal is permanent: the chain no longer breaks, so nothing re-asks. ValuesEqual is therefore load-bearing now. It leans the safe way — structural JsonDeepEquals for JsonElement, ToJsonString for JsonNode, Equals otherwise, and any exception degrades to "changed" (emit) — and SameValuePatchKeepsTheChainTest pins both sides: the no-op frame emits nothing and moves the clock, the genuinely different frame emits AND moves it. Keep it two-sided; an "optimisation" that made everything equal would pass a one-sided test.

🚨 [SYNC_STREAM] Frame loss detected … is a RESYNC counter, not a data-loss counter. Every line is a gap that was detected and answered; the mirror converges on the Full that follows. A raw count therefore means nothing on its own — the two numbers that do are per-stream count and whether a Full ever follows. Thousands spread over hundreds of streams is the recovery working; a stream that logs the line repeatedly and never converges is the defect (that one is #2654 — a layout area stuck on its NamedAreaControl placeholder).

🚨 The driver is almost always upstream of this file. Anything that repeatedly ends and re-establishes a subscriber's server-side stream costs one gap per cycle, so read these lines from the same window before blaming the sync protocol:

That correlation is the recorded disposition of the memex-cloud storm on #2641 (847 lines / 30 min): the frame-loss count was the symptom of an attach latch and the eviction cycle it caused, not a defect of the chain. See also Durable Streams Are Mesh Nodes — the version chain is the durable stream, which is why no durable stream provider is bought to stop these lines.

The convergence contract — what re-opens the resync gate (#2654)

Detecting a gap is only half the protocol. The re-ask travels the same leg that just lost a frame, so the design question is what happens when the re-ask, or its answer, does not arrive. That is the whole of #2654: the detector was right, the recovery was not, and the failure was silent — a layout area on its NamedAreaControl placeholder while the breadcrumb, banner and menus around it rendered fine.

resyncInFlight bounds one re-ask OUTSTANDING, and it is released by that re-ask's round trip:

🚨 Releasing the gate asks for nothing. Nothing polls, retries or runs on a timer: only the next frame that proves the mirror still has no base drives a new re-ask, so the rate is bounded by the round trip and by the owner actually emitting — the same bound JsonSynchronizationStream.Resubscribe's in-flight flag has always lived with. The common case therefore costs at most one redundant round trip per gap, and that redundant re-ask is itself answered with a Full, which ends the cycle.

🚨 The ack must not overtake its own answer (#3058)

That last sentence is only true because of an ordering rule the ack did not originally have. SubscribeAck was posted from DataExtensions.HandleSubscribeRequest the instant the re-subscribe was received — but the alreadyServing branch only queues an UpdateStreamRequest on the stream's action block, so the ack raced ahead of the frame it was supposed to acknowledge, down a different path (a direct hub response) from the one the Full takes (the stream's outbound forwarding subscription).

The gate then bounded one round trip to the owner, not one outstanding answer — and the give-up below, which reads an ack as evidence, counted a promise as a result:

05:14:07.039  Fresh-snapshot request acknowledged by owner   ← re-assert still queued
05:14:07.043  Fresh-snapshot request acknowledged by owner   ← behind the burst's own frames
05:14:07.048  Fresh-snapshot request acknowledged by owner
              Resync gave up: 3 consecutive fresh-snapshot requests to owner
              'FutuRe/AmountType' were acknowledged and none produced a base snapshot

Three acks in 9 ms, one healthy Full still in the queue. During a bulk install the re-assert waits behind the install's own frames, every one of those frames re-proves the mirror has no base and earns a fresh re-ask, and each re-ask is acknowledged in about a millisecond — so the count measured the owner's queue depth and called it non-convergence. It faulted mirrors mid-install and reddened the samples content gate on roughly a third of all runs, on unmodified main.

So CreateSynchronizationStream owns the ack now, and posts it from the re-assert's own update turn via Update(…, applied:) — the callback that runs in the same turn right after SetCurrent, by which point Store.OnNext has already driven the forwarding subscription and handed the Full to hub.Post. Ack and frame leave through the same outbound queue, in that order.

A fresh stream still acks immediately, and the asymmetry is deliberate: that subscribe has no answer to overtake, its first Full comes out of the reduce's own hydration (IO-bound and unbounded), and waiting would hold the subscriber's pending callback across it — risking the very RequestTimeout the ack exists to prevent. It also cannot produce the miscount: a re-ask reaches that path at most once per gap, because RegisterClientSubscription makes the next one a re-assert.

Every path posts exactly one ack, failure arms included — a path that posts none leaves the subscriber's pending callback open until RequestTimeout, which is the wedge the ack exists to prevent.

Three properties this contract needs, each of which was missing:

1. The re-ask is Observed, never Posted. SubscribeRequest is an IRequest<SubscribeAck> and DataExtensions.HandleSubscribeRequest answers every one of them, so a verdict always exists — fire-and-forget threw it away. ResyncRefused applies the same classification the stream's own DeliveryFailure handler does, one policy per type: ShuttingDown is transient and is ridden out; every other verdict is terminal and faults the stream, so the subscriber sees a failure rather than an eternal placeholder. A verdict that never arrives at all (the request was undeliverable) is neither — Warning, recoverable. The Observe is wrapped in Observable.Defer so a synchronous post throw reaches the same arm instead of escaping UpdateStream with the gate already shut.

   🚨 Classify on ErrorType, never on TargetUnserved. That stamp is the owner-side eviction gate (DataExtensions.HandleTargetUnservedFailure, #2426/#2546), and the router deliberately puts it on both of its "nobody serves that address" verdicts — the terminal no-live-subscriber refusal (RefuseNoSubscriber, NotFound) and the transient pod-hub refusal a rolling deploy produces while a silo's claim has not landed (AnswerPodHubNotHere, ShuttingDown, #2745). Reading the stamp as "terminal" faults every mirror in that overlap window. RoutingGrain states the rule itself: the stamp is the eviction gate, the ErrorType beside it says whether the sender keeps its recovery armed, and the two are independent.

2. A mirror holding no cached JSON accepts a Full at any frame version. RequestFreshSnapshot discards the snapshot before re-asking, so there is nothing a rebased Full could clobber, and refusing it leaves the mirror with nothing at all. This matters because an owner that has to rebuild the server-side stream to answer (the subscriber was evicted on the router's TargetUnserved verdict, #2620; the owner grain recycled) stamps that stream's first Full on a reset clock — so §3's monotonicity guard used to throw away the very snapshot the mirror had asked for.

3. The gate cannot wait on the chain to notice a lost answer. A re-assert Full carries the version of the state it re-asserts (BuildReassertFrame, §6 / #945), not a new one — so it shares a version with the frame before it, and the BasedOnVersion chain reads v4 → Full v4 → v5(basedOn 4) exactly like v4 → v5(basedOn 4). Losing a re-assert Full is invisible to the chain. Measured, not assumed (StreamResyncConvergenceTest).

The operator signal is [SYNC_STREAM] Resync has not converged for {StreamId}: asking {Owner} for a fresh snapshot again (attempt N) at Warning. Attempt 1 is the ordinary recovery and stays at Debug; anything above 1 is a mirror that asked and was not answered. That is the line to grep for when reading a portal log — it separates "gaps that were answered" (the healthy shape above) from "a stream that keeps asking and never converges".

What a re-subscribe is owed by the owner. CreateSynchronizationStream's alreadyServing branch re-asserts the current snapshot as a Full. When there is nothing to assert yet — the initial subscribe is still hydrating, Current is null — it returns without sending (acknowledging immediately, since there is no answer to overtake), and that is correct rather than a hole: Current and the outbound JSON cursor are set by the same emission, so a stream with no Current has an empty cursor, and ToDataChanged's currentJson is null branch makes its first frame a Full by construction. The subscriber is therefore always answered with a Full; if that Full is lost in transport, the gate above — not the chain — is what recovers it.

Pinned by StreamResyncConvergenceTest: the answer lost in transport, the answer arriving on a rebased clock, the re-ask refused terminally (must fault), and the re-ask refused transiently with the identical TargetUnserved stamp (must be ridden out and still converge). Each fails on the pre-#2654 tree.

The termination contract — when the mirror stops asking and faults (#1384)

The contract above recovers a mirror when the leg loses one thing. It does not terminate when the leg keeps losing this stream's snapshots, and that residual is a silent forever-wedge: every re-ask is acknowledged, every acknowledgement re-opens the gate, every answering Full dies on the way back, and the mirror asks again for the rest of the process's life. The Resync has not converged Warning above is the only trace, and a log line is not an API — nothing downstream of the stream is told anything, so nothing can re-establish.

Measured on memex-cloud, 2026-09-01, on Event/SavGeneralversammlung2026/Talk:

[SYNC_STREAM] Frame loss detected for G4LJWZjBXkWdULLsT-5N5g: incoming Patch v13 chains onto v12
  but the last applied frame is v11 — a frame was lost in transport; requesting fresh snapshot
Layout area 'Present' on Event/SavGeneralversammlung2026/Talk was torn down having never rendered
  — the subscriber only ever saw the "awaiting first data" placeholder.

Plain node reads on that path answered instantly, so the owner was healthy; recycling the pod that held the activation did not clear it, because the wedge lived entirely in a subscriber that had not been told anything was wrong.

So the count of consecutive acknowledged-and-unanswered re-asks is bounded, and at SynchronizationStream.MaxUnansweredResyncs (3) the mirror stops asking and calls OnError with a StreamNotConvergingException.

🚨 The count is of ACKS, not of asks, and that is the #2745 policy in one line. The increment lives in the ack arm of RequestFreshSnapshot and nowhere else. An acknowledgement means the owner sent the snapshot answering the re-subscribe — it is posted from the re-assert's own update turn, after the frame is already in the owner's outbound queue (see The ack must not overtake its own answer, #3058) — so a base snapshot that never follows died on the leg. That ordering is what makes an ack evidence at all: an ack that precedes its own answer makes "acknowledged and unanswered" true of every ask the moment it is made, and the bound then fires on a burst rather than on a wedge. A re-ask refused transiently (ShuttingDown, the rolling-deploy overlap window) is deliberately ridden out by the table above, and counting it would fault every mirror in that window, which is worse than the bug this fixes. An increment that races an answering Full is undone by that Full's own reset a moment later. A re-ask that gets no verdict at all (undeliverable, so the hub's own request/response terminal fires) is likewise not counted — "we could not find out" is not an answer about the owner, and the table above already keeps that case recoverable and loud, one Warning carrying the exception per attempt. Widening the count to cover it would be a separate decision with a separate justification, not a free extension of this one.

🚨 Mirrors only. The give-up applies where StreamIdentity.Owner is another hub — the same predicate OwnerVersion() uses. PatchDataChangeRequest reaches UpdateStream on the owner's server-side stream as well, where a subscriber write that races that stream's first outbound frame takes the same "Patch before base Full" branch; there the counter never resets (the Full that would reset it is one that stream sends), so it measures nothing and faulting on it would kill an owner's stream for a subscriber-side race.

🚨 This is a bound on EVIDENCE, not a retry budget. Nothing here retries and nothing polls. An increment costs a full round trip to the owner — the gate suppresses every re-ask while one is outstanding, and only the answer releases it — plus a subsequent owner frame that proves the mirror still has no base. Raising the bound buys a longer silence, not a better chance; that is the opposite of a widened timeout, and it is why the number is small.

🚨 A give-up counter must count outstanding FAILURES, never attempts — #3058, and it is the general lesson. The two are the same number only when each attempt has been given its answer's round trip. The moment the counter can be incremented by something that has not yet had a chance to succeed, it stops measuring the condition it names and starts measuring load: a bound written against "the owner is not answering" fires hardest exactly when the owner is busiest — during an install, a burst, a cold start — which is when a healthy system looks most like a broken one. The cure is never a bigger bound (that just moves the burst size that trips it, and re-hides the underlying frame loss); it is to make each increment cost a completed failure. Here that meant fixing the ordering of the evidence, not the arithmetic: the ack had to start following its own answer.

Why 3. The one healthy way to spend an attempt without converging is a Patch overtaking the answering Full — the re-assert was already in flight when the re-ask landed. With the ordering rule above, that is bounded by one redundant round trip, twice over at the very worst; without it, it was bounded by nothing at all. The two failure directions are also not symmetric:

The fault is the recovery signal, not the end of the road. A faulted stream is the one state every consumer already knows how to act on: the store's terminal error reaches every reader, Workspace drops it from _remoteStreamCache and closes it, and OnError also calls Hub.FailStartup + Hub.OpenGate(SynchronizationGate) — which releases whatever was deferred behind that gate. Re-establishing stays the subscriber's decision, taken because it was finally told.

The operator signal is [SYNC_STREAM] Resync gave up for {StreamId}: N consecutive fresh-snapshot requests to {Owner} were acknowledged and none produced a base snapshot at Warning, followed by the stream's own OnError line. Seeing it means the owner→subscriber leg is losing frames systematically — read it together with the query pressure work in Cross-Schema Fan-Out Elimination: a leg drops frames when the process behind it is saturated, and that saturation is the thing to fix. This bound exists so the saturation cannot cash out as a view that is blank forever.

A note on the ordering inside RequestFreshSnapshot. The method establishes that it can ask — Reference is WorkspaceReference — before it nulls the cached JSON or takes the gate. That order matters (the reverse closes the gate on a re-ask that was never made, permanently by construction) and it is already how the code reads since #2654. It is not what produced the incident above: LayoutAreaReference is a WorkspaceReference, so the Present-area stream took the ordinary path, asked, was acknowledged, and was never answered.

Pinned by two tests, one per direction, and neither passes without the other's code:

• StreamResyncGivesUpTest — one mid-burst Patch eaten to start the resync, then every fresh snapshot eaten, one write per proven gap. The mirror must fault with a StreamNotConvergingException, and a fresh subscriber must then get a new stream that converges on the owner's complete state. It hangs on the pre-#1384 tree, and it passes unchanged across #3058 — which is the evidence that correcting the ack's ordering did not weaken the termination it enables.
• StreamResyncAnswerInFlightTest — the answer is held, not destroyed: the re-assert Full and the ack the owner posts once that frame is on its way, together, because on a real leg they leave through the same queue. The burst's own patches keep flowing. The mirror must ask once, never fault, and converge the moment the answer is released. On the pre-#3058 tree the ack escapes the hold and the mirror asks again per frame.

7. Minimal bytes on the wire

We move a lot of state, much of it large strings. The transport sends only what changed:

• Owner → subscriber: a JSON patch (RFC 6901 / merge-patch RFC 7396) for deltas (ToJsonPatch); a Full for the initial snapshot and roll-backs.
• Big strings, owner → subscriber → a splice operation, NEGOTIATED. A replace of a string leaf at or above PatchStringSplice.MinSpliceLength travels as {"op":"splice","path":…,"value":{"$sd":[start,removed,"inserted"],"$sdb":[baseLength,"fingerprint"]}} — the changed span plus a fingerprint of the text it was diffed against, so a streaming cell costs O(chunk) per frame instead of O(length) per frame per subscriber (measured: a 20 kB answer over 200 frames, 1.93 MB → 42 kB). The subscriber applies it only when the fingerprint proves its text IS that base; otherwise it refuses and takes the ordinary stale-patch route, RequestFreshSnapshot() (§6) — never a blind splice. 🚨 Emitted only to a subscriber that set SubscribeRequest.AcceptsStringSplice. Unlike the write direction, this fan-out is consumed by hand-rolled appliers in clients/grpc-web, clients/react and clients/python that this repo's CI does not build, and each of them fails silently on a shape it does not know — the JS ones skip an unknown op, the Python one applies it as a replace. So the capability is declared, not assumed, and everyone who does not declare it receives byte-identical bytes to before. (PatchStringSplice.Compress; test FanOutStringSpliceTest.)
• Subscriber → owner: a DataChangeRequest carrying the changed entities only (per (Collection, Id)), not the whole store.
• Big strings → EntityDeltaUpdate (recursive string-delta): a changed string field travels as its splice ({ "$sd": [start, removed, "inserted"] }) — recursively, so a string buried in a nested object splices too ({ "$nd": {…} }, e.g. the markdown inside MeshNode.Content.Content) — never the whole value. A 100 KB body that gained one character is a few bytes on the wire. Wiring: the subscriber's ToDataChangeRequest emits an EntityDeltaUpdate (carrying Collection, Id, Partition, and the splice) in place of the full entity — gated to entities ≥ EntityDelta.MinDeltaSize whose delta is actually smaller and whose partition resolves (so the owner routes it to the same stream; otherwise it falls back to a full re-send, unchanged whole-replace). The owner (WorkspaceOperations.ResolveDelta) replays the splice onto its CURRENT value before the normal apply, so a disjoint concurrent edit on the owner survives (same merge semantics as §5). (StringDeltaPatch, EntityDelta; tests StringDeltaPatchTest, EntityDeltaTest, StringDeltaTransportTest.)

8. Reading & writing a mesh node (the public surface)

Application code never touches GetRemoteStream<MeshNode> (forbidden — it does not converge; see CqrsAndContentAccess.md). The one API is hub.GetMeshNodeStream(path) / workspace.GetMeshNodeStream(path), which routes every cross-hub read and write through the shared IMeshNodeStreamCache — one process-wide upstream per path, so reads and writes share the same live mirror and the convergence rules above hold.

9. Invariants (the test ledger)

10. Single source — the owning hub, and why there is no dedup

Every synchronized value has exactly ONE authoritative source: its owning hub.

• Mesh nodes → the per-node hub at the node's path address (§1 Owner).
• Layout areas → their own sync hub.

A synced type (agents, language models, any live collection) is sourced only from those owning hubs' sync streams. It is not also loaded from persistence, not re-published by routing, and not returned as a second authoritative copy by mesh queries. A query may tell you which paths are in a collection (membership), but the content of each comes from that path's owning hub — never a parallel persistence/query mirror.

Why this matters: it removes the need for dedup. When the same entity arrives through two sources (its sync stream and a query/persistence mirror), the workspace sees two value-equal frames and something downstream must suppress the redundant one. That suppression — a value-equality check in SetCurrent — is a band-aid, and it once swallowed a legitimate re-assertion: a roll-back Full whose value happened to equal what an upstream stream still held, stranding a subscriber that had optimistically diverged (§6). That specific hole is closed — SetCurrent now value-dedups patches only, and a Full always applies — but the dedup itself is still the symptom of a double-source. With a single source there are no value-equal redundant frames at all.

Rule. If you find yourself adding (or relying on) a value-equality dedup on a sync stream, you have a double-source — fix the source, not the symptom. Route the read through the owning hub (workspace.GetMeshNodeStream(path)), and keep the synced collection's content single-sourced from there.

11. Mistakes this design exists to prevent

• Stamping Host.Version (or a subscriber's sync-hub) on any frame instead of the owner's stream clock → non-monotonic versions → the guard drops real updates → "view doesn't refresh / blank layout". The trap is the init/base frame: it is easy to stamp it from the surrounding Host while the content frames correctly ride Hub.Version. Funnel every emission through OwnerVersion() (§2). (This is the exact defect behind the 2026-06 layout-area "stuck on Building layout…" non-emission — latent until §3 began guarding Fulls.)
• Guarding Fulls without the one-clock guarantee → a real Full looks stale and is dropped. Guarding Fulls (§3) is correct and necessary, but it is only safe because every owner frame rides OwnerVersion(); break that and the guard turns on you. A genuine roll-back/re-sync Full always carries the owner's current version, so it is never below Current — see §6.
• Sending whole entities / whole strings → bandwidth blowup on large content.
• Letting a subscriber mint versions → two mirrors fight over ordering; last write wins on the whole node instead of a field-wise merge.