MeshNode Versioning

Version counts REVISIONS OF THIS NODE — contiguous, monotonic, and completely independent of hub traffic, of other nodes, and of how many times the node happened to be re-saved. A recycle cannot move it. New node = 1 · never-mutated seeded node = 0 · v(n) → v(n+1) exactly once per real edit.

A write that changes nothing is completed, acked, and dropped — it never reaches the counter.

No Version Without a Change

The gate comes first at every place a write can land; the bump is what happens after it passes. This ordering is load-bearing: the bump would itself manufacture the difference that makes a no-op write look like a change.

Why SerializedEquals and not plain record Equals. MeshNode.Content is an object?. A rebuilt-but-identical typed content, a re-parsed JsonElement (a struct whose default equality compares the parse buffer), or a content record holding collections (compared by reference) all read as "changed" under record equality while the persisted JSON is byte-identical. Every such write used to mint a version and persist a history row — the "v1170 with no edits" report. MeshNode.SerializedEquals(a, b, options) compares the serialized JSON and runs only after the cheap checks already disagreed, so an unchanged node costs nothing.

LastModified is stamped only on a real change. The audit stamp is applied after the diff, never before — otherwise the stamp is the only thing in the patch and every save looks like an edit.

🚨 Never Author Version in a Source File — Use SemVer

Version is the OWNER's persistence clock. It is [Editable(false)], it means "revision N of this node", and it belongs to the hub that owns the node. A node file in a source repo must never carry it.

A committed Version is a snapshot of some other mesh's clock. On a fresh mesh it collides with the durable row and the store correctly refuses the write:

[MonotonicWriteGuard] REFUSED a backward write to Store: incoming Version=1 is BELOW the
    stored Version=249 … forked lineage: stale activation seed or a second writer
MeshWeaver.PluginCatalog.Catalog: Install of Store failed.

What that costs is out of all proportion to the symptom, and the symptom names the wrong thing. The refused write fails the INSTALL; the partition then has no readable root, so every subscribe is denied (799 of them in one observed run); its NodeTypes never compile because nothing readable reaches them; and the harness finally reports "Store/Plugin never reached compilationStatus Ok" — a compile error, five minutes and four causal steps from the actual failure. It reads as a flaky compile for as long as you let it.

Found 2026-08-06 across every node repo — 13 modules carrying counters as high as 4489, each a fork waiting for a fresh mesh.

The rule: a module's authored version is SemVer — manifest.lock's version (and content.version for the series), published as the git tag <Module>/vMAJOR.MINOR.PATCH. The persistence counter is never authored, only minted. Two different things that happened to share a field name, which is exactly how this went unnoticed.

🚨 Do not "fix" a fork by relaxing the guard. Its own message says it: "Find the writer that adopted a stale own-node snapshot; do not relax this guard." A regressing write is a stale snapshot about to destroy acknowledged data. When a fork appears, find the writer — and check first whether the writer is a checked-in file.

Only the Owner Mints

A client/subscriber writing a node it does not own (a cross-hub GetMeshNodeStream(path).Update(...)) carries the base version it last observed and lets the owner assign the fresh value on apply. It never increments client-side. A pre-incremented client version (the old Math.Max(existing, …) + 1) ships a frame whose base is already out of date by the time it lands, and the owner's version-guarded merge mishandles it. See DataSyncAndCrdt §2 ("a subscriber never mints a version").

Write through the live lambda parameter. stream.Update(node => node with { … }) must transform the node it is handed — the live, owner-reconciled value — never discard it and slam a separately-read full node (_ => fetchedNode). The owner computes the diff it applies against that live value; a discarded parameter bases the diff on a stale snapshot and can clobber a concurrent edit.

Each write bumps once. The write path that commits a change is the one that bumps. MeshNodeTypeSource.UpdateImpl re-stamps only a node whose incoming version did not already advance past what it previously carried (a sync-stream value update, a client-carried base version) — re-stamping a node the write path already bumped would count one edit twice.

Not the Hub Clock (#325)

MessageHub.Version increments once per message dispatch — it counts operations the hub processed. MeshNode.Version used to be stamped from it (max(hubVersion, current + 1)), which had two consequences that are now gone:

• Unrelated traffic moved the number. A node touched under message 3 and again under message 47 jumped 3 → 47. The version described the hub's workload, not the node's history.
• A recycle rolled it backward. The hub clock resets to 0 on every activation, so a deactivate → reactivate cycle (idle-release, Recycle/DisposeRequest, a replica restart) stamped the node's next write with the fresh low clock. A caller re-reading the node saw an OLDER version than the writes it had just confirmed — the write-rollback / "v113 read back as v3" of issue #325.

A node-local counter is monotonic across activations by construction: the node loads its persisted Version verbatim on activation (MeshNodeTypeSource.BuildInstanceCollection leaves it alone — a load is a read), and the next write is that value + 1.

The hub clock is untouched, and that matters. Hub.Version also stamps the owning hub's layout-area render Fulls, and the sync-stream frame version rides it. Re-seeding the shared clock from a node version was tried (SetInitialVersion(node.Version)) and reverted: layout Fulls advance per render and run far ahead of a doc/static node's low Version, so seeding it backward made the monotonicity guard drop every later Full — the prod 2026-06-18 "cannot find pinned doc" wedge. Likewise, flooring the frame version at a content baseline broke the normal single-hub data-load frame sequence (the PageLoadingTest / SourceDocumentDataLoadingTest hangs) and the activity/export relay. The two clocks stay separate: node revisions on the node, render/frame ordering on the hub.

The residual cross-silo mirror-drop (#325 symptom 2, "index vs node-resolution split-brain", multi-replica only — the monolith heals it via the heartbeat resubscribe) is fixed on the mirror side, scoped precisely to the resubscribe path so no normal frame is touched. After an owner grain idle-recycles, a mirror on another silo that cached the higher pre-recycle frame version would drop the recycled owner's low post-recycle resubscribe Full under the guard (version < Current.Version) and stay orphaned. But that mirror already detected it is behind — JsonSynchronizationStream.CreateExternalClient's version-gated resubscribe fires only when the change feed announced a higher node version than the mirror holds — and it is asking for a fresh authoritative snapshot. So the resubscribe arms a one-shot latch (SynchronizationStream.ExpectResubscribeFull); the next Full that reaches UpdateStream consumes the latch and is accepted even though its frame version regressed, then the mirror adopts that re-based clock. Only a Full consumes the latch (a stray reordered patch is still dropped), and it is set only when the mirror is genuinely behind (receivedVersion < announcedVersion) — so it can never clobber a newer optimistic write with a stale snapshot. Proven by TwoSiloRecycleConvergenceTest and guarded by PageLoadingTest / SourceDocumentDataLoadingTest / ExportDocumentScriptRelayTest / DataChangeStreamUpdateTest / InlineEditingTest.

The Activation Seed Is Durable Storage — and the Store Refuses a Backward Write

current.Version + 1 only holds if the value it counts from is the node's real persisted version. Two invariants make that true, and both exist because they were once violated — with durable data loss.

1. A (re)activating hub seeds its own node from IStorageAdapter, not from a cache. The routing layer attaches an own-node observable at hub instantiation (WithOwnNodeStream, set by MessageHubGrain / MonolithRoutingService). That stream is the right source for live updates — and on Orleans it is the only source of the enriched node, whose HubConfiguration delegate storage cannot hold — but it is not durable state: PathResolutionService memoizes the resolved AddressResolution including its MeshNode snapshot, invalidated only by the per-silo change feed, and MeshNodeStreamCache replays its last seen value. A hub that adopted such a snapshot as its live own-node state came up on an arbitrarily old node — which the persistence sampler then wrote back over newer durable data. So MeshNodeTypeSource.Initialize merges a one-shot durable read with the routing stream. Merge, not replace: a slow or faulted storage read never delays activation (the routing stream still seeds, and a node that has never been persisted reads back null and is simply dropped), while a stale routing emission loses to the durable one on version.

2. A hub never adopts an own-node emission whose Version regresses. MeshNodeTypeSource keeps a per-hub floor raised by every state it adopts — the durable seed, a routing emission, and every local write committed through UpdateImpl (the last is load-bearing: the durable read is asynchronous on a real backend and can land after a local write already advanced the in-RAM node). Only a strictly lower version is dropped; equal passes, because a never-mutated node sits at its seed version forever. The one legitimate rewind — a same-path recreate restarting at Version = 1 — is recognised through the existing delete tombstone (RecentlyDeletedRegistry) and resets the floor.

3. The store itself refuses a backward write. MonotonicWriteGuardStorageAdapter is the outermost IStorageAdapter decorator (composed alongside the version writer, so every consumer that resolves IStorageAdapter from DI gets it). Since the counter only ever moves forward, a write whose Version is below the stored one is never a newer state — it is a stale snapshot about to destroy acknowledged data, and it is refused with an Error log naming both versions; the write emits the stored (winning) node rather than throwing, so a data-integrity save cannot fault a create or dispose-flush chain. A per-path in-process high-water mark (fed by writes and reads, so it costs no extra I/O) is only a cheap filter: a suspected regression is verified against a real read of the current row before anything is refused, so a stale mark — another replica deleted and recreated the node, the store was restored out of band — can never refuse a legitimate write. Re-persisting an unchanged node at its existing version is accepted: the guard refuses only a strictly lower version. There is deliberately no bypass hatch: every framework rewind already writes forward (version restore re-stamps Version = 0 so the owner mints a new top version; imports and GitSync go through the owner's stream.Update; a delete drops the row, so a recreate faces no stored row at all).

Pinned by StaleActivationSeedRollbackTest (deterministic: recycle the owner, advance the durable row out of band, assert the reactivated hub serves durable state and never rolls the store back) and MonotonicWriteGuardTests.

One Change, One Durable Write

An own MeshNode has exactly two ways to reach storage, and for any given change only one of them may run.

For a patch, both used to fire. That is not merely a wasted write: the two are never ordered against each other — the flush writes from an emission thread, the sampler through the inbox — so under a sustained write rate the row advances while the sampler's message queues, and its write lands as a strict version regression. The guard above then reports a CONFLICT for a strictly sequential writer and resolves it by merging; with no common ancestor that merge keeps the string superset and the array union, so a deletion the newer write made is silently re-added. Resurrection is a deliberate trade-off for a genuine conflict (MeshNodePatchMerge.TryMergeTwoWay) — never for one the framework manufactured against itself, and the noise also devalued the guard's alarm into background chatter (issue #1249).

PostCommitFlushRegistry is what keeps it to one. The flush records path → durable version on the write's emission; HandleSaveMeshNode and the dispose-time FlushPendingOwnSave drop any sampled state at or below that mark. Three properties make the predicate sound:

• A version, not a snapshot stamp. The sampler's gate chain runs in the same synchronous fan-out as the flush and — having subscribed at hub init — runs first, so nothing the flush stamps can be visible to it. The mark is therefore read at handler time, after the flush settled. OwnNodeCache.PersistedSnapshot cannot serve here — and, it turned out, could not serve the initial-load echo it was written for either: see below.
• <= cannot suppress newer content. Two distinct own states never share a version — MeshNodeTypeSource.UpdateImpl re-stamps any own update arriving at or below its previous version with NextVersion.
• It fails open. The mark is raised only on a write that actually emitted (a failed flush, or the try-then-claim null sentinel, leaves the sampler as the writer of record), and it is dropped on delete so a same-id recreate at Version = 1 is never read as already-persisted.

It is registered at the mesh root (MeshBuilder, next to RecentlyDeletedRegistry) because the flush is a mesh-level singleton while its reader runs on the per-node owner hub; a hub-level registration would hand each side its own instance and neither would see the other's writes.

Pinned by PatchWriteRouteCollapseTest: one patch ⇒ exactly one durable write; a sequential writer's deleted text is not resurrected; and a genuine second writer still trips the guard, merge and _Activity/write-conflict-* record.

The activation seed is an observation, and it goes on the same mark

A hub's durable activation seed — the row MeshNodeTypeSource.DurableSeed reads at startup — is state the hub did not mint, so it must never reach either write route. That was supposed to be the job of the sampler's initial-load gate, a reference comparison against OwnNodeCache.PersistedSnapshot. It does not hold, and never did. PersistedSnapshot is one slot, while Initialize builds two collections back to back — the durable seed, then the routing-supplied leg — before the workspace hands either to the sampler. The slot therefore holds the routing instance, the seed emission fails the reference test, and the sampler dispatches the row the hub had just read as though it were a local edit.

On a quiet node that is an equal-version rewrite and invisible. On a node another replica is writing it is the regression this whole page is about: the guard refuses the echo, AdoptDurableTruth correctly rebases the refusing owner at durable + 1, and a hub that never edited anything ends up holding a revision the store never held, with an _Activity/write-conflict-* record to match (issue #2008 — it surfaced as an intermittent cross-process test failure at ~1.3% of CI runs, because whether the echo or the real write reached the store first was a coin flip).

So DurableSeed raises the same high-water, through PostCommitFlushRegistry.RecordObservedDurable — a read-sourced raise, distinct from Record: it never resolves an in-flight Claim, because a read is not a completed write and answering a waiter "persisted" on a write that may still fail would trade this duplicate-write bug for a lost-write one. MeshNodeStreamHandle.AdoptPersisted already did the equivalent for the other durable-observation path (a storage change notification); the activation seed was the one that did not. A real edit is unaffected — UpdateImpl mints it strictly above the seed, so it is above the mark and still writes.

Pinned by CrossProcessChangeFeedTest.AMirroringProcess_NeverWritesTheNode_NotEvenItsActivationSeed: a process that only mirrors a node performs zero writes to it.

Never-Mutated Nodes Keep Their Seed Version

A node loaded from persistence — or seeded via AddMeshNodes / IStaticNodeProvider — and never written through Update keeps whatever Version it was created with, typically 0. The HandleSaveMeshNode path persists the node's Version verbatim; it does not synthesise a bump on save. So a static config node legitimately reads back as Version == 0.

The same holds for a hub reactivating: its already-durable node is re-added to a fresh in-memory collection, but re-persisting it is not a change, so it keeps its version.

Created Nodes Start at Version 1

HandleCreateNodeRequest stamps new nodes as follows:

Version = node.Version > 0 ? node.Version : 1

A freshly created node gets Version = 1 unless the caller explicitly supplied a higher value (for example, an import flow replaying historical versions). The reason is serialisation, not semantics: the hub's JsonSerializerOptions uses DefaultIgnoreCondition = WhenWritingDefault, so Version = 0 would be omitted from the persisted JSON entirely. Starting at 1 guarantees the field is always present on the wire and in storage.

Version Semantics at a Glance

🚨 A Version Row's Timestamp Is Not the Write's Clock

GetVersions reports each row's lastModified, and that column is the node's LastModified field — not when the row was written. Only IMeshService.UpdateNode re-stamps it (NodeUpdatePipeline sets LastModified = DateTimeOffset.UtcNow inside the apply lambda, and only when the write really changes something). workspace.GetMeshNodeStream(path).Update(...) does not. A caller writing node with { Content = … } carries the field through untouched.

So a burst of control-plane writes — the shape every RequestedX watcher has, where each pass stamps progress onto the node it watches — produces several versions that all carry the timestamp of whichever earlier write last went through UpdateNode. Measured on a production request (Systemorph/MeshWeaver.Plugins#1320): versions 3–6 all read 11:15:32, while the content those versions carry says startedAt: 15:59:11 and the child node the same pass created carries createdDate: 15:59:11.286Z. Four hours and forty-four minutes of the story are invisible in the version list, and reading it as a write log says the opposite of what happened.

To date a write, read something the framework stamps at write time: the content's own timestamps, or a node the pass CREATED (CreateNodeRequest always stamps CreatedDate). Use the version list for what changed and in what order, never for when.

What This Is Not

This is in-mesh change tracking for the live MeshNode graph. It is entirely unrelated to data versioning of the content held by NodeTypes — historical queries, time-travel, and the {path}@V{n} snapshot convention are a separate concern covered in DataVersioning (which is a guide to backend mechanisms, not a framework API).