Setting Up Data Sync

This is the how-to manual for getting MeshNodes into a partition by syncing them from a source. For the synchronization protocol (versions, conflict resolution, the monotonicity guard) see DataSyncAndCrdt.md; for the static-repo import mechanism (fingerprint, content-addressed Activity lock) see StaticRepoImport.md. This page tells you how to set one up and the one rule you must not break.

1. The model: source → target

        SYNC SOURCE                         SYNC TARGET
  (transient, init-only)                (persisted partition)
  ┌────────────────────┐   seed/sync    ┌────────────────────┐
  │ platform static repo│  ───────────▶  │ partition nodes in │
  │ node on another mesh│   (gated by    │ the DB — the owning │
  │ MeshNodes in GitHub │    version)    │ hub is authoritative│
  └────────────────────┘                └────────────────────┘
        ▲ NOT queried                          ▲ queried + persisted
        ▲ NOT persisted                        ▲ served to clients
        ▲ discarded after sync                 ▲ the single runtime source

These are MeshNodes we ship/own — built-in agents, language models, documentation, sample graphs. They are authored outside the live mesh and synced into it.

2. 🚨 The golden rule

A sync source participates in SYNC ONLY — never in query, never in persistence, at runtime. Only the sync target participates in query and persistence.

Today a synced collection's source and target both answer queries (and both get persisted). That double-source is why a value-equality dedup exists on the sync stream at all — a band-aid that once also swallowed a legitimate roll-back Full (that hole is closed: SetCurrent value-dedups patches only — see DataSyncAndCrdt.md §6, §10). Fix the source, not the symptom:

Role Sync Query Persistence
Sync source (static repo / remote node / GitHub) ✓ ✗ ✗
Sync target (persisted partition, owning hub) ✓ ✓ ✓

Single-source ⇒ no redundant value-equal frames ⇒ no dedup needed.

Declare participation per type/source on its storage-adapter registration ("I am only a sync source" vs "I participate in the mesh query / persistence"). The query provider and the persistence write-back each skip sources whose participation excludes them.

Status. The participation flag (excluding the source from query + persistence) is the agreed design; today the static-repo source is consulted only at import time and the importer writes through the target's canonical pipeline, so the source already isn't a runtime serving copy — wiring the explicit flag is the remaining step that lets the dedup be deleted.

3. Version gating — sync only when out of date

A sync must be idempotent and cheap on the hot path (every boot). What records "this exact source content is already installed" is a content-addressed Activity node — {Partition}/_Activity/import-{fingerprint} — whose Succeeded status is the durable checkpoint. The id is the fingerprint, so the marker doubles as the cross-replica lock: concurrent replicas racing the same import converge on one execution.

On init, compute the source's fingerprint and look for that marker:

The fingerprint is order-independent and changes iff a node is added, removed, or modified — PartitionSourceFingerprint.Compute(nodes, versioned). Versioned sources hash (path, version); unversioned ones hash (path, contentHash). Full mechanism in StaticRepoImport.md.

4. Source kinds

The source is an abstraction (IStaticRepoSource: a Partition, a Versioned flag, a SyncMode, and EnumerateSourceNodes() returning authored nodes with content). The same target pipeline accepts any source that can enumerate nodes. SyncMode decides what the import prunes after upserting: FullReplace (default — mirror the partition to the repo) or Additive (leave unmatched nodes alone, which is what lets a user's own skills survive a re-import).

a. Platform static repo — available today

MeshNodes shipped in an assembly (models, skills, harnesses, docs). Implement IStaticRepoSource, enumerate from your in-memory provider:

public sealed class SkillStaticRepoSource(BuiltInSkillProvider provider) : IStaticRepoSource
{
    public string Partition => SkillNodeType.RootNamespace;
    public bool Versioned => false;            // skill .md has no version → hash content
    public PartitionSyncMode SyncMode => PartitionSyncMode.Additive;  // user skills survive
    public IReadOnlyList<MeshNode> EnumerateSourceNodes() =>
        provider.GetStaticNodes()
            .Where(n => !n.Segments.Skip(1).Any(s => s.StartsWith('_'))) // content only; skip _Access governance
            .ToArray();
}

Real examples: ModelStaticRepoSource, SkillStaticRepoSource, HarnessStaticRepoSource (registered together by AiContentSources.AddBuiltInAiContentSources) and DocumentationStaticRepoSource. There is deliberately no Agent source in the framework: the built-in agents moved to the Agent plugin, so the Agent partition is served from the DB but filled by the plugin, not by this binary.

b. A node/partition on another instance — same abstraction, planned

The source enumerates nodes pulled from a remote mesh (another portal/instance) instead of an embedded provider — e.g. GetRemoteStream<MeshNode> / a mesh query against the remote address. Everything downstream (fingerprint gate, canonical upsert into the target, prune) is identical. Versioned = true when the remote nodes carry meaningful versions.

c. MeshNodes in a GitHub repo — the "sync from anywhere" target

The source enumerates nodes read from a public GitHub repository over HTTP — list the tree, fetch each authored MeshNode file's content, map file→node. Pin the ref to the commit the binary was built from — every CI assembly carries it as AssemblyMetadata("MeshWeaverCommitHash") — or, on a clean release, to the immutable tag v$(PlatformVersion) (e.g. v3.0.0), which names the same tree. Set Versioned = true, so the fingerprint changes exactly when the source commit changes and a boot at the same commit is a no-op (§3). No clone, no working copy: the GitHub REST API (git/trees/{ref}?recursive=1 for the listing) + raw.githubusercontent.com (for content) is enough for a public repo, and all HTTP goes through IIoPool (never Observable.FromAsync — see ControlledIoPooling.md).

🥚 Which ref: the stamped commit, or the release tag. A binary CAN know its own commit: CI stamps $(GITHUB_SHA) / SourceRevisionId into every assembly (Directory.Build.props → MeshWeaverCommitHash), because the hash names the tree being built, not the build. A continuous build (3.0.0-ci.<n>) therefore syncs from that commit — there is no tag for it, and there never will be. A clean release (3.0.0) may sync from v$(PlatformVersion) instead; it resolves to the same tree, because a release is a promotion of a continuous build, never a rebuild (ReleaseProcess.md). A release tag must be immutable (annotated, never force-moved) so the fingerprint is sound.

This is the goal: sync from anywhere. Once docs (and samples, agent/model templates) are synced from GitHub into the partition, the platform no longer needs to compile/embed them — the MeshWeaver.Documentation embedded-resource build step becomes unnecessary; the partition is seeded from the repo and served from the DB like any other node.

⚠️ The "stop compiling docs" cutover is gated, not global. Adding a GitHub source is additive and safe. Demoting the embedded doc-serving path is the separate Phase-4 cutover in StaticRepoImport.md: the monolith serves docs in-process from the embedded overlay today and must keep working, while the distributed/PG path is the one that needs the DB-materialized copy. So switch serving to the partition opt-in on the distributed path, verified end-to-end — never a global demote in one step.

5. Setup steps

  1. Implement the source. A class implementing IStaticRepoSource for your target Partition, enumerating the authored nodes with content.
  2. Register it in DI (it's discovered via hub.ServiceProvider.GetServices<IStaticRepoSource>()): 
    builder.ConfigureServices(s => s.AddSingleton<IStaticRepoSource, SkillStaticRepoSource>());
  3. Let init run it. StaticRepoImporter.ImportAll(hub) ("sync context init") imports every registered source on boot — no-op when none is registered, and a no-op per source when the fingerprint already matches (§3).
  4. Keep the source out of the runtime read/write path (§2): serve the partition from the DB target, not from the source provider/overlay. Governance nodes you intend to keep in-memory (e.g. _Access policy) are simply excluded from EnumerateSourceNodes().

That's it: implement → register → boot. Idempotent, distributed-replica safe, served from the DB like any other node.

6. Declarative sync config — what exists, and the proposed shape

Hard-coding sources in DI (§5.2) is the bootstrap path, and it is what the IStaticRepoSource pipeline described above uses today.

What exists today. Config-as-data already works for the pull-from-elsewhere sync engines, one config node per source, edited through the standard node-content editor: GitSync's config satellites (gitsync-cfg:{path}) and instance sync's {space}/_Sync/{sourceId} registrations (InstanceSync). Both surface on the platform-admin Partitions page through the IPartitionSyncSourceProvider seam (PartitionSyncAdminLayoutArea), which is also where a partition is flipped between Synced and Not synced — that flip sets the partition root's SyncBehavior to ExcludeThisAndChildren, decoupling it from static-repo import.

What is proposed (design, not implemented): the same treatment for the static-repo pipeline — a PartitionSync-shaped config node in the admin partition, so you add, change, or stop a partition sync at runtime with no redeploy:

// PROPOSED — MeshNode { Namespace="Admin", Id="sync-doc", NodeType="PartitionSync" }.Content
{
  "targetPartition": "Doc",
  "source":  "github",
  "url":     "https://github.com/Systemorph/MeshWeaver",
  "ref":     "v3.0.0",       // immutable release tag (or the stamped commit) = the version gate (§3, §4c)
  "path":    "src/MeshWeaver.Documentation/Data",
  "enabled": true
}

Pin an immutable ref — the release tag v$(PlatformVersion) on a clean release, the stamped commit on a continuous build (§4c) — never a moving branch, so the fingerprint (§3) is exact and a re-boot at the same ref is a guaranteed no-op. The ref resolves to a single canonical GitHub tree URL:

https://github.com/Systemorph/MeshWeaver/tree/v3.0.0/src/MeshWeaver.Documentation/Data

Bumping the sync to a newer release is one edit to the config node's ref (or, for the default, it falls out of the deployed binary's PlatformVersion) — the next boot sees a new fingerprint and re-syncs; everything in between is a no-op.

In that shape the sync engine would query the admin partition for PartitionSync nodes and run each through the same source→target pipeline (§1), fingerprint-gated (§3). A config node says "synchronize this partition from there" — it is itself an ordinary target node (queried + persisted in the admin partition); it configures a source, it is not one.

Breaking the sync — taking over a partition

Because a declarative sync is data, it is revocable. To take over a partition (own it locally, stop tracking the upstream), break the sync:

This is the clean ownership switch: ship a partition synced from a repo, and any deployment can break the sync and make it its own.

7. Pitfalls

A repo with no webhook to a portal syncs ONCE, then never again

The portal pulls a partition when the repository's green main build arrives as a workflow_run webhook (GitHubWebhookProcessor); a push only logs that a build is coming. A repository created without a hook to a portal therefore syncs exactly as often as somebody triggers it by hand — and nothing measures the gap. MeshWeaver.Crm (created 2026-08-28, no hooks) last synced to memex.systemorph.com on 08-30; by 09-07 the repository was 54 commits ahead, the portal still held files the repository had retired, every prebuilt Crm bundle was refused by the source-fingerprint gate (the bundle was built from the newer files), and every Crm page compiled the stale copy on first use. The fix is a hook per live portal with that portal's own secret, proven by a ping delivery that reads 200; the setup runbook is /new-repo §12 — .claude/skills/new-repo/SKILL.md in the repository. Since MeshWeaver#3583 a module's content in a partition the portal does not track at all (no _GitSync naming a repository) no longer compiles from the leftover copy: it settles at a named error instead (NodeType Compilation → A module's content this mesh does not TRACK).

8. See also

Reconnecting…
The server was updated. Reloading the page to pick up the latest version.