Module pins — how a shipped fix stays invisible, and what now sees it

A store-installed module does not follow its registry. It is pinned at the generation it was installed at and loads from /tmp/meshweaver-pinned-modules/<pid>-…/<Module>@<generation>/, and nothing moves that pin by itself — not a rolling restart, not visiting the package cover, not an enrollment.

On memex.systemorph.com (#959) that cost two days. AddHubPermissionRule(Read, IsAuthenticated) (#845) was merged, published, and present in every released 1.1.x; /app/Northwind/Dashboard denied every signed-in user anyway. Three control-plane restarts were spent discovering that the deployment was still running bytes from before the fix. Every version string anyone looked at was current. Only the bytes were old.

Three numbers, and only one of them is about the bytes in memory

The third one is the only honest answer to "what is this mesh serving", and even it is only honest when the loaded assembly actually came out of the generation the entry names. A newer generation can sit on the volume while this pod still runs the previous one — the entry is then perfectly current and describes bytes nobody in this process has loaded.

What #959 shipped, and what it did not

• (a) the typed InstallManifest read — no defect behind it; every reader was already shape-tolerant.
• (b) the cure: the RefreshModules maintenance task (#1001), which re-runs the platform's own registry install so the pin moves.
• (c) the alarm — nothing that could SEE the state the cure cures. RefreshModules is a remedy: somebody still has to suspect the pin and go run it. That was #1074, and it is what ModulePinAudit is.

The alarm — Store/Maintenance → ModulePinAudit

For every Plugins/{id} record that declares a module, it compares the version behind the bytes this process loaded against the version the configured registries currently publish for that package, off the same /api/plugins/bundles index the landing lane itself reads — so the alarm and the updater can never disagree about what "published" means. It runs as cases in the Store/Maintenance Tests area, so CI executes it on every change and an operator can render it against a live mesh.

A finding names the version on both sides, the generation, the loaded MVID, the registry, and the remedy:

❌ Module pins: 1 of 1 store-pinned module(s) are NOT provably at the version their registry publishes.
- ❌ `Northwind` (`MeshWeaver.Northwind.Application`): serving 1.0.3 (generation
  MeshWeaver.Northwind.Application@dd91b292, mvid 554cf7e2) while https://memex.meshweaver.cloud
  publishes 1.1.1 — the store pin is BEHIND the registry, and the install record already claims
  1.1.1 — the RECORD moved, the BYTES did not. Re-land it: create a `Store/Maintenance` node under
  `Admin/Maintenance` with task `RefreshModules`, then RESTART the deployment so the landed
  generation loads.

The two rules the design turns on

1. 🚨 A source that cannot be READ is not "everything is current." An unreachable registry, a 401, a timeout or a denied Plugins/* listing FAILS the audit. All three of "nothing is behind", "I could not reach the registry" and "I was not allowed to list the installs" produce zero findings; only the failure line tells them apart. This is ModuleRefresh.Unreadable's rule, applied one layer up.
2. 🚨 A version string on a record is not evidence about the bytes. The comparison never reads releasedVersion; it reads the activation entry for the generation the loaded assembly came out of, and refuses to name any version when those two disagree. The record's number is carried only so a finding can say "the RECORD moved, the BYTES did not" — the sentence that turns "why is the fix not live" into an answer without a second investigation.

What is deliberately NOT a finding

• A module loaded from the image or the Modules:Assemblies baseline — it has no store pin, and comparing it would red every deployment whose image is not the registry's newest publish.
• A disabled activation entry — a completed uninstall; an alarm must not fight the operator.
• A registry serving older than what is landed — a pinned or lagging registry, never a stale pin (the same call ModuleUpdateDecision.SkipOlder makes).
• 1.2 against a landed 1.2.0 — the comparison is SemVer, so a two-part publish is not an eternal "behind".

Everything else that cannot be resolved is reported, never skipped.

Fixing a mesh the alarm has flagged

1. Create a Store/Maintenance node under Admin/Maintenance with task: RefreshModules (blank packageId = every recorded install; a named one must actually be recorded). Global-admin only.
2. Watch state go Requested → Running → Done/Failed; the summary names every package.
3. Restart the deployment — landing writes a new generation and moves the activation pointer, and the module loads at the next restart (restart-as-activation). Until then the audit correctly still reports the process as serving the old generation.