Decentralised Tests

Goal, from the maintainer: eliminate MonolithMeshTestBase and OrleansTestBase. Not by rewriting 463 classes onto a new base — by decentralising tests to the module they are about, so that nothing is left inheriting them and the two files can be deleted.

"refactoring goal: eliminate monolith.testbase and orleans.testbase" "anything which is a plugin surely does not stay xunit. test process is 1) take image, 2) install plugin, 3) execute test code" "the substrate ones are testable in-mesh ==> test container must boot with this plugin enabled" "each repo should only use itself" · "decentralize" … "also tests"

Tests are the third artefact to decentralise, after skills and docs:

This is not a new mechanism. 78 NodeTypes in MeshWeaver.Plugins already carry a <Type>/Test/ folder, 76 of them with a Tests layout area, executed by mw-plugin-test against a pinned image. Those were never xunit. The xunit estate is the centralised exception, not the norm.

Measured against Systemorph/MeshWeaver@f8e208567 (core origin/main) and Systemorph/MeshWeaver.Plugins@origin/main, 2026-09-01. Re-measure before acting; the numbers move weekly.

1. What is actually there — the measured estate

🚨 Correct a number that has been repeated wrongly. "68 test projects under test/" is Plugins' count (69 test csproj), not core's. Core has 44 project directories under test/ — 40 with cases, 4 support libraries (MeshWeaver.Fixture, MeshWeaver.Hub.Fixture, MeshWeaver.TestDomain, MeshWeaver.Data.TestDomain) plus the two test bases this page is about. The 6,817 cases figure IS core's; comment-stripped the count is 6,841.

🚨 A second number to correct: the estate is ~13,900 declarations, not ~10,760. Plugins carries slightly MORE than core (7,030 vs 6,841) across 67 projects — and seven of its suites share a name with a core suite (Hosting.Monolith.Test, PluginCatalog.Test, Query.Test, Security.Test, GitSync.Test, Auth.Test, Content.Test), which is the carve-out's footprint. Its largest single suite is MeshWeaver.AI.Test at 1,354.

🚨 Every case count on this page is a count of [Fact]/[Theory] DECLARATIONS, not of executed cases. [InlineData] multiplies them: MeshWeaver.Persistence.Test declares 99 and executes 1,137 (measured on this branch, 46 s). The executed estate is several times 6,841. That gap is not bookkeeping — it is the whole argument of §5b, and it is why an in-mesh runner without data cases is not a substitute.

Base-class fan-in (core, direct derivations)

⁽¹⁾ Re-measured by RUNNING the assembly during step 3: OrleansSharedTestBase reaches 27 cases, not 24, and the OrleansTestBase<T> row's twelfth "class" is the non-generic OrleansTestBase declaration — a base, not a suite. Eleven suites, 23 cases. Both bases together: 23 suites, 50 cases. Counting [Fact] declarations undercounts; only a run is evidence.

🚨 The first honest correction: the two bases in scope carry 2,139 of 6,841 cases — 31%. 4,048 cases (59%) inherit no test base at all — they are plain xunit unit tests that never boot a mesh. Deleting MonolithMeshTestBase and OrleansTestBase does not touch them, and no plan should claim their number as progress. HubTestBase (611 cases) is a third base, not in the maintainer's scope, and it is the natural landing place for anything that needs hubs but not a mesh.

2. Per-project classification

Buckets are the maintainer's: plugin (its subject ships as an installable module ⇒ install-and- execute), substrate (the mesh runs on it ⇒ in-mesh with the container booted with that backend enabled), framework primitive (below any mesh boot), not-a-mesh-thing, and — added by measurement — harness (the code that boots the mesh, which cannot run inside one).

Bucket totals (cases):

🚨 The second honest correction: "anything which is a plugin does not stay xunit" removes 515 cases, 7.5% of core's estate. Core is a framework repo — almost nothing in it ships as an installable module. The volume is in framework primitive and substrate, and the maintainer is right that substrate is migratable: booting a mesh on PostgreSQL and asserting from inside is a more faithful test than a mock, not a less faithful one. That is what the bootstrap seam exists to say.

3. The 463 subclasses reduce to FOUR knobs

The elimination is only tractable if the 439 + 24 classes are not 463 different setups. They are not. Measured by parsing every class body that transitively reaches each base and recording which virtual it overrides:

MonolithMeshTestBase — 471 class bodies (439 direct + generic/nested)

Four knobs cover 96%: ConfigureMesh (~285), ShareMeshAcrossTests (~74), ConfigureClient (~53), SetupAccessRightsAsync (~34). The long tail is two deadline properties and DisposeAsync.

OrleansTestBase / OrleansSharedTestBase — 23 class bodies

🚨 OrleansTestBase is not a design, it is a hosting choice. 20 of 23 subclasses override nothing. That is exactly the shape the bootstrap seam replaces: which mesh do I boot is a class-level parameter, not a second base class.

✅ DONE (step 3). The two bases are now one, OrleansMeshTestBase. Re-measured while doing it, because two of the numbers above are wrong: the two bases carry 23 classes / 50 cases, not 24 / 47 — OrleansTestBase's family is 11 classes / 23 cases (the "12th class" the parser counted is the non-generic OrleansTestBase declaration itself, which is a base, not a suite) and OrleansSharedTestBase's is 12 classes / 27 cases, not 24. The whole assembly executes 207 cases, not the 192 the [Fact]-counting survey reported. Verdict before and after the conversion: 207 passed / 0 failed, 30 s → 26 s.

🚨 And the step as written does not work. "Convert the subclasses to Bootstrap => MeshBootstrap.Orleans(…)" reads as make them MonolithMeshTestBase subclasses. They cannot be: a cluster is deployed ASYNCHRONOUSLY, in InitializeAsync, whereas ConfigureMeshBase — where Bootstrap.Bootstrap(builder) is called — runs in the CONSTRUCTOR, before any cluster exists; and 18 of these suites assert on Cluster or SiloServices, surfaces a base that must not reference Orleans cannot expose. What the seam unifies is the declaration, not the boot. So the win is the one the measurement actually supports: the two Orleans bases were the same machinery written twice (deploy, client hub + AccessContext + routing registration, tracked teardown), differing in three VALUES — silo configurator, silo count, leased-or-dedicated — and values do not need a second base class.

And what is inside ConfigureMesh?

322 ConfigureMesh bodies, tagged by what they do (a body can do several):

4. The blockers, with counts

In Plugins the blocker was ServiceProvider.GetRequiredService<T>() — 169 files, because in-mesh area code is compiled node source running in a NODE hub and cannot reach the host container.

Core's equivalent: 384 files, 1,145 sites. The histogram is far narrower than the raw count:

🚨 Eleven types account for 803 of 1,145 sites (70%). The facility question is therefore "can a decentralised test be handed a context exposing about a dozen mesh services?" — not "can it reach an arbitrary container?".

And the answer differs by how the test ships, which is the single most important design choice on this page:

Decentralising tests must take the second route. Then GetRequiredService<T> is not a blocker at all: it needs the runner to hand each test the mesh's IServiceProvider. That is one facility, and it retires all 1,145 sites — and §5 shows the mechanism already half-exists as StaticTestRunner's Outcome.NeedsMesh, which routes a case that takes a parameter to the mesh lane.

The shipping route is settled by that choice too. Plugins#1060's ModuleSkillProvider is the precedent for per-module content: <EmbeddedResource Include="MeshSkills/**/*.md" /> — one csproj line, no marker attribute, no reference to the consuming module — read by an IStaticNodeProvider over IEnumerable<InstalledModuleAssembly> (what MeshBuilder.InstallAssemblies records). A MeshTests/{Package}/… analogue would be a direct structural copy, and it is the wrong route for core. That mechanism serves source the mesh compiles; a framework assembly's tests are already compiled, already have InternalsVisibleTo, and StaticTestRunner already loads the built assembly and reflects over it. 🚨 It also carries a cost that matters here: a statically served path cannot also hold a durable row, so an embedded-resource route would forbid a test node ever being written to. Ship core's decentralised tests as types in (or beside) the module assembly, discovered by reflection — not as embedded source.

The second blocker: registering test doubles into the mesh

119 ConfigureMesh bodies register into the host container. What they register:

🚨 This is the real facility gap, and it is ~50 sites, not 1,145. A decentralised suite must be able to say "boot the mesh with THIS extra registration". Because the test ships as an assembly, the registration is just a delegate — the bootstrap needs a WithServices(Action<IServiceCollection>) escape hatch, applied once per suite. IConfiguration and ILoggerProvider (35 of the 119) are infrastructural and belong on the runner, not in test code.

Other blockers, measured (files / cases)

The filesystem is not a blocker. A test running inside the portal container has a filesystem; the paths just live in the container. The AssemblyLoadContext cases are a blocker, because they are the compile/emit pipeline the in-mesh runner itself depends on (§6).

There is no mocking framework to port. Zero Moq sites; four NSubstitute sites. Test doubles here are hand-written classes, which ship with the module like any other type.

5. What the in-mesh runner is today — TWO lanes, not one

Before asking what a runner must replicate, be precise about what exists. mw-plugin-test (tools/MeshWeaver.PluginTester/) drives two different lanes, and they discover tests differently. Conflating them is the main way this design goes wrong.

Lane A — the mesh gate (renders the Tests area)

PluginGateRunner + AreaProbe: stands up a mesh, imports content, renders each type's Tests layout area over the ordinary client sync stream, and classifies the frames (AreaProbe.ClassifyTestsFrame): compile-progress and "Area not found" are transient; ❌ is red; N/M passed is green iff N == M; no verdict inside the timeout is RED — "a Tests area that reports nothing is a broken gate, never a silent pass."

The suites it runs are hand-rolled. Store/Catalog/Test/CatalogTestsArea.cs is the canonical shape:

var cases = new (string Name, Action Body)[]
{
    ("Packs: the backfill trigger is exact", StoreCatalogTests.StandardPacksBackfill_TriggerIsExact),
    …
};

— a manual (name, delegate) array, bodies that are public static void and signal failure by throw, and a StringBuilder markdown table emitted as ONE frame at the end. A separate (string, Func<IObservable<Unit>>) array carries hosted cases through RunLive (45 s budget, DefaultIfEmpty so "completed without an outcome" is a named row rather than silence); only 3 of 74 *TestsArea.cs files use it.

🚨 A method that is not listed in the array is never run. There is no discovery in Lane A.

Lane B — the build lane (mw-compiler build, no mesh)

CascadeBuild → StaticTestRunner. This lane is already xunit-shaped, and it is the one to build on:

• loads the just-emitted assembly into a collectible AssemblyLoadContext;
• discovers by reflection and convention — static class whose name ends Test/Tests/TestsArea, public static methods, not compiler-generated;
• runs each case on its own thread with Thread.Join(timeout), so a hang is reported by name and the build continues;
• Outcome is Passed | Failed | NeedsMesh — a method that takes parameters is classified NeedsMesh, counted and named, never dropped;
• Run.IsGreen => LoadError is null && Failed == 0, with the explicit rule "No cases is green, not a pass."

🚨 NeedsMesh is the existing answer to "how does a test get a mesh handle": it takes one as a parameter. Today that parameter is a LayoutAreaHost. Generalising it to a test context that also exposes the mesh's IServiceProvider is an extension of a mechanism that already ships — not a new one.

The xunit shim is already in the tree, and is not wired up

tools/MeshWeaver.PluginTester/Testing/TestContext.cs (namespace MeshWeaver.Testing) already defines, verbatim for this purpose:

• TestContext.Current → TestContextData(CaseName, CancellationToken), [ThreadStatic], installed by TestContext.Enter(…) — "the shape xUnit's TestContext.Current gave the suites this replaces, so a migrated body keeps reading TestContext.Current.CancellationToken";
• SkipException(string reason) — "The runner reports it as skipped — never as passed, never as failed";
• TestLog.WriteLine — "the role ITestOutputHelper played in the suites this replaces."

🚨 StaticTestRunner does not reference any of it. There is no TestContext.Enter call, no SkipException catch, no Outcome.Skipped. The shim is landed and unwired — the cheapest first facility on the list, and a gate that silently never skips is worse than one that cannot.

5b. The xunit features that must be replicated, with counts

🚨 [Theory] + [InlineData] is the requirement that decides the plan, and the raw count understates it. Measured on a real suite: MeshWeaver.Persistence.Test declares 99 [Fact]/[Theory] and executes 1,137 cases — an 11× expansion. Neither lane can express a data case. Today the in-mesh idiom is an in-body foreach over a local array, which collapses N logical cases into ONE row and aborts the rest at the first failure. Migrating 431 theories onto that would turn ~2,000+ per-case verdicts into 431 opaque ones and lose the "which row failed" information that makes a red run actionable.

Build the data-case facility before migrating any suite.

6. The honest floor — what cannot decentralise

Hard floor: 535 of 6,841 cases (7.8%). These cannot execute inside a booted mesh by construction — not for want of a facility, but because they are about the boot itself.

Plus a soft floor of ~115 cases with nothing to gain: Json.Test (78), Reactive.Assertions.Test (23), Serialization.Test (14) are pure functions over pure data. Nothing blocks moving them; nothing is learned by doing so. Leave them.

So the honest arithmetic for core:

🚨 The last row is the one to read. Not one of the 6,190 can move today. Five facilities are missing or unwired, and until each exists the count of migratable cases is zero:

Only #3 and #4 are genuinely small. An estimate of "90% can move" that does not carry all five preconditions is an optimistic number — and #1 alone is a larger piece of work than any single suite migration on this page.

7. The bootstrap seam — where it lives

IMeshBootstrap is the mechanism by which a base class disappears: a base class can be deleted when its setup is expressible as a bootstrap configuration and its subclasses select one.

protected override IMeshBootstrap Bootstrap => MeshBootstrap.Orleans(o => o
    .WithClustering(ClusterProvider.AdoNet, "Server=…")
    .WithGrainStorage(StorageProvider.Redis, "localhost:6379")
    .WithSilos(2));

It lives in core. Maintainer, 2026-09-01: "we can migrate it to core for now." That is settled, and it is the right way round — core is not allowed to depend on Plugins, and 439 of the 463 inheriting classes are in core.

🚨 "For now" is doing real work in that sentence. Its final home is with the substrate module it describes, once tests are decentralised: MonolithBootstrap with MeshWeaver.Hosting.Monolith, OrleansBootstrap with MeshWeaver.Hosting.Orleans, and the IMeshBootstrap contract in the messaging/mesh contract layer everything already references. So the seam is deliberately kept relocatable: it depends on MeshBuilder and nothing else, OrleansBootstrapOptions is a pure record (validated without a cluster), and the Orleans applicator is a static delegate the Orleans-referencing assembly registers — which is why MeshWeaver.Hosting.Monolith.TestBase never takes an Orleans dependency.

Retiring Plugins' copy — and a defect the survey found

MeshWeaver.Plugins/src/MeshWeaver.Hosting.Monolith.TestBase is a full fork of core's test/MeshWeaver.Hosting.Monolith.TestBase: byte-identical except MeshBootstrap.cs, the 12-line Bootstrap route, and $(MeshWeaverRoot)-relative ProjectReferences.

🚨 The fork is already inert, and nobody noticed. Every Plugins test project references the base with a conditional pair:

<ProjectReference Include="$(MeshWeaverRoot)/test/MeshWeaver.Hosting.Monolith.TestBase/…"
                  Condition="Exists('$(MeshWeaverRoot)/test/…')" />
<ProjectReference Include="$(MeshWeaverRoot)/src/MeshWeaver.Hosting.Monolith.TestBase/…"
                  Condition="!Exists('$(MeshWeaverRoot)/test/…')" />

Core has carried test/MeshWeaver.Hosting.Monolith.TestBase since the "fixture to test" move (2026-08-30), so the first arm always wins and the fork is never referenced by any real suite. Its only consumer is MeshWeaver.Mesh.TestBase.Test, which references it by relative path — so the bootstrap seam was pinned by its own unit test and reachable from nothing else. This is the two-homes failure mode in its purest form: the copy that got the new feature is the copy nothing builds against.

Retirement, in order:

1. (done in this change) core gains MeshBootstrap.cs and the 12-line Bootstrap route; MeshBootstrapTest moves to test/MeshWeaver.Hosting.Monolith.Test. Behaviour is unchanged — MonolithBootstrap.Instance applies exactly UseMonolithMesh().AddInMemoryPersistence(), which is what ConfigureMeshBase inlined before.
2. Plugins deletes src/MeshWeaver.Hosting.Monolith.TestBase/ and src/MeshWeaver.Mesh.TestBase.Test/, and drops the !Exists(...) fallback arm from all ~50 consumers. There is no delta left to preserve.
3. MeshWeaver.Hosting.Orleans.TestBase registers OrleansBootstrap.Applicator at module init, so MeshBootstrap.Orleans(…) can actually stand a cluster up.

Until step 2 lands, the fork must not receive changes — it is the drift hazard, not the seam.

8. The order to do it in

Each step is chosen so the next one is cheaper, and so that nothing is migrated before the facility it needs exists.

Steps 1–3 eliminate one of the two named base classes for 50 cases of blast radius (measured; the survey's 47 counted [Fact] declarations, not executed cases). That is the whole argument for this ordering: the Orleans half is 2% of the estate and 100% of the second goal, and it needs none of the five facilities in §6 — the bootstrap seam alone is enough. MonolithMeshTestBase cannot follow until step 5c is green.

9. What this change actually contains

• test/MeshWeaver.Hosting.Monolith.TestBase/MeshBootstrap.cs — the seam, moved from Plugins.
• MonolithMeshTestBase.ConfigureMeshBase routes through Bootstrap.Bootstrap(builder); a new protected virtual IMeshBootstrap Bootstrap => MonolithBootstrap.Instance.
• test/MeshWeaver.Hosting.Monolith.Test/MeshBootstrapTest.cs — 9 cases pinning the fluent API without a cluster, moved from Plugins' MeshWeaver.Mesh.TestBase.Test.
• this page.

No behaviour change: the default bootstrap emits the same two builder calls that were inlined. All 22 core consumers of MonolithMeshTestBase are untouched.