A test that flips may be a platform skew, not a race
The rule. Before calling a satellite test non-deterministic, reproduce each verdict against a named core commit. A suite in this repo compiles against
$(MeshWeaverRoot)— a localSystemorph/MeshWeavercheckout, at whatever commit and whateverbin/state that checkout happens to be in — so "the same binary" is a claim about the test assembly, never about the platform assemblies it links. Two runs minutes apart can straddle a core commit.
What was reported (MeshWeaver.Plugins#2109)
src/MeshWeaver.Auth.Test/ApiTokenServiceTests.cs → DeleteToken_NonexistentPath_Completes
answered differently on two consecutive runs:
| Run | Assertion in the binary | Failure |
|---|---|---|
| 1 — 16:00:10Z, full suite of 160 | result.Should().BeFalse() |
Expected value to be false, but found True. |
| 2 — 16:03:15Z, rebuilt | result.Should().BeTrue() |
Expected value to be true, but found False. |
The other 159 cases passed on both runs, and the test was identical to main. Read as one
instrument that is the signature of a race: the service returns either verdict.
What it actually was
Both verdicts are deterministic, and they are the two sides of one core commit —
757c9d1ad4, "fix(auth): DeleteToken reports what it removed, not that it ran".
ApiTokenService lives in core (memex/Memex.Portal.Shared/Authentication/), not in this
repo. Before that commit it read:
var primary = nodeFactory.DeleteNode(tokenNodePath)
.Select(_ => true) // ← the constant
.Catch<bool, Exception>(ex => { logger.LogWarning(...); return Observable.Return(false); });
That projection was harmless only while deleting an absent path faulted. MeshWeaver#4668 made
it a SUCCESS that emits false (DeleteNodeResponse.AlreadyAbsent on the wire,
IMeshService.DeleteNode returning !AlreadyAbsent), so from #4668 until 757c9d1ad4 the
projection overwrote exactly the bit the new contract had added — and every absent path reported a
removal that never happened. After 757c9d1ad4 the delete's own value passes through, and
505a416f10 then took the .Catch out too, so a REFUSAL faults instead of masquerading as
"already gone".
The control (measured 2026-09-18, this machine)
One case on each side of the change, same worktree, same test source, only
-p:MeshWeaverRoot= differing:
| Platform under test | Runs | Result |
|---|---|---|
core 757c9d1ad4^ (pre-fix) |
5 | 5/5 red, Expected value to be false, but found True. — the issue's run-1 message, verbatim |
core main (73e0519eda) |
8 isolated + 3 × full suite (160) | 11/11 green, and 480/480 cases green |
So neither verdict is timing-dependent; the observed pair is fully explained by which
Memex.Portal.Shared build the test host had loaded. No re-run, under any load, produced a second
value from one platform.
The verdict DeleteToken returns, and why
false — it removed nothing. The delete verb's postcondition is "no node exists at that
path", and an absent node already satisfies it, so the call SUCCEEDS; the bool is the honest
half, saying which of the two successes happened. Core pins the same contract twice —
test/MeshWeaver.Graph.Test/DeletingAnAlreadyDeletedNodeIsNotAnErrorTest at the framework level
and test/Memex.Portal.Shared.Test/DeleteTokenReportsWhatItRemovedTest at the service level — so
flipping this repo's assertion to BeTrue would have pinned a value core actively tests against.
What this repo changed
Not the service — core's is correct and covered. What was wrong here is that the test could not tell a contract from a constant, and so could not say which side it had measured:
- a lone
result.Should().BeFalse()over one call passes just as happily under a hardwiredfalse, and carries nobecause, so its red says nothing; - it is now
DeleteToken_ReportsWhatItRemoved_SoAbsentAndRemovedCannotBeConfused— a live token's delete answerstrue, the same path's second delete answersfalse, and a never-minted path answersfalse, all on one mesh through one service instance. No constant satisfies all three, and the second leg's failure text names the.Select(_ => true)shape, so the next reader gets the mechanism instead of a flip.
How to read the next one
- Record the platform commit before you rerun anything, and rerun only against a NAMED one. It is the unpinned rerun that decides nothing — a second green says nothing about which platform produced the first red. Repeated runs are not the problem and steps 4-5 require them; an unstated platform is. Pinning also keeps the other verdict reachable: one fixed, named build that really is flaky shows up as a mixed result on that pin.
- Ask what the failing call's implementation is, and where it lives. If it is under
$(MeshWeaverRoot), the suite's answer is the platform's answer. git log -S'<the projection or literal>' -- <that file>in core; the commit that introduced or removed it is the candidate boundary.- Build the one test project twice —
-p:MeshWeaverRoot=<worktree at BOUNDARY^>and-p:MeshWeaverRoot=<worktree at BOUNDARY>— and run it. Two deterministic, opposite results close the question; one flaky result means you have a real race and the repro is still owed. - Say the denominator in the finding: N runs each side, not "it reproduces".
Related: AdvisoryFloorsInADependent.md · ATestFakeIsScopedToItsSubject.md · PlatformSourcePin.md