A regression test that passes proves the code passes the test. It does not prove the test can detect the defect it is named for. Those are different claims, and only one of them is what a regression test is for.

A pin is only a pin if it fails against the defect. Revert only the fixing line(s), run the test, confirm it fails with the message you expect, restore, confirm it passes. Record the failure output in the change that introduces the test.

This is not extra rigour for hard cases. It is the cheapest available check on the one property a regression test must have, and it costs a minute. The alternative is a test whose failure mode is a false pass: the guard silently disappears and the green tick does not.

The procedure

  1. Write the test against the fixed code. Green.
  2. Revert only the fix. Not the test, not the signature, not a nearby refactor — the specific lines that change behaviour. If you cannot isolate them, that is itself a finding: the fix is entangled with something else.
  3. Run and read the message. It must be the failure you predicted, in the words you predicted. "It failed" is not enough — a test that fails for the wrong reason is still not a pin.
  4. Restore, re-run, confirm green.
  5. Paste the red output into the PR body. That output is the evidence; the green run is not.

Two things this also buys you, beyond confirming the pin:

State plainly where you could not produce a difference. One control broke coalescing "the other way" — each track creating its own node — and both the old and new test shapes went red. The report said so, and named the remaining exposure as timing-dependent and therefore not demonstrable without rigging the clock. A negative control that comes out neutral is a result, not a failure to report.

Four tests that proved nothing

Each of these was written in good faith, passed, and was believed. What broke them is not carelessness — it is that the assertion measured something adjacent to the property.

1. The count was 1 either way

TrackActivity_ConcurrentSamePath_DoesNotRaceAlreadyExists fires five concurrent posts at one path and asserts they coalesce into one node. It waited for the first node to surface on the eventually-consistent query index, then asserted HaveCount(1).

The path is the storage key. The node count is 1 whether or not the race is coalesced — with the coalescing fold removed, four of five creates throw Node already exists and their increments are simply lost, and the count stays 1. The test could not detect the race it was named for, and the blind spot is structural, not timing.

Demonstrated rather than argued: a temporary replica of the old shape ran side by side with the migrated test against the same deliberately broken handler, in the same run.

TEST FAILED: Expected value to be 5 ... but found 1     <- migrated test          (RED)
Passed                                                  <- old PollForFirst shape (GREEN)
Failed: 1, Passed: 1

The assertion that actually pins the defect is AccessCount == 5 — every track folded its increment onto the live record. The waiting had to change too: the wait now covers all five writes (WhenSettled(path, writes)), because a wait that returns while writes are still in flight makes even a correct assertion vacuous.

The lesson to carry: ask what the assertion's value would be if the defect were present. If the answer is "the same", the assertion is not measuring the property. Counting is the usual culprit, because a count is so often invariant under exactly the merging or deduplication you are trying to test.

2. Something else was doing the blocking

PixelRenderIsolationTests covers the pixel export's server-side-request-forgery surface: a headless browser renders user-authored slide HTML inside the server's trust boundary, so it can reach internal services the deck's author never could. There are two independent defences — a Content-Security-Policy meta tag in the composed document, and Chromium's own process-level network denial flags.

The first draft asserted the CSP with both layers armed. It passed. It would have passed with no CSP at all, because the process flags were doing the blocking.

"No request arrived" is a claim with many explanations, and most of them are not the one you want: the browser never had a route, the listener was misconfigured, the markup was wrong. Testing two layers at once hides a hole in either.

The rebuilt suite makes each test single-variable, and demands a leak before believing a block:

// ── Control: without the policy the slide MUST reach the listener. ──
probe.Reset();
await renderer.Render(WithoutPolicy(html)).Should().Emit();
var leaked = probe.Connections;
leaked.Should().BeGreaterThan(0,
    "the control must demonstrate the vector is real — a slide CAN otherwise make the "
    + "server's browser open a connection of its choosing. At 0 the assertion below would "
    + "prove nothing, which is exactly how the first draft of this test fooled itself.");

// ── Protected: the composed document must reach nothing. ──
probe.Reset();
var pdf = await renderer.Render(html).Should().Emit();
probe.Connections.Should().Be(0, "…this is the SSRF surface…");

The_policy_alone_stops_a_slide_from_opening_a_connection runs with the process-level denial neutralised, so the CSP is the only thing left that could block. The_process_flags_stop_it_too_even_with_no_policy_at_all strips the CSP entirely. Two layers, two tests, one variable each.

Note the third safeguard: the helper that strips the policy throws if the meta tag is ever renamed, rather than silently returning an identical document. A control that quietly stops being a control turns the whole suite green-and-meaningless — which is the same failure class one level up.

The lesson to carry: when an assertion is negative — nothing happened, no request arrived, nothing leaked — a live positive control is mandatory. Disable one layer at a time and require the control to fail.

3. It samples, so a green run is not evidence

MeshHubDisposalLeakTest walks GC roots looking for a MessageHub that survived disposal. It is an excellent discovery probe and it found nine real leaks. It is not a pin, and a green run from it is not evidence:

The pin for a leak is a timing-free ownership test next to the code that owns the subscription: after the owner is disposed, the handle holding the pending timer is disposed too — a property that holds whether or not the timer has already fired, so the test can neither flake nor pass by accident. A WeakReference probe would be a sampling test of a 100 ms window, i.e. the thing that pins nothing. Subscription Ownership carries the full treatment, including the negative-control table for the two live sites.

The carve-out, which proves the rule rather than weakening it (#3321). What makes a WeakReference a sample is that its truth depends on when it is evaluated. Remove the when and it stops being one. A hub's Dispose() returns while its teardown is still running, so the window a naive probe samples is "did the shutdown happen to finish yet" — but that window has an explicit end signal, DisposalCompleted. A probe that joins the signal and only then collects is asserting something timing-free: after the owner's teardown is complete, nothing but a kept reference can hold it. That is a property, not a snapshot, and StreamReleasesItsHubTest pins it.

So the test is not "is it a WeakReference?" but "does the assertion name the moment it becomes true?" A probe with no such moment — a 100 ms log flush, a 1 s watcher re-establish — has none to name and stays a discovery tool. The same page's own lesson applies unchanged, and the measured cost of getting this wrong is in Writing Tests: without the join, that assertion passed in a 485-test suite and failed alone on the same binary.

The lesson to carry: if the assertion's truth depends on when it is evaluated, it is a sample, not a proof. Sampling probes belong in discovery, never in the regression suite as the guard for a specific defect.

4. The control was served by the transport the test then destroyed

The first three measured something adjacent to the property. This one is worse, and it is the shape to learn: the control was guaranteed green by construction, and it was actively preventing the state the test went on to assert.

PodHubTransportTest.CrossSiloNack_ReachesASenderWhoseStreamSubscriptionIsGone erases a sender's Orleans stream subscription and asserts that a router's NACK still reaches it — over the directed pod-hub transport, which is the whole point. Before erasing, it proved the address was live:

// "Prove the pod-hub claim is LIVE before erasing anything — by a cross-silo delivery ARRIVING"
await WaitUntil(async () => {
    SiloMeshHub(cluster, 1).Post(new PingRequest(), o => o.WithTarget(sender));
    return inbox.Any(d => Describes(d, nameof(PingRequest)));
}, …);

That comment states the intent exactly. The code does not do it. RegisterStream establishes two things on different clocks — the local route synchronously, and the cluster-wide pod-hub claim asynchronously, on a capped backoff with no give-up. The probe asserts reachability, and during the window that matters reachability is served by the stream — the transport erased three lines later. So the probe was green whether or not the claim had landed.

And it fought the claim it appeared to prove. The probe posts from the other silo in a loop. Each post whose directed call finds no claim mints a throw-away IPodHubGrain activation on that silo — [PreferLocalPlacement] places an unactivated grain on the caller — and the owner's next Attach lands on that activation, answers false, and restarts its backoff. The louder the control ran, the longer the condition took to become true.

The failure was therefore intermittent and independent of the diff: two heads of one pull request differing only in a string-assertion test in another assembly disagreed on it, which is what got it filed as a platform race (#3298).

Measured. With one bounce forced and the claim's backoff pinned long, the probe goes green in under a second while PodHubClaimSettled has demonstrably not completed — six runs, six times. That experiment was deliberately not committed: its setup depends on an activation-lifetime ordering that cannot be enforced, so as a permanent test it would have become the next flake. An experiment that pins a cause and a test that guards it are different artifacts, and the first does not have to become the second.

The cure is to wait on the condition itself — PodHubClaimSettled, the positive signal for "the claim stopped attempting" — which a sibling test already did. See Orleans Test Routing Pattern → "Reachability is not a claim".

The lesson to carry: when a test isolates one mechanism in order to assert another, ask what serves the precondition probe while the isolation is not yet in place. If the answer is "the thing I am about to remove", the probe cannot fail, and its greenness is not evidence of anything. The generalisation is broader than tests: any control whose green is guaranteed by construction — a detector that still passes with the fix removed, a gate that skips when its input is missing, a watcher that renders "I cannot see" as "nothing is wrong" — is the same defect wearing a control's clothes.

What a good control looks like in a PR

Name the test, show the reverted state, show the message.

`CreateNodeAlwaysAnswersTest` (4 tests). With the fix: 4/4 pass in 411 ms.
With ONLY MeshExtensions.cs reverted:

  Create_WhenAdapterDeclinesTheWrite_AnswersWithFailure [FAIL] [15 s]
    Expected the observable to emit a value within 15s … but it did not.
    The observable emitted nothing at all.
  Create_WhenSaveCompletesWithoutEmitting_AnswersWithFailure [FAIL] [15 s]
    … The observable emitted nothing at all.
  Failed! - Failed: 2, Passed: 2

The other two pass in both states, which is the point: they are guards, not the pin.

A hang is the pre-fix behaviour there, so a timeout is what the assertion reports — and saying so explicitly is part of the evidence, because otherwise a reader cannot tell a real pin from a slow test.

When it is harder than reverting one line

See also

Reconnecting…
The server was updated. Reloading the page to pick up the latest version.