The Merge Queue

A merge queue builds the combination that is about to land, before it lands. With strict: false branch protection every pull request is tested against the main it branched from, so a burst of merges lands a tree no run ever compiled. That happened twice in one week — CS0246: MeshOperations on 2026-08-26, and on 2026-08-30 a new guard (#2782) plus an independently-landed gate primitive (#2792) that were green alone and red together, holding main red for five consecutive runs and catching a documentation-only PR in the blast. The queue is the structural fix (#2412): each entry is built on top of the entries ahead of it, and the tested commit is the commit that lands.

This page is the operating manual: what the first outing taught, the settings that answer it, and the steward — the lane that acts on every ejection so a person does not have to.

What the first outing measured (2026-08-30 → 09-01)

The queue was enabled on 2026-08-30 by #2799 (merge_group trigger on Build and Test) plus a merge_queue rule on the main pr protection ruleset — ALLGREEN, up to 3 entries built and merged, SQUASH, 60-minute check timeout — and removed on 2026-09-01. Two failure modes, both measured:

1. The churn window. With three entries built speculatively, every membership mutation — an enqueue, a dequeue, a push to a queued branch — rebuilt the speculative stack and restarted every in-flight build. Several sessions each acting reasonably kept the stack in permanent rebuild: for over an hour nothing landed, and nothing failed. Zero reds, zero merges — the run list showed only cancellations, which nothing alerts on (the same shape as Reading CI Signals § "Delivery can stop for hours with every dashboard green").
2. Hand re-queues. An entry a flake ejected simply left the queue. Nothing re-queued it; the maintainer did, by hand, after noticing. Of the 83 red Build-and-Test runs between 08-30 and 09-02, 51 were merge_group runs — most of them ejections that then needed a person.

Two things did work and are kept: queue merges are ordinary pushes, so main's push lanes ran on every one of them (40/40 on 08-31 — the GITHUB_TOKEN non-triggering trap does not apply to the queue); and Build and Test already fires on merge_group, with Consolidate test results produced unconditionally on that event (collect-results is if: always() with no event filter, and the one gate that needs a pull-request body — cross-repo-pair — is exempted on the event, so its fail step cannot fire on a queue run).

The settings, and why each one

Two properties of dotnet-test.yml matter for the queue and were checked rather than assumed:

• Concurrency cannot cancel a queue build. The group is build-test-${{ github.ref }}, and a queue entry's ref is gh-readonly-queue/main/pr-<N>-<base sha> — unique per entry per base. Two runs share a group only when GitHub rebuilds the same entry on the same base, and then cancelling the older one is correct.
• Consolidate test results is present on every queue run. See above; a queue whose required context can be absent ejects entries with nothing to point at (CI_TIMEOUT), which is what the missing merge_group trigger looked like before #2799.

Enabling it

The rule is added to ruleset 2128472 (main pr protection) with the REST rulesets API. PUT replaces the whole ruleset, so the existing rules are read back and the queue rule appended:

gh api repos/Systemorph/MeshWeaver/rulesets/2128472 \
  --jq '{name, target, enforcement, conditions, bypass_actors,
         rules: (.rules + [{type: "merge_queue", parameters: {
           merge_method: "MERGE", grouping_strategy: "ALLGREEN",
           max_entries_to_build: 1, max_entries_to_merge: 3,
           min_entries_to_merge: 1, min_entries_to_merge_wait_minutes: 3,
           check_response_timeout_minutes: 45}}])}' > /tmp/ruleset-2128472.json
gh api -X PUT repos/Systemorph/MeshWeaver/rulesets/2128472 --input /tmp/ruleset-2128472.json

Read it back — mergeQueue(branch:"main") is null while disabled:

python3 .github/scripts/merge-queue-steward.py status --repo Systemorph/MeshWeaver

status prints the live configuration, warns on every parameter that drifts from the table above, and lists the entries currently queued.

The steward — the hand that re-queues, on evidence only

.github/workflows/merge-queue-steward.yml fires on pull_request with action: dequeued; the event carries a reason. The workflow is thin; .github/scripts/merge-queue-steward.py decides, mints nothing itself, and proves its own decision table with --self-test before every real decision. It uses two tokens, split by direction: every read (the failed run's jobs and artifacts, the PR head's check-runs, commits, comments) goes through the job's own GITHUB_TOKEN with actions: read + checks: read; every write (comment, label, enqueue) goes through the App installation token minted the way auto-arm.yml does — the org grants that App exactly contents: write + pull_requests: write, so it cannot read Actions, and a re-queue performed with GITHUB_TOKEN would merge as the bot and start no run on main (#2916). The queue-rejected label is a repository fixture (creating a label needs issues: write, which the App does not hold; applying an existing one needs only pull_requests: write).

How it reads the failure: the newest merge_group run of MeshWeaver Build and Test whose head branch is gh-readonly-queue/main/pr-<N>-…; its failed jobs; for each failed shard the testResults-shard<N> artifact — the same evidence Consolidate test results downloads — parsed for <UnitTestResult outcome="Failed"> with message and stack, and for non-TESTFAIL exit markers in test-results.log. The group's PR list is the first-parent chain of the queue commit back to a commit on main; "own run green" is Consolidate test results on the PR's head commit.

Every attempt is recorded in a hidden marker on the PR — <!-- steward: requeued=N head=<sha> kind=<kind> --> — and caps are counted from those markers per head sha: a new push is a new question. A rejection spends nothing.

🚨 The steward never re-runs a workflow. A re-run of the same tree hides the bug the failing run found (and the run's failing transcript is the control arm for the flake). A re-queue builds a new tree against the main that has moved — a different measurement.

The flake catalogue

.github/known-flakes.json is the only thing that turns a red assertion into a re-queue, so each entry is a temporary, evidence-bearing allowance:

{
  "id": "graph-late-nack-timeout",
  "assertionPattern": "TimeoutException : The operation has timed out\\.[\\s\\S]*LateNackReenqueueTest\\.cs:line 131",
  "testName": "MeshWeaver.Graph.Test.LateNackReenqueueTest.LateOwnerDisposingNack_AfterOptimisticEmit_ReenqueuesAndLands",
  "issue": "https://github.com/Systemorph/MeshWeaver/issues/NNNN",
  "evidence": ["https://github.com/Systemorph/MeshWeaver/actions/runs/33630685580"],
  "addedOn": "2026-09-02",
  "expires": "2026-10-02",
  "addedBy": "rbuergi"
}

• assertionPattern is a regex over the failure message and stack trace — never the test name. A catalogued flake must be evidence-bearing: the same test failing on a different assertion is a different defect and must reject. The loader refuses a pattern that matches the empty string.
• issue tracks the root cause; evidence lists the failing run(s), ideally beside a passing run of the same tree (the control arm).
• expires is at most 30 days after addedOn. An expired entry is inert — the steward treats it as uncatalogued — and it reds MergeQueueStewardGuard until it is deleted or renewed with fresh evidence. A catalogue is a ledger of tolerated defects; it may not rot silently.

The catalogue was seeded empty on 2026-09-02, and deliberately. The 700 Build-and-Test runs since the queue was first enabled hold 83 reds; the repeated signatures were MeshWeaver.Hosting.Monolith.Test.HOST_CRASHED (×5, exit=124 at the 8-minute cap) and CompileFinishAndDisposeTest (×4) — both in a project that has since moved to MeshWeaver.Plugins — and TestTimeoutLiteralRatchetGuard (×6 on 09-01), which was the queue working: two PRs each moved a ratchet count, green alone and red together. Every remaining candidate was a single occurrence, or its subject changed after the failing runs (ScopeTeardownRenderTest, #2877 on 08-31). A catalogue seeded on weaker evidence than that would be an automatic re-runner with a JSON file in front of it.

Guards

• MergeQueueStewardGuard (test/MeshWeaver.Documentation.Test): every catalogue entry carries an issue URL, run-URL evidence, a compiling assertion regex, and an unexpired date within 30 days of addedOn — with a negative control proving the validator refuses each defect; runs the Python self-test; and pins the workflow's shape (types: [dequeued], every job capped at 10 minutes, no workflow re-run anywhere).
• dotnet-test.yml's CI's own shell job runs the steward's --self-test on every pull request, so a classifier regression cannot merge on a branch that never dequeued anything.

🚨 Auto-merge waits on the BASE's protection — so a stacked PR is armed to merge NOW

--auto does not mean "merge when this pull request is green". It means "merge when the contexts branch protection requires of the BASE are satisfied" — and in this fleet protection is configured on the default branch and nowhere else. A stacked pull request, opened against another pull request's feature branch, therefore has a base that is protected by nothing, an EMPTY required set, and a condition that is met the instant GitHub first reads it.

Measured 2026-09-11. MeshWeaver.Plugins#1685 was opened against the feature branch of #1681. auto-arm.yml armed it and GitHub merged it at 21:03:23Z — 61 seconds after it was opened, before its own CI had started a single job. Nothing failed and nothing was bypassed; the stack collapsed unreviewed, exactly as configured. auto-arm.yml is the fleet's only copy of the arm lane and every satellite reaches it through workflow_call, so the hole was open in every repository at once.

🚨 But "the satellites get it for free" is true only where the caller pins @main, and one does not. Measured 2026-09-11 over the contents API (never a local clone — satellite checkouts here run days stale): all seven callers exist, and six pin auto-arm.yml@main — MeshWeaver.Plugins, .Reinsurance, .SocialMedia, .Education, .Crm, .Manufacturing — so a core fix lands there on merge. Systemorph/Memex pins a SHA (@c7fef7a2, 971 commits behind core's main when measured), so it picks up nothing until that line moves. The lesson generalises past this fix: before claiming a reusable-lane change reaches the fleet, read every caller's uses: ref — a fleet-wide fix and a six-of-seven fix are indistinguishable from core, and the odd one out is silent, not red.

The lane now reads github.event.repository.default_branch off the event — never a literal main, which would be a copy of a fact every repo happens to share today and the exact drift the single-copy design exists to end — compares it with github.event.pull_request.base.ref, and arms only on a match.

🚨 One self-clearing window, named so it is not misread as the fix failing. The lane runs on pull_request_target, and that event runs the workflow file as it exists on the pull request's BASE branch — not on main. So a stacked pull request opened onto a feature branch that was cut before this fix landed still runs the old, unguarded copy and is still armed immediately. Nothing in core can reach that: the branch carries its own copy by the event's definition. It clears as soon as the branch is cut from, or catches up with, a main that has the fix. If you see a stacked pull request merge instantly in the days after this lands, check the age of its base branch before concluding the guard is broken.

🚨 The decision is announced, and the JOB is never what skips. Moving the base test onto the job's if: is the tidy-looking version of this fix and deletes the message with it: a skipped job renders like a passed one and carries no warning, no summary and no comment, so "the lane declined" and "the lane is broken" become the same page. Instead the job runs, a ::warning:: plus a step summary name the base, and one idempotent comment says so on the pull request itself. Draft and fork stay on the job condition, because for those there is genuinely nothing to say.

The other lanes were checked and are clean structurally, not by luck — which is why none is exempted by name in the guard: arm-credential.yml mints and inspects a token and arms nothing; this steward acts only on a dequeued event, which cannot occur for a pull request never admitted to a queue, and a queue exists only on a branch configured to have one; release.yml and node-repo-platform-ref-bump.yml OPEN pull requests (--base main, explicitly) without arming them.

ArmedMergeMustTriggerMainsPushLanesGuard holds all three properties: the arming step is governed by an if: consuming a step output whose producer reads default_branch (a condition moved into env:, or one that consults only the token mint, breaks the chain and fails); no workflow in the directory arms without that chain; and the non-default path still warns and still comments.

Working with the queue

• gh pr merge <n> --auto enqueues — but only onto the default branch. With a queue enabled, "auto-merge" means enqueue when the PR's own required checks are green. auto-arm.yml does this for every non-draft PR whose base is the default branch; see the section above for why a stacked PR is deliberately left unarmed. Marking a PR draft is the opt-out.
• 🚨 DISARMING auto-merge does NOT hold a pull request. Converting it to DRAFT does. This is the single most likely way to believe you have stopped a merge and be wrong — see the section below.
• A push to a queued branch ejects it. GitHub removes the entry (reason MANUAL-shaped from the steward's point of view: it comments once and takes no action); auto-arm re-arms on the synchronize event, so the new head re-enters the queue once its own run is green. Do not push to a queued branch expecting the queue to pick up the new commit in place.
• Dequeue via GraphQL, never by re-ordering. gh api graphql -f query='mutation($id:ID!){dequeuePullRequest(input:{id:$id}){clientMutationId}}' -f id=<PR node id>. There is no "jump the queue" here: enqueuePullRequest's jump is not used by the steward and should not be used by hand — the queue's order is its correctness argument.
• Never re-queue by hand after an ejection. The steward does it, on evidence, and records the attempt. If it rejected — the PR carries queue-rejected and a comment naming the assertion and the run — the PR needs a fix or, with evidence and an issue, a catalogue entry. Remove the label when you re-arm.
• Reading the queue: gh pr view <n> --json mergeQueueEntry, or the status command above.

🚨 Disarming auto-merge is not a hold — DRAFT is the only durable one

Measured: MeshWeaver.Plugins#1683 was deliberately disarmed during a merge window and merged anyway. Nothing malfunctioned. auto-arm.yml fires on synchronize among other events, so the next push re-armed it — and a pull request being held is exactly a pull request someone is still pushing to. A push is what happened to re-arm that one; it is not the only thing that would have.

The asymmetry is the whole point, and it is structural rather than a bug to fix:

Read off the merged lane: types: [opened, reopened, ready_for_review, synchronize] and a job condition of github.event.pull_request.draft == false. A draft is never armed no matter how many times it is pushed to, and marking it ready arms it immediately — so draft is a two-sided hold the lane honours on both edges, while a disarm is a one-shot the next event overwrites. (GitHub also disables auto-merge when a pull request is converted to draft, so the conversion does both halves in one act. That last clause is GitHub's documented behaviour rather than something measured here.)

So: to hold a pull request, convert it to draft. Never rely on --disable-auto — and if you find a PR merged that you thought you had stopped, look for one of this lane's trigger events after the disarm before looking for anything else. A push is the first thing to check because it is the most frequent, but a reopen or a ready-for-review re-arms just as completely and leaves no new commit to notice, which makes those the harder two to spot afterwards.

#4057 does not change this. That fix stops the lane arming a pull request whose base is not the default branch; for an ordinary pull request onto main, re-arming after a disarm is unchanged.

The option considered and NOT taken

Making the lane refuse to re-arm a pull request a human explicitly disarmed is attractive and is not small, so it was left alone rather than half-built:

• Honour a hold label. Needs the label to exist in all eight repositories and a convention nobody has agreed to. It would at least fail toward today's behaviour — a missing or misspelled label arms, as now — and the refusal would be visible on the pull request, so it would not be a silent skip. But a fleet-wide convention is a maintainer's call, not a side effect of a CI fix.
• Read the timeline for auto_merge_disabled and decline to re-arm after it. This needs no convention and rests on real evidence rather than a marker, which makes it the better design — but it turns a read failure into a policy decision: fail open and the hold is not a hold, fail closed and one flaky API call silently stops arming the whole repository. That trade-off, and a paginated timeline read, is engineering with a decision in it, not a small patch.

Either way the rule that must survive is the one this document already states: a decision not to arm has to be SAID, on the pull request. A hold that works by something quietly not happening is the same defect as a gate that skips on a missing input — silence and "still running" look identical.

Related

Reading CI Signals · The Continuous Delivery Contract · The Cross-Repo Pair Gate · Writing Tests