Why pull requests here took hours to land, and what changed

"pr in plugins not draining" … "work on making this smooth" — maintainer, 2026-09-11

This page records one measurement, what it found, what was fixed, and what is left for the maintainer to decide. Every number below can be re-derived from the GitHub REST API and a clone. The method is at the end.

The denominator

• 50 pull requests. The 42 most recently merged (#1544 … #1663, merged 2026-09-08 22:10Z … 09-11 17:17Z, 6 of them Dependabot) plus the 8 open non-draft ones at 2026-09-11 18:39Z.
• 133 Plugin Catalog CI runs on those pull requests: 58 success, 21 failure, 33 cancelled (superseded by the author's next push, as designed), 14 startup_failure, 7 still running.
• Time to land. From ready (or created) to merged, over the 36 non-Dependabot merges: median 1.3 h, p75 3.7 h, max 20.7 h (#1607). Open: #1575 at 47.8 h.
• One green run (first attempt) takes a median 39 min of wall clock, p90 58 min. That is the floor. Everything above it is one of the causes below.

The causes, counted

(d) DIRTY churn — the dominant cause

GitHub's own merge (driver-free, see the method) called a pull request conflicting 40 times across 16 of the 50. 37 of the 40 conflicted on nothing but */manifest.lock. The locks that conflicted most: Hosting 16, Edu 12, Observability 7, Mcp/AI/Mail 6 each, Chat 5, and every AI-provider lock at once whenever src/MeshWeaver.AI moved (it rides into ten bundles).

Why every lock conflicts. A lock's trailer — moduleVersion, sourceCommit, version — changes on every content change to its module. Two pull requests that touch the same module, even in different files, therefore always rewrite the same three lines to different values, and a text merge cannot reconcile them. Since 2026-09-04 the most-touched file on main is Hosting/manifest.lock (143 commits), then Mail/manifest.lock (91). ci.yml is third, at 51.

Why the local merge driver did not help — the root cause, named. .gitattributes declares */manifest.lock merge=mw-manifest-lock, but a merge driver is configuration, not content: it exists only where scripts/setup-git-merge.py has installed it, per checkout. GitHub has no such driver, so its plain text merge stops on a lock that a correctly-configured clone merges cleanly.

That asymmetry is the whole mechanism, and it produces a contradiction that reads like a broken instrument: git merge-tree says CLEAN while the API says mergeable=false for the same two commits. Measured on the 40 real DIRTY states: with the driver armed locally, a local merge-tree reported no conflict at all in 37 of them. It is also why #1676 and #1670 "went DIRTY three times after being fully green" on 2026-09-11: every merge to main regenerates the locks, so any two pull requests touching one module collide pairwise, again and again.

🚨 And the second trap, which the machinery's own note warns about: a conflicting pull request can silently drop its armed auto-merge. Checking that the arm survived one cycle is not a guarantee that it survives the next, so a PR can sit green, unarmed and unnoticed. Read the arm state itself (auto_merge), never the fact that an arm ran.

Why it cost hours, not minutes. The DIRTY windows sum to 116 PR-hours, and 83 of those are windows longer than eight hours: an agent session opened the pull request, moved on, and nobody pushed the one merge that was needed. #1575 alone was DIRTY for 39.6 h across seven windows, every one of them lock-only.

The pull request that added the resolver hit this exact case while it was still in CI. #1678 went DIRTY on Hosting/manifest.lock alone the moment #1673 (a Hosting change) merged at 2026-09-11 ~19:3xZ. Its own classifier answered ['Hosting/manifest.lock'] -> generated-only. It was resolved by hand, because the lane was not on main yet: a merge, the post-merge regeneration, both --check copies, and a push. That restarted a ~40-minute check set over content nobody had changed.

The devtools/ PR (#1679) hit the same case about an hour later. When #1625 (the Edu learning roadmap) merged, #1679 went DIRTY on Edu/manifest.lock alone. The only thing #1679 changed under Edu/ was a one-line doc reference to the moved script. It was resolved the same way, by hand, and its CI restarted from the queue. Two of this change set's own PRs lost a full CI cycle each to cause (d) on the day they were written.

(a) and (a′) — the platform moved under every open pull request

A window runs from the seal of the first set that carried the regression (or the merge of the lane change) to the seal or merge that ended it. "PR-hours exposed" is window × open non-draft pull requests: each one that ran the affected shard in that window went red for a reason its diff could not reach, and a narrowed PR that did not run the shard was spared. Only 10 red runs are directly attributable, fewer than the exposure suggests, because the PR matrix is narrowed to the affected scope. The cost shows up as the wait: an author who sees the red either re-runs it or waits for core. Neither is something their diff can fix.

The lane class (a′) now has a platform-side gate. Core's check-workflow-permission-pairing.py (core 05577b3cd, "a caller must grant what its lane demands — the check that would have caught #3933") runs in node-repo-validate.yml against every caller.

(b), (c), (e) — read, not re-run

• (b) #1614 changed CI itself (validate red, no bundles assembled). #1628 added nullable warnings (CS8600/CS8602) to the Orleans test files it edited. #1666's new test expected the rendered title but got the bare key logIncident.refused.commentTitle. #1624 landed red. test-repos was not yet a required context, so auto-merge merged it at 18:25Z over a red Edu/Module case. That reddened #1622 and #1625 until #1627 at 18:59Z. #1644 made test-repos required the next morning, which closes this route.
• (c) Two races, each seen once and each green on the same platform set for other pull requests: NodeTypeOnDemandAdoptionTest.AForcedRelease_NeverConsultsTheBundleSources_AndCompilesTheLiveSource (#1625, set 8329, shard 3) and RetiredNodePruneTest.SharedSourceChange_FallsBackToFullInstall_AndPrunesTheRetiredNodeType (#1544, a Dependabot npm bump that cannot touch .NET, shard 1). They are real races and are named here for whoever takes them. Nobody re-ran them to "see".
• (e) Actions artifact service 403 Forbidden (#1664, set 8357, module bundles), artifact ETIMEDOUT (#1661), and one Resolve the released platform failure whose log is no longer readable (#1623). Per job, from created to started: median 22 s, p90 530 s, max 31 min (the module-test matrix on #1670). The p90 is the real capacity cost: nine minutes a job, and it stacks along needs: chains.

(g) and (h) — a run that rebuilt what it had already proven

Peer session meshweaver-plugins-a6 measured both on run 34618468550 (#1668).

(g) Over-selection. scripts/ is the gates' directory, so a change there counts as EVERYTHING, and correctly so for a gate. scripts/run-node-tests.py, though, is a local dev loop that no workflow references. A one-line edit to it selected 52 of 52 compiled projects and every module suite. The same diff without that file selected 1 of 52. NOOP_FILES is a closed list of repo-root names by design, so it offered no sanctioned way to say "this script reaches no lane".

The fix is a declared no-op directory, devtools/, in NOOP_DIRS on both sides. The dev tool now lives there and still borrows scripts/compile-check.py. Why these choices:

• Not tools/. This repo's tools/ is read by src/Directory.Build.targets, Memex.Portal.Gui.Test and MeshWeaver.Observability.Test (the Monaco bundle and the Memex template pack), so it is not inert.
• Core landed first (core #4050). No fleet repo had a devtools/ directory, so check-noop-scope-parity.py reported the new name as dormant everywhere. The reverse order is measured red: this repo calls devtools inert and the platform does not.

It also joins both package enumerators' SKIP and gen-manifests.config.json. Without that, gen-manifests.py minted a devtools/manifest.lock and treated the directory as a module.

Control on this repo's real tree, using the platform's node-repo-project-scope.py against .github/compiled-validation.json:

The same over-selection, one gate further on — in flight by another session. The required gate Compile every NodeType (vs core) already compiles each NodeType on its own, from its declared source set. But it compiles all of them on every run. Core's .github/scripts/compile-check.py has no subset option (only --refs, --image, --self-test, --gen-allow), and ci.yml hands it the whole checkout. So a pull request that reaches one module still compiles every NodeType in the repository.

The fix has the same shape as test-repos: on pull_request the gate takes the affected closure from scripts/affected-modules.py, and stays FULL on push, dispatch and schedule. An unknown module name is refused, never compiled as nothing. An empty closure still publishes the required context as a success that says "0 NodeTypes affected", and never as skipped.

This work belongs to session meshweaver-plugins-a6, which Roland asked to build it on 2026-09-11. It is in flight as #1681, "ci(compile-check): one leg per atomic unit". It builds on the devtools/ fix above (#1679), because the selector is only worth something once scripts/ stops forcing ALL. It is not part of this change.

(h) The ledger is off on PR runs. The modules-floor call passes no ledger: input, and the lane defaults to off. So every push to a PR branch re-runs every selected module suite from scratch, even when a Tested record for the identical key exists. The fix is ledger: required plus ledger-token. Two facts shape it:

• No ledger-token secret exists in this repo today. Minting one is a provisioning step: a mw_ ApiToken for the registry portal's CI user on Admin/ModuleBuilds.
• Dependabot and fork PRs must fail RED at preflight, never skip. That means either provision the token in both secret stores, or add a reasoned pr-secret-preflight-allow entry with the exemption expressed on the event.

This change sits inside the modules-floor call that #1676 rewrites, so it was handed to that PR's owner as a comment (#1676) instead of being built beside it.

What changed: resolve-generated-conflicts.yml

Status. This page is the record; the lane's own code lands in #1678. It was split out so that the measurements, the method and tonight's drain events are not held behind an investigation into a delete-then-recreate access-grant race that #1678's run hit and its diff cannot reach.

On every push to main (a merge landing is what makes other pull requests conflict), and every three hours as a safety net, the lane reads every open pull request with GitHub's own text merge. It emulates that merge by running git merge-tree with attr.tree=<empty tree>, so even a clone with the local driver armed sees the conflict GitHub sees. It skips drafts (the opt-out, as for auto-arm), forks, and Dependabot's branches. For each pull request whose conflict set is non-empty and entirely */manifest.lock, it:

1. merges main into the pull request's head in a throwaway worktree, with no hooks;
2. regenerates the locks with the repo's own self-tested gen-manifests.py --resolve, which refuses if any source path is unmerged and stages every lock the regeneration moved;
3. requires both gen-manifests.py --check copies to pass: this repo's, and the platform's canonical one that Validate node repos believes;
4. pushes a plain fast-forward with the meshweaver-cloud App token, so CI and auto-arm fire on the new head. If the author pushed meanwhile, the push is refused and the next run re-reads the branch. It never force-pushes;
5. writes one comment on the pull request, edited in place, saying what it did and that the author's local branch is now one merge behind.

A pull request with any source conflict is left untouched for its author. scripts/resolve-generated-conflicts.py --self-test runs in Repo policy gates on every PR and in the lane before it acts. It pins four things:

• GitHub's view sees a lock conflict that the armed local driver hides. This is the control: if it ever stops being true, the emulation is no longer what makes detection correct.
• A lock-only conflict resolves to a merge whose lock describes the merged tree.
• A source conflict is refused, even when resolve is called directly.
• An author's newer push reads as a race, and the Workflows refusal is named.

Verified on real history. Replayed over the 40 measured DIRTY states, the classifier saw all 40 (37 generated-only, 3 source). For comparison, the locally armed driver hid 37 of them. A full resolution of #1575's newest lock-only state (main@2026-09-11T17:17Z) produced a merge that passed both checkers.

The one gap: the App's Workflows grant

GitHub refuses a GitHub App push that brings .github/workflows/** changes onto a branch unless the App holds Workflows: write. The meshweaver-cloud installation holds only contents, metadata and pull_requests (gh api orgs/Systemorph/installations). So when main changed a workflow since the pull request's base, which was true for 13 of the 37 measured lock-only states, the push is refused. The lane reports needs-workflows-grant in its summary and in the PR comment, with the three local commands. The token mint deliberately does not ask for the grant: an installation that lacks it fails the mint outright, which would take down the other 24 cases too. See core's Doc/Architecture/PlatformRefBumpLane for the same refusal on another lane.

The governing rule: no job spans unrelated dependency networks

"we must not have any job going across the atomic unit", where the atomic unit is "all dependency patterns in repo". Roland, 2026-09-11, relayed by session meshweaver-plugins-a6.

The unit is therefore a dependency network: the changed units plus their transitive dependents. It is not a single package. A job may cover one whole network. It must not span two unrelated ones, because then a red in one network fails, or re-runs, the other. The compile gate is being reshaped to fit this: one job per affected network, with a receipt-count aggregate as the required context. That work is in flight by meshweaver-plugins-a6.

Read against this repo's lane, two job families span networks today. Both are recorded here as proposals only:

Neither of this change set's PRs restructures these jobs, and none adds a job that spans networks. The resolver lane is one job, but it builds and tests nothing and produces no verdict about any network. It acts on each pull request independently and reports each on its own line.

Timing baseline (2026-09-11)

"once we have atomic build jobs, let's measure timings." — Roland, 2026-09-11. "Atomic" means one job per dependency network. This is the before, taken while today's job shapes still run, with the exact instrument recorded below so that the after is the same measurement.

Window: Plugin Catalog CI runs created 2026-09-08T19:00Z → 2026-09-11T19:00Z (72 h). This is fixed, so the drain waves that began after 19:00Z are not in it.

Denominator: 597 runs were created in the window. 223 are kept: 199 pull_request runs and 24 pushes to main, each concluding success or failure. Excluded:

Cross-check. Session meshweaver-plugins-a6 measured one run independently: main run 34626892604, per job and per step. This instrument gives the same answer on it:

What the baseline says

• The portal hosts barely narrow. 166 of the 199 PR runs (83 %) ran all four shards, 33 ran one, and none ran two or three. The partition is min(4, moved suites), so any diff that reaches four or more moved suites takes all four shards, and in practice most do. The portal hosts account for 15,457 of 43,551 PR runner-minutes (35 %), and 14,999 of those were on four-shard runs, about 91 minutes each at the median. The peer's step split of shard 0 shows ~15 of its minutes are build: hosts, moved projects, surface manifests, the no-GUI build. The other three shards build too, invisibly: only shard 0 runs those steps, while shards 1–3 compile their slice's closure inside dotnet test (940–1150 s against shard 0's 748 s), so a shared project compiles up to four times per run. The restructure into per-network legs is claimed by session meshweaver-plugins-a6, stacked on #1681; this page measures it, it does not propose it.
• Module suites that run as their own job are the other 40 %: 17,393 PR runner-minutes. The median is 152 min per PR run that ran any, over n = 164. MeshWeaver.AI alone takes a median 14.6 min on each of its 113 runs. On PR runs these suites run with the build ledger off, which is cause (h). 🚨 And most of an AI leg is not tests. Profiled by another session and written up in Hosting/AiSuiteProfile.md: of 936 s per AI leg, 476 s re-compiles the platform from source — already compiled seven steps earlier, in 7 s, from the image — and 438 s is tests, of which only 40 % is test bodies. Enabling a mesh-sharing setting that 36 classes had already declared, and that was inert for want of one assembly attribute, took 672 meshes down to 363 with identical coverage: about 70 s off every AI leg. That 476 s re-compile is the largest single remaining item in this repository's CI bill.
• Module-bundle narrowing works. A PR run builds a median of 7 bundles (p90 37). Suites follow at a median of 5. It is the compiled half, the portal hosts, that barely narrows.
• Per run: a PR run costs a median 159 runner-minutes (p90 359) and 38 min of wall clock (p90 54). A push to main costs a median 350 runner-minutes. On a push most module suites run inside their bundle legs, so read module-bundles and module-tests together.
• Per pull request (118 PRs, every kept PR run mapped): a median of 291 runner-minutes to land, p90 948, max 1,427 (#1575, 4 runs). 43 of the 59 PRs above the median (73 %) had a newest run that reached all 37 bundles and all 4 shards.
• Queue wait at run level is small: median 0.2 min, p90 1.7 min from the run start to its first job. That is not the job-level queueing measured earlier, where p90 was ~9 minutes per job and the module-test matrix legs waited up to 31 min. The first job starts quickly; the wide matrix behind it waits for runners.

The numbers

(38 module suites in all; the 12 largest by total runner-minutes shown.)

Per pull request (118 PRs; 199 of the 199 kept PR runs mapped to a PR by head branch; 0 run(s) unmapped): runner-minutes per PR median 291, p90 948, max 1427.1.

(The 20 costliest of 118 shown.)

Drain event during the drafting of this page

Drain event: the undraft wave of 2026-09-11 evening. At Roland's request ("undraft a few", "let's drain"), session meshweaver-plugins-a6 undrafted and auto-armed Plugins drafts whose stated hold reason no longer applied. It worked in batches of three, grouped so that no two PRs in a batch touch the same manifest.lock, and spaced the waves while #4051 (the bake race fix) and module-load tolerance were undeployed. #1679, #1678, #1575 and #1536 were left alone. The wave raises runner load, and DIRTY churn is likely. The baseline window closes at 2026-09-11T19:00Z, before the wave began, so no baseline run is affected.

• Wave 1, before the batching rules applied: #1634, #1653 and #1650 were undrafted, all green on their heads with no stated hold.

  • #1653 and #1650 went DIRTY on lock conflicts right after being armed. Cause (d) reproduced live: a draft that sat while main moved came back conflicting on nothing but generated locks.
  • #1634's arm did not take, because its mergeable state was still unknown.

• Later waves run under lock-disjoint batching, spaced out with /health checks, after verifying from the content that each hold reason no longer applies. A DIRTY PR is brought current with git merge origin/main plus gen-manifests.py --resolve. Results, as reported by the drain agent after 19:00Z:

  • Merged: #1637 (19:52Z, lock-free). #1634 was already merged.
  • Brought current and armed: #1587 and #1653. For each, main was merged in, the locks regenerated, gen-manifests --check and validate-repos passed, and the result was pushed (aa8c5e65 / ec46d96d). #1650's conflict was only a difference in JSON key order; main's side was taken and pushed as 58299fd6.
  • Undrafted and armed, with each hold reason checked against the content: #1638, #1645 and #1541. #1541 was waiting for "portals must run an image carrying MeshWeaver.Deployment.Contract"; both portals run core 74d4c85, so that is satisfied.
  • 🔁 The Hosting/manifest.lock cascade. #1653, #1650, #1638, #1645 and #1541 all collide on Hosting/manifest.lock and nothing else. Each one that lands makes the next one DIRTY. This is cause (d) as a chain, and exactly the case the resolver lane handles on each merge without anyone pushing by hand.
  • Held, each for a reason inside its own diff:
    • #1636: real content conflicts in InstanceActionPlan.cs / RepairPlan.cs; it needs its author.
    • #1635: production acceptance is pending, it has real conflicts, and it duplicates #1587's fix.
    • #1654: "draft on purpose until a human has looked". CI never fired on cfb8eed6, with no skip token and no workflow change.
    • #1646: red with timeout-class failures. Its own resolve-platform.py change resolved platform 8340 where main resolved 8369, so the red is within its diff's reach.
  • Both portals answered 200 after the wave.
  • Flagged for follow-up: four Plugins main push runs since 17:13Z ended cancelled (9ffa1677, bd212d0c, 5c2a838c, 6d82cf22). Whether the superseding runs completed and published is being checked. This is #1676's area (per-module deploy, nothing on main cancelled).

• After #1678 lands, the resolver lane acts on the same lock-only DIRTY PRs on every merge to main. Two writers can then reach one branch. Each push is a plain fast-forward, so the second is refused: the lane reports it as raced, and a human sees rejected (fetch first) and pulls. Nothing is overwritten. An after-measurement whose window overlaps the wave, or any later mass undraft, must mark those runs. It should report queue wait with and without them, because a wave's queue wait is not the steady state.

• Wave 2, after main went green at c18b02fe:

  • #1587 re-merged with main and pushed b673edd7 (Store lock); #1653 re-merged and pushed edb41775 (Hosting lock, the head of the cascade). Both armed, both green on the gates: locks current, 271 nodes across 63 repos valid.
  • 🚨 #1650 was CLOSED as superseded, not merged. #1625 (d9221e68) already carries the same work — its history merges d07ee196, and the GitSync test files are byte-identical to main. This is a second instance of the duplicate-fix cause recorded under (b): a stale draft duplicating work that landed separately. Both instances were found by a pre-check against the content, not by a red run. A duplicate that is merged costs a revert; a duplicate that is detected costs one git diff.
  • Re-checked against c18b02fe and not duplicated, so they proceed: #1638 (main still has @key="@routeData"), #1645 (the readiness tests are absent on main), #1541 (main's Hosting/Deployment.json does not bind MeshWeaver.Deployment.DeploymentContent).
  • Order from here: #1653 → #1638 → #1645 → #1541, one lock cycle at a time, then a full main run and the Hosting/v1.17* tag before wave 3. Portals 200/200 throughout.

• The hold of 21:4xZ, and what it shows about the coupling. Session meshweaver-plugins-85 asked for a clear window for #1676 and #1670, which had gone DIRTY three times that evening after being fully green. #1678 and #1679 were disarmed (gh pr merge --disable-auto) rather than allowed to land inside it, and re-armed afterwards; their CI kept running. #1676 is what stops main runs cancelling each other — nothing had published since 17:17Z — so it is worth the wait for every PR here. The arithmetic is cause (d) in one line: each lock-only re-merge costs a full ~40-minute check set, and three of them is a working evening. That is exactly the cost the resolver lane removes, and the reason it is worth landing before the queue is drained rather than after.

• The single-workspace-build chokepoint, measured 22:14Z (session meshweaver-plugins-85). The clearest statement of what the per-network work removes:

  • #1676: 56 checks green, one queued — Module bundles / Build the workspace (one global build), queued since 21:57Z and never started.
  • #1670: 50 green, the same job queued since 22:02Z. The pack legs and the required All selected bundles built get their check-runs only after that single workspace job completes, so both pull requests owed an entire module fan-out behind one job that had not started in 17 minutes. #1676 pays the full fan-out, because its diff touches .github/. This is the cross-network job of the rule above, seen from the queue rather than from runner-minutes: one job that has not started is indistinguishable, on the wall of ticks, from a suite that has nothing left to do.

Method: re-run exactly this for the after-measurement

The window is fixed: runs created in 2026-09-08T19:00:00Z..2026-09-11T19:00:00Z (72 h). An after-measurement uses a window of the same length, ending at a stated timestamp. REST only, at most one call per second, no GraphQL.

W='2026-09-08T19:00:00Z..2026-09-11T19:00:00Z'      # the after-run: same length, new end
mkdir -p baseline/jobs
# 1. every Plugin Catalog CI run created in the window (workflow id 308964643 = ci.yml)
for page in $(seq 1 20); do
  gh api "repos/Systemorph/MeshWeaver.Plugins/actions/workflows/308964643/runs?per_page=100&page=$page&created=$W" \
    > "baseline/runs-$page.json"; sleep 1
  [ "$(jq '.workflow_runs | length' "baseline/runs-$page.json")" -lt 100 ] && break
done
jq -s '[.[].workflow_runs[]]' baseline/runs-*.json > baseline/runs.json
# 2. KEEP: pull_request, and push to main; conclusion success|failure. Everything else is counted
#    as excluded (cancelled, skipped, startup_failure, still running, release-lane events).
jq -r '.[] | select((.event=="pull_request" or (.event=="push" and .head_branch=="main"))
                    and (.conclusion=="success" or .conclusion=="failure")) | .id' baseline/runs.json > baseline/keep.txt
# 3. the LATEST attempt's jobs of every kept run (page 2 for runs over 100 jobs)
while read -r id; do
  gh api "repos/Systemorph/MeshWeaver.Plugins/actions/runs/$id/jobs?filter=latest&per_page=100" > baseline/j1.json; sleep 1
  if [ "$(jq .total_count baseline/j1.json)" -gt 100 ]; then
    gh api "repos/Systemorph/MeshWeaver.Plugins/actions/runs/$id/jobs?filter=latest&per_page=100&page=2" > baseline/j2.json; sleep 1
  else echo '{"jobs":[]}' > baseline/j2.json; fi
  jq -n --slurpfile r <(jq --argjson id "$id" '.[] | select(.id==$id)' baseline/runs.json) \
        --slurpfile a baseline/j1.json --slurpfile b baseline/j2.json \
        '{run: $r[0], jobs: ($a[0].jobs + $b[0].jobs)}' > "baseline/jobs/$id.json"
done < baseline/keep.txt
# 4. the PR list, to map a run to its PR by HEAD BRANCH: a run's pull_requests array is EMPTY once
#    its PR has merged or closed (measured: 161 of 199 kept PR runs carried no number)
for page in 1 2 3; do
  gh api "repos/Systemorph/MeshWeaver.Plugins/pulls?state=all&sort=created&direction=desc&per_page=100&page=$page" \
    > "baseline/pulls-$page.json"; sleep 1
done
jq -s '[.[][] | {number, ref: .head.ref, created_at, closed_at}]' baseline/pulls-*.json > baseline/pulls.json
#    (widen the page range until prs.jq reports unmapped_runs: 0)
# 5. aggregate: one row per run, then medians / p90 per event, then per PR
jq -s -f rows.jq baseline/jobs/*.json > baseline/rows.json
jq -f stats.jq baseline/rows.json
jq --slurpfile pulls baseline/pulls.json -f prs.jq baseline/rows.json

When GitHub answers with a server error. During the baseline pull (2026-09-11, about 19:58–20:09Z), three job-list calls failed, each on a different, older run and each after real progress: an HTTP 504, an HTTP/2 stream CANCEL and an HTTP 502. The policy applied, which the re-measurement must copy:

• Retry only server and transport errors (HTTP 502/503/504, stream error, "couldn't respond to your request in time"): at most 3 attempts, with a logged backoff of 10, 30 and 60 s. Any other error aborts the pull.
• A run whose jobs still cannot be fetched after that is excluded by name, with the error text, and counted in the exclusions below. It is never silently dropped, and its numbers are never estimated.
• After a refusal, stop for at least 5 minutes before resuming. Keep the combined rate, including any PR watchers running at the same time, under one call per second.
• /rate_limit reading 5000/5000 does not rule out the secondary limit, so it is not a go-ahead.

In the loop above, a failed gh api call is the point to apply this policy.

Definitions, as the jq below computes them:

• Queue wait: from run_started_at to the earliest job started_at. On a first attempt, run_started_at equals created_at.
• Wall clock: from run_started_at to the latest job completed_at.
• Runner-minutes: the sum over jobs of completed_at − started_at, excluding skipped jobs. These are raw minutes, not billed minutes, which GitHub rounds up per job.
• p90: nearest rank, lower: the sorted value at index floor((n−1)·0.9).
• 🚨 A required context can read ABSENT because the listing was TRUNCATED. commits/<sha>/check-runs pages at 100, and a run here produces more than that. Measured 2026-09-11 while merging #1679: the first read showed two required contexts missing, and paginating showed all seven literally success. An absent required context and a truncated page look identical, and they mean opposite things — one blocks forever under classic protection, the other is a green wall. Paginate to exhaustion before concluding that a context never reported, in this measurement and in any readiness check.
• Units reached: the selector's output as it shows up in jobs. That is Module bundle (X) legs (bundles built), separate Module tests (X) jobs (suites run as their own job), and Portal hosts (shard N) legs.
• 🚨 A module's tests can run INSIDE its bundle leg. The lane records the owner per module: none, inline or lane. On push runs most suites run inline, so those minutes count under module-bundles, not module-tests. For example, on main run 34626892604 the MeshWeaver.AI bundle leg spends 936 of 1,001 s in "Run the module's tests". Compare like with like, and read the two families together.

rows.jq, with the job-name to family mapping as its first definition:

# One row per kept run. Input: `jq -s` over baseline/jobs/*.json, each {run, jobs}.
def family:
  if   test("^Portal hosts \\(shard")              then "portal-hosts"
  elif test("Build the workspace")                 then "workspace-build"
  elif . == "Compile every NodeType (vs core)"     then "compile-check"
  elif test("Module tests \\(")                    then "module-tests"
  elif test("Module bundle \\(")                   then "module-bundles"
  elif test("^test-repos / ")                      then "test-repos"
  elif test("Validate node repos$")                then "validate"
  elif test("^publish-bake / ")                    then "publish-bake"
  else "other" end;
def ts: sub("\\.[0-9]+Z$"; "Z") | fromdateiso8601;
def mins(a; b): ((b | ts) - (a | ts)) / 60;
[ .[] | . as $f
  | ($f.jobs | map(select(.started_at != null and .completed_at != null and .conclusion != "skipped"))) as $j
  | select(($j | length) > 0)
  | { id: $f.run.id, event: $f.run.event, conclusion: $f.run.conclusion, attempt: $f.run.run_attempt,
      created_at: $f.run.created_at,
      pr: ($f.run.pull_requests[0].number // null), branch: $f.run.head_branch,
      queue_wait_min: mins($f.run.run_started_at; ($j | map(.started_at) | min)),
      wall_min:       mins($f.run.run_started_at; ($j | map(.completed_at) | max)),
      runner_min:     ($j | map(mins(.started_at; .completed_at)) | add),
      family: ($j | group_by(.name | family)
                  | map({key: (.[0].name | family), value: (map(mins(.started_at; .completed_at)) | add)})
                  | from_entries),
      modules: ($j | map(select(.name | test("Module tests \\(")))
                   | map({key: (.name | capture("Module tests \\((?<m>[^)]+)\\)").m),
                          value: mins(.started_at; .completed_at)}) | from_entries),
      units: { bundles: ($j | map(select(.name | test("Module bundle \\("))) | length),
               suites:  ($j | map(select(.name | test("Module tests \\("))) | length),
               shards:  ($j | map(select(.name | test("^Portal hosts \\(shard"))) | length) } } ]

stats.jq:

# Medians and p90 (nearest-rank, lower: index floor((n-1)·p)) per event, from baseline_rows.jq's output.
def pct(p): sort | if length == 0 then null else .[((length - 1) * p | floor)] end;
def r1: if . == null then null else (. * 10 | round / 10) end;
def stat: map(select(. != null)) | {n: length, median: (pct(0.5) | r1), p90: (pct(0.9) | r1), total: (add // 0 | r1)};
["portal-hosts","workspace-build","compile-check","module-tests","module-bundles","test-repos","validate","publish-bake","other"] as $fams
| group_by(.event) | map(. as $rs | {
    event: $rs[0].event, runs: ($rs | length),
    queue_wait_min: ($rs | map(.queue_wait_min) | stat),
    wall_min:       ($rs | map(.wall_min) | stat),
    runner_min:     ($rs | map(.runner_min) | stat),
    families: ($fams | map(. as $f | {key: $f, value: ($rs | map(.family[$f]) | stat)}) | from_entries),
    module_tests: ($rs | map(.modules | to_entries[]) | group_by(.key)
                  | map({key: .[0].key, value: (map(.value) | stat)}) | from_entries),
    units: { bundles: ($rs | map(.units.bundles) | stat), suites: ($rs | map(.units.suites) | stat),
             shards: ($rs | map(.units.shards) | stat) } })

prs.jq:

# Per pull request: kept runs, total runner-minutes, and the units its NEWEST kept run reached
# (the selector's output materialised as jobs: module bundles built, module suites run, portal-host shards).
#
# 🚨 A run's `pull_requests` array is EMPTY once its PR has merged or closed, so it cannot identify
# the PR. Map by HEAD BRANCH instead, through the PR list (`--slurpfile pulls baseline/pulls.json`,
# built from `pulls?state=all`), choosing the PR on that branch that was open when the run was
# created. A run no PR accounts for is counted and printed, never dropped silently.
def r1: (. * 10 | round / 10);
def ts: sub("\\.[0-9]+Z$"; "Z") | fromdateiso8601;
($pulls[0]) as $P
| map(select(.event == "pull_request"))
| map(. as $r
      | ($P | map(select(.ref == $r.branch
                          and (.created_at | ts) <= ($r.created_at | ts)
                          and (.closed_at == null or (.closed_at | ts) >= ($r.created_at | ts))))
            | sort_by(.created_at) | last | .number // null) as $n
      | . + {pr: ($r.pr // $n)})
| (map(select(.pr == null)) | length) as $unmapped
| group_by(.pr)
| map(select(.[0].pr != null)
      | sort_by(.id)
      | {pr: .[0].pr, runs: length, runner_min: (map(.runner_min) | add | r1),
         last_units: (.[-1].units), last_conclusion: .[-1].conclusion})
| sort_by(-.runner_min)
| {unmapped_runs: $unmapped, prs: .}

When to measure the after, and how to mark what distorts it

Trigger — all of it on main, not part of it. Re-measure once both of these are on Plugins main:

1. per-module deploy (#1676) — landed 02:37Z, 2026-09-12. It changes what a push run does and removes the cancellation, so the push column of the before table describes a pipeline that no longer exists;
2. the per-network Portal hosts legs and compile gate (#1681, session meshweaver-plugins-a6) — still red in CI at the time of writing.

Then run the identical method above, from the same REST queries through the same three jq programs, over a window of the same length (72 h) ending at a stated timestamp, and add a before/after table with the same rows. Measuring in between describes a half-changed pipeline: state which parts had landed, because the portal hosts are 35 % of PR runner-minutes, the compile gate is ~2 %, and #1676 moves the push numbers rather than the pull_request ones.

Mark, don't average, anything that is not steady state.

• The drain waves of 2026-09-11 evening (after 19:00Z, see Drain event above).
• Any later mass undraft.
• A core-lane window like 09-10 18:57–23:47Z. Its runs are startup_failure and are excluded anyway, but the runs just after it queue behind the backlog.

Report queue wait and wall clock with and without the marked runs. A wave's queue wait is not the steady state, and a before/after that mixes the two measures the wave, not the job shape.

Proposed, for the maintainer to decide

Proposal 1: which platform set a pull request tests against

Today every run resolves the newest sealed core set (scripts/resolve-platform.py, see PlatformResolution.md). MW_PLATFORM_REF is an incident freeze, never a pin. So a sealed set that regresses a Plugins suite reds every open pull request that runs the shard, at once, until core ships the next set. Measured above: four such windows in 24 hours, 8.6 h in total, 91 PR-hours exposed, and nothing any author could do inside their own diff.

The options, each with the cost it moves:

Recommendation: B. It keeps "never pinned, never a literal". The resolved set is still computed per run, only from a different record. It turns "every PR red" into "main red", and a red main is already watched and owned. The lane class (a′) is not addressed by any option, because uses: cannot take an expression. That class is now gated in core by check-workflow-permission-pairing.py.

✅ DECIDED and IMPLEMENTED — option B (Roland, 2026-09-12)

A pull request resolves the newest sealed set that this repository's own main has already passed. main, the release dispatch and the daily poll are unchanged: they take the newest sealed set, which is what makes main the place a regression appears.

The failure mode is the point, not a side effect. When core seals a set that regresses this repo, main goes red on it and every open pull request keeps building on the last set main passed. One owner, one place. What it replaces is measured above: four such windows in 24 hours, 91 PR-hours exposed, and nothing any author could do inside their own diff.

How it works, in scripts/resolve-platform.py:

• --passed-on-main <owner/repo> is passed by ci.yml on pull_request and merge_group only.
• The ceiling is each run's own answer, not a second bookkeeping mechanism: every run already publishes ::notice title=Platform for this run::<set> — core <sha9>, and an annotation survives with its check run. The resolver reads that back from the newest successful push runs of ci.yml on main (12 by default, deep enough to survive a red patch) and takes the highest core CD run number among them.
• A sealed set newer than that ceiling is passed over, with the reason printed per set.
• 🚨 The lag is said out loud, with both set ids. When the chosen set is not the newest sealed one, the run prints ::notice title=Platform lag (pull requests follow main)::this run resolved <chosen>, not the newer sealed <newest>: pull requests follow main, and main has not passed on it yet, and the same sentence lands in the job summary and in the step's lag output. "Why did my core fix not show up in my PR?" is answerable from the log alone: it shows up once main's own run goes green on the set carrying it.
• 🚨 "main has passed nothing" is a REFUSAL, never a fallback. If no ceiling can be established from those runs, the job is RED naming why and pointing at MW_PLATFORM_REF. Falling back to the newest sealed set would put every pull request back on an unvouched set, which is the behaviour this rule exists to remove.
• A freeze still wins. MW_PLATFORM_REF is an instruction, so it overrides the ceiling.

What it costs. A core fix is not available to pull requests until main has passed on the set carrying it — normally about one main run. A Plugins half that needs a brand-new core API therefore waits for the seal plus one main run. A pull request merged on set S−1 is re-proven by main on S, and same-MAJOR compatibility already holds (PlatformResolution.md).

The resolver's self-test carries the rule in both directions, including the two cases that must go RED: a sealed set main has not passed is passed over and the lag names both sets; the newest set is taken with no lag once main has passed it; a freeze overrides the ceiling; and an unestablishable ceiling refuses. Disabling the ceiling check turns the first case red, which is what makes it a test rather than a decoration.

Proposal 2: grant Workflows: write to the meshweaver-cloud App

This closes the 13-of-37 gap above with no code change. It is an org-admin action on the App's settings, and the new permission must be accepted on the installation. The trade-off: any lane that mints this App's token can then edit workflow files. Today that means auto-arm and this resolver, both of which act only on same-repo branches.

Proposal 3: fix it upstream of the lane — the lock itself

🚨 The resolver lane is a STOPGAP, and now a stopgap with a named root cause (see the root cause, named, above): the repository declares a merge driver that only exists where someone installed it, and GitHub — the one place mergeability is decided — never has it. The lane removes the wait on main. It does not remove the conflict. Two durable fixes sit upstream of it, and neither was built tonight.

Option A — make the generated content merge cleanly under a plain text merge. Then GitHub's own merge does what a configured clone does, and the driver becomes unnecessary rather than load-bearing. It means: deterministic ordering, one fact per line, and no field that every change rewrites. Today three trailer lines do exactly that — moduleVersion, sourceCommit and version — so any two pull requests touching one module collide on them whatever else they touched. sourceCommit is informational; moduleVersion is a pure function of the files map beside it. A lock carrying only files plus version merges line-wise whenever two PRs touch different files. Cost: a cross-repo schema change — core's canonical .github/scripts/gen-manifests.py is the verdict Validate node repos believes, scripts/tag-modules.py and the trunk-baseline derivation read moduleVersion, and a runtime consumer reads sourceCommit (src/MeshWeaver.PluginCatalog/ModuleManifest.cs). Core first, then every satellite.

Option B — stop committing what every merge regenerates. The lock is derived from the tree, and gen-manifests.py can recompute it at any time; a file that is a pure function of its inputs does not have to be a tracked file. Derive it in CI and at pack time, and the conflict class disappears with the committed content. Cost: the committed lock is currently the second witness the version derivation rests on (the trunk's committed lock beside the published tag — the thing that stops two branches computing the same released number for different trees), and installers diff it cheaply. Removing it needs that witness rebuilt somewhere else first, which is a larger design change than Option A and touches the release path rather than only the schema.

Both are maintainer calls. Until one lands, the lane is what stands between the fleet and the churn measured on this page.

Proposal 4: a per-project unit key and Tested record for compiled suites

The module ledger lets a re-push skip a bundle whose inputs did not move. The compiled test projects and the portal-host shards have no equivalent, so the ~89 portal-host minutes in (g) would recur on every re-push that reaches them. The design question is the key: what a compiled test project's verdict is a function of, meaning its project closure, the resolved platform set, the tester image and the slice. It sits in the same place as #1676's per-module publication, so it belongs with that owner. Nothing is built here.

Left alone, on purpose

Session meshweaver-plugins-85 rewrote the per-module publish path in #1676: modules-floor, publish-modules, publish-newest-only and Hosting/BuildReleaseSeparation.md. Nothing here touches them. The lane's own change adds one self-test step to Repo policy gates, far from #1676's hunks.

✅ The publish starvation — measured here, fixed by #1676 (merged 02:37Z, 2026-09-12)

The measurement stays because it is the evidence for why the change was needed: 78 main push runs were cancelled between 09-08 and 09-11, and in the window this page measures nothing published between 17:17Z and 23:00Z, because every main run was cancelled by the next merge before it could publish. The clearest casualty was the WhatsApp module (#1664): merged, and then served by nobody.

From #1676's commit on main:

• No push run on main is cancelled any more. The supersede lane call is gone, and check-main-runs-not-cancelled.py refuses any attempt to re-add it.
• A push builds only the dependency network changed since the last SUCCESSFUL main run, walking past failed, cancelled and in-flight ones.
• Each module leg publishes after its own build and suite, so one red module or one red shard no longer holds back the rest.
• A leg stands down when a newer main commit reaches its module, and the bake refuses to seal an older commit over a newer one.
• 🚨 Main runs therefore OVERLAP by design. Two in-flight runs on main are the intended shape, not a fault to diagnose. Under the old lane, overlap was impossible because the newer run killed the older one — which is precisely what caused the starvation above.

This is the same rule as the rest of this page, applied to publication rather than to gating: a unit is a dependency network, and one network's failure must not decide another's.

Method

• REST only, at most one call per second: pulls?state=closed|open, issues/{n}/events, actions/workflows/308964643/runs (created ≥ 2026-09-08), actions/runs/{id}/jobs, check-runs/{id}/annotations (the resolved set is the Platform for this run notice that is not the resolver's self-test fixture), and actions/jobs/{id}/logs for every failed leaf job.
• DIRTY states were reproduced, not guessed. The analysis used a fresh bare clone with no merge driver configured, which is GitHub's view. Along each pull request's lifetime, at every first-parent commit of main, the PR head was taken as the newest own commit no younger than that main commit, and git merge-tree --write-tree --name-only gave the conflict set. A window lasts until the author's next push or the merge. Windows are unioned per PR so that overlapping states are not double-counted.
• The workflow-touch count is git diff --name-only $(git merge-base main head) main -- .github/workflows at each lock-only state.