Agent admission — one rule for every lane

The rule (maintainer, 2026-10-03/04): "must generally use up the queue" · "if we have empty queue ⇒ utilize" · "we can configure relative load but not absolute number" · "it applies only when we have exhausted pool" · "we don't exhaust in one species and then block" · "slowly floods rather than sudden flooding … waits for seeds to grow and spread" · "if a lot is free ⇒ should add with high likelihood" · "we must be very tolerant and try to always find an alternative. however, must also provide possibility to hard pin model."

Every agent round in a process asks ONE admission before it starts: the dispatch pool (ThreadDispatchPool, src/MeshWeaver.AI/Supervision/). Its state is AdmissionState (immutable, pure transitions) and its rules are LaneScheduler (pure). The lanes that start work — the bug pool, the triage issue sweep, the review steward — pace themselves by the same admission's Allowance(lane), so none of them keeps an absolute cap of its own. There is no number of concurrent rounds anywhere in configuration: what is configured is relative (floors, weights) or a declaration (alternatives, pins).

The layers

1. Bulkheads — one per model pool

A bulkhead is the measured state of one model pool (the model a round runs on). Exhaustion is measured per bulkhead, so an exhausted Opus pool never holds back work on GLM or Kimi. A bulkhead carries its estimate K̂ (the concurrency it serves now), the carrying capacity K (the load at which it last said "too much"), when and why, Thompson's success/failure counts, and the mean round time and cost.

What says "too much" — reported by the engine where it happens:

A clean round reports ReportClean; every round's price reports ReportCost.

2. Seed-and-grow — the estimate of one pool

3. The probability of a marginal admission rises with the free fraction

A round beyond its lane's floor is admitted with probability p(φ) where φ = 1 − running/K̂: certain while at least 30 % is free (HighFree), falling smoothly to 0 as the pool fills, and 0 at or above the estimate. A light lane competes less eagerly for the last slots (the probability is raised to heaviest weight / its weight, between 1 and 4). Upward probing happens by growing the estimate — never by letting each waiting round gamble above it. Refused rounds re-ask on every release and on the pool's 15-second clock, so a refused draw is retried within seconds.

4. Lanes — guaranteed floors, borrowing, weighted fair share

A thread's lane is its Threads-app group (Lanes.Of, AI/ThreadGroups); LaneOfAgent on Admin/Threads overrides it per agent. Floors, weights, alternatives and pins are all live configuration on Admin/Threads (laneFloors, laneWeights, modelAlternatives, pins, laneOfAgent); the old maxConcurrentAgents is retired and only warned about.

In one pool, in this order (LaneScheduler.Decide):

1. Within its floor — a lane with fewer running than its guaranteed slots (its floor of K̂, allocated by largest remainder among the lanes that HAVE work; a lane with waiting work and a non-zero floor gets at least one) is admitted while a slot is free. On a pool never measured exhausted it is admitted even when the pool is full: the estimate is then a conservative start, not a measured limit.
2. Reserved — a borrower is refused while the free slots are owed to other lanes below their floors that have work waiting.
3. Cost guard — a lane whose spend today ON THIS BULKHEAD'S PROVIDER reached its floor share of that provider's daily budget borrows nothing more there (it keeps its floor). Spend is kept per lane per provider, so a lane's spend on one provider never stops it borrowing on another; the budget is the limit the platform's budget gate last read for the provider today (AdmissionState.Budget) — with no reading today the guard does not apply.
4. Fair share — the borrowed part goes to the waiting lane with the least borrowed per weight (deficit-round-robin in effect; ties to the higher-priority lane).
5. Probability — §3.

Idle lanes lend everything: a single flooding lane fills the whole pool.

5. Reclaim at round boundaries

Every round asks again (a slot is released when the thread leaves its executing states), so every round is a boundary. A borrower is refused at its next boundary when a lane below its floor waits — nothing is killed, and the lane short of its floor starts within one round (LaneScheduler.MustYield names the borrowers that will yield).

6. Aging inside a lane

Inside a lane the oldest waiting round goes first: a newcomer does not take a free slot while older rounds of its lane wait for it. Where a lane chooses WHICH item to start (the bug pool), urgency is (1 + severity) × (1 + age / AgingUnit) (LaneScheduler.Urgency); the bug pool weights a red main 32, sev:B 16, H 8, M 4, L 2, unlabelled 1, with an aging unit of a day — so an eight-day-old low bug comes before a fresh blocking one, and nothing waits for ever.

7. Starvation signal

A lane below its floor with work waiting longer than StarvedAfter (15 min) — and every hard-pinned round waiting that long — is listed under starving on Admin/Threads. The stuck watchdog reads it as data.

8. Fallback to an alternative, and the hard pin

A round its own pool refuses moves to a declared equivalent model (modelAlternatives: same tier, route rules respected — declared, never guessed) that admits it now. Among those, Thompson sampling picks: each alternative's success rate is drawn from Beta(successes + 1, failures + 1), discounted by its mean cost; an unexplored alternative draws from Beta(1, 1), so the choice keeps exploring. The move is written onto the round's drained messages in the claim write, so the round runs on the alternative.

A hard pin (pins: a lane or an agent) disables the fallback: the round waits for its model, the wait says hard-pinned to … on the pool's census row, and past StarvedAfter it is a pinned starvation signal.

8b. Fallback is ON by default — default alternatives per tier

"Very tolerant — always find an alternative", a hard pin the exception (maintainer, 2026-10-04). With nothing configured, a model in one of the default tiers (LanePolicy.Equivalences) falls back to the other models of its tier that the deployment's catalog actually has (every LanguageModel under Provider/, read live by the supervisor — ThreadDispatchPool.SetCatalog):

Order (LaneScheduler.DefaultAlternatives): the same model on a DIFFERENT provider first (a direct Anthropic or Azure Foundry model beside an OpenRouter route — a key's daily cap is per key, so another provider is what keeps the lane alive), then the tier's other models in token order, own provider before others. Never off the EU route: from an EU-routed provider only another EU route or a direct provider, never the non-EU router (LaneScheduler.MayMoveTo). A model in no tier has no defaults; an explicit modelAlternatives entry on Admin/Threads replaces its defaults. A hard pin still waits.

9. The provider's DAY — closed pools and the budget spent by priority

On 2026-10-04 at 14:47Z the control instance's provider key hit its daily limit (HTTP 403: Key limit exceeded (daily limit)), and every lane stopped at once — reviews, triage, bug fixes, embeddings. Two rules answer it:

• A refusal no load change cures CLOSES the provider (AdmissionState.Closed, ThreadDispatchPool.FailureOf): the key's daily limit until the next UTC midnight; credit or a rejected/forbidden key for ClosedFor (35 min); the provider's daily budget reached at the gate — until midnight. Every pool of that provider closes (the limit is the key's, not one model's). Nothing is admitted on a closed pool: each round moves to its declared alternative on another provider (within the declared, EU-respecting equivalences) or — hard-pinned — waits. A closed provider with work waiting is the top item of starving (provider-closed). A closed pool lets one probe round through every ProbeEvery (10 s) while nothing runs on it; a probe that ends cleanly reopens it — a topped-up key does not wait for midnight, and a refused probe costs a fast 403. A rate limit (429), a stall or the round cap is LOAD: it shrinks the estimate and closes nothing.
• The day's budget is read BEFORE the limit and the remainder is spent by priority (BudgetReserve): the budget gate's reading (spent / daily limit) is reported on every round (ReportBudget) to every pool of that provider. Each lane leaves a share of the day for the lanes above it: triage stops at 30 % left, bug fixes at 20 %, the rest at 15 %, reviews at 10 %, the babysitter and the watchdog at 5 %, express never. A reading from another UTC day stops nobody.

Measured the same day: from 16:31Z to ~20:01Z every review round of every pull-request head on the control instance ended within a second on that 403 (this PR's own seven rounds included) — the reviewer is bound to one model on one key, with no declared alternative to fail over to. Default alternatives per tier, including another provider first, follow in the stacked change #2875.

Not wired yet: OpenRouter publishes a key's own limit and remaining credit (GET /api/v1/key: limit, limit_remaining, usage_daily); this reads the platform's own ledger against the provider node's DailyLimit instead. A key whose limit is set only at the provider is therefore learned from its first refusal (and closed at once), not ahead of it.

10. Delegated rounds

A sub-thread of a round that holds a slot ({parent}/…) is part of that round's work: it is admitted at once on its parent's lane — refusing it could only stall the parent (a parent holding a slot while waiting on a child that waits for a slot is a nested-resource deadlock). It does NOT ride the parent's slot: it is COUNTED as a running round of that lane, because the provider sees it, and it is marked (Seat.DelegatedBy; delegated per lane × bulkhead on Admin/Threads) so a fan-out is visible. Nothing in admission bounds the fan-out: a bound there would reintroduce the deadlock. A bound belongs where the delegation is made (the delegate call refused visibly to the parent), not here.

11. The read model and the dispatch entry point (for a coordinator)

Admin/Threads carries, per bulkhead: capacity (K̂), carryingCapacity, exhausted, exhaustedAt, why, slowStart, free, running, waiting, successes, failures, meanSeconds, meanCostUsd; per lane × bulkhead: running, waiting, guaranteed, borrowed, bound, oldestWaitingSince, spentTodayUsd (on that bulkhead's provider), delegated; per bulkhead also closedUntil, budgetSpentUsd, budgetLimitUsd, budgetAt; and starving (a closed provider first). A coordinator decides what to start and how much: ThreadDispatchPool.Plan(lane, model, wanted) answers how many of wanted rounds the admission would start now (floors, fair share, pins and alternatives honoured, nothing seated). Enforcement stays in the admission: each round still asks.

The audit — every cap the lanes had (2026-10-04)

Read on origin/main fd3dbd1e1. (a) = a throughput throttle, replaced by the shared admission; (b) = a safety bound, kept, with why.

The Monte Carlo comparison — why seed-and-grow

src/MeshWeaver.AI.Test/AgentAdmissionSimulationTest.cs runs the admission that ran until 2026-10-04 (step: everything admitted until a refusal, then a tenth of the measured load for 35 minutes, one bound for everything) against seed-and-grow (the shipped AdmissionState) over the load measured on the control instance on 2026-10-02 (Hosting/ReviewRoundCapacity): 35 review heads in two minutes (40,000 tokens each), the bug backlog (100 rounds of 20,000), three red mains five minutes in, triage every four minutes, the babysitter every six. The provider model is fitted to the measured rates — 61 / (1 + (n − 1)/(0.53·K)) tokens/s per round with n in flight (K = 18: 61 alone, 39 at five, 20 at twenty); a round started above K is refused with a 429 within 15 s with probability min(0.9, (n − K)/10); a round unfinished at 30 min is cut; a round slower than its tokens / 1,200 s reports the pool exhausted. Ten seeded runs per row, means; the test asserts the relations below and prints the table, so a run reproduces it exactly.

The reviews of the last scenario drained in 28 min under seed-and-grow against 81 min under step (its one bound held them behind the flood).

What it shows:

• The burst: step floods the pool at once — hundreds of 429s and rounds cut at the cap, and the load does not drain within ten simulated hours. Seed-and-grow drains it, with a handful of refusals, and starts the red mains within minutes.
• An idle pool: a remembered pool is used exactly as fast as step; a never-measured one costs about a minute of ramp.
• Isolation: a bug flood that exhausts a small pool does not hold back reviews on another pool (step's single bound did). The flood itself drains slower than under step on that small pool: step overloads it and pays in refusals, and this provider model rewards overload with throughput, which a real provider's rate limit does not.

What this does not establish

• Multi-pod. The pool is per process; on a multi-silo mesh every silo admits its own rounds and Admin/Threads shows whichever silo wrote last. The review steward's ledger stays the cross-pod record for reviews.
• The provider model's shape beyond the three measured points (1, ~5 and ~20 rounds in flight).
• Equivalence beyond the default tiers. The three default tiers name today's models; a new model joins a tier only when its id carries a tier token or modelAlternatives names it.
• Cost guard inputs. The guard reads each lane's priced spend; the provider's daily limit is applied by ProviderBudgetGuard itself (a reached budget is an exhaustion signal here).

The tests — one per rule

Every test below fails when its rule is removed. Checked 2026-10-04 by mutating each rule in LaneScheduler / AdmissionState / ThreadDispatchPool.FailureOf (27 mutations: the floor, the overdraft, reclaim, fair share, the probability, slow start and its interval, logistic growth, the decrease and its once-per-burst hold, aging, the cost guard, starvation, spreading, Thompson, the pin, the fallback, oldest-first, bulkheads, the closure and its provider-wide reach, the budget reserve and its day, delegation, the midnight reset, rate-limit-is-load, closed-first) and running the three suites: every mutation turned at least one named test red.

Where the code is

• src/MeshWeaver.AI/Supervision/LaneScheduler.cs — the lanes, the policy, every rule (pure).
• src/MeshWeaver.AI/Supervision/AdmissionState.cs — the pool's state and its transitions (pure).
• src/MeshWeaver.AI/Supervision/ThreadDispatchPool.cs — the live pool: admission, signals, clock, census, Allowance, Plan.
• src/MeshWeaver.AI/ThreadSubmission.cs — the watcher asks Admit with the thread's lane and writes a fallback model onto the round.
• src/MeshWeaver.AI/ThreadExecution.cs, ProviderBudget/ProviderBudgetGuard.cs — the signals.
• Hosting/Deployment/Source/{BugFixPool,TriageIssueSweep,ReviewAdmission}.cs — the lanes that start work pace by Allowance.

Related: Review round capacity · Queues · The activity execution model · Thread groups.