The /api/content 503 β three causes, two discriminators
/api/content/{node}/{collection}/{file} answering 503 after ~10.3 s is not a class of failure.
It is one failure with three possible causes, and the fix for one does nothing for the others. This
page is the elimination: what the 10 s is, what it is not, and the fields that decide.
Filed as #2901 β "a freshly uploaded content file 503s for minutes" β and reopened from a different direction as #3931 β "every COLD read 503s after ~10.2 s", which is Cause C.
π¨ The one sentence
The 10 s is
ReadBudget.Defaultexpiring on ONE request β the collection-configGetDataRequestβ and it expires only when no notification reached the ISSUING hub's callback in time. Three things do that. The reply could not be DELIVERED there (a lost pod-hub claim, Cause A); it was delivered and not DISPATCHED (that hub's action block was saturated, Cause B); or the OWNING hub had not finished STARTING, so the request was still parked in its deferred queue when the budget lapsed (Cause C). Nothing else on the route produces 10.3 s.
π¨ Cause C is the one that reads as Cause A, because the documented Queue(buffer=β¦)
discriminator snapshots the READER, and under Cause C the reader is idle. It has its own section and
its own discriminator below; read them before applying either of the other cures.
π¨ The issuing hub used to be portal/nodeops-{meshId} β the mesh's node-CRUD EXECUTION hub β
which is what made the second case reachable from an ordinary upload. It is now
portal/reads-{meshId}, a hub that registers no handlers β see "Cause B's cure" below. The
elimination
below is written against the pre-fix code, because that is what the production occurrences ran, and
because it is still the map of the route.
The read path, and where the 10 s lives
| # | Step | Where | Bounded? |
|---|---|---|---|
| 1 | decode + traversal guard | BlazorHostingExtensions.ResolveContentFile (Plugins, :791-806) |
n/a |
| 2 | resolve the owning node β IPathResolver.ResolvePath |
ContentFileResolver.cs:114-117 |
NO β unbounded |
| 3 | read the collection config β GetDataRequest(ContentCollectionReference) to the owner |
ContentFileResolver.cs:183-193 |
ReadBudget.Default = 10 s (ReadBudget.cs:92) |
| 4 | mount check (IsStatic) |
Plugins :823-824 |
n/a |
| 5 | permission gate, tri-state | Plugins AllowContentRead :595-605 / GatedContentRead :631-646 |
fold-local |
| 6 | open the collection + read the bytes | ContentService.GetCollection β ContentCollection.GetContent, on the collection's IIoPool |
pool-bounded |
Step 3 is the only 10-second bound on the path. FailIfNoFirstEmission (ReadBudget.cs:120-134)
is an Observable.Amb of the request and a 10 s timer; when the timer wins it throws
HubUnreachableException, a TimeoutException subclass, which
BlazorHostingExtensions.ContentFailure maps to 503 (Plugins, :746-748). So a measurement of
~10.2β10.8 s then 503 is ResolvePath (a few hundred ms) plus the budget, exactly.
Step 3 is issued from, and answered onto, a hub that is not the caller's own. The
/api/content endpoint holds the DI-injected IMessageHub, which in the mesh's root container is
the router, and the router must be neither end of a delivery β so ContentFileResolver hops onto a
dedicated off-router hub. Until this page's fix that hop landed on portal/nodeops-{meshId},
the mesh's one node-operation hub (hub.NodeOperationIssuingHub()); it now lands on
portal/reads-{meshId} (hub.ReadIssuingHub()). Either way the reply must be dispatched by that
hub's single-threaded action block before hub.Observe emits (MessageHub.cs:451 registers
HandleCallbacks as a delivery rule, and every delivery β responses included β goes through
EnqueueTurn, MessageService.cs:906). Which hub it is decides what else can be in that block's
way.
What is provably NOT the cause
Each of these was a candidate on #2901. Each answers, and answering is not silence.
| Candidate | Why it cannot produce 10.3 s |
|---|---|
| The config handler is slow / propagating | HandleCollectionConfigRequest (ContentCollectionsExtensions.cs:300-346) is a pure in-memory dictionary read followed by hub.Post. There is nothing to settle. |
| The owning node is not content-enabled | The generic workspace read answers anyway: GetDataResponseObservable<TReference> returns GetDataResponse(null, 0) when the reduce manager resolves no stream (DataExtensions.cs:3092-3104). ReadCollectionConfigs then yields null β 404, in milliseconds. |
| No handler matched at all | MessageHub.FinishDelivery NACKs any unhandled IRequest<> with a typed DeliveryFailure(NotFound) (MessageHub.cs:936-965). |
| The permission fold reached no verdict | AccessControlPipeline answers DeliveryFailure with Unavailable / ShuttingDown (AccessControlPipeline.cs:618-641) β a fast 503, not a silent one. |
| A storm breaker latched by the first early read | The per-key breaker trips at 2000 messages/second on one key (MessageStormBreaker.cs:92) with a 2 s cooldown (:100); the aggregate shed only touches [CanBeIgnored] traffic (:505) and GetDataRequest is not (Messages.cs:318-319). A browser cannot reach either. |
| Per-node hub warm-up | π¨ Refuted for #2901's occurrence ONLY β this is Cause C and it is real. The #2901 measurement rules it out there: AgenticPrimer/content/og.png answered in 0.9 s while AgenticPrimer/content/videos/skills.mp4 β the same node, the same GetDataRequest β took 10.3 s in the same window, and an activating hub cannot answer one and drop the other. That argument turns on the two files sharing an owner; it says nothing about a read whose owner has not started at all. See "Cause C" below. |
| A settle window on the uploaded file | Nothing on the path is keyed to the file. The bytes are read directly from the store; the blob provider has no monitor at all (AzureBlobStreamProvider.AttachMonitor returns null), and the file-system monitor drops every non-.md event at the watcher callback (FileSystemStreamProvider.cs:272-277), so 32 binaries neither ingest nor lengthen collection init. |
The per-file appearance is a cache key, not a settle window
The one per-file thing on the whole path is PathResolutionService's value cache, whose key is the
full joined path including the file name (PathResolutionService.cs:442-505). So every distinct
file URL is its own entry and its own storage query on first request, and a warm URL replays
synchronously (:456-458) while a cold sibling pays a live cross-schema query. That produces
per-file latency differences. It does not produce the 503 β step 3 is identical for both files.
Cause A β the reply cannot be delivered
The owning hub produces the GetDataResponse and the router refuses to carry it, because the
grain directory holds no claim for portal/nodeops-{meshId}.
portal is a stream-routed address type (MeshConfiguration.cs:69-70), so RoutingGrain takes the
directed pod-hub route. On a refusal it reaches AnswerPodHubNotHere
(RoutingGrain.cs:610-652), and ClientHostedAddressTypes is empty in production by explicit
design (MeshConfiguration.cs:78-82) β so there is no stream fallback. The transient
DeliveryFailure is posted to the sender (RoutingGrain.cs:650), i.e. to the responding node
hub. The requester is told nothing and can only wait out its budget.
This is #2938 / #2915, root-caused on
#2901 on 2026-09-02 from production logs
(33 occurrences in 24 h, every one on a single pod, onset in the same hour as a silo death). The
cure is The Pod-Hub Claim Must Be Re-Asserted, merged as f37f3fc87.
Signature. The HubUnreachableException message carries the reader's own snapshot
(ReadBudget.cs:234-245 β MessageHub.GetPendingRequestDiagnostics, :2116-2136):
Reader: Hub portal/nodeops-β¦ RunLevel=Started Queue(buffer=0,deferred=0,exec=0)
PendingCallbacks=1[β¦=GetDataRequest@DoublePendulum(10003ms)]
Empty queue, nothing executing. The reader processed everything delivered to it; the silence is
upstream. Peer pods carry the matching [ROUTE] Directed delivery to pod hub 'β¦' was refused line.
π¨
exec=in the captures on this page is the pre-#3593 name of a field that was a hard-coded literal0β it read the same in every state and carried no information. Today the same position printsdrainsInFlight=, a real count. Nothing on this page depended on it: the discriminator below isbufferand theExecuting(β¦)line. See Reading a Disposal Stall Verdict.
Cause B β the reply is delivered and not dispatched
portal/nodeops-{meshId} is also the mesh's ONE node-CRUD execution hub. Every
CreateNodeRequest / CreateOrUpdateNodeRequest issued from the router runs there and is
serialised on the same action block (MeshService.cs:93 issues on it, :104-108 targets
NodeOperationTarget, whose fallback is that same hub, MeshExtensions.cs:325-326). The turn loop
runs exactly one turn at a time and does not advance until the current turn's observable completes
(MessageService.cs:945-1006).
That this hub stops draining for tens of seconds under a bulk node-CRUD burst is measured, from
a different direction, in Bake Seal β NodeOps Saturation: queue
latency bimodal at β€ 3.2 s / 33β49 s, and one capture of
Queue(buffer=45,β¦) Executing(CreateNodeRequest, 24888ms).
A content upload burst is a node-CRUD burst. With the indexing pipeline registered, each
uploaded file starts its own Activity β and until 2026-09-10 ContentIndexingObserver.OnUploaded
fired them unbounded and fully parallel, one per file: it subscribed the per-file activity
inline, the instant the seam was raised, while its own sibling walk sequenced the identical work
with .Concat(). Per file, on portal/nodeops-{meshId}:
| Work | Deliveries on that action block |
|---|---|
meshService.CreateNode for the _Activity node (ContentIndexingActivity.cs:123) |
request + response |
MeshDocumentSink.WriteDocument β CreateOrUpdateNodeRequest for _Documents/{slug} (MeshDocumentSink.cs:64-65) |
request + response |
β¦plus one fresh per-node hub activation for {owner}/_Activity/{id}, routed through the mesh
router's own single-threaded block, and ~3 activity-log stream.Updates on cache/{meshId}.
Thirty-two files is on the order of a hundred deliveries. The burst does not drain when the files
are written β it drains when the slowest indexing leg finishes, and for image posters that leg is a
vision-model round trip on the Http pool, capped at 16 (ContentIndexingService.cs:215-224 β
ChatClientImageDescriber.cs:72; IoPoolOptions.cs:171). That is where minutes comes from, and
why it heals untouched.
The fan-out that produced the burst β re-measured, and bounded
π¨ Re-read on 2026-09-10 at the same coordinates: the mechanism above was still live, verbatim.
ContentIndexingObserver.OnUploaded (Plugins, ContentIndexingObserver.cs:107-116) was
IndexFileActivity(collectionPath, filePath).Subscribe(...) β one activity per file, subscribed the
moment the seam fired, with no bound of any kind, while ReindexCollection sequenced the identical
per-file work with .Concat() at :270. It is now bounded
(MeshWeaver.Plugins#1616): uploads
push onto an instance channel the observer drains with
.Merge(ContentIndexingObserver.MaxConcurrentUploadIndexing), so at most four indexing
activities are ever in flight. OnUploaded still returns immediately β the surplus is queued by
Merge, not by a gate of our own.
π¨ The asymmetry that decides the bound is not where a reader first looks. The expensive leg is
the model round trip, and it is already bounded β 16, by the shared Http pool, as the paragraph
above says. So fan-out past that cap buys no throughput at all: the surplus activities exist
only to queue at the Http gate, and every one of them has ALREADY paid its node-CRUD cost in full,
up front, before reaching the model. The burst on portal/nodeops-{meshId} was therefore pure cost
with no compensating benefit β which is exactly why bounding the fan-out is not a slowdown.
Why not sequence, and why not a pool slot. Both were candidates; each is refused by a measurement rather than a preference.
| Candidate | Why not |
|---|---|
Sequence it β .Concat(), the sibling walk's own shape |
ReindexAll is ONE activity for a whole collection, so ordering its log costs it nothing. Here each file is its own activity, so a bound of 1 does not reduce the node-CRUD count at all β it only paces it, which a bound of 4 already does β while giving up a 4x on work the Http pool is happy to run concurrently. |
Hold an IIoPool slot for the whole activity (the Ai pool's stated "runaway-fan-out STOP" shape) |
The right instinct, refused on two counts. The activity's own inner legs take FileSystem, Http and Query slots, so the outer bound must be a pool the inner work never re-enters β a nested acquisition of a pool you already hold is the classic gate deadlock the framework names at IoPoolOptions.cs:55-59. A new pool name is available to a plugin, but its cap falls to IoPoolOptions.Default = Environment.ProcessorCount, which is a fallback rather than a reasoned bound, and giving it a real one is a change to IoPoolOptions here in core that the plugins repo cannot reach behind its pin. |
.Merge(n) on an instance channel (chosen) |
The fleet's established bounded fan-out for this exact shape: .Merge(4) wherever the per-item work writes mesh nodes (AiSourcesInstallHook, AiContentDiskWriter, IssueService.cs:83, GitHubWebhookProcessor.cs:154), .Merge(8) where it does not (GitHubSyncService.cs:214, OctokitGitHubRepoClient.cs:223). .Concat() is Merge(1), so the sibling walk and the new pump are one operator at two settings - nothing hand-woven, no SemaphoreSlim, no pacer, no queue of our own. |
The control, both directions. UploadIndexingFanOutBoundTest (Plugins,
MeshWeaver.ContentCollections.Indexing.Graph.Test) runs on a monolith mesh with a real
file-system collection and no sleeps. Its instrument is the summarizer: ContentIndexingService
calls it exactly ONCE per document, from inside that file's activity, so concurrent calls are
concurrent activities β and the test's summarizer PARKS, which holds every admitted file inside the
pipeline long enough for the number to be read. Twelve files raised through RaiseContentUploaded
in one burst, measured 2026-09-10:
| Observer | Peak concurrent indexing activities | Verdict |
|---|---|---|
.Merge(4) (bounded) |
4 β 4 of 12 admitted while parked, all 12 indexed after release | green |
.Merge() (the pre-fix shape) |
12 β the whole burst at once | red: "the upload to index fan-out must be bounded at 4: 12 files were uploaded in one burst and 12 indexing activities ran at once" |
The green run also asserts the lower half β the peak must REACH 4, so a future change that over-serialises the pump is red too β and the positive control that every one of the 12 files is still indexed, because a bound that indexed nothing would satisfy a ceiling assertion on its own.
Signature.
Reader: Hub portal/nodeops-β¦ RunLevel=Started Queue(buffer=N>0,deferred=0,exec=β¦)
Executing(<message>, <thousands>ms) PendingCallbacks=β¦
Cause C β the owning hub had not finished STARTING
Nothing is lost and nothing is saturated. The read is the FIRST thing to address that node since the
process started, so the GetDataRequest is what creates the owning hub β and until that hub's last
initialization gate opens, MessageService parks every arriving delivery in the deferred queue
instead of dispatching it (MessageService.cs:1541-1544, enqueue :1687-1692). The read therefore
does not measure the owner's health. It measures the owner's START-UP, and reports a node that is
present, permitted and about to answer as unreachable.
Filed as #3931.
The bounds are inverted, and that is the whole defect. ReadBudget's own remarks state the rule:
"a bound nested inside another bound must be able to fire FIRST, because it is the only one that
knows WHICH read starved." On this path every inner bound is LARGER than the budget waiting on it:
| Bound on the work the read triggers | Value | Where |
|---|---|---|
| the reader's budget | 10 s | ReadBudget.cs β Default |
| deferred delivery, per message | 30 s | MessageService.cs:151 β DeferralTimeout |
| the hub's own initialization turn | 120 s | MessageHub.cs:210 β DefaultInitializationTimeout |
DataContext initialization |
120 s | DataContext.cs:144 β InitializationTimeout |
| monolith path resolution on the routing path | 30 s | RoutingServiceBase.cs:306 |
| Orleans first-node resolution | 30 s | MessageHubGrain.cs:88 β FirstNodeResolutionTimeout |
| NodeType slow path (no progress) | 30 s | NodeTypeEnrichmentHelpers.cs:69 β SlowPathTimeout |
So the 10 s ALWAYS fires first, no inner bound can ever deliver its diagnosis to this caller, and every Cause C occurrence is reported in the outermost, least informative terms. See Bounds Must Be Ordered.
Why it is invisible until it is not. A cold start costs ~0.1 s on a quiet replica β measured
2026-09-10 on memex.meshweaver.cloud, where five previously-untouched owning nodes (Doc/DataMesh,
Doc/Architecture/MessageBasedCommunication, β¦/UserInterface, β¦/AccessControl,
β¦/BusinessRules) each served their first asset in 0.14β0.20 s. The budget is 100Γ that, so
nothing is ever seen. It becomes visible only while the replica is doing something that makes a
start-up cost tens of seconds β a roll, a restart with its boot bake (PreWarmCompletion's own
measurement: ~10 min of sequential rebuilds, across three production portals), or a bulk import into
the partition being read.
Production capture, 2026-09-10
The Prod synthetic probe failed twice, 16 minutes apart, and both runs read the same asset three
times with a 2 s gap:
14:46:43 round 1: platform=503 absent-control=404
14:46:56 round 2: platform=503 absent-control=404
14:47:08 round 3: platform=503 absent-control=404
15:02:28 round 1: platform=503 absent-control=404
15:02:40 round 2: platform=503 absent-control=404
15:02:43 round 3: platform=200 absent-control=404
The 15:02 run is the whole shape in six lines: two consecutive budgets burned, then the same URL served in 0.2 s about 24 s after the first attempt β a start-up that completed on its own, inside every one of its own bounds and outside the reader's. The absent control answered 404 in every round of both runs, so path resolution (step 2) was never the problem.
What that replica had been doing is in the mesh: Doc/_Activity/import-bfcebef8e3a0b6ac β
"Import Doc (1446 nodes)", 14:35:19β14:35:26Z, eleven minutes before the first failure β and a
fleet-wide NodeType bake wave whose _Activity/compile-state nodes are stamped 15:04:58β15:05:23Z
across Reinsurance, Edu, Planning, Store, Chess and two dozen more partitions. By 15:31Z
every cold read measured 0.14β0.20 s again, and the probe has been green since.
The mechanism, and the two facts that identify it
- The failing unit is the OWNING NODE, exactly once. Every later read of any file under that node is fast, because the hub is now hosted; a different node read afterwards pays its own start-up. That is neither Cause A (a per-hub claim, which would fail every read alike until it is re-asserted) nor Cause B (a burst, which is per-window rather than per-node).
- The impossible path keeps answering 404 in milliseconds, because an unmatched reference returns
at step 2 and never reaches the bounded read (
ContentFileResolver.csβNotFound("No matching node found for path")).
π¨ Cause C wears Cause A's signature β the reader clause cannot see it
This is the trap, and it cost two sessions on #3931 before it was named. The discriminator below
reads the READER's queue. Under Cause C the reader β portal/reads-{meshId}, a hub that
registers no handlers β is completely idle, so it prints Cause A's signature exactly. Verbatim,
from ContentReadPaysTheOwningHubsStartupTest run against the pre-#3931 code, on a mesh where the
owning hub was in the same process and answered milliseconds later:
Reading content collection config from 'TestData/ColdProbe' gave up after 10s β the owning hub
never answered. β¦ Reader: Hub portal/reads-gNtsgKCSMUOiivUSuHw-ag RunLevel=Started
Queue(buffer=0,deferred=0,drainsInFlight=0)
PendingCallbacks=1[β¦=GetDataRequest@TestData/ColdProbe(10002ms)]
Target: NO LOCAL HUB at 'TestData/ColdProbe' β it never activated in this process (or it is owned
by another silo and the reply was lost in transit).
Queue(buffer=0,β¦), nothing executing β "Cause A: the reply never arrived" by the rule below, on a
run where the reply arrived fine. Two things in that line were wrong, and both are fixed:
- "the owning hub never answered" is a claim the reader cannot support. A budget knows one thing: that no notification arrived in time. It now says that instead.
Target: NO LOCAL HUBwas a CONSTANT, not evidence.GetHostedHubsearches one hub's own collection β it walks neither the hosted tree nor the parent chain β and every per-node hub is hosted by the mesh hub (MonolithRoutingService.CreateHub,MessageHubGrain.CompleteActivation). The probe asked the READER, which hosts nothing, so it answered "NO LOCAL HUB" for every read on this seam whether or not the hub was sitting right there. It now asks the mesh hub, and when the answer is a hub that has not reachedRunLevel=Startedit says so in words.
Cause C's discriminator β read the TARGET clause
Target: STILL STARTING β '<owner>' β¦ RunLevel=Starting Queue(β¦,deferred=N,β¦)
β Cause C: the read was DEFERRED behind the owner's start-up. It is not lost.
Target: Hub <owner> RunLevel=Started Queue(buffer=0,deferred=0,β¦)
β the owner was up: fall through to the reader clause and decide A vs B there.
Target: NO LOCAL HUB at '<owner>'
β the owner is not in THIS process. Not decidable from this line β use the black-box table.
The black-box discriminator β no logs required
π¨ This is the durable one, because it needs no log at all β everything below is curl against
the public route, from anywhere, with no credential and no portal.
π¨ It is NOT durable because the log line is unreachable. That reading β "
search 'nodeType:Hosting/LogEntry'returns no content-route entries, so the ingest carries nothing from this route" β was measured on 2026-09-10 and is a wrong inference from a correct observation.Hosting/LogEntryis not a feed: it is the output of oneLogsHosting/InstanceAction, and that portal's whole population is 13 rows from one query about assessment routing. The log line IS reachable through the sanctioned API β see "What the discriminator costs to obtain" below and Log Entries Are a Query Result, Not a Feed. Prefer the black-box table when you want an answer with no portal access; prefer the line when you want the run level and the queue depths, which the black box cannot give you.
| Observation | Cause C | Cause A | Cause B |
|---|---|---|---|
| unit of failure | the OWNING NODE, once | the reader's hub β every read alike | whatever lands inside the burst |
| the same URL, read again | fast, and stays fast | fails identically | fails while the burst lasts |
| a DIFFERENT node, read after the first healed | pays its own start-up and can fail too | already cured once the claim is re-asserted | tracks the burst, not the node |
| the impossible path | 404 in ms | 404 in ms | 404 in ms |
| when | within minutes of a roll, a restart, or a bulk import into that partition | after a silo death | during a bulk node-CRUD burst |
| how it clears | on its own, per node, permanently | needs the pod-hub claim re-asserted | when the burst drains |
π¨ What Cause C does NOT have: a cure
Causes A and B each have one. Cause C does not, and picking one is a decision rather than a patch, because every exit is either forbidden or changes what the route is:
- Wait for the start-up. The read budget would have to be ordered outside the target's own
envelope (β₯ 30 s). That is widening
ReadBudget.Default, which AGENTS.md forbids β "over budget means STUCK, not slow" β and the measurement above says the read is not stuck, so the honest version of this exit is "derive the budget fromMessageService.DeferralTimeoutand pin the ordering with a guard". It is still a bound change and it is still a decision. - Answer without the owning hub. The collection config is deliberately uncached and deliberately
gated by the owner's
[RequiresPermission(Read)]β see CQRS and Content Access. Caching it is the short-circuit this page's own class remarks refuse. - Keep interactive traffic off a replica that cannot yet serve it. Readiness, not budget: a replica whose boot bake has not settled stays out of the load balancer. This is the only exit that removes the user-visible symptom without widening a bound or retrying, and its cost is that a roll takes as long as the bake β a fleet-level trade-off, not a code change.
Until one is taken, what #3931 buys is that an occurrence is now identifiable in one line instead
of being misfiled as Cause A. The repro is ContentReadPaysTheOwningHubsStartupTest
(test/MeshWeaver.Graph.Test): one node's reactive initialization is parked on a subject the test
completes, the content read must say what it observed, and the identical read resolves the moment the
gate opens β which is what makes the 503 a false negative rather than a measurement.
π¨ The discriminator
π¨ Read the TARGET clause FIRST, then this one. The rule below separates A from B and
cannot see Cause C at all β it snapshots the READER, and a start-up leaves the reader idle, so
Cause C satisfies the Queue(buffer=0,β¦) test perfectly. Rule out Cause C on the target clause
("Cause C's discriminator" above) before reading a word of this.
A and B are then separated by one field in a line the framework already prints β no new instrumentation, no cluster access beyond the log:
Queue(buffer=0,β¦) and no Executing(β¦) β Cause A: the reply never arrived β IF the owner was Started.
Queue(buffer>0,β¦) or Executing(β¦, Nms) β Cause B: the reply arrived and waited.
What the discriminator costs to obtain
π¨ Reading content collection config from β¦ is not a log statement. Nothing in this repo calls
Log* with it. It is the MESSAGE of the HubUnreachableException that ReadBudget.Unreachable
builds (ReadBudget.cs), and it reaches a log only because BlazorHostingExtensions.ContentFailure
(Plugins) does logger.LogWarning(ex, "Content read timed out for {Path}", path) β the exception
rides along as the warning's detail. Three things follow, and each has misled a reader:
- It is greppable, and on ONE line β BOTH clauses.
MessageHub.GetPendingRequestDiagnosticsis a single-line snapshot by contract, soReader: β¦ Queue(buffer=β¦) [Executing(β¦, Nms)] PendingCallbacks=β¦and theTarget: β¦clause that decides Cause C both sit on the same physical line as the sentence above. "Read theQueue(on the same line" is therefore literally true, and one grep hit carries the whole three-way verdict β which is exactly why the line is worth fetching rather than reconstructing.ContentRoute503DiscriminatorGuard(core,test/MeshWeaver.Documentation.Test) fails if a newline ever gets into it. - No level or category filter can select it. It is a continuation line: the
warn:header and theMeshWeaver.Hosting.Blazorβ¦category are on the PRECEDING line, which the log store keeps as a separate entry. - π¨ The obvious query returns the 503 without the discriminator. Filtering for
Content read timed out formatches the warning's own message line and stops there. Filter forReading content collection config fromβ the line that carries the answer.
Through the API, not by break-glass. A Logs Hosting/InstanceAction on the control instance
takes the LogQL, so this is one node and no cluster credential:
{ "namespace": "Ops/Actions", "nodeType": "Hosting/InstanceAction",
"content": { "$type": "InstanceActionContent", "deployment": "Deployments/memex-cloud",
"requestedAction": "Logs", "query": "Reading content collection config from",
"sinceMinutes": 240, "limit": 200,
"reason": "Read-only β read the Target clause for Cause C, then the Reader clause for A vs B." } }
Read logQl, entryCount and truncated back off the run; the matched lines land as
Hosting/LogEntry nodes under Ops/Logs. π¨ Do NOT instead search the Hosting/LogEntry nodes
that are already there β they are the result of whatever somebody last asked, not a feed, and
reading them as one produced a wrong conclusion on
#3931. See
Log Entries Are a Query Result, Not a Feed.
The break-glass form of the same read β az aks command invoke β¦ curl lokiβ¦/query_range, then
grep -o "Queue(buffer=[0-9]*" bucketed over a window β still answers it for a whole day at once,
and is the fallback when the control plane itself cannot act.
Do not apply Cause A's fix to a Cause B occurrence. Re-asserting a claim that was never lost changes nothing, and a single-replica or restarted portal removes the exposure to Cause A without touching Cause B β which is why a 12/12 green probe proves neither. And do not apply either fix to a Cause C occurrence: neither one touches how long a per-node hub takes to start.
π¨ Cause B's cure β the read does not belong on that block at all
The fix is not to make the node-CRUD hub drain faster. It is that a bounded read with a person waiting on it must never be issued on the hub that EXECUTES the mesh's node CRUD, whatever that CRUD costs. A serial execution hub occupied by a bulk write burst is that hub working as designed; a ten-second interactive read queued behind it is not.
MeshExtensions.ReadIssuingHub() is the seam. It hops a router-held caller onto
portal/reads-{meshId} β a hub wired exactly as portal/nodeops-{meshId} is (the mesh's own type
registry, the mesh hub's permission evaluator, registered with the routing service so replies land
on it cross-silo) with one deliberate difference: it registers no handlers. Nothing executes
there, so the only thing its action block ever dispatches is the reply to a read issued on it. Two
call sites moved:
| Read | Was | Now |
|---|---|---|
ContentFileResolver.Resolve β the collection-config GetDataRequest (step 3 above) |
NodeOperationIssuingHub() β portal/nodeops-{meshId} |
ReadIssuingHub() β portal/reads-{meshId} |
MeshNodeStreamExtensions.GetMeshNodeOutcome β the one-shot node read |
same | same |
NodeOperationIssuingHub() is unchanged and still correct for a write: a target-less
CreateOrUpdateNodeRequest posted there executes on the node-CRUD hub, and a write ack that waits
behind the writes queued ahead of it is the ordering that hub exists to impose. The two seams differ
because reads and writes want opposite things from the same block.
The repro
ContentReadIsNotQueuedBehindNodeCrudTest (test/MeshWeaver.Graph.Test) β a monolith mesh, no
sleeps, no cluster. An INodeValidator parks the create of ONE node (matched by path, so nothing
else in the mesh is slowed), which holds the node-CRUD execution hub's turn exactly as a real write
does; with the block held, ContentFileResolver.Resolve must still answer. Reverting the two call
sites to NodeOperationIssuingHub() reproduces this page's Cause B verbatim, including the
discriminator:
Reading content collection config from 'TestData/ContentProbe' gave up after 10s β the owning hub
never answered. β¦ Reader: Hub portal/nodeops-GMUrd3FU90aD8lxUt_XUlw RunLevel=Started
Queue(buffer=1,deferred=0,exec=0) Executing(CreateNodeResponse, 10004ms)
PendingCallbacks=1[β¦=GetDataRequest@TestData/ContentProbe(10002ms)]
Queue(buffer=1) and an Executing(β¦, 10004ms) line β Cause B by the discriminator above, on a
run where the owning hub answered promptly. Note what is executing: the create's continuation is
running inside the turn that delivered an intermediate CreateNodeResponse to that hub, so the
chain's remaining legs are charged to a node-CRUD delivery turn. That is the mechanism by which a
single create can occupy the block far longer than any one of its own steps.
What it does NOT fix
Nothing about how long a node-CRUD turn takes. #2543
β the same hub seen from the write side, where a CreateNodeRequest turn was captured at 24 888 ms
and queue latency is bimodal at β€ 3.2 s / 33β49 s β is untouched and stays open. What this change
does take off that block is the router-issued reads: the `PendingCallbacks=26[GetDataRequest@Store/Core,
services that hold the DI root hub, and every one of them now registers on portal/reads-{meshId}
instead. That removes a contributor and, more usefully, removes a confound: pending callbacks left on
portal/nodeops after this change were issued by something running on that hub, not by a
router-held caller.
Recorded, not fixed
Two unbounded seams on this exact path. Neither causes the 503; both make an occurrence harder to read.
ResolvePathis unbounded here (ContentFileResolver.cs:114-117) while the routing path bounds the identical call at 30 s (RoutingServiceBase.cs:304-306). A resolution that never emits β a documented shape, pinned byPathResolutionCachePoisonTest.HungFirstQuery_DoesNotPoisonCacheβ never even subscribes step 3's budget timer, so the request hangs until the client aborts with no 503 and no log line. Strictly worse than the failure this page is about.- The issuing hub is re-resolved per request (
ContentFileResolver.cs, nowReadIssuingHub()) rather than cached asMeshService.cs:93does. During mesh teardown it returns the router (MeshExtensions.MeshReadHubreturns null pastDisposeHostedHubs), reinstating the router-as-both-ends hang the long comment above that call exists to prevent.
A third lived in the plugins repo β the per-file indexing fan-out had no concurrency bound while its own sibling walk did. Re-measured 2026-09-10, found still live at the same coordinates, and fixed: see "The fan-out that produced the burst" under Cause B above.
On answering "not ready" with 409/425
Half of #2901's second ask is already true: the 503 is not generic. It is the deliberate answer for
"no verdict was reached", kept distinct from the 404 a missing or refused file gets β see
ContentUnavailable and PermissionApi β "The anonymous gate is tri-state too". What stands is
that 503 is also the code for down, so a monitor cannot separate the two. Under the A/B analysis that complaint had no
"still settling" case left to serve: both causes are genuine unavailability of a dependency, and both
are correctly 503.
π¨ Cause C reopens exactly that question, because it IS a ready-later state. The owning
hub is starting, it will answer, and the answer is usually seconds away β which is what a 503 cannot
express and a monitor cannot separate from down. That is one more input to the decision recorded
under "What Cause C does NOT have: a cure", not a change to make on its own: a distinct status is
only worth introducing alongside whichever exit is taken there, because the browser treats an
<img> the same way whatever the code is.
Related
- Log Entries Are a Query Result, Not a Feed β how to fetch the discriminator through the API, and why the log nodes already on a portal are not a feed.
- Bake Seal β NodeOps Saturation β the same hub, measured from the write side, with the open question of which rule holds its block.
- The Pod-Hub Claim Must Be Re-Asserted β Cause A's mechanism and cure.
- Action-Block Wedge Prevention β the invariants a single-threaded hub must satisfy.
- Bounds Must Be Ordered β why an inner bound just under an outer one destroys the outer one's diagnosis.
- Controlled I/O Pooling β where per-file work belongs instead of on an action block.
- CQRS and Content Access β why the collection-config read is a request/response and deliberately uncached.