The catalog's first frame β what it may wait for, and what it may not
/Store on memex.meshweaver.cloud showed Loading the store⦠for over two minutes before a
single card appeared (#1177). The same page on a local install painted immediately. Nothing was
broken: the NodeType compiled, the hub was healthy, the packages were there, and every other area on
the same node rendered in about two seconds.
The rule this cost us
π¨ Every arm of the catalog's first-frame join is SEEDED, unconditionally. A feed may take as long as it takes; the page frame may not wait for it. "Still loading" is something a feed says β it is never inferred from a clock, and never from the fact that some earlier render registered a query.
What was measured, and what each measurement eliminated
Read-only, against memex.meshweaver.cloud, 2026-09-02, plus the portal logs from memex-cloud.
| Measurement | Result | What it rules out |
|---|---|---|
GET /Store (page shell) |
200, TTFB 0.18 s | the HTTP path, the shell, TLS, routing |
get @Store warm |
instant | cold grain activation, hub health |
get @Store/area/Plans |
complete in ~2 s | the node, the hub, the process, the NodeType assembly, the cluster |
get @Store/area/Catalog |
timed out at 35 s, three separate attempts | β this is the defect |
get @Store/data/StorePackage |
132 packages, instant | the package feed, and with it the whole "the GitHub enumeration is slow" theory |
StoreManifestSource git polls |
fail in 2β50 ms (Octokit.AuthorizationException: Bad credentials, all five repos within 50 ms of each other) |
"the cloud's git sources are live and the first render waits for the fetch" |
StoreManifestSource registry poll |
fails, 503 instance-key resolution | same |
[CrossSchema] SLOW on nodeType:Store/Plugin |
1.0β7.7 s per pass across 196 partition schemas, re-running every 2β5 s | nothing β this is the cost that was on the critical path |
Two things in that table are worth stating plainly, because the first diagnosis of this bug was built on their opposites:
- The package feed is not slow here β it fails fast. On the cloud every git poll dies on
Bad credentialsin milliseconds and the registry poll 503s; theScan/DistinctUntilChangedloop turns each failure into an immediate empty snapshot. TheStorePackagecollection is fully populated and reads instantly. Seeding that arm was the proposed fix; on its own it would have changed nothing. - The cloud has 196 partition schemas, not 96. The count was compared against a local install to argue that partition fan-out could not be the differentiator. It is roughly double, over tables that are not empty, on a mesh that is also running 2.5-second GC pauses and routing back-pressure.
The arm that was not seeded
StoreCatalogLayoutAreas.Catalog joins six arms with CombineLatest, which emits only once every
arm has emitted. Five were seeded (isAdmin, installStatus, provisionRun, the grace timer, and β
via its own StartWith β the inner real-node batch). The plugin feed was seeded conditionally:
var warm = IsWarm(hub, PluginsQueryId); // hub.GetQuery(id) is not null
var plugins = Seeded(ObservePlugins(host), warm); // warm ? feed : feed.StartWith([])
IsWarm asks whether a synced query is registered under that id. It is not a question about
data, and three facts turn it into the bug:
MeshNodeStreamCache.GetQueryRawinserts the stream into its registry before anything subscribes β registration precedes the first emission.- The registry entry then persists for the life of the process (
Replay(1).AutoConnect(1); the entry is only ever removed by fault eviction). So after the very first render on a pod,IsWarmanswerstruefor every visitor, forever. warmwas also the branch that withheld the seed.
So from the second render onward, the catalog's first frame was gated on ObservePlugins
delivering β and delivering means a mesh-wide nodeType:Store/Plugin query (1.0β7.7 s across 196
schemas) whose shared snapshot is then handed to each subscriber through
WrapWithPerUserRls β FilterByReadPermission, which probes CheckPermission(node.Path, user, Read)
once per node in the snapshot, sequentially, and emits nothing until the whole snapshot is
filtered. Eighty-seven plugin nodes, each probe potentially its own AccessAssignment scope:Subtree
and PartitionAccessPolicy scope:Children fan-out β the two most frequent fanned-out queries in the
whole fleet log (1,857 and 458 occurrences in ten minutes). That is the two minutes, and the page
was blank for all of it.
The warm branch was not careless: it was written to stop a "Nothing published yet" flash on a store that is full. That is a real problem with a better answer β see below.
Why this is NOT a location-declaration problem
nodeType:Store/Plugin fans out across every schema because every plugin root is its own partition
root. There is no path or namespace to pin it to; the query is legitimately mesh-wide. Declaring an
instance location for it would be the same false premise that #1127 was closed on. The fix is not to
make the query cheap β it is to keep it off the first frame.
What the fix is
Seeded(feed)always seeds, and marks the seed as not an answer:(IReadOnlyList<MeshNode> Nodes, bool Answered). Thewarmparameter andIsWarmare gone.packagedis seeded on its outer hop too. An empty seed is safe by the tri-state contract: no entries means no cards, so no Install can be offered against an unknown resolution.IsLoading(pluginsAnswered, packagedAnswered, graceOver)replaces "the grace timer said so". The empty message now waits for a real answer instead of a clock β which is strictly better at the job the warm branch was doing, because an answer that arrives in 300 ms ends the progress bar in 300 ms, and one that never arrives is still bounded by the 12-second backstop.GraceOver()always starts the timer. It is a backstop for a feed that never answers, not a statement about warmth.
The page now paints its frame, hero and category tiles on the first frame; the cards fold in when the feed answers. Nothing about the feed's cost changed.
What is still slow, and where it belongs
The per-user RLS filter on a shared synced query is O(nodes) permission probes per subscriber,
serialized ahead of that subscriber's first emission, and the probes themselves fan out across every
schema. That is MeshWeaver.Graph.Contract/SyncedQueryDataSourceExtensions.FilterByReadPermission
and it is core's, not this repo's. It is filed separately. Until it is cheaper, the rule at the top of
this page is what keeps it off a user's screen.