Where a model's configuration lives

A model's configuration used to live in three places that could not see each other, and the symptoms — a pinned model declared unusable, a config key no template rendered, duplicate provider nodes nobody could repair, a funded key on a node and a dead one in Key Vault — were four faces of that one gap rather than four bugs. This page is the resolution: one administered home per fact, and a named path from every other place into it.

Read this before touching BuiltInLanguageModelProvider, ProviderCredentialSeed, ChatClientCredentialResolver, ModelStaticRepoSource or ModelProviderService. The operational procedures — set a key, rotate one, decouple a partition, the AKS Key Vault layering — are the /provider-keys skill; this page is why they are shaped that way.

The decision

The NODE is the administered home. Deployment configuration is SEED-ONLY. Maintainer, 2026-08-22.

resolve:  the node, and only the node
seed:     configuration → node, converging, never read again at resolve time

One place to look, one place to rotate. That is not a preference, it is what removes a whole class of failure: two live sources that disagree silently, which is exactly what cost an afternoon on 2026-08-21 when a funded OpenRouter key sat on the node while env and Key Vault carried a stale unfunded one, and every full round on ns memex returned HTTP 402.

One administered source per fact

Fact Administered home Bootstrap seed
Provider endpoint Provider/{name} → endpoint {Section}:Endpoint, fill-if-absent
Provider API key Provider/{name} → apiKey (enc: at rest) {Section}:ApiKey, fill-if-absent
The provider's model list the LanguageModel children of Provider/{name} {Section}:Models[], fully synced
Model order / price / tier label the LanguageModel node catalog defaults
Reasoning effort sent on every call the LanguageModel node → reasoningEffort none — provider default until set
Tier registry, tier → model Provider/Tier/{id} ModelTierDefaults, create-if-absent
A user's own key {user}/_Memex/{provider} none — the user pastes it

A deployment's {Section}:* values are the bootstrap fallback for a FRESH deployment, and nothing else. Nothing in the resolution path reads them. ChatClientCredentialResolver walks node rungs only — providerRef → Provider/{name} → the legacy per-model fields — and a model whose provider node has no key is honestly unusable. (A driver factory still applies its own IOptions fallback when it actually calls out; that is the driver's business, not a source of truth this platform arbitrates.)

The seam that had to change: a seed allowed to run TWICE

The catalog's static-repo import marks each Provider/{name} SyncBehavior.ExcludeThisOnly, so the importer CREATES it once and never revisits it. That claim is what protects an admin's key edit from the next boot's re-seed — and it is also why a key added to a deployment AFTER the node existed reached every factory and never reached the node.

That is measured, not hypothetical: on memex.systemorph.com, Provider/Anthropic was created keyless by system-security on 2026-08-14, Anthropic__ApiKey was configured afterwards, and the node stayed keyless until a human pasted the key in by hand on 2026-08-21. For that week the deployment could serve claude-* rounds and the platform believed it could not.

A seeder that only runs at CREATION cannot converge that. So the credential is no longer that seam's business at all: the import strips it (ModelStaticRepoSource.StripApiKey) and ProviderCredentialSeed runs on every boot of a DB-synced deployment. Its rules:

Two replicas booting together both run it, and that is fine: whichever writes first wins and the other reports AlreadyAdministered. In the narrow window where both decide to write, both values decrypt to the same key.

The read boundary — cryptographic, because nothing else can hold

Moving a secret onto a node changes its security class. A Helm value is a deployment secret; a node's content is application data, readable by whoever can read the node — through the GUI, the REST API, MCP, exports, sync targets and any GitSync binding on the partition.

And the Provider partition must be publicly readable. Its _Policy sets publicRead: true because the model picker queries namespace:Provider nodeType:LanguageModel under the user's own identity; without it RLS filters out every model and the picker is empty. On memex.meshweaver.cloud that policy was measured still set on 2026-09-01, over seven ModelProvider nodes.

So the question is not "who may read the node" but "what does the node hand back". Three candidate controls were examined and two of them cannot work here:

Candidate Why it does not hold
An access rule that hides the field Access control in MeshWeaver is per-NODE, never per-field, and NodeTypeGate only OPENS surfaces. A policy that opens a provider's LanguageModel children opens the provider node too — the models are its children, and a policy grants at its scope and below. Structural.
Redaction at the read boundary There is deliberately no per-caller content-projection seam in the mesh: a single-node read is a SHARED subscription — the owning hub broadcasts ONE node value to every subscriber on a silo, so it cannot project per reader (see UserPiiRedaction's class doc, which says exactly this and is applied at exactly one call site). Redacting only the identity-carrying surfaces would leave QueryNodes, CatalogDownload, GetVersion, Export and the whole sync path untouched — and a partial redaction is worse than none, because it reads as safety. Nor can it be done in the serializer: hub.JsonSerializerOptions is the same instance the storage adapters persist with, so a converter that dropped apiKey would blank it in Postgres.
Ciphertext at rest ✅ Holds on every surface at once — MCP get, the REST read, the sync stream, export, catalog download, version history — without enumerating any of them, because the master key is deployment configuration and never node data.

The rule, stated once: a secret's administered home may be a node; its VALUE may never be what a read returns. On this platform that is enforced by encryption, not by an access rule and not by a projection.

[Browsable(false)] on ModelProviderConfiguration.ApiKey is not part of this. It suppresses the GUI property grid and nothing else — its own doc comment says the value still serializes for the factory and for persistence. A GUI that stops rendering a field is not a read boundary.

The invariant, and how it is kept

ModelProvider.apiKey is enc:-tagged at rest. Always, on every path.

Two halves, and refusing to write plaintext only ever covered the first:

  1. Nothing new writes plaintext. Protect THROWS with no master key rather than passing the value through — the passthrough is what put a live OpenRouter key in cleartext into production on 2026-08-24. NoPlaintextCredentialFallbackGuard fails the build on the protector?.Protect(k) ?? k shape.
  2. What is ALREADY stored converges. Refusing future writes says nothing about the past, and the platform wrote plenty before the refusal existed: on memex.meshweaver.cloud, Provider/Azure and Provider/AzureFoundry have carried the same key in the clear since 2026-08-03/04. So ProviderCredentialSeed's second pass encrypts a stored plaintext key in place — the same credential, a different storage form, so Unprotect returns exactly what it returned before and the resolver's next read serves the same key.

That second pass walks every provider node in the catalog, whether or not a catalog source still names it. The qualifier is load-bearing: the duplicate Provider/Azure node is precisely one a source-driven pass walks past, and duplicates are why a rotation's blast radius is not fixed but grows with the duplication.

Its boundary is explicit. With no master key the pass can neither repair the node nor pretend it is fine, so it logs at Error naming the exposure and leaves the credential working — a repair that took chat down to fix a storage form would be worse than the exposure, and the actual answer to an exposure is a rotation.

🚨 A key that has been exposed is rotated — encrypting it is not the remedy

Re-protecting repairs the storage defect. It does not undo the window in which the value was readable. Rotate any provider key that has been stored in the clear on a publicly-readable partition, on the exposure window alone — evidence of use is not the test. The re-protect pass logs at Warning saying exactly this, per node, so an operator finds it rather than having to know.

Thinking level — a per-model datum, sent on every call

A reasoning model's provider decides how long every round thinks before its first token, and that default is the provider's, not ours. z-ai/glm-5.3 on OpenRouter ships with reasoning always on at max; an Executive Assistant round of nine cheap tool calls took 3 min 11 s on it, 74 s of thinking before the first tool call, four times over — and the platform had no knob at all. The composer's "effort" reaches the CLI harnesses only; nothing sent a reasoning effort to a provider.

The knob is reasoningEffort on the LanguageModel node — none, low, medium, high or max (aliases minimal, xhigh). The round reads it through ChatClientCredentialResolver.ReasoningEffortFor and sets ChatOptions.Reasoning once, in the one place the round builds its options, so every provider call of the round — including the tool-loop iterations the function invoker makes — carries it; the OpenAI adapter puts it on the wire as reasoning_effort, which OpenRouter forwards to the model. Absent, nothing is sent and the provider's default stays in force, which is exactly the pre-field behaviour. An unknown spelling parses to "send nothing" and never throws, for the reason tier gives: a typo on a node costs the setting, never the model.

Tuning a model down is therefore a mesh edit, not a deploy:

patch @Provider/OpenRouter/z-ai/glm-5.3  {"content":{"reasoningEffort":"low"}}

Provider support varies: a model without a reasoning phase ignores the field; one whose reasoning cannot be disabled treats none as its lowest level. Pinned by ReasoningEffortRoundTest (node → request) and ReasoningEffortWireTest (request → wire).

What is still open

This page describes the seam. Three pieces of the administered surface are not built yet, and they are independent of everything above:

Two known gaps in the invariant itself, both narrower than the above and both stated here so they are not rediscovered:

Reconnecting…
The server was updated. Reloading the page to pick up the latest version.