Where every value lives

Every configuration value an instance runs with has exactly one home, and from that home it travels one path to the pod. If you cannot name the home of a value you are looking at on a live pod, it is drift, and the next helm upgrade will either delete it or — worse — leave it racing the rendered copy. This page is the map: read the table, follow the path, and never edit a stop along the way instead of the home.

The path

Hosting/Deployment record            Systemorph/Memex  mesh/Deployments/<instance>.json   ← THE home (git, reviewed)
        │  HelmValues.Render (the Hosting plugin; pure, tested in Deployment/Test)
        ▼
values.<release>.public.yaml         Systemorph/Memex  deployments/aks/<env>/               ← RENDERED, committed, never hand-edited
        │  helm upgrade  -f <vault half>  -f values.<release>.public.yaml  --set portal.image=<running>
        ▼
chart templates                      Systemorph/MeshWeaver  deploy/helm/templates/memex-portal/
        │
        ├─ config.yaml               → ConfigMap  memex-portal-config      → envFrom → env  (non-secret keys)
        ├─ secrets.yaml              → Secret     memex-portal-secrets     → envFrom → env  (the captured vault half)
        ├─ secretproviderclass.yaml  → SecretProviderClass → CSI mount → synced Secret → envFrom → env  (Key Vault secrets, by NAME)
        ├─ deployment.yaml           → the pod: image, resources, volumes, the CSI mount, the envFrom order, probes, drain
        └─ ingress.yaml / pdb / scaledobject / rbac / serviceaccount

Three files, three repos, one direction. The record is the only thing a human edits; the rendered values file is a diff of intent (its header says so); the chart is code reviewed in the platform repo. helm get values <release> on the cluster must equal the layering of the vault half and the committed file — the drift gate (deploy-drift.yml in Systemorph/Memex) says when it does not.

One home per kind of value

Two rules of the renderer that decide what a blank means:

• Null renders nothing; a stated empty string renders "". An unstated property writes no key, so the vault half (or the chart default) decides; "" is how a record turns a sign-in scheme or a provider OFF. A bool always renders, so a false never falls back to somebody else's true.
• A keyVaultSecrets block with no secrets renders nothing, and so does the chart for it — an environment that has not declared its secrets is byte-identical to before.
• Keys are one key in ANY letter case. .NET folds environment keys case-insensitively — which is exactly what made EMAIL__ENABLED and Email__Enabled race on 2026-08-30 — so the renderer treats EMAIL__CLIENTID, Email__ClientId and a typed field of that name as the same key: a declared secret that the ConfigMap also renders, two extras differing only by case, or an extra naming a typed field in another case are all named problems, never a second source.
• A record with a named problem does not render. HelmValues.Problems is the preflight and HelmValues.Render refuses on it, listing every problem — and Render is the one call the launch path makes, so an InstanceAction fails at Launch operator job naming what to fix before anything reaches helm. The setup dialog's review shows the same list.

Log levels: the one value whose home is the IMAGE, and Information is an ALLOW-LIST

Every other row above travels record → overlay → ConfigMap → env. Log levels do not. The deployed portal's verbosity is set entirely by src/Memex.Portal.Distributed/appsettings.json in MeshWeaver.Plugins — the image's own file. Neither mesh/Deployments/*.json nor deployments/aks/*/values.*.public.yaml carries a Logging__* key on any instance, and memex-portal-config on both memex and memex-cloud holds none either (measured 2026-09-07). That is deliberate: what an operator must be able to read is the same on every deployment, so putting it in the overlay would mean every new instance silently starts blind.

And the shape of that file is not "a default plus a few caps" — it is the reverse. Default, MeshWeaver, Memex, Microsoft, Orleans, Azure, Npgsql and System are all Warning, and a short, explicit list of categories is raised back to Information. So the Information channel is an allow-list, and a category that is not on it is invisible — not quiet, invisible, with nothing to grep and no error anywhere saying so.

Measured on one memex-cloud replica's complete retained log, 2026-09-07, counting <level>: <category> headers:

Two rules follow, and both were learned the expensive way (Memex#233, whose two silent categories were StoreManifestSource and MeshWeaver.GitSync.GitHubAppTokenService):

• An un-dotted category inherits Default ONLY. ILoggerFactory.CreateLogger("Foo") gets a name with no namespace, so no prefix entry can ever reach it — it is Warning until it is named literally. The prefix rule is longest-match, so naming the TYPE (MeshWeaver.GitSync.GitHubAppTokenService) raises exactly that one logger and leaves its chatty siblings where they are; raising the namespace would take all of them.
• A LogInformation an operator is expected to read is not finished until its category is on the list, in the SAME change. The call and the filter are two halves of one decision, and the half that is missing is the one nobody can see. Memex#165 spent four days on three wrong root causes, and every discriminator that finally settled it was a count on a category like these — the Error half was reaching stdout to be counted against, and the Information half was not.

🚨 The class is bigger than the two categories that prompted this. Measured 2026-09-07 over this repo: 43 distinct un-dotted logger categories are created with ILoggerFactory.CreateLogger("Name") — BatchBake*, PreWarm*, every *ControlPlane, PlatformBuildInboxWatcher, Migration, TornSnapshot, StoreManifestSource — and none of them appeared in the image's Information allow-list. Each inherits Default and nothing else, so every Information line any of them writes is invisible on a deployed portal today. That is not a reason to raise all 43: each one owes the volume argument the entries above make, and a blanket raise is the cost model changing by accident in the other direction. It is a reason to know the number, and to check yours before assuming a channel exists:

grep -rhon 'CreateLogger("[^".]*")' --include='*.cs' . | sed 's/.*CreateLogger("//; s/")//' | sort -u

An obvious next step, not built: a gate that fails when an un-dotted category with a LogInformation call is neither on the list nor in a reasoned allow file — the ratchet shape this repo already uses for record-overlay.allow and friends.

To turn verbosity up for a debugging session, edit the appsettings.json in the test's or host's bin/ output (reloadOnChange: true flips it mid-run). The committed file is the production cost model — Information ships to Loki — so a change there is a permanent decision that states its volume, which is what every comment in that file does.

Secrets: the Key Vault naming rule

The pod reads a secret as the env key Section__Key. Its vault object is named by the fleet rule

<keyVaultSecretPrefix><Section>-<Key>          every "__" becomes "-", behind the record's prefix
Email__ClientSecret                      ↔     memexcloud-Email-ClientSecret          (memex-cloud, prefix memexcloud-)
Authentication__Microsoft__ClientSecret  ↔     memexsystemorph-Authentication-Microsoft-ClientSecret   (memex, prefix memexsystemorph-)

So a record following the rule names each secret once, by key:

"keyVaultSecrets": {
  "tenantId": "3a01d7ac-3330-444d-942d-975eb491b5d6",
  "identityClientId": "<the Key Vault Secrets Provider add-on's user-assigned identity client id>",
  "secrets": [
    { "key": "Email__ClientSecret" },
    { "key": "Authentication__Microsoft__ClientSecret" },
    { "key": "GitHub__App__PrivateKey", "vaultSecret": "github-app-privatekey" }
  ]
}

vaultName falls back to the record's keyVault; name (the SecretProviderClass) falls back to memex-portal-keyvault, syncedSecret to that name, volumeName to kv-secrets, mountPath to /mnt/secrets-store. An object that does not follow the rule — memex's unprefixed legacy names such as github-app-privatekey or ai-keyprotection-masterkey — states vaultSecret explicitly. HelmValues.VaultSecretName is the derivation; the rendered file always shows the resolved name, so "which vault object does this pod read for X" is answered by reading the file.

identityClientId is the identity that reads the vault — the AKS Key Vault Secrets Provider add-on's user-assigned identity (az aks show -g <rg> -n <cluster> --query addonProfiles.azureKeyvaultSecretsProvider.identity.clientId -o tsv), which needs Key Vault Secrets User on the vault. It is an identifier, committed on purpose.

Three facts that do not change because the declaration moved into the chart:

1. A declared object the vault does not hold fails the whole mount. The new pod stays ContainerCreating, the old pod keeps serving, the rollout stalls and nobody is paged. Put the value in the vault before the record names it: az keyvault secret set --vault-name Systemorph --name memexsystemorph-Email-ClientSecret --value '…'.
2. A new or changed vault value reaches env only on a pod restart, and the first restart may read the old synced Secret — the mount of pod A populates the Secret pod B's envFrom reads (DeploymentAKS → "KeyVault CSI env timing"). Verify in the NEWEST pod, by hash, never by value.
3. Never kubectl exec … printenv to check a secret. It prints values into your terminal and into CI logs. Compare NAMES: the record, the rendered keyVaultSecrets, the SecretProviderClass's secretObjects, the Deployment's envFrom.

Adopting a hand-made SecretProviderClass (memex-cloud's memexcloud-portal-ai-secrets, memex's memex-kv → memex-kv-secrets): state its live name and syncedSecret in keyVaultSecrets, list every object it maps, delete the matching secretMounts entry in the same change (a mount naming the same volume, class or Secret is a named problem), re-render, run helm-release adopt so helm stamps ownership on the existing objects, then deploy. The pod keeps reading the same Secret name throughout, so the switch is a metadata change followed by one ordinary roll.

Why NOTHING may live only on the cluster — 2026-08-30

The memex install (memex.systemorph.com, namespace memex) crashed at boot with EmailConfigurationGuard: Email:Enabled=true but Email:ClientId unset. Its Email configuration existed twice on the live pod:

.NET's environment configuration provider is case-insensitive, so Email__Enabled and EMAIL__ENABLED are one key, and which copy won was decided by enumeration order — per process start. Some boots saw the hand-made Enabled=true and the chart's empty ClientId; the guard refused, correctly. Nothing in any repository could see either half, because the SecretProviderClass objects were hand-made too.

This was the third time the same shape bit, not the first:

• 2026-08-23 — the memex-cloud CSI volume, mount and envFrom were live kubectl patches; a helm upgrade would have dropped them and detached every AI provider key (hence extraEnvFrom).
• 2026-08-25, MeshWeaver#2210 — the GitHub App identity committed one section off; every deploy rendered the keys "" and helm applied the blanking as a deletion, fleet-wide.
• MeshWeaver#2235 — a hand-set env: on the live Deployment permanently shadows whatever the ConfigMap renders for the same name, which is why webhookInboxTargets states both slots.

So the rule, with no exceptions and no "just this once":

No kubectl set env. No kubectl patch of the Deployment. No hand-made Secret, ConfigMap or SecretProviderClass. A value the pod needs is a record field (or, for a value, a vault object under the rule's name). If the record has no field for it, extraPortalConfig is the temporary home and a typed field is the follow-up — never the cluster.

When someone breaks glass anyway, the Break glass section of Guide.md applies: the half-done operation gets a same-session reconcile — record repo-first, an action node as the audit, the status sample corrected. A live-only change is reverted by the next roll; a record that contradicts the cluster is a lie the fleet board repeats.

How to add a new key

A non-secret key (Section__Key), in dependency order — platform first:

1. Chart (Systemorph/MeshWeaver): add the default under config.memex_portal in deploy/helm/values.yaml AND the line in templates/memex-portal/config.yaml. Two gates hold this pair together: check-values-are-read.sh fails a values key no template reads, and check-config-key-coverage.py (run by the Memex repo's Config key coverage workflow) fails an overlay key the chart renders nowhere.
2. Record (this plugin): a property on the right section in Deployment/Source/InstanceShape.cs (or DeploymentContent.cs), with [Description] and [Translation("de", …)] — every field is a form label.
3. Renderer: HelmValues.PortalConfig emits it; add it to HelmValues.TypedPortalKeys so an extra cannot shadow it; pin it in Deployment/Test/HelmValuesTests.cs and register the case in DeploymentTestsArea. Merge the Plugins PR; the registry serves it.
4. Record change (Systemorph/Memex): set the field on mesh/Deployments/<instance>.json, re-render values.<release>.public.yaml, PR. The blank-guard in helm-release deploy refuses a render that would EMPTY a key that is non-empty live — if that fires, the key is mis-nested, not the guard wrong.
5. Deploy — requestedAction: HelmRelease, helmAction: deploy on the record (or the workflow by hand). Merged is not rolled: values reach a container only when the release rolls.

A secret:

1. Put the value in the vault under the rule's name (az keyvault secret set …). Before step 2, always — a declared object the vault lacks stalls the rollout.
2. Add { "key": "Section__Key" } to keyVaultSecrets.secrets on the record (with vaultSecret only when the name breaks the rule). Re-render, PR, deploy.
3. Expect the value in env after the roll; if the first pod read a stale synced Secret, one more restart. Verify by name and by hash in the newest pod.

Until the record has a typed field for a secret's use (e.g. email.enabled for Email__ClientSecret), declaring the secret alone renders a key nothing reads — harmless, but turn the feature on in the same change so the pair is reviewed together.

capture, adopt, deploy — what each is for

All three are helm-release.yml in Systemorph/Memex, dispatched from the record (requestedAction: HelmRelease + helmAction) or by hand; confirm must repeat the release name for the two that mutate.

The image is a no-op of a deploy by design: the in-portal self-updater owns that field the rest of the time (pinnedImageTag blank), and inputs.image is the explicit opt-in to roll.

Where the operator runs

• The control instance is memex (memex.systemorph.com, namespace memex, helm release memex, vault Systemorph, prefix memexsystemorph-). The Deployments Space there holds the records, two-way GitSynced to Systemorph/Memex → mesh/Deployments.
• Instance actions (Hosting/InstanceAction: provision, suspend, back up, tear down) run as short-lived Kubernetes Jobs in memex-ops under the hosting-operator identity, launched by the control instance's portal with the job-runner token the chart mounts when the record's operator.enabled is true. Off everywhere else, as a security property. The executed runbook and the probe that proves it is live: OperatorEnablement.md.
• HelmRelease actions need no operator — they dispatch the config repo's workflow under the fleet's GitHub App credential, which is how the operator itself was enabled.
• The names are crossed — the namespace called memex is systemorph.com and the vault called Systemorph serves meshweaver.cloud. Confirm against the RECORD (get @Deployments/<name>), never against a name. README.md carries the full table.

What the two records carry today

memex-cloud (mesh/Deployments/memex-cloud.json) already has the email section and a secretMounts entry pointing at the hand-made memexcloud-portal-ai-secrets. To finish: move the class into a declaration and drop the mount —

"keyVaultSecrets": {
  "tenantId": "3a01d7ac-3330-444d-942d-975eb491b5d6",
  "identityClientId": "<csi add-on identity client id>",
  "name": "memexcloud-portal-ai-secrets",
  "secrets": [ { "key": "Email__ClientSecret" }, "…every object the live class maps, by key…" ]
},
"secretMounts": []

memex (mesh/Deployments/memex.json) had no email section at all — the whole section lived in the hand-made Secret. It needs the section AND the declaration; the vault object memexsystemorph-Email-ClientSecret must exist first:

"email": {
  "enabled": true,
  "clientId": "<the Graph sender app's client id>",
  "tenantId": "3a01d7ac-3330-444d-942d-975eb491b5d6",
  "mailboxAddress": "<the shared mailbox>",
  "useManagedIdentity": false,
  "inboundEnabled": true,
  "webhookBaseUrl": "https://memex.systemorph.com"
},
"keyVaultSecrets": {
  "tenantId": "3a01d7ac-3330-444d-942d-975eb491b5d6",
  "identityClientId": "<csi add-on identity client id>",
  "name": "memex-kv",
  "syncedSecret": "memex-kv-secrets",
  "volumeName": "kv-secrets",
  "mountPath": "/mnt/kv-secrets",
  "secrets": [
    { "key": "Email__ClientSecret" },
    { "key": "ConnectionStrings__memex", "vaultSecret": "memex-connectionstring" },
    "…every object the live memex-kv class maps; the unprefixed legacy names state vaultSecret…"
  ]
},
"secretMounts": []

and, on the cluster, the hand-made memex-email-secret and the explicit EMAIL__* env on the live Deployment are deleted — the next deploy renders the pod spec from the record, and a kubectl patch that survives it is the coin toss again. helm-release adopt first (the class memex-kv becomes helm-owned), then deploy.