Deployment env layers: what a record must be able to hold

A Hosting/Deployment record is supposed to be the ONE source of an instance's deployment: change the record, re-render, deploy. That claim is only as good as the record's ability to describe what the instance actually runs — and on 2026-09-06 it could not. Re-rendering memex from its own record and diffing the result structurally against the deployed overlay gave 41 differences, and every one of them was a silent loss. This page records the five layers, what each one is, and the record shape that can hold all of them.

The five layers, lowest precedence first

A portal pod's configuration is assembled from five sources. Kubernetes keeps the last envFrom entry when two supply the same key, and an inline env: entry outranks every envFrom. So the list is a precedence order, and the last source that carries a key is the value the portal reads.

# Layer Rendered by Recorded as
1 ConfigMap memex-portal-config the chart, from config.memex_portal the typed fields + extraPortalConfig
2 Secret memex-portal-secrets the chart, from the Key Vault values half vaultValuesKeys (names only)
3… Secret per SecretProviderClass the chart, from keyVaultSecrets / keyVaultSecretClasses keyVaultSecrets + keyVaultSecretClasses
last inline env: on the Deployment nothing — kubectl put it there inlineEnv (declarative)

Layers 2 and 5 are the ones a record could not previously see at all. Layer 2 comes from a values file captured into Key Vault (helm-values-memex), which no repository holds; layer 5 comes from somebody's kubectl set env, which no file renders. HelmValues.EnvPrecedence folds all five into one answer per key: every layer that supplies it, and therefore which one wins.

Why one Key Vault class was not enough

Until this change the record had exactly one slot for a SecretProviderClass (keyVaultSecrets), plus a legacy escape hatch that could only point at a hand-made one by name. memex runs two classes:

With one slot the record had to CHOOSE, and whichever it named, a re-render destroyed the other. It named memex-kv while the deployed overlay named memex-portal-keyvault, so a re-render would have:

  1. deleted both PluginCatalog mappings — the fix that had made the registry poll work that same morning after 740 consecutive 401s;
  2. dropped extraEnvFrom: memex-kv-secrets, detaching all 13 keys of the hand-made class from the pod;
  3. added a mapping for a vault object that does not exist (memexsystemorph-PluginCatalog-RegistryToken, derived from the record's prefix) — and a declared object the vault does not hold fails the whole CSI mount, so every new pod stays in ContainerCreating and the rollout stalls.

None of that is visible from the record; it is a data-loss bug wearing the shape of a formatting change.

One vault object may serve several keys

The projection used to refuse maps vault object X onto more than one key — state distinct vault objects. That rule was wrong, and it refused the shape that fixes the registry poll: memex reads one credential under two names because two code paths look for two names — PluginCatalog__RegistryToken (the legacy single-registry key) and PluginCatalog__Registries__0__Token (the per-registry key of the named registry) — both from the one vault object PluginCatalog-RegistryToken. Nothing about that is ambiguous: both land, both are read, and rotating the object rotates both. A rotation writes THAT object — the one the declaring class names — never the one the prefix rule would derive: memex's prefix is memexsystemorph-, and memexsystemorph-PluginCatalog-RegistryToken is an object nothing reads (Registry-key rotation).

The same rule was silently losing a key on memex-cloud, where one object (memexcloud-AzureAIS-ApiKey) has served both AzureAIS__ApiKey and AzureFoundry__ApiKey since before the record existed: the record could hold only one of the two.

The correct rule is the other way round. A KEY has exactly one home — two sources for one key means envFrom order alone decides what the portal reads, which is the 2026-08-30 EmailConfigurationGuard crash re-created. An OBJECT may have as many keys as the portal reads it under. HelmValues.Problems now checks the first, across classes as well as within one, and permits the second.

Recording an inline env: entry does not create one

inlineEnv and vaultValuesKeys render nothing, and that is the point.

An inline entry is out-of-band by construction: the chart never emits one, and helm upgrade does not remove one either — three-way merge removes only what helm previously owned. That was measured on helm v3.21.1 and v4.2.4 with a positive control; see Chart Drift — what a deploy actually does. So an inline entry survives every deploy and keeps outranking the ConfigMap and every synced Secret.

Recording one therefore neither creates nor deletes it. What recording buys is that the record stops silently disagreeing with the pod — and once the known, deliberate entries are declared, anything left over is a surprise rather than noise. Each entry says what it stands over (shadows), whether that source is known to agree (agreesWithShadowed), why it is still there, and what retires it.

Three things the fleet's own entries show, none of which was written down anywhere before:

Never a credential's value

These records sync to a git repository. An inline entry that carries a credential — both portals hold a plugin-registry instance key inline, in plaintext, readable by anything that can get deploy — sets isSecret and leaves value unset. Stating both is refused by HelmValues.Problems, so the one shape that would put a live token into a git-synced node fails the render instead of shipping.

Retiring a shadow takes two steps, and the SECOND one is what changes the pod

Recording a shadow does not clear it. Clearing one is two acts in a fixed order, and either alone leaves the pod exactly where it was:

  1. give the key a declared home — a keyVaultSecrets mapping, so the vault-synced Secret carries it. This changes nothing observable: the inline entry still outranks every envFrom.
  2. remove the inline entry — mark the record's inlineEnv entry retiredBy and run Reconcile; its RetireInlineEnv remedy removes the key from every container (see "Step 2 is a Reconcile" below). This is the step that changes which value the pod reads, and it rolls the Deployment. Until 2026-09-11 it was a break-glass kubectl -n <ns> set env deployment/<name> <KEY>-.

The trap is doing step 2 first, or doing step 2 without measuring what the pod falls through to. PluginCatalog__RegistryToken is the worked example (MeshWeaver#3201). Its inline entry stood over a chart Secret carrying a different, also-valid instance key — a key registered to another instance, with strictly more scope. Removing the inline entry before step 1 would not have restored a status quo; it would have silently switched the portal onto a different registered identity on one portal and left the other with no key at all.

So the precondition for step 2 is a measured EQUALITY, not a hope. Before removing an inline credential entry, compare it against the source it will fall through to and require the answer EQUAL:

🚨 Be precise about where the value travels — this is the part that is easy to state wrongly. kubectl is a client: both the inline env: value (read from the Deployment spec) and the Secret data cross the Kubernetes API to wherever kubectl runs. What the snippet below buys is that neither value is ever printed, logged or persisted — only a verdict and a length are emitted. That is a real and worthwhile property, and it is a different one from "the value stayed in the cluster".

The in-cluster claim is true only of the route this cluster actually allows — and that route is break-glass: the record's inlineEnv / extraPortalConfig / key-vault classes on the control instance plus an Audit Hosting/InstanceAction are the API-side answer to "what does the pod actually run" (OperatingFromThePortal). It is private, so kubectl is reachable only through az aks command invoke, which uploads the script, runs it in a pod on the cluster, and returns that pod's stdout. Run that way the credential is read API-server-side and only the verdict crosses back. Run the same snippet from a laptop and the credential lands in that laptop's shell process — same commands, different boundary. Say which one you used.

# Run through `az aks command invoke -f eq.sh --command "sh eq.sh"`, so the read happens in-cluster
# and only these verdict lines come back. Nothing here prints a value under any route.
inline() { kubectl -n "$1" get deploy <deployment> -o go-template='{{range .spec.template.spec.containers}}{{if eq .name "<container>"}}{{range .env}}{{if eq .name "<KEY>"}}{{.value}}{{end}}{{end}}{{end}}{{end}}'; }
sec()    { kubectl -n "$1" get secret "$2" -o go-template="{{index .data \"$3\"}}" | base64 -d; }
A=$(inline <ns>); B=$(sec <ns> <synced-secret> <KEY>)
[ "$A" = "$B" ] && echo "EQUAL (len ${#A})" || echo "DIFFER (a=${#A} b=${#B})"

DIFFER means step 1 is not done — promote the in-use value into the vault first, never the other one, and never mint a replacement as part of a cleanup.

The remedy runs this same comparison itself, in-cluster, immediately before it writes, and refuses on DIFFER — so a recorded agreesWithShadowed: true that has gone stale cannot switch a portal onto another credential; it produces a refusal. What no InstanceAction offers yet is a read-only equality measurement, so recording agreesWithShadowed for a new credential entry still takes the break-glass read above.

Three further things this key showed, each of which generalises:

🚨 Step 2 rolls the Deployment, so it is subject to whatever else is rolling. Removing an entry — the remedy's patch or a break-glass set env alike — mutates the pod template, which creates a new ReplicaSet and supersedes an in-flight rollout. Read kubectl rollout status first and hold if a deploy is already in progress — a cleanup that ejects a release roll costs more than the shadow it clears. hosting-inline-env-retire makes exactly that check itself and refuses mid-rollout.

Step 2 is a Reconcile — and why it needed a remedy of its own

Under the 2026-09-08 operating directive every operation is a Hosting/InstanceAction the control instance's operator executes in-cluster, and a cluster command is break-glass (OperatingFromThePortal). Until 2026-09-11, step 2 was the one act on this page that had no such action — and no configuration change could stand in for one. Four sources said so, and they agreed — measured 2026-09-10:

So until 2026-09-11 the audit named the drift and nothing could repair it, and kubectl -n <ns> set env deployment/<name> <KEY>- was the only instrument. Under rule 2 of OperatingFromThePortal an audit finding with no remedy is a gap to file against the Hosting package, not a recipe to promote back into a procedure — filed as MeshWeaver.Plugins#1593, and closed by the remedy the record's shape had anticipated all along: Reconcile now carries RepairRemedy.RetireInlineEnv, which reads InlineEnvOverride.RetiredBy and removes exactly the entries it names. The procedure — mark the entry retired, file a Reconcile, drop the entry once the audit is clean — is in OperatingFromThePortal → "Retiring an inline env: entry". It needs an operator image carrying hosting-inline-env-retire and the Hosting module version that plans it; until both are on the control instance, the break-glass line is still the only one that works.

Why this matters more than one duplicated variable. MeshWeaver#3201 has outlived three merged PRs. Every deferral until 2026-09-08 was about rollout timing — a fleet freeze, then the newly armed readiness gate (#3404, #3395), then the bake-gate stall (#3663). All three closed by 2026-09-08. What is left is not a schedule and not a risk: the last step's instrument is the one the operating model withdrew, and the entry's retiredBy had no lane to travel — until the remedy above gave it one.

What else the record gained

The chart half

keyVaultSecrets in values still holds one class, unchanged, so every existing environment renders byte-identically. Additional classes go in keyVaultSecretClasses, a list of the same shape; templates/memex-portal/_keyvault.tpl resolves both into one ordered list that the SecretProviderClass template, the envFrom, the volumes and the mounts all read, so the four can never disagree about a name.

On an entry in the plural list name, volumeName and mountPath are required: a second class falling back to the chart's defaults would collide with the first on all three at once, and two pod volumes of one name is an invalid spec while two classes syncing into one Secret race each other. syncedSecret still defaults to the class's own name, and the vault coordinates fall back to the singular block's — a namespace's classes normally read one vault with one add-on identity.

Two workloads, one secret set — and the escape hatch has to reach both

The portal Deployment is not the only pod that reads the environment's secrets. The migration Job reads them too, because MeshNodeEmbeddingBackfill — the only thing that embeds rows written before a provider existed — needs Embedding__ApiKey. So the invariant is not "the portal gets the keys", it is both workloads carry the same secret set, by whichever mechanism delivers it, and that has now failed twice in two different ways:

#3548 #3595
mechanism chart-owned class (keyVaultSecrets) hand-made class (extraEnvFrom)
what the Job had the value, from before the rotation no value at all
defect staleness absence
remedy the Job mounts the SPC, so the driver writes the Secret before the container starts the Job renders the escape hatch — env source, volume and mount

Both produced the same observable outcome, which is why neither was caught by watching the deploy: the backfill logs-and-skips per row rather than throwing, so the Job's exit code says nothing, and it reported Database migration completed in both cases. #3548's run authenticated 1,260 times with a stale credential; #3595's would have embedded nothing at all on an environment whose keys arrive through the escape hatch.

The escape hatch is three keys and they are a set. extraEnvFrom names a Secret; extraVolumes is the CSI volume behind it; extraVolumeMounts is the mount — and the mount is what makes the Secrets Store CSI driver fetch the vault objects and write that Secret. A pod that names the Secret without mounting the class free-rides on some other pod's mount, and envFrom is resolved once, at container start. Render all three or none.

Why the #3548 guard could not see #3595. KeyVaultCsiFreshnessGuard's original fact is keyed on $class.syncedSecret / $class.mountPath — the markers of a class the chart owns. A hand-made SecretProviderClass carries neither, so the guard was blind to the escape hatch by construction, and that blindness is measurable: removing the Job's extraEnvFrom render leaves the original fact green while the two facts added for #3595 both go red. Those two are:

Each asserts its own denominator, including that values.yaml still declares the three keys, so a rename there fails the guard instead of silently making it match nothing.

The falsification test

The shape is only worth having if rendering the record reproduces what is actually running. Rendering memex's extended record through HelmValues and then through helm template, and comparing the result against the live cluster objects, reproduces: the same two SecretProviderClass objects with the same vault, tenant and identity; all 2 + 13 object→key mappings in the same order, duplicates included; the same four envFrom sources in the same order; the same CSI volumes and mount paths; and the same three containers. memex-cloud reproduces the same way, including its one-object-two-keys AzureAIS mapping.

One entry does not reproduce, and it is a real defect rather than a modelling gap: both committed overlays declare an Embedding-ApiKey secret whose vault object does not exist. It is not on either live SecretProviderClass because it has not been deployed yet — and when it is, the missing object will fail the whole mount. The record states it because it states the overlay's intent; provisioning the vault secret is what unblocks the next deploy.

Reconnecting…
The server was updated. Reloading the page to pick up the latest version.