Deploying to AKS

This is one of two deploy routes for MeshWeaver. Use it for the shared portals on the AKS cluster <aks-cluster> (resource group <aks-resource-group>, region swedencentral) — the memex namespace, backed by the Postgres Flexible Server, with container images in ACR meshweaver.azurecr.io. For the Azure Container Apps route (Aspire test/prod modes via tools/deploy.sh), see DeploymentContainerApps.md. These are different routes to different targets, not old-vs-new — pick the one that matches where you're deploying.

🚨 This runbook is the bootstrap / break-glass form of a Roll. Since 2026-09-08 the rule is that operations go through the control instance's Hosting API: the instance is a Deployments/<name> record, its image pin is the roll, and a Roll (or Restart, Suspend, Audit, Reconcile) Hosting/InstanceAction is what the in-cluster operator executes — running exactly the commands below for you. Read them as what happens, not as what you type. Policy, and what the API does not answer yet: OperatingFromThePortal.

The cluster is private. kubectl is not reachable directly — where a break-glass command is unavoidable it runs through az aks command invoke -g <aks-resource-group> -n <aks-cluster> --command "…", which executes inside the cluster's API-server-side runner.

A code update is three steps: build the images, point the Deployments at the new tag, restart. It is not tools/deploy.sh and not aspire deploy — those are the Container Apps route.

Steady state is self-update, not this runbook. Once an environment runs, it rolls itself to new images per Admin/UpdatePolicy (default Stable — clean releases; Continuous + a pattern such as 3.0.0-ci* follows a line's continuous builds) — the portal patches its own Deployment from inside the pod. This manual runbook is the bootstrap / break-glass path (first install, or to force a specific tag). See ReleaseStrategy.md, which also covers the one-time RBAC + workload-identity (AcrPull) setup the in-pod updater needs.

1. Build + push the images

az acr login -n meshweaver

# Portal — needs the prebuilt custom base image. MULTI-ARCH: never pass `-r linux-x64`.
dotnet publish ../MeshWeaver.Plugins/src/Memex.Portal.Distributed/Memex.Portal.Distributed.csproj -c Release \
  --no-self-contained -t:PublishContainer -p:PublishProfile= \
  -p:ContainerRuntimeIdentifiers='"linux-x64;linux-arm64"' \
  -p:ContainerRegistry=meshweaver.azurecr.io \
  -p:ContainerRepository=memex-portal-ai -p:ContainerImageTag=<tag> \
  -p:ContainerBaseImage=meshweaver.azurecr.io/memex-portal-ai-base:latest

# Migration — this is what creates the schema, the partition_access table, AND the
# public.top_level_index materialized view. A schema/index change ships in THIS image:
dotnet publish <MeshWeaver.Plugins>/src/Memex.Database.Migration/Memex.Database.Migration.csproj -c Release -p:MeshWeaverRoot=<this checkout> \
  -t:PublishContainer -p:ContainerRegistry=meshweaver.azurecr.io \
  -p:ContainerRepository=memex-migration -p:ContainerImageTag=<tag>

🚨 The image must be MULTI-ARCH. memex-portal-ai-base:latest is built for linux/amd64 and linux/arm64 (x86 cloud nodes and Apple-silicon local k3s both pull it), so the app image layered on it must be too. Do not pass -r linux-x64 — a single RID produces a single-arch image, and a node of the other architecture gets an ImagePullBackOff that reads as "the deploy hung". Three details make the flags above non-obvious, all of them silent when wrong:

• -p:PublishProfile= (empty) is required to override the csproj's <PublishProfile>DefaultContainer</PublishProfile>.
• RuntimeIdentifiers is declared project-local in Memex.Portal.Distributed.csproj, never as a global -p: — a global one propagates to every ProjectReference and fails them with NETSDK1083.
• The '"a;b"' quoting (single-quoted double-quotes around a real ;) is load-bearing. Writing %3B instead gives MSBuild an escaped semicolon, so the value stays the single bogus RID linux-x64;linux-arm64 and the build fails with MSB4115 / NETSDK1083.

🚨 <tag> must be dotted SemVer (3.0.0, 3.0.0-ci.749) — a descriptive tag gets reverted. The in-pod self-updater only recognises tags matching ^\d+\.\d+\.\d+([-+].*)?$ (VersionSelect.PlatformVersionTag); a hand-picked bugfix-2026-06-05 is not a candidate, so the poller picks the newest CI tag instead and patches the Deployment straight back off your image — and it polls once immediately on pod start (StartWith(-1L)), so the revert lands within moments of the roll you just did. A hand-built image is therefore only usable with the updater paused (see "Hard pause" below); the normal way to ship code is a merged PR whose CI build produces a ci.<N> tag.

CI also builds images on push, but it lags — check az acr repository show-tags -n meshweaver --repository memex-portal-ai --orderby time_desc --top 5 before assuming your commit is built. If only portal code changed (no migration/schema change), you can reuse the live memex-migration tag and skip the migration build.

Business rules / scopes are NOT in the image — they ship as a plugin

The published image contains no MeshWeaver.BusinessRules assembly and no scope source generator. Business rules and scopes were removed from the platform (commit f7a8c086c); they ship as the MeshWeaver.Plugins/BusinessRules plugin, which carries the scope runtime as a shared-source library node (pulled into a consumer's compilation via shared=@BusinessRules/Scope/Source) and carries the ScopeCodeGenerator source for the generator-injection seam. The platform and the Doc partition start with zero business-rules dependency — stated in the comment blocks in MeshWeaver.Graph.csproj and Memex.Portal.Distributed.csproj, and asserted against the built image by PortalImageFacility.

Consequences for a deploy:

• An IScope<,> node needs the BusinessRules plugin installed on the target environment. It does not compile "just by declaring the interface" — neither the IScope<,> surface nor the generator is in the image's reference set.
• MeshNodeCompilationService.BuiltInGeneratorPaths is EMPTY on a deployed image. It only fills when a MeshWeaver.BusinessRules.Generator.dll happens to sit next to the app (a dev/self-host tree that put one there) — kept as graceful degradation, not as the shipping story.
• The legacy-#r strip is keyed off that same list, so on a deployed image it does nothing. StripBuiltInScopeGeneratorRef removes a legacy #r "nuget:MeshWeaver.BusinessRules.Generator" only when builtInPresent is true. On an image where it is false the #r survives and is resolved through NuGet. ⚠️ Unresolved: the XML doc on that method still describes the built-in generator as shipping with the platform, and warns that resolving the legacy #r hard-fails on a deployed image once the mesh-local feed was gone. Those two statements can no longer both hold. Treat a deployed node still carrying that #r as suspect and verify it compiles on the target environment rather than assuming it is filtered.

The dotnet publish -t:PublishContainer command above is self-contained — there is no pack step and no mesh-local feed. The former BakeMeshLocalFeed target and the CI "Pack mesh-local #r packages" step were both removed in #395; nothing resolves MeshWeaver.* from dist/packages during a deploy. nuget.config is still copied into the image (so a Code node's third-party #r "nuget:…" can resolve), and it still declares the mesh-local source plus a packageSourceMapping pinning MeshWeaver.*/Memex.* to it — the mapping is what stops a typo'd #r "nuget:MeshWeaver.X" pulling a same-named package from another publisher on nuget.org. The directory that source points at does not exist in the image.

2. Roll out (NS = memex)

Portal-only code update (no schema change):

az aks command invoke -g <aks-resource-group> -n <aks-cluster> --command "\
  kubectl -n <NS> set image deployment/memex-portal-deployment memex-portal=meshweaver.azurecr.io/memex-portal-ai:<tag>; \
  kubectl -n <NS> rollout restart deployment/memex-portal-deployment; \
  kubectl -n <NS> rollout status deployment/memex-portal-deployment --timeout=300s"

The portal container is memex-portal in Deployment memex-portal-deployment.

🚨 The migration is a run-once Job in the chart, NOT a Deployment — set image cannot roll it. Older copies of this runbook also passed deployment/memex-migration-deployment to set image and rollout restart; that step is removed above because it does not run a migration. Since 2026-09-03 the SELF-UPDATER also mints one (memex-migration-su-<tag>) before every automatic roll — see Database Migration Procedure for the routine, the one-time RBAC grant it needs, and the recovery when a pod refuses on db_version. deploy/helm/templates/memex-migration/job.yaml renders a batch/v1 Job named memex-migration-<Release.Revision> with restartPolicy: Never; a fresh one is created by helm upgrade, and ttlSecondsAfterFinished cleans it up. memex-migration-deployment is a legacy resource: per deploy/aks/SELF-UPDATE.md, live AKS clusters still carry it scaled to 0, so set image / rollout restart against it changes a pod spec that never runs, and on a chart-only environment the command simply 404s. Either way it is silent — the commands appear to succeed and no migration happens. A schema change therefore has to go out via helm upgrade (which mints a new Job), not via this set image roll-out. Confirm which of the two shapes the target namespace actually has before you rely on either (kubectl -n <NS> get deploy,job | grep migration).

3. Verify

• Migration ran: find the Job first — kubectl -n <NS> get jobs -l app.kubernetes.io/component=memex-migration — then kubectl -n <NS> logs job/memex-migration-<revision> --tail=40 → expect Database migration completed. Version: N. A Job that reports Complete with that line is the success signal. 🚨 A Job still Running past ten minutes is a FAILURE in progress, not a slow success: the run has a budget (migration.budgetMinutes, default 10 — the runner fails RED naming the step that outlived it, and activeDeadlineSeconds ends the Job one minute later). A deployment that needs more writes the number into its overlay explicitly; a step that needs more is rewritten as bulk work (one set-based statement per partition, never a request per row — measured 2026-09-07: a row-at-a-time embedding backfill held memex-cloud's Job 4 h 30 min with its deploy long since reported failed).
  • A CrashLoopBackOff on a migration Deployment is NOT benign. That is the legacy shape, and it is exactly the failure the Job replaced: the process exits 0, the Deployment restarts it, and every run rebuilds public.top_level_index across every partition schema. The chart records 310 restarts in a day pegging a full core. If you see it, the namespace is still on the legacy Deployment — do not wave it through.
• Portal serves: curl -sS -o /dev/null -w '%{http_code}' https://<portal-host>/ → 200. The host is not derivable from the namespace — namespace memex serves memex.systemorph.com (the DNS record deploy/aks/README.md → "Public ingress + TLS + DNS" creates), while memex.meshweaver.cloud is the memex-cloud namespace. Read the host off the namespace's own Ingress (kubectl -n <NS> get ingress -o wide) rather than templating it, or you will happily verify a portal you did not deploy to.
• Schema/index applied (when the change was a migration): spot-check via az aks command invoke … "kubectl -n <NS> exec deployment/memex-portal-deployment -- …" or an MCP query.

The cluster runs what the chart describes — check it, don't assume it

A green rollout says the pods came up. It says nothing about whether they came up with the configuration the chart declares. An image-tag roll does not apply a chart change — only a helm upgrade does — so a fix committed to the chart can sit unapplied indefinitely, and a setting applied by hand (kubectl set env) runs until the next helm upgrade silently deletes it. Four such divergences surfaced on one day, none of them detected by anything:

deploy/aks/scripts/check-chart-drift.sh -n <NS> -r <release> \
  -f <values.yaml> [-f <values.env.yaml>] \
  --via aks-invoke -g <aks-resource-group> --aks <aks-cluster> \
  --expect-patch deploy/aks/envs/<env>/portal-patch.json

It renders the chart, reads the live memex-portal-config and memex-portal-deployment plus the namespace's PodDisruptionBudget and ScaledObject, and reads the last-deployed release manifest (helm get manifest), then classifies every difference into the five classes below — COLLIDES and SHADOWS (an inline env: entry that overrides envFrom, so the value the chart supplies is dead or resolved at random per pod start), then CLUSTER-ONLY (hand-applied, and in no committed source), CHART-ONLY (described but never applied — nobody is getting it), and DIFFERS.

🚨 CLUSTER-ONLY does NOT mean "the next helm upgrade deletes it". This page said that until 2026-09-03 and it is false — measured, and contradicted four paragraphs below by this same page. Helm's three-way merge removes only what Helm previously owned, so drift applied out of band survives every deploy. Ranking the backlog by "what a deploy would destroy" ranked it by a hazard that does not exist and buried the two classes that are wrong now.

Except for one narrow half of it, which the check now computes. There are three sides here, not two: D the chart as CI renders it, L the live objects, M the release manifest. A key that is in L and in M but no longer in D is a chart retirement — helm owned it, so the merge removes it, and the next helm upgrade genuinely does delete it. Such a finding stays in the CLUSTER-ONLY class and reads PENDING DELETION, with a count in the summary line, and it states whether the live value is empty (the removal is a no-op) or not (a real removal) — the value decides that, not the key name. The other half now says helm never owned it rather than asking you to take "a helm upgrade PRESERVES it" on trust. The manifest is a required input: if it cannot be read the check fails RED, because with M missing every live-only key would read as the surviving half and the report would answer "nothing is about to be deleted" at the moment it could not tell.

Secret values are never printed, and neither are inline env values — but both are compared: an entry present on both the chart and the pod with a different value is reported as DIFFERS without printing either side.

The availability shape — spec.replicas, the budget, the autoscaler — is compared because without it the check cannot see an outage. On 2026-08-14 memex-cloud served every request from ONE pod, and none of the three reasons was a ConfigMap key: a hand-applied minAvailable: 2 budget (the chart renders maxUnavailable: 1) sitting at disruptionsAllowed: 0, and a ScaledObject annotated autoscaling.keda.sh/paused-replicas: "1" — which pins the replica count, deletes the HPA, and silently reverts kubectl scale. A live disruptionsAllowed: 0 is now reported on its own terms, whether or not the chart agrees with it: agreeing would not make it survivable.

Two things to know before you trust a green run:

• Render with the SAME -f list the deploy uses. Values files are git-ignored and live outside this repo (see "First-time environment setup"). A different -f list diffs against a chart nobody deployed, and every unset key reads as drift.
• --expect-patch is how a post-helm patch is declared. An env's deploy.sh applies portal-patch.json after helm upgrade (the CSI envFrom, extra volumes); pass it so those additions read as intentional. Anything cluster-only and not in that file is undeclared drift.

It fails RED when it cannot compare — an unreachable cluster, a failed render, a rendered ConfigMap with no keys, or an unreadable release manifest — rather than reporting "no drift" on no evidence.

Two callers, and neither may skip. The script's inputs are cluster credentials and the per-env values, so it cannot be a pull-request gate: a gate that skips when a credential is absent renders the same tick as one that passed, which would rebuild the very defect it catches. It runs from .github/workflows/chart-drift.yml on a daily schedule (plus workflow_dispatch) behind a preflight job that asserts every external input and fails red naming what to provision — never an if: that decides whether to run. Expect it red until the CI identity has AKS runCommand rights and each environment's SECRET-FREE values.<env>.public.yaml is committed to the private deploy repo; that red is the honest report that drift detection is not wired up yet.

Reading its report. The findings are ranked worst-first and the classes mean different things — and the two that matter most, COLLIDES and SHADOWS, are wrong on the cluster right now rather than at the next deploy. A helm upgrade neither deletes nor overwrites drift (measured; Helm's three-way merge removes only what Helm owns), so nothing on that list resolves itself. Chart Drift — what a deploy actually does has the measurement and the per-class triage.

Volume capacity is a record property — volumes[].size, applied by the operator

🚨 Never kubectl patch pvc … storage on a live portal. The size of every persistent volume is volumes[].size on the instance's Hosting/Deployment record, and the operator applies it:

What the command does, and refuses, is the whole contract (deploy/aks/operator/bin/hosting-pv-resize):

• it grows one claim to the declared size and reports the capacity it read back from the claim's status, never the request (::hosting:: pv_capacity=<quantity>, pv_resized=0|1);
• it never shrinks — a record that declares LESS than the claim holds is a wrong record, and the refusal says to correct the record to the measured capacity;
• it never creates — an absent claim is the chart's job (persistence.<name>.create on a record-driven Provision), so the refusal names the claim rather than provisioning one on a guess;
• it refuses a storage class without allowVolumeExpansion BEFORE writing anything (a patch on such a class sits in Pending forever). azurefile-memex allows it, and Azure Files expands online — no pod restart. A block volume whose filesystem resize waits for a pod re-mount is reported as ::hosting:: pv_resize=filesystem-pending, and the rollout that follows completes it;
• a claim already at size is a successful no-op, so the step is idempotent from the top.

Why this exists: on this fleet the portal's claims are NOT helm-managed (they were applied by hand once from portal-pvcs.yaml; helm upgrade never touches an object it does not own), so a bigger size on the record re-rendered a bigger number into the values file and changed nothing on the cluster. Measured 2026-09-08 13:51Z: memex-data in namespace memex was 16Gi with 3 MiB free while memex-cloud's ran 128Gi. The portal-pvcs.yaml captures in the config repo are descriptive; the record is what the operator applies.

The chart must also agree with ITSELF — check-chart-invariants.sh

Drift is only half of it. The memex-cloud outage above needed no cluster to detect: the chart in git described an impossibility. deploy/aks/values.aks.yaml asked for keda.minReplicas: 2, scaledobject.yaml rendered a floor of 2, pdb.yaml budgeted for two pods — and deployment.yaml hard-coded replicas: 1, so replicas.portal: 2 had sat in values consumed by nothing for a month. helm template emits that set happily, because helm validates syntax, not sense.

deploy/aks/scripts/check-chart-invariants.sh renders every values combination the repo ships and asserts the ones that matter: spec.replicas absent under KEDA (helm and the HPA must not both own it), a replica floor above 1 implying AdoNet clustering and ReadWriteMany on every portal claim, AdoNet implying a rendered ConnectionStrings__orleans, a budget using maxUnavailable rather than the not-scale-invariant minAvailable, a budget implying a floor above 1, and strategy.maxUnavailable: 0 under KEDA. Every input is in this repository, so it runs unconditionally on every pull request (.github/workflows/chart-gate.yml) — no secret to be absent means no condition under which it may decline to run.

Node drains — the disruption budget and where the replicas sit

An AKS node-pool operation (node image upgrade, OS patching, autoscaler consolidation) cordons a node and evicts its pods through the eviction API, one node after another. Two facts decide what the portal's users see while that happens:

• The eviction API honours a PodDisruptionBudget, and nothing else. The chart renders memex-portal-pdb with maxUnavailable: 1 whenever the replica floor is two or more — keda.minReplicas under autoscaling, replicas.portal otherwise. Under it a drain evicts one pod, waits until the Deployment has a replacement Ready, then evicts the next. cluster-autoscaler.kubernetes.io/safe-to-evict: "false" speaks only to the autoscaler's bin-packing; a drain ignores it.
• A budget cannot help when "the other pod" is on the same node. The pod template declares a preferred podAntiAffinity on kubernetes.io/hostname, so two replicas land on two nodes when two are schedulable and still schedule when only one is (a drained node's replacement has to land on the survivor).

🚨 Measured 2026-09-09 (MeshWeaver#3772): memex ran replicas.portal: 2 with no budget — pdb.yaml was gated on keda.enabled, and memex's helm-release lane renders the vault values plus values.memex.public.yaml, never deploy/aks/values.aks.yaml where KEDA is on. Both pods sat on one node; a drain at 01:05:01Z shut both down in the same second and the portal answered 503 until a replacement passed the startup gate (~90 s). Reading it needed the cluster-wide log — pods in three namespaces stopping together — because inside the memex namespace it looked like a rollout that changed nothing. check-chart-invariants.py now refuses a floor above one with no budget (invariant 14) or no spreading (invariant 15); before, it only asked whether a budget that exists has replicas under it.

Loki loses whatever its ingester had not flushed when loki-0 itself is drained (the 01:05Z lines were readable at 01:08Z and gone at 01:11Z) — read a cluster-wide incident promptly, and see the header of deploy/aks/scripts/values.observability.yaml for what the loki-stack chart can and cannot persist.

Self-update ops — pausing, pinning, and the rules that bite

Operational facts about the in-pod updater (learned the hard way — each cost a debugging session):

• 🚨 Check availability BEFORE a manual roll. kubectl set image bypasses the poller, and with it the release availability gate — so it can put an environment on a release whose content bake has not been published, and every pod then Roslyn-compiles the whole content set at boot (a type that fails to compile parks its hub for the full activation budget). Ask the target portal first:

  curl -s -H "Authorization: Bearer $MWI_KEY" \
    "https://<portal>/api/plugins/is-updatable?version=<tag>" | jq
  

  isUpdatable: false names the blocking packages in holdReason. indeterminate: true means the check itself could not run — an availability problem to fix, never a release to re-bake, and never clearance to proceed. If the portal is unreachable, .github/scripts/check-release-availability.sh <tag> asks the artifact store the same question directly (needs the storage role). A held update needs no manual roll anyway: publish the missing bake and the poller applies it on its next tick.

• Tags must be dotted SemVer (3.0.0 / 3.0.0-ci.749). VersionSelect.PickTarget keeps only tags matching ^\d+\.\d+\.\d+([-+].*)?$ (and drops per-RID suffixes like -linux-arm64), then picks the highest. It never inspects the tag you deployed — it compares its pick against the running build's stamped version (ShippedReleaseSeed.InstalledPlatformVersion). So a hand-built myfix-<sha> image whose build stamped an older version is simply overtaken: the poller finds the newest ci.<N> tag, judges it newer, and patches the Deployment off your image. Manual rolls therefore only stick with CI-built ci.<N> tags — ship code via a merged PR, or pause the updater first.

• Pause switch = the Admin/UpdatePolicy node: patch content.policy to None (Continuous+pattern/Stable/None). BUT a freshly booted pod races the policy read: CreatePolicySource emits the configured default (Stable since 2026-09-08) via StartWith before the node's live value arrives, and the poll timer fires immediately (StartWith(-1L)). The live None then switches the poller off, but a check may already have fired. None alone therefore does not reliably protect a roll that restarts the pod.

• Hard pause (break-glass, e.g. pinning a diagnostic image): delete the RoleBinding memex-portal-self-update (namespace-local; role + SA are both named per chart) — the updater's Deployment PATCH then fails closed. Recreate the RoleBinding to resume. Always restore promptly.

• KeyVault CSI env timing: a new/changed KV secret needs two rollout restarts — the first pod's mount populates the synced k8s Secret, but that pod's envFrom snapshot predates it; the second restart reads the populated Secret. Verify with printenv <key> | md5sum in the NEWEST pod (sort by creationTimestamp).

• 🚨 A workload that READS a synced Secret must MOUNT the class that feeds it. The driver fetches and rotates for the pods that mount the SecretProviderClass volume. A pod that only lists the synced Secret in envFrom is a free rider: it reads whatever some other pod's mount last wrote, and envFrom is resolved once, at container start. The migration Job was exactly that until #3548 — it mounted nothing and starts the instant helm upgrade applies, i.e. before the portal pods that own the rotation have rolled. So on the memex release of 2026-09-07, the deploy that repointed Embedding__ApiKey from the Azure object to the OpenRouter one ran the embedding backfill with the PREVIOUS key: 1,260 × HTTP 401, 1274 upserted (0 embedded), and Database migration completed — a green Job that authenticated with a stale credential and embedded nothing, while a portal pod started minutes later held the same key name and got 200s. The cure is structural, not a retry: a CSI mount is set up before ANY container in the pod starts and fetches from the vault at that moment, so a mounting pod's envFrom resolves against a freshly written Secret by construction. The chart now renders the volume + volumeMount for the Job from the same memex.keyVaultClasses block that renders its envFrom, and KeyVaultCsiFreshnessGuard fails any pod-bearing template that reads a class without mounting it.

• 🚨 Namespace ↔ instance mapping: this cluster hosts several instances whose Deployments all share names (memex-portal-deployment): namespace memex = the systemorph.com company portal, memex-cloud = memex.meshweaver.cloud (SPC <database>-portal-ai-secrets, KeyVault Systemorph, <database>--prefixed secret names), prod = the customer portal. Before ANY kubectl change, confirm the namespace matches the instance you mean — e.g. run a diagnostic on the target portal that prints its pod hostname and kubectl get pods -A | grep <hostname>.

Portal self-update — Workload Identity for ACR polling

Steady state is self-update (see ReleaseStrategy.md): the portal polls ACR and patches its own Deployment to a newer image. The in-cluster PATCH uses the memex-portal-sa service-account token (RBAC ships in the Helm chart and works everywhere). Listing the ACR tags to discover a newer image needs an Azure credential — that is wired with AKS Workload Identity, mirroring the existing pgBackRest wiring (deploy/aks/infra/modules/storage.bicep).

What the Helm chart already does (no edits needed): when selfUpdate.azureClientId is set it annotates memex-portal-sa with azure.workload.identity/client-id, labels the pod azure.workload.identity/use: "true", and sets AZURE_CLIENT_ID. The self-updater (AcrTagLister) then uses ManagedIdentityCredential(AZURE_CLIENT_ID) → AAD token → ACR token.

What the Azure side provides (deploy/aks/infra/modules/portal-identity.bicep, wired from deploy/aks/infra/main.bicep): a single shared user-assigned managed identity (<namePrefix>-portal-mi) with one federated credential per portal namespace — subject system:serviceaccount:<ns>:memex-portal-sa, issuer = the cluster OIDC issuer, audience api://AzureADTokenExchange — for every namespace in the portalNamespaces param (memex, memex-cloud, and any customer portal namespaces). The UAMI gets AcrPull on meshweaver.azurecr.io (AcrPull includes the metadata_read the tag-list call needs). One UAMI → one AcrPull grant → the same portalIdentityClientId wired into selfUpdate.azureClientId for every namespace.

One-time setup

1. Provision the UAMI + federated credentials — included in the infra deploy (deployPortalIdentity defaults true). Read the client id back:

   # --name is whatever the infra deploy was created with; deploy/aks/README.md uses memex-aks-infra.
   az deployment sub show --name memex-aks-infra-sc \
     --query "properties.outputs.{clientId:portalIdentityClientId.value, principalId:portalIdentityPrincipalId.value}" -o jsonc
   

2. Grant AcrPull on the shared registry. The ACR (meshweaver.azurecr.io, RG meshweaver-shared) is cross-RG from <aks-resource-group>, so — exactly like the cluster kubelet's AcrPull — grant it out-of-band:

   PORTAL_MI_OID=$(az identity show -g <aks-resource-group> -n <portal-identity> --query principalId -o tsv)
   az role assignment create --assignee-object-id "$PORTAL_MI_OID" --assignee-principal-type ServicePrincipal \
     --role AcrPull --scope $(az acr show -n meshweaver --query id -o tsv)
   

   (IaC alternative: deploy with grantSharedAcrPull=true — authors this via infra/modules/acr-role-assignment.bicep in the registry's RG; needs User Access Administrator on meshweaver-shared. A per-deployment ACR instead of the shared one is granted in-bicep automatically.)
3. Set selfUpdate.azureClientId to portalIdentityClientId for each environment (the in-pod patch works without it; this only authenticates the tag-list). Same value everywhere:
   • memex → the git-ignored values.deploy.yaml in the staging dir (template: deploy/aks/scripts/values.deploy.example.yaml), or helm upgrade --set selfUpdate.azureClientId=<clientId>.
   • memex-cloud / customer portals → the git-ignored deploy/aks/envs/<env>/values.<env>.yaml.

Adding a new portal namespace? It needs its own federated credential on the shared UAMI — add the namespace to portalNamespaces and re-run the infra deploy (idempotent), or az identity federated-credential create (see OnboardingNewEnvironment.md). The subject must be exactly system:serviceaccount:<ns>:memex-portal-sa.

Migration under self-update

When an install rolls itself to a new tag (per Admin/UpdatePolicy), the in-pod updater patches exactly one workload: memex-portal-deployment (container memex-portal). It does not touch the migration, and it says so in its own success line:

[SelfUpdate] patched memex-portal-deployment to <tag>. The database migration is a helm-run Job (memex-migration-<revision>) and is NOT rolled from here; DbVersionGate refuses to serve if this image is ahead of the schema.

This section used to say the updater patched two Deployments — the portal and memex-migration-deployment — and that the second 404'd and threw, poisoning the verdict of the portal roll that had already succeeded. That was true until #2797 and is not true now. The second PATCH is gone, and a guard fails the build if anyone re-adds one — SelfUpdatePatchesOnlyPatchableWorkloadsGuard, in MeshWeaver.Plugins (src/Memex.Hosts.Test/), which is also where the updater itself lives (src/MeshWeaver.SelfUpdate.Aks/KubernetesDeploymentUpdater.cs). Neither is in THIS repo, so git grep here finds only this page. Left recorded because the stale text sent at least one investigation looking for a 404 that no longer happens.

🚨 The gap is not that the migration leg is broken — it is that there is no migration leg at all. A self-update moves the image and nothing else, so a release that bumps the schema is, by construction, un-takeable by self-update. CD does build and push a correctly-tagged memex-migration image on every run (tag-for-tag with memex-portal-ai); nothing in the continuous path ever runs it. main-cd.yml says as much: "a migration that can never run surfaces only at the deploy that needs it." Only helm upgrade mints the Job.

This is a property of the fleet, not of one namespace, and the remedy is an OPEN decision. An install rolls itself forward until it meets its first schema-bumping release and stops there; an install that has not stopped has not arrived yet, not been configured differently (memex-cloud served ci.7621 healthily the same day for exactly that reason). Clearing one instance at one tag clears today and re-arms for the next bump. Why that matters for the control instance, what the three candidate remedies trade, and how to pick a target when an operator does carry an install across, are on The Self-Update Schema Wall.

What that looks like when it fires (memex, 2026-09-03 — MeshWeaver#3207): self-update rolled the portal to three successive builds needing db_version 55 against a database at 54. Each new pod hit DbVersionGate, logged Critical, and exited; the ReplicaSet never went Ready and the rollout recorded ProgressDeadlineExceeded. The cost was a pod crash-looping on a 5-minute back-off, each attempt writing a ~685 MB core dump.

🚨 Read the exit code carefully if you meet this. The container reports exitCode 139, which reads as SIGSEGV — but the dump records NT_SIGINFO … signo 6, i.e. SIGABRT. It is a managed fail-closed shutdown, not a native crash, and the OrleansProvisioningGate cancellation and Hosting failed to start that follow are the shutdown's own noise rather than the fault.

The service stayed up, and that is not luck. maxUnavailable: 0 with maxSurge: 1 kept the previous pods serving while the new ReplicaSet failed to become Ready, so a fail-closed boot became a stalled rollout instead of an outage. Verify that setting before any roll — it is the only reason this class of failure is recoverable rather than an incident.

So a portal stuck behind its schema is a SAFE state, and the fix is not a hand roll. Rolls go through CD; a hand-pinned tag is what the roll-via-CD directive exists to prevent. Where the gate must hold a roll, the decision belongs BEFORE the portal is patched — that is the shape that guard prescribes, and re-attempting a migration PATCH after the portal has already rolled is the defect #2797 removed.

🚨 And where a helm upgrade genuinely is the sanctioned path, choosing its target is a separate problem with its own trap. A tag whose three platform images are all promoted and all pullable can still have no plugin modules for its framework identity — Promote: tag the full set and Verify every image shipped both go green in that state, and the job that discriminates is Plugins: bake + seal the publication for this identity. Check the seal, the ancestry of the fix you need, and a memex-migration image at the SAME tag, before naming any build a target: The Self-Update Schema Wall → "What makes a tag a safe target".

Startup ordering — what actually happens if the portal outruns the schema. Two mechanisms exist, and only the first is a wait:

• the portal pod's wait-for-postgres initContainer genuinely blocks startup until Postgres accepts TCP connections; and
• the portal's DbVersionGate hosted service does a one-shot check at startup, and does not wait. It reads admin.mesh_nodes.db_version once and, if it is below the ExpectedDbVersion constant compiled into the build, logs Critical and calls lifetime.StopApplication().

So a portal that starts against an un-migrated database fails closed and exits — it does not hold and then go live. Kubernetes restarts it, and that pod recovers only once something else has bumped db_version.

Whether the NAMESPACE still serves depends entirely on the rollout strategy, and this page used to say flatly that it does not. On a rolling update with maxUnavailable: 0 and maxSurge: 1 — what the chart ships and what memex ran on 2026-09-03 — the previous ReplicaSet keeps serving because the new pod never becomes Ready, so the failure is a stalled rollout (ProgressDeadlineExceeded) and not an outage. Lose that setting, or hit this on a fresh install / a full restart where there is no healthy ReplicaSet to fall back on, and the namespace genuinely has no serving portal. The gate protects the database from a half-migrated portal; it does not make the ordering safe on its own, and the rollout strategy is what decides whether "unsafe ordering" costs you a stalled rollout or an outage. Treat "the migration ran" as a precondition you verify, not one the roll guarantees.

The migration Job IS the evidence — so it must outlive the observer

helm upgrade mints memex-migration-<revision>, and that Job object is the only durable record that this revision's migration ran. Memex's helm-release.yml reads it directly: its Observe the rollout step polls kubectl get deploy memex-portal-deployment and kubectl get job memex-migration-<revision>, and reports DONE only when every replica is updated and available at the new generation and the Job succeeded.

🚨 ttlSecondsAfterFinished reaps a FINISHED Job whether it Completed or FAILED, and at the chart's original 600 s that happened well inside the reader's own window — 25 minutes inside a deploy, 40 on an action: observe re-run, with the lane's error message explicitly telling the operator to "re-run with action=observe" later. Measured 2026-09-07 on the live cluster:

An absent Job is indistinguishable from a Job that never existed, and a migration that failed reaps exactly the same way. The chart now sets ttlSecondsAfterFinished: 86400 so the artefact outlives every window that reads it. That is one half; the other is the verdict refusing to call an absent Job a success (Memex#188). Neither half alone is enough — a longer TTL without the verdict change only moves the cliff, and the verdict change without the longer TTL makes a legitimate observe re-run permanently red.

A memex-cloud migration is measured in HOURS, not minutes, and that is not a deadlock. memex-migration-28 ran 4 h 47 m on 2026-09-07 and was healthy throughout: the tail of its log is [EmbeddingBackfill] <schema>: N embedded, walking ~137 partition schemas alphabetically. The portal serves normally while it runs (2/2 Ready on 3.0.0-ci.8009), because DbVersionGate gates on the schema version, not on the backfill. So "the rollout finished" and "the Job finished" are two different clocks in that namespace, minutes apart from hours — do not read a still-running migration Job as a stuck deploy.

Secrets the chart renders that no in-cluster Postgres backs

postgres.enabled: false on every AKS namespace — the mesh lives on the Azure Postgres flexible server and is reached through ConnectionStrings__memex. There is no memex-postgres-statefulset and no memex-postgres-service in either production namespace (measured 2026-09-07: "No resources found" in both).

Two keys in memex-portal-secrets / memex-migration-secrets address that absent Service, and until 2026-09-07 they rendered there unconditionally (Memex#204). Read by base64 length only — never by value:

🚨 The URI is the instructive half. Its template default fires on the empty password and derives a value, so the key is present, non-empty, plausible and wrong — it survives a keys[] audit and the base64-length audit that exists because keys[] is insufficient. The shape that catches it is neither: it is a per-key statement of whether empty is legal, plus an expected form.

Both are now gated on postgres.enabled (or on a value supplied explicitly, so nothing an operator sets is ever silently dropped). Nothing read either key — swept across MeshWeaver and MeshWeaver.Plugins, all *.cs: every match in either repository is a manifest that sets it (compose ×4, ACA bicep ×2, helm ×2). They still render for compose, local k3s and e2e, where an in-cluster Postgres genuinely exists and the same values are correct.

The general rule, which is the reusable part: a secret that reads as configured and is not fails at connect time with a credentials error rather than at startup with "not configured", and the second is the one an operator can act on at 3am. Prefer absent to empty, and never let a default manufacture a plausible value out of an unconfigured input.

Key Vault secrets are DECLARED in values — keyVaultSecrets

Since 2026-08-30 the chart renders the SecretProviderClass itself, from one block in the values file, together with the three halves the pod needs (the CSI volume, its mount, the envFrom on the synced Secret):

keyVaultSecrets:
  vaultName: "Systemorph"
  tenantId: "<entra tenant id>"
  identityClientId: "<the Key Vault Secrets Provider add-on's user-assigned identity client id>"
  name: "memexcloud-portal-ai-secrets"       # the SecretProviderClass; blank ⇒ memex-portal-keyvault
  syncedSecret: ""                            # the synced k8s Secret; blank ⇒ the same name
  secrets:
    - vaultSecret: memexcloud-Email-ClientSecret          # the vault OBJECT'S name — never a value
      key: Email__ClientSecret                            # the env key it lands as

Empty secrets renders nothing, so an environment that has not opted in is byte-identical to before. With any entry the vault, tenant and identity are required and a half-declared block fails helm template naming the key. Names only: no value is ever in values or in the render.

An instance that reads more than one vault-secret set — memex runs the hand-made memex-kv alongside the chart-owned memex-portal-keyvault — declares the rest under keyVaultSecretClasses, a list of the same shape. Order is precedence, and one vault object may serve several keys: that is how one credential lands under both PluginCatalog__RegistryToken and PluginCatalog__Registries__0__Token. Why a single slot was a data-loss bug, and the other two layers a record could not previously see, are in Deployment env layers.

🚨 Why it moved into the chart. Until then every SecretProviderClass in the fleet was a hand-made object — kubectl apply-ed once from a laptop, present in no repository, rendered by nothing — and the values file could only point at it by name (extraEnvFrom / extraVolumes, now the legacy escape hatch). Which vault objects a pod carried was knowable only from the cluster. On 2026-08-30 the memex install crashed at boot with EmailConfigurationGuard (Email:Enabled=true, Email:ClientId unset): its Email configuration existed TWICE on the live pod — the chart's ConfigMap rendering the defaults, and a hand-made Secret patched onto the live Deployment as explicit env entries in a different letter case — and .NET's case-insensitive environment provider let enumeration order pick the winner on every pod start. A coin toss per boot, invisible from any file.

The fleet's records drive this block: a Hosting/Deployment record's keyVaultSecrets section is projected onto it by the Hosting plugin's HelmValues, with the vault name derived from the key by the convention <keyVaultSecretPrefix><Section>-<Key> when the record leaves it blank. The operator's page for the whole path — record → render → chart → ConfigMap / SecretProviderClass / synced Secret → env — is Hosting/Configuration in MeshWeaver.Plugins.

Two things that do not change: a declared object the vault does not hold still fails the whole mount (the pod stays ContainerCreating, the old pod keeps serving, the rollout stalls) — create the vault secret before declaring it; and a new or changed vault value still needs the pod to restart (see "KeyVault CSI env timing" above).

First-time environment setup ≠ code update

deploy/aks/envs/<env>/deploy.sh provisions a new environment: helm upgrade --install of the chart, PVCs, the Key Vault SecretProviderClass (legacy hand-made shape — a new environment declares its secrets under keyVaultSecrets in values instead), ingress, and the connection-string patch. Do not run it for a code update — it re-applies the whole chart and can reset live ConfigMaps (e.g. the email config). Use it only when standing up a brand-new namespace.

Only the reference env deploy/aks/envs/example/ is in this repo; per-tenant env directories are git-ignored (.gitignore: "directory names are tenant identities and must not enter this public repo"), so the real ones live outside it. Copy example/ as the template.

Note the tension with §2: because the chart's migration is a Job created per Helm revision, helm upgrade is also the only in-repo path that runs a migration. A schema change consequently needs this script (or a bare helm upgrade) even though a plain code update must not use it.

Diagnostics — through the memex API first

🚨 Maintainer directive, 2026-09-08: operations and diagnostics go through the memex API, not through az/kubectl. The control instance's Hosting module answers the two everyday questions as nodes, with no cluster credential on the caller:

Both observations read the cluster's monitoring stack (kube-state-metrics via Prometheus, Loki) from inside the cluster, where it is credential-free; the roll runs as the in-cluster operator Job. The fleet guide (get @Hosting/Guide, "Roll, restart, observe") carries the full table. What follows is the break-glass form, for when the control plane itself is what is broken.

Diagnostics (private cluster — break glass)

• Logs: az aks command invoke … --command "kubectl -n <NS> logs deployment/memex-portal-deployment --tail=120". Note: the Azure CLI can crash on non-ASCII (→) in log output on Windows (cp1252) — pipe through tr -cd '\11\12\15\40-\176' inside the --command so az only receives printable text.
• Intermittent hangs while most requests succeed (portal recently synced or baked): suspect a degraded-but-Ready replica, not a global wedge — after startup, readiness and liveness both watch the light /alive, so a GC-bound pod never leaves rotation on its own. Run az aks command invoke … --command "kubectl top pods -n <NS> --no-headers"; one or two pods far above their siblings in BOTH memory and CPU is the superseded-NodeType-build (ALC) accumulation of issue #2194 — kubectl delete pod the outliers (grace-drain; the Deployment replaces them). Mechanism and the convergence key that prevents it: NodeTypeCompilation.
• A MESHWEAVER_MSG_TRACE=1 env var on the portal Deployment turns on the message-flow trace (/tmp/meshweaver-msg-trace.log in the pod). Toggling it restarts the pod; remove it (kubectl set env … MESHWEAVER_MSG_TRACE-) when done — it writes per-message and adds lock/IO overhead.

For Azure AD app registration and secrets (shared across both routes), see Deployment.md.