Deploying the Memex Portal to a Private AKS Cluster

This guide explains how to stand up the Memex portal on a private Azure Kubernetes Service cluster — with everything except the portal itself kept off the public internet. It covers what runs where, how each layer is provisioned, and how to operate it in production.

Conventions. Examples use placeholder names — domain portal.example.com, registry meshweaver, resource group <aks-resource-group>. Substitute your own values. Sensitive values (IP addresses, tenant/app GUIDs, passwords, client secrets) appear as <placeholders>; never commit real ones — keep them in Key Vault.

The exact, ordered command sequence lives in deploy/aks/README.md (in the repository) — the public, self-contained AKS deployment sample, which DEPLOY-RUNBOOK.md was folded into. This document is the architecture and operations layer around it.

Deployment model. One Aspire AppHost (deploy/aspire/Memex.Deploy.AppHost) describes the workload from published images. The Aspire Kubernetes publisher generates the Helm chart (deploy/helm). The AKS platform — cluster, Postgres, VPN, TLS — is Bicep plus a thin overlay.

1. Architecture at a Glance

The cluster has a single public surface: the portal on port 443. Everything else — the Kubernetes API server, Postgres, Grafana — stays private and reachable only over the P2S VPN.

Deployment topology: only the portal is public-facing; all data, admin, and observability stay on the private VNet, reachable only over the P2S VPN.

The ingress public IP is assigned by Azure. Retrieve it with kubectl get svc -n app-routing-system, then point your domain's A-record at it (see §5).

2. Images (Shared ACR)

These images are pushed to the shared ACR. Grant the AKS kubelet AcrPull on the registry (cross-RG if needed) so nodes can pull. Separately, the in-pod self-updater lists ACR tags under a portal Workload Identity (a shared UAMI federated to system:serviceaccount:<ns>:memex-portal-sa, granted AcrPull) — provisioned by deploy/aks/infra/modules/portal-identity.bicep and wired via selfUpdate.azureClientId. See DeploymentAKS → Portal self-update.

Build and push the portal (no Dockerfile — the SDK's PublishContainer pushes straight to the registry):

az acr login --name <registry>
dotnet publish ../MeshWeaver.Plugins/src/Memex.Portal.Distributed/Memex.Portal.Distributed.csproj \
  -c Release --no-self-contained -t:PublishContainer -p:PublishProfile= \
  -p:ContainerRuntimeIdentifiers='"linux-x64;linux-arm64"' \
  -p:ContainerRegistry=<registry>.azurecr.io -p:ContainerRepository=memex-portal-ai \
  -p:ContainerImageTag=<tag> -p:ContainerBaseImage=<registry>.azurecr.io/memex-portal-ai-base:latest
kubectl -n memex set image deployment/memex-portal-deployment memex-portal=<registry>.azurecr.io/memex-portal-ai:<tag>

The set image line is what a Roll Hosting/InstanceAction runs for you on the control instance; typing it yourself is break-glass (OperatingFromThePortal).

-r linux-x64 was removed deliberately. Pinning one RID builds a single-arch image; the other architecture then gets an ImagePullBackOff. Dropping -r and setting ContainerRuntimeIdentifiers makes the SDK publish per-RID and assemble an OCI image index. -p:PublishProfile= is required to override the csproj's DefaultContainer profile, and the '"a;b"' quoting must use a real ; (a %3B is an escaped literal and yields one bogus RID). This mirrors what main-cd.yml does. Prerequisite: memex-portal-ai-base:latest must itself be multi-arch, or the arm64 leg has no base layer.

Use a distinct tag per build (not :latest) so the rollout is guaranteed to pull the new image. On an environment running the self-updater, that tag must also be dotted SemVer — see DeploymentAKS.

3. Platform (Bicep)

deploy/aks/infra/main.bicep provisions the cluster, the VNet-injected Postgres Flexible Server, the VPN gateway, RWX storage, and (optionally) a per-deployment ACR. Set useSharedAcr=true to point at your shared registry instead.

Parameters live in deploy/aks/infra/main.parameters.json:

• Region and node size/count — stay within your vCPU quota.
• postgresHighAvailability — enable for production.
• gatewaySku — use an AZ SKU such as VpnGw1AZ.

The Postgres connection uses the server FQDN + password + SSL (Host=<pg-server>.postgres.database.azure.com;…;SslMode=Require;Trust Server Certificate=true). The servers moved to the FQDN on 2026-08-24.

🚨 This page previously said the opposite — "do not use the public FQDN form, a database.azure.com hostname routes the portal into its managed-identity-token branch" — and that would now talk an operator out of the correct configuration. It stopped being true with 9e468aa70: AzurePostgres.UsesManagedIdentityAuth (MeshWeaver.Plugins/src/MeshWeaver.Hosting.PostgreSql/AzurePostgres.cs) takes the Entra-token path only when the host carries the Azure suffix AND no password is present, so FQDN + password deliberately takes plain Npgsql. AzurePostgresAuthSelectionTests pins exactly that case.

4. Workload (Helm + Overlay)

deploy/aks/scripts/deploy.sh runs via az aks command invoke against the private cluster. In order, it:

1. Creates the namespace and RWX PVCs.
2. Runs helm upgrade --install with deploy/helm + values.aks.yaml + values.deploy.yaml.
3. Scales the chart's in-cluster Postgres to 0 (you use the Flexible Server).
4. Runs kubectl set image on the portal Deployment to the shared ACR.
5. Patches the portal to 1 replica (the Azure Files mounts render from the chart itself — persistence: in values.aks.yaml — so there is no volume patch any more).
6. Patches the connection-string secret to the external Postgres.

Known chart gaps (the chart renders from the Deployment record — ConfiguringAnInstanceFromAspire; it is not generated from Aspire, that direction is retired):

• The chart's secrets.yaml hardcodes the in-cluster Postgres connection string → deploy.sh patches it post-install.
• deploy.sh used to carry TWO commands against a memex-migration-deployment — a set image and a rollout restart — and only the first was guarded with || true; the second printed an error on every documented deploy. Both are gone (#1788): the chart renders the migration as a run-once Job (memex-migration-<Release.Revision>), so there is no such Deployment. The migration runs from the helm upgrade in step 2; override its image through the chart (--set migration.image=…), never with kubectl set image.

🚨 A migration CrashLoopBackOff is NOT harmless. Earlier revisions of this chart rendered the migration as a Deployment, which restarted the process after each clean exit 0. Every run rebuilds public.top_level_index across every partition schema, so this produced 310 restarts in a day pegging a full core — a CPU storm that previous versions of these docs described as benign. That is exactly why it is a Job now. If you see a migration Deployment crash-looping, the namespace is on the legacy shape; treat it as a live problem.

5. TLS, Ingress, and DNS

deploy/aks/scripts/tls.sh installs cert-manager, a Let's Encrypt ClusterIssuer (HTTP-01), and the portal ingress. HTTP→HTTPS redirect is automatic once TLS is active.

After the script runs:

1. Add a DNS A-record in your DNS zone pointing to the nginx LB public IP.
2. Verify Blazor Server's sticky sessions — the ingress sets a cookie-affinity cookie. Confirm the SignalR /_blazor WebSocket upgrade returns HTTP 101 through managed nginx.

To expose another private tool publicly (e.g. Grafana), create a second Ingress with the same ingressClassName + cert-manager.io/cluster-issuer annotation, its own host and service, and a matching A-record at the same ingress IP. Note that this gates the tool only by its own login — weigh that against the "only the portal is public" stance.

6. External Sign-In (OAuth)

Sign-in is the record's signIn block (AddMemex(…).WithSignIn(microsoftClientId: …, googleClientId: …)), rendered as the portal keys Authentication__<Provider>__ClientId/TenantId and Social__LinkedIn__ClientId by Helm and by Aspire alike; the client SECRETS are Key Vault names on the record (keyVaultSecrets) or WithSecret(…) under Aspire — never record fields (ConfiguringAnInstanceFromAspire). A provider is offered in the sign-in UI only when its ClientId is set.

Microsoft / Entra Register an app (<entra-app-client-id>) in your tenant (<tenant-guid>). Use single-tenant (AzureADMyOrg) for an internal portal. Set the redirect URI to https://<your-domain>/signin-microsoft. Also set Authentication__Provider=Custom and Authentication__EnableDevLogin=false.

Google / LinkedIn Create the OAuth apps (redirects /signin-google, /signin-linkedin) and supply ClientId/Secret to enable each provider.

Sign-in flow /auth/login?provider=Microsoft → the provider → /signin-microsoft (OIDC middleware signs the cookie) → /auth/callback/Microsoft (ExternalAuthController normalises claims; ObjectId = email) → /.

7. Onboarding and First Admin

OnboardingMiddleware (after UserContextMiddleware) intercepts an authenticated request whose email has no backing User node and redirects to /onboarding. UserOnboardingService then writes the partition-root User node + a User-catalog mirror, then grants self-Admin and (for the first user only) platform-Admin at Admin/_Access. All onboarding writes self-impersonate as System — PostPipeline fails closed without an identity context, and the user doesn't exist yet.

First-admin bootstrap (operator tool)

BootstrapController (POST/GET /bootstrap/first-admin) seeds the first admin server-side via the same UserOnboardingService write path. It is gated by the Bootstrap:Secret config value and disabled when that value is unset. Use it when the interactive /onboarding flow can't be driven, then unset the secret:

curl -sS "https://<your-domain>/bootstrap/first-admin?secret=<bootstrap-secret>&email=<admin-email>&username=<admin>"

8. Observability

deploy/aks/scripts/install-observability.sh installs the grafana/loki-stack chart (Grafana + Loki + Promtail + Prometheus) into the monitoring namespace.

• Promtail scrapes every pod's stdout into Loki — no portal-side configuration needed.
• OTLP traces/metrics: the record's telemetry block (WithTelemetry(otlpEndpoint)) renders OTEL_EXPORTER_OTLP_ENDPOINT (not needed for logs).
• Grafana defaults to ClusterIP (private). Reach it via the VPN (§9) + port-forward, or expose it publicly behind its own login (§5).

The observability stack is folded into the standard deploy: export GRAFANA_PW alongside MEMEX_PG_CONN and deploy.sh brings it up automatically.

9. Admin Access — The P2S VPN

Everything except the portal is private, so kubectl (private API server) and Grafana go through the point-to-site VPN — an AZ gateway SKU, OpenVPN + IKEv2, with a client address pool of your choice.

# A P2S root cert is uploaded to the gateway; the matching client cert lives in the operator's cert store.
az network vnet-gateway vpn-client generate -g <rg> -n <gateway> -o tsv   # download URL
# install + connect, then:
az aks get-credentials -g <rg> -n <cluster>
kubectl -n monitoring port-forward svc/loki-grafana 3000:80   # http://localhost:3000

az gotcha: Recent CLI versions read --public-cert-data as a file path — pass the path to a base64 file, not the inline string and not @file.

10. Operations

All admin commands target the private API server. Without the VPN, run them via az aks command invoke -g <rg> -n <cluster> --command "<kubectl…>". With the VPN active, plain kubectl works directly.

10.1 Update the Portal to a New Image

Build and push the image (§2), then repoint the deployment and wait for the rollout:

kubectl -n memex set image deployment/memex-portal-deployment memex-portal=<registry>.azurecr.io/memex-portal-ai:<tag>
kubectl -n memex rollout status deployment/memex-portal-deployment --timeout=220s

Use a fresh tag each build so the pull is guaranteed. Roll back by setting the previous tag.

10.2 View the Logs

• Quick (no VPN): az aks command invoke -g <rg> -n <cluster> --command "kubectl -n memex logs deployment/memex-portal-deployment --since=10m"
• Dashboard (Grafana, via P2S VPN — §9): kubectl -n monitoring port-forward svc/loki-grafana 3000:80 → http://localhost:3000 → Explore → {namespace="memex"} (add |= "error" to filter).

10.3 Enable / Configure AI Providers

Providers are gated by feature flags. Set them on the deployment once; each user then supplies their own credentials via Settings → Models.

kubectl -n memex set env deployment/memex-portal-deployment \
  Features__Ai__Providers__AzureFoundry=true Features__Ai__Providers__AzureOpenAI=true \
  Features__Ai__Providers__Anthropic=true   Features__Ai__Providers__OpenAI=true \
  Features__Ai__Clis__ClaudeCode=true       Features__Ai__Clis__Copilot=true \
  Ai__KeyProtection__MasterKey='<base64-32-byte-key>'    # encrypts stored provider credentials at rest

API providers (Anthropic, Azure OpenAI, Azure AI Foundry, OpenAI) work via bring-your-own-key — users add their endpoint + key per provider. The co-hosted CLIs (Claude Code, GitHub Copilot) require the per-user Connect flow (Phase 1 — see §11); the CLI binaries ship in the portal-ai image but the per-user login is not yet wired.

The master key should live in Key Vault, not as a plaintext env var — see §10.6.

10.4 Restart, Scale, and Inspect

kubectl -n memex rollout restart deployment/memex-portal-deployment   # clears in-memory caches + any wedged hub
kubectl -n memex scale deployment/memex-portal-deployment --replicas=1  # >1 needs Orleans AzureTables clustering (§11)
kubectl -n memex get pods -o wide

10.5 Postgres: Query and Reset

Postgres is private (VNet IP only). Run SQL via a throwaway pod inside the cluster:

kubectl -n memex run pg --restart=Never --rm -i --image=postgres:17 \
  --env=PGPASSWORD=<pw> --env=PGSSLMODE=require --command -- \
  psql -h <pg-private-ip> -U <admin> -d memex -c "SELECT count(*) FROM auth.mesh_nodes WHERE node_type='User';"

To reset to the post-initialize state, drop the per-user partition schemas and truncate content (keep admin.db_version), then restart the portal — direct SQL bypasses the workspace cache.

10.6 Secrets via Key Vault (CSI Secrets Store)

Production secrets live in Key Vault (access-policy mode) and are projected into the pod by the AKS CSI Secrets Store add-on — no plaintext env vars; the vault is the source of truth.

One-time wiring: grant the CSI add-on's identity get/list on the vault, store each secret, then create a SecretProviderClass that maps Key Vault secret names → env-var keys and syncs them into a k8s Secret the deployment mounts (CSI volume) and reads via envFrom:

# CSI identity object id: az aks show -g <rg> -n <cluster> --query addonProfiles.azureKeyvaultSecretsProvider.identity.objectId -o tsv
az keyvault set-policy -n <key-vault> --object-id <csi-identity-objectid> --secret-permissions get list
az keyvault secret set --vault-name <key-vault> --name ai-keyprotection-masterkey --value '<value>'   # dashes only in KV names
# SecretProviderClass `memex-kv` maps ai-keyprotection-masterkey -> Ai__KeyProtection__MasterKey (and the
# PG conn / Microsoft secret / Bootstrap secret) and syncs them into the `memex-kv-secrets` k8s Secret;
# the portal has a CSI volume for `memex-kv` + `envFrom: secretRef: memex-kv-secrets`.

To rotate a secret: az keyvault secret set (creates a new version) → kubectl -n memex rollout restart deployment/memex-portal-deployment (the CSI driver re-reads on the next mount).

The SecretProviderClass + the CSI volume/envFrom were applied post-deploy.sh by hand until 2026-08-30. The chart now renders all of them from the keyVaultSecrets values block (names only); see DeploymentAKS → "Key Vault secrets are DECLARED in values". The hand-made memex-kv object above is the shape the record adopts by stating its live names.

Windows az gotcha: the CLI's console writer cannot encode non-ASCII characters in cp1252 and crashes on a raw log dump. Pipe the cluster-side command through tr -cd '\11\12\15\40-\176' to strip non-ASCII before az prints it.

11. Known Issues and Follow-ups

Route-derived spurious partitions Visiting an auth-flow route (/onboarding, /login, /welcome) can create a same-named partition schema. The router then tries to activate that empty partition and its hub deadlocks (messages time out at 30s and retry). Fix: drop the spurious schemas and trace the code path that maps a route to a partition address.

Static/seed user shadowing onboarding If a static node provider seeds a User for the admin email, a fresh CreateUser fails with "Node already exists" and the interactive form shows "user exists" even with 0 DB users. Remove the seed so real onboarding can persist the partition root.

Secrets in Key Vault (done) The master key, PG connection string, Microsoft client secret, and Bootstrap:Secret live in Key Vault; a SecretProviderClass + the AKS CSI Secrets Store add-on sync them into a k8s Secret the portal reads via envFrom (see §10.6). The SecretProviderClass + CSI volume/envFrom are the record's keyVaultSecrets / keyVaultSecretClasses / vaultValuesKeys (rendered by the chart from the record — ConfiguringAnInstanceFromAspire), so a fresh deploy wires Key Vault from the record. Remaining: the Grafana admin password (monitoring namespace).

Multi-replica HA Needs Orleans AzureTables/AdoNet clustering wired on the Filesystem backend.

Migration as a Job (done) The chart now renders the migration as a run-once Job (deploy/helm/templates/memex-migration/job.yaml, restartPolicy: Never, named per Helm revision with ttlSecondsAfterFinished). Remaining: the in-pod self-updater and its RBAC still target a memex-migration-deployment, so a self-update does not run the migration — see DeploymentAKS → Migration under self-update.

Release image Replace any temporary debug image tag with a clean latest/release tag before treating the deployment as final.