Enabling the operator — the executed runbook

Steps 1–5 of this page were executed live on 2026-08-22 against the production cluster and committed back the same session. It exists so the next enablement is a checklist, not an archaeology dig. The generic halves live in the platform repo (deploy/aks/INSTANCE-LIFECYCLE.md, deploy/aks/manifests/hosting-operator/README.md); this page records what an ACTUAL enablement looks like, with the real names, and what only doing it reveals.

🚨 A merged enablement is not an enabled operator — step 6 is the enablement. The values PR (Memex#86) merged 2026-08-22 13:24Z; the last Helm release run was 05:21Z the same day — eight hours EARLIER. Nothing red anywhere, five green checklist items, and on 2026-08-23 a Suspend action still failed at Launch operator job with "instance actions are disabled on this installation", because the Hosting__Operator__* env had never reached a running container. Config that is merged but not rolled is the same silent state as a gate that skipped: the evidence it did not happen is the absence of evidence. Do not tick this runbook off until the verification probe below passes.

Why an operator at all

The mesh half of instance lifecycle (records, requests, plans, phases) is complete and unit-tested — but a lifecycle run legitimately needs az, kubectl, helm and pg_dump, and the portal image carries none of them, on purpose. So every run executes as a short-lived Kubernetes Job in memex-ops under its own identity. Three principals, deliberately separate:

The checklist (as executed)

1. Identity + backup store — backups.bicep deployed to memex-aks-rg: storage memexdbbackups (ZRS, 365d lifecycle), identity hosting-operator (client 280fd351-…), federated to system:serviceaccount:memex-ops:hosting-operator.
2. Grants — for BOTH instances' resources, not just the control instance's: DNS Zone Contributor on meshweaver.cloud and systemorph.com; Key Vault Secrets Officer on Systemorph and memex; plus the bicep's own Postgres Contributor and backup-account Blob Data Contributor.
3. RBAC — kubectl apply of manifests/hosting-operator/ (client-id annotation edited first).
4. Image — az acr build of hosting-operator:1.0.0 with build context deploy/ (the image carries the chart; building from operator/ produces one that fails at "Deploy release").
5. Chart values — the hostingOperator: block in the control instance's PUBLIC overlay (values.memex.public.yaml, Memex#86). 🚨 These render into the ConfigMap (Hosting__Operator__*) — they are NOT free-form config entries; a key set anywhere else reaches no container and every action refuses as "disabled on this installation". 🚨 Since 2026-09-08 the configuration is the FALLBACK, not the input. Every run reads the Deployment record's operator block first — image, namespace, service account, environment — and the Hosting__Operator__* configuration only where the record is silent (HostingOperator.ResolveIdentity). Measured the other way round the same day: Memex#255 moved operator.image to hosting-operator:00dc0e8 (record v50) and the next reconcile (memex-reconcile-20260908-ci8131, 20:23Z) still ran the old image and failed at the exact step the new image fixes, because a ConfigMap reaches a pod only at a helm deploy. An operator fix is now a record edit and the next action. Hosting__Operator__Enabled stays configuration-only: whether this installation may run actions at all is not any record's call.
6. Token mount + THE ROLL — the jobrunner token volume in portal-patch.json, then the Helm release workflow (deploy) rolls the portal. This step is the enablement: everything before it edits repos and cloud resources; only this run puts Hosting__Operator__Enabled=true and the token mount into a running pod. It can itself be triggered from the mesh — requestedAction: HelmRelease + helmAction: deploy on the control instance's own record — because a HelmRelease action dispatches the config repo's workflow under the fleet's GitHub App credential and does not need the operator that is being enabled.

Verify it is actually live

The checklist having been executed is not evidence the config reached a container — values land in a ConfigMap only when the release rolls, and a portal that later rolled by self-update keeps whatever env its Deployment spec had. The only honest probe is a real action — ⚠️ NOT a dry run: a dryRun: true plan deliberately ends at "Dry run — nothing changed" BEFORE the Launch operator job phase, so it proves nothing about the operator. A real Backup is the harmless probe — its worst case is that it works and you gain a verified dump:

{ "id": "operator-probe", "namespace": "Deployments", "name": "Operator probe",
  "nodeType": "Hosting/InstanceAction",
  "content": { "$type": "InstanceActionContent",
    "deployment": "Deployments/<any-recorded-instance>",
    "requestedAction": "Backup", "backupStore": "<a recorded store>",
    "reason": "probe: is the operator live?" } }

Read the Launch operator job phase of its log. Started and beyond → live. "instance actions are disabled on this installation" → step 6 has not landed, whatever the repos say. (That is exactly how the gap above surfaced: a real Suspend on 2026-08-23 refused at that phase while every repo read as done.)

The two defects only a live enablement found

1. The jobrunner Role had no RoleBinding (MeshWeaver#2070). A Role without a binding is furniture: the portal's token 403s on its first CreateJob, and from the mesh that reads as a misconfigured operator, not a missing binding.
2. The jobrunner SA + token Secret were unmountable by construction. They shipped in memex-ops, but a pod can only mount Secrets from its OWN namespace. They belong in the portal's namespace; the RoleBinding in memex-ops names the foreign SA — which is exactly what scopes the portal-held credential to jobs-only power there while staying mountable here. Verified with SubjectAccessReviews: create-jobs true, delete-namespaces false.

Both are the same lesson as the CI invariants: a control surface nobody has exercised is not a control surface. Every green unit test passed while the whole thing was unlaunchable.

Who owns a run — the attribution rule

Every Hosting/InstanceAction run has an OWNER, and it is never a guess:

• The owner is the node's framework-stamped lastModifiedBy (else createdBy) — never a field on the content, which a requester could simply type.
• A destructive action (teardown, restore) with no named owner is refused — deliberately stricter than package provisioning, where an unattributed write is trusted infrastructure context. Installing a package unattributed is fine; dropping a database unattributed is not.
• A system-initiated run (the idle lifecycle: suspend after idleSuspendDays, teardown after idleTeardownDays) is owned by the System principal — never attributed to a human, for the same reason provisioning never grants its invoker ownership: attribution must not become entitlement, in either direction.

After enablement: the mesh IS the admin surface

From here, cluster administration goes through nodes on the control instance — Hosting/Deployment records as intent, Hosting/InstanceAction runs as execution (including HelmRelease with capture/adopt/deploy), Hosting/InstanceRequest for self-service. Raw kubectl/az remains a break-glass stopgap whose every use must land back in a repo the same session.