Operating the fleet

Everything the Hosting plugin does follows one rule: a deployment is a RECORD, and changing one is a RUN. The record is intent, version-controlled and reviewable as a diff. The run is a Hosting/InstanceAction node that moves the world toward it, with phases, a log and an identity it ran as.

⚠️ Read this first: nothing here can run until the operator is wired. Instance actions are OFF by default. Until Hosting:Operator:Enabled=true plus an operator image, namespace and token exist on the control instance, every action REFUSES and names what is missing. That is deliberate — the identity these runs use can delete namespaces and drop databases — but it means a fresh installation can read every page here and execute nothing. See Before anything runs.

Start here: the Fleet Console

/Hosting/Console is this package's app — the page an operator opens first, and the entryPoint its tile lands on:

🛠️ The console REPORTS the fleet; it does not command it. It reads the Hosting/Admin node's readiness at the top of the page, so "instance actions are OFF here" is something you learn from a page rather than from a refused teardown.

Two lines the console is careful about, because both are silently wrong on most fleet boards: a suspended instance being down is the point, not an alert (health is judged only where the instance is meant to be running), and never sampled is not healthy — a blank sample time is its own answer and is labelled as one.

🗺️ Where does a value live, and how does it reach the pod? — Configuration.md. One home per kind of value; the path record → HelmValues → values.<release>.public.yaml → chart → ConfigMap / SecretProviderClass / synced Secret → env; the Key Vault naming rule; why nothing may live only on the cluster (the 2026-08-30 coin toss); how to add a key; what capture / adopt / deploy each do. Read it before touching a record, a values file, or a namespace.

The pages

Self-service: a business user asks for an instance

The Hosting/InstanceRequest node is the request form — over MCP or the GUI. The requester chooses what a business user can meaningfully choose; everything else is derived or cloned:

{ "id": "acme-request", "namespace": "Ops/Requests", "name": "Instance for ACME",
  "nodeType": "Hosting/InstanceRequest",
  "content": { "$type": "InstanceRequestContent",
    "name": "acme",
    "template": "Deployments/atioz",
    "modules": ["Essentials", "Edu"],
    "addOns": ["log-triage"],
    "idleSuspendDays": 30, "idleTeardownDays": 60,
    "reason": "pilot for ACME" } }

• name — a short lowercase label; it becomes the namespace, database, host and helm release.
• template — an existing Hosting/Deployment whose INFRASTRUCTURE coordinates are cloned (cluster, database server, key vault, image, DNS zone, backup store, config repo). The requester never types infrastructure, and the control plane never guesses it.
• modules + addOns — what the instance carries. Add-ons are curated package sets (log-triage = log ingestion + automatic issue triage); their packages merge into preInstall.
• idleSuspendDays / idleTeardownDays — the lifecycle: unused instances suspend, then go away (below).

A global administrator decides: set requestedAction: Approve (or Refuse + refusalReason). On approval the control plane composes the record, creates it, opens the write-through PR against the config repo (the mesh→repo half of the bijective record sync — the repo stays the source of truth), creates the provisioning Hosting/InstanceAction, and follows it to Live. The request node is the audit: who asked, who decided, what ran.

The setup dialog: the full new-instance configuration, in one place

The Fleet Console's Request page carries Set up a new instance… — a guided dialog for the operator-grade setup the quick form deliberately hides. Five sections, then a review:

1. Identity — name, subdomain, the template to clone, idle policy, reason.
2. Main database — server, database name, the Key Vault secret NAME for the connection string, and optionally the connection string itself.
3. Main storage — the storage account, its secret name, and optionally its connection string.
4. Boot modules — the module assemblies the instance must come up with (the database driver MeshWeaver.Hosting.PostgreSql, the AI engine MeshWeaver.AI, the MCP endpoint MeshWeaver.Mcp, the view packs). Rendered into Modules:Required, so a missing module STALLS the rollout instead of booting green without the feature. 🚨 A non-empty list is the COMPLETE required set — configuration arrays override the image's list BY INDEX, never append.
5. Packages & repos — the plugin repositories to mount (one per line, Name https://url [ref] [source]; source marks a repo this instance SERVES) and the packages a fresh boot seeds itself with (PluginCatalog:InstallByDefault, source-scoped).

The review step composes the record through the SAME InstanceComposition the approval uses, so what it previews — the field table the fleet's config visualizer shows for live instances, the vault secret names, the full boot configuration (PluginCatalog:* + Modules:Required:*) — is byte-for-byte what an approval writes. Every section may stay blank: blank means "the template's value", exactly like the quick form.

🔐 Where the connection strings go

The two connection-string fields are write-only, and three rules make them safe to offer:

• The Deployment record NEVER carries a value — it syncs to a git repository, so it records secret NAMES only ({prefix}db-connection, {prefix}storage-connection by convention; databaseConnectionSecret / storageConnectionSecret to override).
• The request node stores them ENCRYPTED or not at all. Filing runs the value through the platform key protector (IProviderKeyProtector, AES-256-GCM under Ai:KeyProtection:MasterKey) and refuses the whole filing when encryption is unavailable — a plaintext credential at rest is the provider-key incident, not a convenience. InstanceRequestContent.Refusal() enforces the same rule against hand-written MCP requests.
• Left blank, nothing is stored: the provision's hosting-kv-ensure step mints the vault secrets, or an operator sets them out of band (az keyvault secret set --vault-name … --name …).

One-click add-ons on a live instance

requestedAction: InstallAddOn + addOn: log-triage (+ the confirmation) on an Hosting/InstanceAction node extends the record's preInstall with the add-on's packages and re-runs the provision runbook — idempotent, so existing packages are untouched and the new ones land. The record change is written through to the config repo in the same run.

The instance goes away when nobody uses it

The record's idle policy drives the SAME lifecycle machinery a human would:

Fail-safe by construction: no policy → nothing; no recorded activity signal → nothing (a status sample must carry lastActivityAt — unknown is never idle); a suspension with no recorded start → nothing. The policy is a REQUESTER, not a bypass: it creates ordinary action nodes and every gate (backup answered, confirmation, grace period) still holds.

Roll, restart, observe — the ops actions

🚨 Every operation an operator needs is a Hosting/InstanceAction on the control instance — no az, no kubectl, no Loki curl, no cluster credential on the caller. The mesh is the surface AND the implementation: a mutating action runs as an operator Job in-cluster (the operator's service account and workload identity are the only credentials, and they never leave the cluster); a read-only observation runs in the control instance's own process against the cluster's monitoring stack. Maintainer directive, 2026-09-08: "pls shoot for doing all the operations through the memex api … no direct access of aks etc".

Roll, Restart and Reconcile carry the name-the-instance confirmation; Audit, Sample and Logs READ and ask nothing. None asks the backup question (no database is touched).

Rotating and revoking a registry key

Where it is requested. Like every instance action: a Hosting/InstanceAction node on the control instance (memex.systemorph.com), e.g. { "deployment": "memex-cloud", "requestedAction": "RotateRegistryKey", "confirmation": "memex-cloud" } — first with "dryRun": true, whose plan must show --registry-url and a Retire the previous key at the registry step.

How it reaches the registry. Not through the control instance. Every Memex portal hosts an IInstanceKeyRegistry, but the instances live only in the REGISTRY's store (memex.meshweaver.cloud); the first version of this action adopted the new key on the control instance's own store, after Key Vault had been written, and a failed adoption left the portal one restart away from a 401 on every catalog read (MeshWeaver#2802). The plan now names the registry the instance's key authenticates at — its one consumer pluginRepos mount, or its own host when the record is isPluginRegistry — and the operator job talks to THAT registry itself, authorised by possession of the instance's key:

1. hosting-kv-rotate reads the key the token's synced Secret carries and asks the registry (GET /api/instances/self) whether it accepts it, as the record's registryInstanceId. A 401 — the answer of a registry that does not hold the instance — a 404 (a registry too old to rotate), or a key of another instance: refused, nothing minted, Key Vault unchanged.
2. It mints a key, stages its hash at the registry (both keys now authenticate), proves the new key there, and only then writes Key Vault — the object the token's Key Vault class declares, not the prefix rule's name — and waits for the synced Secret.
3. The portal restarts onto it; the rollout is waited for.
4. hosting-registry-key commit presents the key the synced Secret carries — the one the restarted pods read — and only now does the registry retire the previous key.
5. hosting-verify.

Nothing before step 4 retires any key, so a run that stops anywhere leaves a portal that still authenticates; its failure line says what Key Vault holds and which keys the registry accepts. A run that stopped after the vault write is RESUMED by the next RotateRegistryKey (key_resumed=1), not repeated. The log facts of a clean run: registry_instance, key_hash, key_staged=1, kv_rotated=1, key_committed=1, verify.

Revoking. RevokeRegistryKey with revokeSecret and revokeSecretKey presents the key found there to the registry's revoke and reads it back as refused (key_revoked=1, or already for a key the registry no longer accepts). It refuses the token class's own Secret/key — the key the pods present is rotated, never revoked.

What must be deployed first — three halves, each refusing before anything is minted until all are there: the core image on the REGISTRY (it serves /api/instances/self*; an anonymous request answers 401, a 404 means not yet), the operator image on the control instance (it carries hosting-registry-key; the image is Hosting:Operator:Image, moved by memex's helm-release deploy, never by a record), and this module at ≥ 1.18 on the control instance. The design and the full runbook are on the platform page Doc/Architecture/RegistryKeyRotation.

🚨 …and something takes the observation WITHOUT being asked

Every action above is a request a person makes. Measured on the control instance 2026-09-10, that was the whole of fleet observation: Ops/Status/memex had 23 versions in 29 days, every one traceable to a person creating a Sample action or running refresh-status by hand, with gaps of 22 days and 6 days — and Ops/Issue held zero nodes, ever. Which is why a workload sat at desired 1 / ready 0 in two production namespaces for 2 d 22 h and nothing reported it (Systemorph/Memex#220).

FleetWatch is the sweep that runs anyway. Armed on Hosting/PlatformBuildInbox beside OperationalSpaceProvisioning — the same always-activated hub, for the same #777 reason — it samples every recorded deployment every 5 minutes through the SAME ObservationQueries the Sample action uses, writes Ops/Status/<id>, reconciles Ops/Issue/*, and raises a notification in the platform bell for every Critical condition that is new or whose evidence changed. A healthy fleet raises nothing; a deployment it could not READ raises Unobserved — Critical — because "I did not check" must never be deliverable as "checked and clean". Full reference: FleetWatch.md.

Before the first call: the space exists, and you may write to it

🚨 Measured 2026-09-08 on the control instance: Ops did not exist, no instance action had ever been created, and the maintainer's own identity was refused (Access denied: Create permission required) — the lane below had never run on production. The Hosting module now provisions the operational space itself: Hosting/PlatformBuilds (the always-activated inbox hub) seeds Ops and its containers at start and mirrors every Admin on Admin/_Access — the platform-admin definition — as an Admin on Ops/_Access. A global administrator can create an action through the API or MCP the first time; a newly appointed one is picked up on the next activation. No root _Access, no public grant: Ops holds the fleet's records, and only platform administrators and the control plane write there. If you are refused, read @Ops/_Access/* — your subject is missing from Admin/_Access, which is the thing to fix.

A roll is Done when the OLD process is gone

🚨 Measured 2026-09-08 on memex.systemorph.com: helm reported the 8059→8079 rollout a success at 10:34Z; twenty minutes later @PartnerRe/area/SettingsAbout still answered 3.0.0-ci.8059, no plugins are installed while @Admin/area/SettingsAbout answered 8079 with 63 plugins — a previous-generation pod was still a cluster member hosting hubs, and a process with no plugins registered renders every control as fallback text (the user-visible outage). "The new pods are Ready" is therefore not the outcome. Every action that cycles pods (Roll, Restart, Reconcile, Reactivate) now ends with Verify one generation: it samples every 30 s until the deployment is ONE generation (one ReplicaSet in kube_pod_info.created_by_name), nothing is terminating and every desired replica is ready — then records that sample — and REFUSES to say Done after 20 minutes, naming the previous-generation pods that are still members. A Sample shows the same facts on demand: per replica the phase, terminating, generation, and what the process says about itself on /health (version, frameworkIdentity, pluginCount — a serving process with 0 plugins degrades the verdict on its own).

The three calls an operator makes, as nodes

(a) Roll memex onto a pinned tag. Pin it on the record (a PR against the config repo, or a patch that the record's two-way sync carries back), then converge:

{ "nodeType": "Hosting/InstanceAction",
  "content": { "$type": "InstanceActionContent",
    "deployment": "Deployments/memex", "requestedAction": "Reconcile",
    "confirmation": "memex", "reason": "roll onto 3.0.0-ci.8079 (sealed set carrying #3694)" } }

The run renders the record (pinnedImageTag → portal.image and the migration on the same tag), runs hosting-deploy in-cluster, waits for the rollout under the bake gate, and ends with the audit. Roll + imageTag is the same outcome for the portal image alone, without the record.

(b) Read the last hour's error lines.

{ "nodeType": "Hosting/InstanceAction",
  "content": { "$type": "InstanceActionContent",
    "deployment": "Deployments/memex", "requestedAction": "Logs",
    "query": "fail:|crit:|Exception", "sinceMinutes": 60, "limit": 300 } }

The run node answers with logQl (exactly what was sent), entryCount and truncated; the lines are Hosting/LogEntry nodes named memex-<ns-timestamp>-<pod> under Ops/Logs, so get @Deployments/memex/area/Logs (or the Deployment page) shows them. Zero entries with a logQl on the node is an answer; zero entries with no logQl means the run did not get that far — read its error.

(c) Which replica cannot load a NodeType.

{ "nodeType": "Hosting/InstanceAction",
  "content": { "$type": "InstanceActionContent",
    "deployment": "Deployments/memex", "requestedAction": "Sample" } }

Ops/Status/memex then carries replicas[] — pod, image, ready, restarts, started-at, and each pod's OWN /health verdict and detail. A replica whose /health says Degraded degrades the sample's verdict even when Kubernetes calls it Ready, because readiness is what the probe gates and Degraded is what the probe lets through. The per-replica detail is whatever the portal reports on /health — as the platform's health detail grows (which types this replica cannot load, which sealed sets it adopted), it shows up here with no change to the sampler. warnings[] names what the sample could NOT see (a metric Prometheus did not answer, a pod whose /health was unreachable): a number that is unknown is never written as zero.

Where the observations read from

Both are reachable from the control instance's pod with no credential because the monitoring stack sits on the same cluster network — which is the honest reason refresh-status and ingest-logs always read them, and why Sample and Logs need no operator Job. Those two scripts still exist (they sample the WHOLE roster in one run); the actions are the per-instance, on-demand, API-shaped form with the denominator on the node.

Growing a volume is an edit to the record, not a kubectl patch. volumes[].size on the Deployment record is what the claim holds: Reconcile (and Provision) carry one Ensure volume capacity: <volume> step per declared claim, running the operator's hosting-pv-resize — which grows only (a record that declares LESS than the claim holds is refused, naming the record as the thing to correct), never creates a claim, refuses a storage class without allowVolumeExpansion before writing anything, and reports the capacity it read BACK from the claim. Azure Files expands online; a claim already at size is a no-op, so the step rides in every run. The Audit reports a claim that has fallen below its record under volumes below record. Why: on this fleet the portal's claims are hand-applied and helm never touches them, so a bigger size on the record used to render a bigger number and change nothing — memex-data measured 16Gi with 3 MiB free on 2026-09-08 with three files all saying 16Gi and none able to grow it.

🚨 Setting up an environment: the database address

Connection strings pin the database server's PRIVATE IP today (10.42.18.5), and the vault is the source of truth for it. Addressing it by FQDN is the intended end state but does not work yet — see below. Do not "improve" a working environment by switching it.

Why not the FQDN (yet)

memex/aspire/Memex.Portal.Distributed/Program.cs selects its Npgsql registration by inspecting the hostname, not by asking whether a password is present:

if (connectionString.Contains("database.azure.com"))
    builder.AddAzureNpgsqlDataSource(...)   // Entra TOKEN auth
else
    builder.AddNpgsqlDataSource(...)        // password auth

An FQDN plus the existing Password= therefore registers a token provider and a password, and Npgsql refuses — NotSupportedException: When registering a password provider, a password or password file may not be set → SIGABRT → CrashLoopBackOff on every pod. Observed in production 2026-08-23.

The fix is the rule SchemaHelpers.BuildSchemaDataSource already applies in the same repo: isAzure && !hasPassword. Until that ships, keep the IP.

(The private DNS zone IS linked to the cluster vnet now — it had zero links, which is why IPs were hardcoded in the first place — so the FQDN does resolve from pods. The blocker is code, not DNS.)

When the IP moves — and it does

Azure relocates a Flexible Server to a new backing instance on any compute scale, failover or maintenance. A D2ds_v5 → D8ds_v5 scale on 2026-08-23 moved it 10.42.18.4 → 10.42.18.5 and took down both portals.

Find the current address:

az network private-dns record-set a list -g memex-aks-rg \
  -z memexaks.private.postgres.database.azure.com --query "[].aRecords[0].ipv4Address" -o tsv

🚨 The address lives in ELEVEN places, across three kinds of store. Fix the VAULT first — miss it and the next sync silently reverts everything else:

🚨 Verify against the NAMESPACE, not the set you already fixed. Re-scanning only the secrets you knew about reports stale=0 while memex-kv-secrets still holds the dead address. That one mistake cost hours on 2026-08-23:

for NS in memex memex-cloud; do
  for S in $(kubectl get secret -n $NS -o name | cut -d/ -f2); do
    for K in $(kubectl get secret $S -n $NS -o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}'); do
      kubectl get secret $S -n $NS -o go-template="{{index .data \"$K\"}}" | base64 -d \
        | grep -q '10\.42\.18\.' && echo "$NS/$S :: $K"
    done
  done
done

The two addressing models — and why neither is finished

The two environments address the same server differently, and both are half-broken. Know which you are looking at before "fixing" one to match the other.

🚨 memex-postgres-service does not actually work, and it is a trap. It carries a selector (app.kubernetes.io/component: memex-postgres) pointing at the retired in-cluster StatefulSet, which is scaled 0/0. Because the selector matches zero pods, kube-controller-manager owns the Endpoints object and keeps emptying it — a hand-written Endpoints entry is reverted, not kept. Measured 2026-08-23: endpoints listed 10.42.18.5, and a TCP connect to memex-postgres-service:5432 from a live pod still failed. memex-cloud is healthy only because its ConnectionStrings__memex uses the IP directly; nothing on the working path goes through that Service.

So do not "give systemorph the same Service" as a fix — that copies a dead path. If you want the Service model (the address in ONE Endpoints object instead of eleven config sites), the Service must first be made genuinely selectorless: remove spec.selector, so the endpoints controller stops managing it and a manually-written Endpoints entry survives. Without that step the model looks tidier and is strictly worse, because the failure is silent.

Ranking the options honestly:

1. FQDN — the only self-healing answer: DNS follows the server through any scale or failover, zero manual steps, zero places to sweep. Blocked only by the Contains("database.azure.com") branch bug above. Fix that one line and this becomes available.
2. Truly selectorless Service — collapses eleven sites to one, but still needs a human to update the Endpoints each time Azure moves the server.
3. Raw IP everywhere — what runs today; eleven sites, and the vault ones revert the others if missed.

Triage — the symptoms point AWAY from the cause

Restarts are not free

Each pod start triggers a NodeType pre-warm/compile sweep with cross-schema queries. Repeated restarts drove the database to 97% CPU even on 8 vCPU, which then timed out DB calls, which crash-looped pods, which restarted them again. If you are firefighting, reduce replicas to 1 to break the loop before doing anything else. PreWarm__DynamicTypes=false makes a pod adopt the pre-baked bundles instead of baking in-mesh — check it before blaming capacity.

Create a new instance (by hand)

1. Record it. Create a Hosting/Deployment node. The fields that decide what the instance is:

{ "id": "acme", "namespace": "Deployments", "name": "acme",
  "nodeType": "Hosting/Deployment",
  "content": { "$type": "DeploymentContent",
    "host": "acme.meshweaver.cloud",
    "namespace": "acme",
    "cluster": "memexaks-cluster",
    "database": "acmedb",
    "databaseServer": "memexaks-pg",
    "dnsZone": "meshweaver.cloud",
    "keyVault": "Systemorph", "keyVaultSecretPrefix": "acme-",
    "backupStore": "Ops/BackupStores/default",
    "owner": "Acme AG",
    "pluginRepos": [
      { "name": "Plugins", "url": "https://memex.meshweaver.cloud" }
    ],
    "preInstall": [ "Edu", "Store" ] } }

pluginRepos and preInstall are which plugin repos it mounts and what a fresh boot installs. They are not new machinery: they render into the portal's own PluginCatalog:Registries / :Sources / :InstallByDefault configuration, which the new instance applies itself on first boot.

⚠️ A bare package id is qualified against the mounts. The catalog matches install patterns source-scoped and fails closed, so a bare Edu matches nothing and the instance comes up without it, silently. Write Edu and it becomes Plugins/Edu; write Plugins/* and it passes through untouched.

2. Run it. Create a Hosting/InstanceAction:

{ "id": "provision-acme", "namespace": "Ops/Actions", "name": "Provision acme",
  "nodeType": "Hosting/InstanceAction",
  "content": { "$type": "InstanceActionContent",
    "deployment": "acme", "requestedAction": "Provision", "dryRun": true } }

Start with dryRun: true. It renders the exact commands and changes nothing. Then set dryRun: false and requestedAction back to Provision.

Watch the same node: state goes Requested → Running → Done, or Failed naming the phase, or Refused with the question you did not answer. The last provisioning phase verifies the plugin catalog — an instance with green pods and an empty Store is the failure worth catching.

Take one down

Planned ──provision──▶ Live ──suspend──▶ Suspended ──teardown──▶ Decommissioned
                        ▲                    │
                        └────reactivate──────┘

Suspend before you destroy. Teardown refuses on a Live instance. Suspending stops the portal, takes a verified dump, and re-points the host at the paywall — nothing is deleted, and Reactivate undoes it. The suspension is the users' notice and their window to export their data.

{ "content": { "$type": "InstanceActionContent",
    "deployment": "acme", "requestedAction": "Suspend",
    "backup": "Backup", "confirmation": "acme",
    "reason": "contract ended" } }

🚨 Two questions with no defaults

A disruptive action (Suspend, Teardown, Restore) will not run until both are answered:

• backup — Backup or Skip. Unanswered refuses. There is no safe default: defaulting to skip destroys data nobody agreed to lose; defaulting to backup hides that nobody decided.
• confirmation — must repeat the deployment's own id, exactly. Not an "are you sure": a checkbox is satisfied by a click on whichever row happened to be open.

After the grace period (30 days from suspendedAt by default) the same node with requestedAction: Teardown finishes the job. skipGracePeriod: true overrides the wait and requires a reason — an override nobody justified is indistinguishable from a mistake.

Break glass — when the control plane cannot act

Sometimes the emergency is real and the proper lane is not available — the operator is not wired, the control instance itself is the thing that is down, or the order is "now". Then, and only then, kubectl scale deploy --all --replicas=0 -n {namespace} (via az aks command invoke when the API server is private) takes the instance down. Know what that DOESN'T do, because every missing half is a debt:

So a break-glass takedown is HALF a takedown, and the other half is a same-session reconcile:

1. The record, repo-first. The fleet space is GitSynced, so stamp status: "Suspended", replicas: 0, suspendedAt, suspensionReason (and a dated note naming the emergency path) in the config repo's record file, PR it, sync the space. A live-only patch is reverted by the next import; a cluster-only action is drift the record actively contradicts.
2. The action node anyway. Create the Suspend Hosting/InstanceAction even though the pods are already at zero — it is the audit (who ordered it, when, why), and when it can run it is the retry vehicle that applies the paywall redirect. Suspend is idempotent: scaling an already-scaled instance to zero is a no-op, the redirect and the stamp are the point.
3. The status board. The DeploymentStatus sample keeps showing the last healthy observation until someone re-samples; refresh it (or correct it by hand, naming the source) so the fleet board does not report a suspended instance as Healthy.

Worked example, 2026-08-23: atioz was taken down with the emergency scale on Roland's order. The follow-up Suspend action failed at Launch operator job (operator not live on the control instance — see OperatorEnablement.md), so the reconcile was exactly the three steps above: Memex#93 stamped the record, Deployments/atioz-suspend stands as the audit and the retry, and the status sample was corrected by hand. The host serves 503 instead of the paywall until that retry runs — that is the visible cost of the break-glass lane.

Backups

A dump is not a backup until it has been read back. Hosting/Backup.state:

Where dumps land is a Hosting/BackupStore record (provider, container, prefix, retention). It NAMES a credential; it never carries one. A backup with no store resolvable is refused, never silently skipped.

Which modules is each instance running

Hosting/ModuleInventory's Fleet area. Each instance reports itself — the control instance cannot read another installation's nodes — by running report-modules, which posts a signed report to the control instance.

An instance that has never reported shows as not reported, never as "no modules": the two look identical in a row count, and one of them is a confident wrong answer about a healthy portal. A module reported with no comparable build counts as differing, not agreeing.

The inventory receiver also preserves adoptedFrameworkIdentities and adoptedFrameworkInventoryComplete when a portal reports them. A portal can be running a new platform while a module still uses an older adopted build, so its running platform identity alone is not sufficient evidence for artifact cleanup. Missing legacy fields remain unknown; a malformed or explicitly incomplete list never becomes a complete inventory. An explicitly complete empty list is different from missing evidence.

This receiver support prepares the report contract. The paired platform reporter and retention reader must be deployed before cleanup can use these fields to protect remote adopted builds.

Logs

Two surfaces, deliberately:

• On the mesh — the ingested Hosting/LogEntry window is vector-indexed, so it is searchable by meaning. It is a projection of about a hundred lines per namespace, not an archive. Refresh it with ingest-logs.
• In Grafana — every deployment's and every infrastructure component's page carries exact Explore links (all, errors only, last 24h). No Grafana recorded → a sentence naming the field to set and the LogQL selector, never a dead link.

Before anything runs

On the control instance only — never a tenant portal, where it would let that pod start a job that can delete any namespace on the cluster:

config:
  memex_portal:
    Hosting__Operator__Enabled: "true"
    Hosting__Operator__Namespace: "memex-ops"
    Hosting__Operator__Image: "meshweaver.azurecr.io/hosting-operator:<tag>"
    Hosting__Operator__Environment__0: "AZ_RESOURCE_GROUP=<rg>"
    Hosting__Operator__Environment__1: "AZ_AKS_CLUSTER=<cluster>"
    Hosting__Operator__Environment__2: "AZ_PORTAL_IDENTITY=<portal-identity>"
    Hosting__Operator__Environment__3: "AZURE_CLIENT_ID=<operatorIdentityClientId>"
    Hosting__Operator__Environment__4: "PAYWALL_URL=https://<control-host>/Deployments/{instance}/area/Suspended"
    Hosting__ModuleReportSecret: "<shared, or per-deployment>"

plus, on the cluster, the operator image and its three-principal RBAC — deploy/aks/manifests/hosting-operator/ and deploy/aks/INSTANCE-LIFECYCLE.md in the MeshWeaver repo. The Systemorph fleet's own executed enablement — real names, the two defects only doing it found, and the verification probe — is OperatorEnablement.md.

🚨 Merged values are not an enabled operator. The hostingOperator: block reaches a container only when the Helm release actually rolls; until that run, every action refuses as "disabled" while every repo looks done (that exact state shipped on 2026-08-22 and was only caught by a refused action the next day). Prove enablement with the probe in OperatorEnablement.md, never with the merge.

🚨 The paywall target must be anonymously readable. A suspended customer is anonymous by construction — they typed their own URL — and the fleet space is admin-only, so without a grant the redirect lands them on a sign-in prompt. Grant Anonymous + Public Viewer on the node PAYWALL_URL points at.

When something refuses

A refusal is an answer, not a failure — it is re-runnable. Answer the question and set requestedAction again.

This instance

The fleet side above runs on the control instance. The instance side — how an installation registers itself at a plugin registry after a platform admin has accepted the privacy statement and the platform terms, what the plan it lands on means, and how to withdraw — is the Hosting plugin's This instance app, documented in Registering this installation.