Hosting

Administer the MeshWeaver deployments you operate — as mesh nodes, not as a spreadsheet and a folder of scripts.

Install

Installing this package

Hosting is preInstalled and requires Store, so on a platform-baseline instance it arrives on its own and its content is open. To bring it onto a mesh that does not have it: /Store → the Hosting card → Provision (global admin), or create a Store/Provision node over MCP — never a hand-created Space (see AGENTS.md: the creator would silently own the partition). Verify with /Hosting/Console rendering the fleet console and search path:Hosting scope:descendants nodeType:NodeType -compilationStatus:Ok returning nothing.

Setting up a NEW instance — the setup dialog

The full configuration a brand-new instance boots with is entered in ONE place: /Hosting/Console → Request → "Set up a new instance…". Six steps:

1. Identity — name (becomes namespace, database prefix, helm release and host), subdomain, the TEMPLATE record to clone, idle policy, reason.
2. Main database — server, database name, the Key Vault secret NAME for the connection string, and optionally the connection string itself.
3. Main storage — the storage account backing the instance's shares/content, its secret name, its connection string.
4. Boot modules — the module assemblies the instance must come up with (the database driver MeshWeaver.Hosting.PostgreSql, the AI engine MeshWeaver.AI, the MCP endpoint MeshWeaver.Mcp, the view packs). A listed module that cannot load STALLS the rollout instead of booting green without the feature. 🚨 A non-empty list is the COMPLETE required set — the configuration overrides the image's list BY INDEX, never appends.
5. Packages & repos — the plugin repositories to mount (one per line, Name https://url [ref] [source]; source = this instance SERVES the repo) and the packages a fresh boot seeds itself with.
6. Review — the composed record, in the same field-table shape the fleet's config visualizer shows for the live instances, plus the vault secret names and the derived boot configuration (PluginCatalog:* + Modules:Required:*). The preview goes through the SAME composition the approval runs, so it is byte-for-byte what gets written.

Every field may stay blank — blank means "the template's value", which is why the template is required: the control plane clones infrastructure, it never guesses it.

Where the connection strings go. The Deployment record syncs to git, so it carries secret NAMES only ({name}-db-connection / {name}-storage-connection by convention). A typed value is encrypted with the platform key protector and stored on the request node — on a mesh without Ai:KeyProtection:MasterKey the filing is REFUSED rather than stored in cleartext. Left blank, the provision's hosting-kv-ensure step mints the vault secrets, or an operator sets them out of band (az keyvault secret set --vault-name … --name …).

Filing is not provisioning. The dialog writes an UNARMED Hosting/InstanceRequest; a global administrator approves it on the node (requestedAction: Approve), and only then does the control plane compose the record, open the config-repo PR and raise the provision — which itself refuses on a mesh where the operator is not enabled (see OperatorEnablement.md).

Playwright coverage: e2e/setup-dialog.spec.ts drives the dialog against the disposable harness mesh (see e2e/README.md for booting one) — the step walk, the review's composed preview, the plaintext-never-stored invariant, and the approval composing the record with stated values winning over the template.

Which modules are deployed

Hosting/ModuleInventory answers it, and the Fleet area answers the question worth asking: where do the instances disagree.

An instance reports itself. The control instance cannot read a tenant installation's nodes, and adding a way for it to would be a far larger permission decision than a fleet inventory deserves. So each instance folds its own coordinates and posts them to the control instance's signed webhook inbox (the PlatformBuilds shape) — nothing inbound is opened on a tenant, and the control instance gains no new reach. An instance that has never reported shows as not reported, never as "no modules": the two look identical in a row count, and one of them is a confident wrong answer about a healthy portal.

🚨 Both shapes, always. A module's coordinate lives either in {Space}/_GitSync or in a Plugins/{id} install record. Reading only the install records reports almost nothing and looks healthy — measured on memex 2026-08-10: 42 sync configs, ZERO install records. The fold is in ModuleInventoryContent.Fold, and when a module appears in both, the sync's commit wins as the build identity while the installer's version is carried alongside.

Silence is not a match. A module reported with no comparable build counts as differing, not as agreeing — two instances that cannot say what they run are not evidence that they agree. Stale reports are labelled with their age rather than rendered as current state.

Browsing logs

Two surfaces, deliberately:

• On the mesh — the ingested Hosting/LogEntry window is vector-indexed, so it is searchable by meaning, which is what the mesh adds and Grafana cannot do. It is a projection of about a hundred lines, not an archive.
• In Grafana — every deployment's Logs area now carries exact deep links (all logs, errors only, last 24h) built from its namespace and grafanaBaseUrl. A deployment with no Grafana recorded gets a sentence naming the field to set and the LogQL selector, never a dead link.

A layout area does not query Loki on render: that would make every page load depend on the log stack being up. Ingest is a script, i.e. an activity, which is the only reason HTTP belongs in it.

The lifecycle

Planned ──provision──▶ Live ──suspend──▶ Suspended ──teardown──▶ Decommissioned
                        ▲                    │
                        └────reactivate──────┘

An instance is suspended before it is destroyed. Suspending stops the portal, takes a VERIFIED dump, and re-points the host at a paywall page served here — nothing is deleted, and reactivate undoes it. The suspension is the users' notice, their explanation and their window to export their data; teardown refuses on anything still Live, and waits out a grace period measured from the suspension.

The redirect is an Ingress patch, not a DNS change: the fleet shares one ingress controller and one public IP, the host's A-record already points at it, and the namespace's TLS secret keeps serving the certificate. A DNS-level redirect would take the cert with it and leave the customer at a TLS error instead of a page explaining what happened.

🚨 The paywall target must be anonymously readable, and it is not by default. The page is the Suspended area on the Deployment node, and the fleet space that holds those records is admin-only — so a suspended customer, who is anonymous by construction (they typed their own URL), would be bounced to sign-in instead of shown the page. Grant Anonymous + Public Viewer on the Deployment node you point PAYWALL_URL at, or point it at a public page of your own. Nothing enforces this: the redirect lands either way, and the failure looks like a sign-in prompt rather than a misconfiguration. The area itself is already written for that audience — it renders the instance's name and its retention window and NOTHING operational (no database, no namespace, no vault, no suspension reason), which is asserted in Test/.

An instance is defined by its record

Hosting/Deployment carries the whole instance spec — including pluginRepos (which plugin repositories it mounts) and preInstall (what a fresh boot installs). Those render into the portal's own PluginCatalog:* configuration, which the new instance applies itself on first boot; the last provisioning phase then VERIFIES the catalog, because an instance with green pods and an empty Store is the failure worth catching.

So creating an instance is: write the record, run the action. Nothing lives in a folder on somebody's laptop, which is what makes this usable by any platform developer.

Two questions, no defaults

A disruptive action (suspend, teardown, restore) will not run until both are ANSWERED: whether to back the database up, and the deployment's own id as confirmation. An unanswered backup question is a refusal — a default either destroys data nobody agreed to lose or hides that nobody decided.

A backup is a backup once it has been read back

Hosting/Backup.state reaching Verified — the archive was downloaded, parsed and found to contain tables — is what the destructive phases are gated on. Stored means bytes were uploaded and nobody looked; a truncated upload exits zero. IsRestorable refuses everything else.

Off by default

Instance actions need Hosting:Operator:Enabled=true plus an operator image, namespace and token, and belong on the control instance only. Without them every action refuses and says what is missing — the identity these runs use can delete namespaces and drop databases.

Enablement is a Helm release ROLL, not a merge: the values block reaches a container only when the release deploys, so "merged and green everywhere" can still refuse every action. Prove it with an action probe, never with the checklist — OperatorEnablement.md carries the executed runbook and the probe. And when the proper lane genuinely cannot run and someone breaks glass with kubectl, that is half an operation with a same-session reconcile attached — the Break glass section in /instance names the three debts (record repo-first, action node as audit + retry, status sample).

The model

A deployment is a node: the intended state, version-controlled through GitSync, reviewable as a diff. What a deployment is doing is a separate node type, sampled from the monitoring stack. Changing a deployment is an activity: one run with progress, cancellation, a log and an identity it ran as. The overview shows records; the board shows samples; the gap between them is the work list.

On the Systemorph fleet this plugin is the admin surface on memex.systemorph.com: the Deployments Space holds the records, two-way GitSynced to Systemorph/Memex (mesh/Deployments) — edit a record in the UI and the next commit carries it back to the repository; edit the repository and the next update brings it to the UI.

Records, not secrets

Deployment nodes are designed to sync to a git repository. They carry identifiers and topology only. The keyVault / keyVaultSecretPrefix fields name where secrets live — they never hold values, and neither do the other fields. A connection string in a record is a leak the moment it commits.

Which secrets a pod reads is declared by NAME in the record's keyVaultSecrets section, and the chart renders the SecretProviderClass, the CSI mount and the envFrom from it — so the record is the only place a secret's name lives, and the cluster holds nothing a repository cannot see. The naming rule (<prefix><Section>-<Key> ↔ Section__Key), the full path from a record field to the pod's env, and the rule that nothing may live only on the cluster are in Configuration.md.

Where the images come from

The fleet has ONE image source it publishes to — ACR — and one registry it pulls from: cr.meshweaver.cloud, a blob-backed OCI registry the public instance hosts beside its portal (MeshWeaver#3353). The rule every record follows:

• The instance that hosts the registry stays on ACR. It declares the service in its registry section (host, images, storage account, the vault objects for the TLS pair and the token-signing key, the publisher's bcrypt hash — names only) and keeps imageRepository on meshweaver.azurecr.io/…, because a registry cannot serve the image that boots it.
• Every other instance names the registry. imageRepository is cr.meshweaver.cloud/memex-portal-ai and imagePullSecret names the dockerconfigjson Secret its pods present — filled by the provision's hosting-pull-secret step from the instance's own plugin-registry key, which is the credential the registry's auth service validates against the portal. A non-ACR repository with no imagePullSecret is a named problem: the first pull would answer 401 and the instance would never start. The self-updater follows the same host (selfUpdate.registry is derived from the repository), so it polls and rolls against the registry the pods pull from.

Roll and Reconcile ensure the pull Secret too, first thing, so an instance provisioned before its record named one is not left rolling onto an image it cannot pull.

Intent is not observation

pinnedImageTag records what someone asked for. What a deployment is running is sampled from kube-state-metrics into DeploymentStatus nodes — deliberately a separate, unsynced, disposable type, so an observed number can never be committed back to git as if it were intent. The board compares the two and says which is which.

Updates themselves are a pull each installation performs on its own: the portal's self-updater polls the registry and patches its own Kubernetes Deployment (see deploy/aks/SELF-UPDATE.md in the MeshWeaver repo). Nothing in this plugin pushes an image anywhere.

Installing

From the Store. It needs Store@^1.0.0 and nothing else — no cluster credentials, no cloud identity. The scripts reach Prometheus/Loki over the cluster network, so they only return data when the mesh runs next to the monitoring stack. Reading live state and applying changes are separate, separately-permissioned steps; see /deployment-activity before granting anything.

How a deployment actually works — with examples

The one rule

Administer an instance THROUGH MEMEX — the MCP or the GUI. Never by hand-editing the config repository on GitHub, and never by patching the cluster with kubectl / az aks command invoke.

The mesh is the control plane. It holds the records, and it is what writes the repo and the cluster; it writes by opening a PULL REQUEST on a FEATURE BRANCH of the config repository, never a direct commit and never a live kubectl apply. So a change is reviewable before it exists, the repo stays a 1:1 description of what is deployed, and the roll is the merge rather than something someone did by hand at 2am.

Two failure modes make this a rule rather than a preference, and both look fine at the time:

• A cluster patch is DRIFT. values.<env>.public.yaml is captured from the live release (helm get values) and compared against the cluster by a chart-drift gate. A hand-patched cluster is a gate failure someone else has to reconcile without knowing what you intended.
• A repo edit skips the checks the record can make. The record knows the vault, the secret prefix and the config path, so it can answer "will the pod see everything it needs?" BEFORE a roll. Edit GitHub directly and that question gets answered by a pod that will not start.

🚨 The names are CROSSED — never infer an instance from a name

The namespace called memex is systemorph.com; the vault called Systemorph serves meshweaver.cloud. Both read backwards. Confirm against the RECORD — get @Deployments/{name} carries host, namespace, cluster, keyVault, keyVaultSecretPrefix, configPath, repository, helmRelease — or, in a cluster, against the ingress host. Never against the name. Getting this wrong restarts the wrong production portal; that is not hypothetical (2026-08-19).

⚠️ MeshWeaver/deploy/aks/** is the CHART DEFAULT plus an EXAMPLE env — not what is deployed. Reading its keyvaultName as live config is exactly how the vaults get crossed.

Example — record a deployment

A deployment is a node. Nothing is provisioned by writing it; it is the git-backed intent.

{ "id": "acme", "namespace": "Deployments", "name": "acme",
  "nodeType": "Hosting/Deployment",
  "content": { "$type": "DeploymentContent",
    "host": "acme.meshweaver.cloud", "namespace": "acme", "cluster": "memexaks-cluster",
    "database": "acme", "imageRepository": "meshweaver.azurecr.io/memex-portal-ai",
    "keyVault": "Systemorph", "keyVaultSecretPrefix": "acme-",
    "configPath": "deployments/aks/acme", "repository": "https://github.com/Systemorph/Memex",
    "owner": "ACME AG", "purpose": "Customer instance." } }

Example — run a lifecycle action

Changing a deployment is an InstanceAction node that a control plane watches. requestedAction is the only field you write; state / phase / message are watcher-owned.

{ "id": "acme-provision", "namespace": "Deployments", "name": "acme-provision",
  "nodeType": "Hosting/InstanceAction",
  "content": { "$type": "InstanceActionContent",
    "deployment": "acme", "requestedAction": "Provision",
    "reason": "onboarding ACME", "dryRun": true } }

Watch the same node: state goes Requested → Running → Done (or Failed / Refused).

🚨 dryRun: true renders the plan and runs NOTHING — the honest first step for anything destructive, and the only mode safe to leave on a node someone else might re-trigger. Drop it only when you have read the plan.

🚨 A destructive action needs confirmation repeating the deployment id EXACTLY. Not a checkbox: a checkbox is clicked on whichever row happens to be open, and this cannot be.

{ "content": { "$type": "InstanceActionContent",
    "deployment": "acme", "requestedAction": "Teardown",
    "confirmation": "acme", "backup": "Backup",
    "reason": "contract ended 2026-09-30", "dryRun": true } }

A secret can exist and still be invisible

Three independent things must line up, and nothing checks them together:

1. the VALUE exists in the deployment's vault, under keyVaultSecretPrefix + its name;
2. the SecretProviderClass lists that objectName and syncs it to the env key;
3. a pod has RESTARTED since — the CSI driver reads the vault only at pod start.

Miss (2) and the feature behaves as though it were never built: MapPublish only mounts POST /api/plugins/bundles/{plugin} when Plugins:Registry:PublishToken is configured, so the route 404s and looks like a stale image. 🚨 And an SPC naming a secret the vault does not hold blocks pod startup (ContainerCreating forever) — the old pod keeps serving, so it stalls the rollout rather than paging anyone.

⚠️ Naming is not a pure transform. meshweaver prefixes every secret with memexcloud-, but Commerce:Stripe:SecretKey → memexcloud-Stripe-SecretKey (the Commerce segment is dropped) while Ai:KeyProtection:MasterKey → memexcloud-Ai-KeyProtection-MasterKey. Exceptions exist, so the mapping must be read, never derived.

🚨 Never kubectl exec … printenv to debug this — it prints secret VALUES into your terminal and into CI logs. Read the SPC's objectName → key mapping instead.

CI — what runs, and what a client must provide

A client operating their own instance needs:

• MW_TEST_IMAGE (repo variable) plus ACR_USERNAME / ACR_PASSWORD (secrets) — the gates take the framework assemblies from that image. A preflight job fails RED naming any that are missing; it must never be routed around with an if:, because GitHub renders a skipped job with the same tick as a passed one.
• A pinned MW_IMAGE_DIGEST, bumped in its own one-line PR. A moving tag makes two runs of identical code disagree; a stale pin caps what the gates can see (it cannot detect the REMOVAL of an API the pinned image still exports).
• A vault and a SecretProviderClass wired as above, plus the deployment record naming both.

The e2e suite

e2e/instance-lifecycle.spec.ts drives exactly this flow — record → verify → change → refuse an unconfirmed teardown → dry-run teardown → delete. It runs only signed in against a non-shared mesh, uses a unique per-run id so it can never name a real deployment, and sets dryRun on every action.