Running memex locally on Colima k3s (Mac)

This page is a step-by-step guide for standing up a prod-like memex portal on a Mac, on a real Kubernetes cluster (k3s) inside Colima. It exercises the same Helm chart, the same Postgres + ingress + OAuth path that the cloud deployments use — but entirely on your laptop, with a local LLM and trusted local TLS, no cloud dependency.

When to use this (vs. the other routes)

Because this is the same Helm chart as AKS, the in-pod self-updater applies here too: a Continuous install patches its own deployment to a newer ACR tag (see ReleaseStrategy.md). The ACR images are now multi-arch (linux/amd64 + linux/arm64) — built that way by CI (see §3) — so on this arm64 VM the self-updater pulls the native arm64 variant of each new tag and it Just Works. A pure local-build loop (images built straight into Colima's Docker store, never pushed to ACR) has nothing for the in-pod poll to list. For that loop, run memex-local autoroll up — a host-side launchd watcher that re-runs the migration (one-shot Job) then rollout restarts the portal whenever a fresh local portal/migration image is built. It's the local stand-in for the in-pod self-updater (same outcome: latest build goes live automatically), documented in deploy/homebrew/README.md. Needs nothing extra installed — launchd is built into macOS, docker/kubectl are already §1 prerequisites.

The Colima k3s route is the closest thing to "prod on your laptop": it runs the deploy/helm chart (the same chart the AKS environments use), terminates TLS at an ingress controller, authenticates through Microsoft Entra, persists Postgres on a PVC, and survives reboots. The trade-off is build time and a one-time setup. For everyday code iteration, prefer the Monolith / Aspire workflow — reach for Colima k3s when you need to validate the deployment shape, ingress/TLS, OAuth redirects, or the self-hosted-LLM path.

Everything below was set up and verified on an arm64 Mac (Apple Silicon). The defaults — hostname memex.localhost, port 8443, a host-native Ollama — are chosen so the whole thing runs without sudo and survives a reboot.

1. Prerequisites

Install the toolchain via Homebrew:

brew install colima kubectl helm mkcert
brew install ollama                     # local LLM runtime (runs on the host — see §11)
# socket_vmnet enables Colima's vmnet networking (host-gateway reachability):
brew install socket_vmnet

You also need the .NET SDK (10.0) to build the portal image — install from dotnet.microsoft.com or brew install --cask dotnet-sdk.

Prefer one command? brew tap systemorph/memex && brew trust systemorph/memex && brew install memex-local — the memex-local CLI automates every step on this page, idempotently: up / down / status / logs / update. The tap is published by this repository's CI on every merge to main (brew upgrade memex-local follows main). Point it at the cloud plugin registry first (§17, memex-local registry https://registry.example.com — no key needed, that is the free tier) and it needs neither a source checkout nor the .NET SDK. See deploy/homebrew/README.md. The rest of this page is the manual reference the CLI follows 1:1.

The work splits across three areas, which the rest of this page walks through in order:

2. Start Colima with k3s

Start Colima with the Docker runtime and Kubernetes (k3s) enabled, sized for the portal + Postgres + observability stack:

colima start --kubernetes --cpu 8 --memory 16

Host sizing

The VM and the local model both take host RAM, and nothing else checks it for you (#6024):

Only the last row is measured; the others are that arithmetic — VM + about 6 GB for macOS + the model + its KV cache — rounded up, not a soak test. up and update pull the default chat model and the portal names it directly (no alias): qwen3.8:27b-mlx on Apple Silicon when at least 32 GiB are left beside the VM, otherwise qwen3:8b. MEMEX_OLLAMA_MODEL pins another tool-capable model, and none skips the pull. See §10.

This brings up a single arm64 VM running both a Docker daemon and a k3s cluster. The key reason to enable k3s with the Docker runtime (rather than containerd) is that k3s then shares Colima's Docker image store — so an image you build and docker tag locally is immediately visible to the cluster's IfNotPresent pull policy, with no registry push. That is what makes the build loop in §4 fast.

colima start writes a kubeconfig context; verify:

kubectl config current-context     # → colima
kubectl get nodes                  # → one Ready node

k3s here ships without Traefik (we install ingress-nginx instead — §6). If you ever recreate the profile, that's expected.

3. Get the portal image (arm64)

You have two ways to get an arm64 image onto the VM. CI now publishes multi-arch images, so for an unmodified portal you can just pull from ACR; build locally only when you're iterating on un-pushed source changes.

Option A — pull the multi-arch image from ACR (no local build)

Both pipelines build multi-arch manifest lists (linux/amd64 + linux/arm64) via the .NET SDK's ContainerRuntimeIdentifiers="linux-x64;linux-arm64" (an OCI image index — supported since SDK 8.0.405, and we build on .NET 10). They do not build the same set:

• CONTINUOUS (main-cd.yml, every green merge to main → ACR) publishes the image set memex-portal-ai, memex-migration, mw-plugin-test, all-or-nothing via its promote job. This is the channel the self-updater watches.
• RELEASE (release.yml, on a v*.*.* tag) publishes nothing new: it promotes the sealed continuous set of the tagged commit — the same three images retagged with the clean version in ACR and mirrored to GHCR. There is no lean memex-portal any more (it existed only in the retired rebuild lane). The hand-authored base memex-portal-ai-base is built multi-arch by base-image-acr.yml with docker buildx --platform linux/amd64,linux/arm64. So on this arm64 VM, Docker/k3s pulls the native arm64 variant automatically — no emulation, and the in-pod self-updater (the blockquote in the intro) can roll forward to new ACR tags on its own.

🚨 This only holds for genuinely multi-arch tags. Tags built before the multi-arch CI change (and any tag hand-built single-arch) are amd64-only; run emulated on the arm64 VM they make .NET's ConfigurationBinder throw a spurious NullReferenceException (InvokeStub_GraphStorageConfig.get_ConnectionString) that crashes the portal on startup. Verify a tag is multi-arch before relying on it:

docker manifest inspect meshweaver.azurecr.io/memex-portal-ai:latest \
  | grep -A1 '"architecture"'        # expect both "amd64" and "arm64"

One-time operator step: the very first multi-arch roll needs the base rebuilt multi-arch before the app build (the app's arm64 leg has no base layer otherwise). gh workflow run base-image-acr.yml --ref main does it; by hand into ACR: az acr build --registry meshweaver --image memex-portal-ai-base:latest --platform linux/amd64 --platform linux/arm64 deploy/base-images/portal-ai (or docker buildx ... --push).

Registry mode updates from public GHCR

A memex-local install in registry mode runs ghcr.io/systemorph/memex-portal-ai (and ghcr.io/systemorph/memex-migration) with no pull secret, and it rolls itself forward from that same public registry (MeshWeaver.Deploy#65). memex-local sets the chart's self-update values for it:

No portal.imagePullSecret, no az login and no managed identity are involved. The old default, the upstream ACR listed through a managed identity, failed every hour on a Mac with ManagedIdentityCredential authentication failed, so a registry-mode install never rolled its platform and held every package whose floor was above it. The --key mwr_… registration key cannot replace GHCR here: mwr_ is a registration key, while cr.meshweaver.cloud checks an mwi_ instance key that is minted inside the pod and never reaches the host.

What rolls is still decided by Admin/UpdatePolicy (Settings → Updates) — see ReleaseStrategy.md. The listing needs a platform that carries SelfUpdate:RegistryIsPublic. An older image ignores the flag and refuses the GHCR listing by name, because no plugin registry is configured on that host, so it sends nothing. Such an install is moved once by hand onto a platform that carries the flag: brew upgrade memex-local && memex-local update, which in registry mode pulls the CI-built image as in Option A. It updates itself after that. How the credential decision is made: The Self-Update Registry Credential.

Option B — build natively (fast inner loop for un-pushed edits)

When you've changed source that isn't in any pushed tag, build straight into Colima's Docker store. The verified rebuild loop (a few minutes on an M-series Mac):

🚨 The portal host lives in MeshWeaver.Plugins, not here. The GUI extraction (#2169 / #2293) moved Memex.Portal.Distributed, Memex.Portal.Monolith, Memex.AppHost and the login pages (Memex.Portal.Gui/Pages/{Login,DevLogin,DevLoginConfirm}.razor) into the plugins repo. The commands below publish from a checkout of it beside this one; memex-local resolves the same path and fails loudly if it is missing (override with MEMEX_PLUGINS_REPO). The migration image stays here — it is not GUI.

Two consequences worth stating, because each presented as something else when they were missed (#2367): the login UI is compiled INTO the image (it is shell, not a pack — a portal serving 404 on /login is running an image built before the move, and the fix is to rebuild). View rendering needs both the module assemblies and the catalog packages. The image can ship module DLLs under /app/modules (as measured for Graph and EntityViews in #6025), while the DefaultViews/GraphViews/EntityViews packages may be installed by boot reconcile. A portal missing a renderer can show controls as their ToString(), and the Plugin Catalog UI may itself be unusable. memex-local up/update verify the files and the running process's [ModuleLoad] report; memex-local verify repeats that check on a running install.

# 1. Publish a native arm64 container image straight into Colima's Docker store.
#    ../MeshWeaver.Plugins is the sibling checkout; $(MeshWeaverRoot) defaults to this repo.
dotnet publish ../MeshWeaver.Plugins/src/Memex.Portal.Distributed/Memex.Portal.Distributed.csproj -c Release \
  -t:PublishContainer -p:ContainerRepository=memex-portal-ai-local

# 2. Tag it to the name the chart expects (IfNotPresent then finds it locally).
docker tag memex-portal-ai-local:latest ghcr.io/systemorph/memex-portal-ai:latest

# 3. Roll the deployment so the pod picks up the new image.
kubectl rollout restart deploy/memex-portal-deployment -n memex

# 4. Restart the port-forward (a rollout invalidates the old pod binding — see §8/Troubleshooting).

Do the same for the migration image if you changed schema/migrations:

dotnet publish <MeshWeaver.Plugins>/src/Memex.Database.Migration/Memex.Database.Migration.csproj -c Release -p:MeshWeaverRoot=<this checkout> \
  -t:PublishContainer -p:ContainerRepository=memex-migration-local
docker tag memex-migration-local:latest ghcr.io/systemorph/memex-migration:latest

Notes:

• The local build uses the default mcr.microsoft.com/dotnet/aspnet:10.0 base image, not the custom memex-portal-ai-base (that base bundles Node / Claude Code / Copilot, is now multi-arch in ACR, and is unneeded for the local inner loop).
• Because k3s runs the Docker runtime (docker://), the retag is visible to the cluster immediately — there is no docker push / registry step.
• This is the only step that is slow. Once the image exists, config-only changes (§5) don't need a rebuild.

4. Deploy via Helm

The chart lives at deploy/helm. It is neutral by default — every self-host-only feature (ingress, external Ollama, instance identity) is off unless an overlay turns it on. Keep your machine-specific secrets and toggles in a local overlay outside the repo so nothing sensitive is ever committed:

mkdir -p ~/.memex-local
# Generate strong secrets once and write them into the overlay (do this with your
# own values — examples shown as placeholders).

A minimal ~/.memex-local/values.local.yaml looks like this (placeholders — fill in your own generated secrets and OAuth values):

secrets:
  memex_postgres:
    memex_postgres_password: "<generated-pg-password>"
  memex_migration:
    ConnectionStrings__memex: "Host=memex-postgres-service;Port=5432;Database=memex;Username=postgres;Password=<generated-pg-password>"
    memex_postgres_password: "<generated-pg-password>"
  memex_portal:
    ConnectionStrings__memex: "Host=memex-postgres-service;Port=5432;Database=memex;Username=postgres;Password=<generated-pg-password>"
    memex_postgres_password: "<generated-pg-password>"
    Ai__KeyProtection__MasterKey: "<generated-ai-master-key>"
    Authentication__Microsoft__ClientSecret: "<entra-client-secret>"   # see §10

config:
  memex_portal:
    Authentication__Provider: "Microsoft"
    Authentication__Microsoft__ClientId: "<entra-client-id>"
    Authentication__Microsoft__TenantId: "<entra-tenant-id>"
    Portal__InstanceName: "Local"           # tab title + favicon badge — see §13
    Portal__InstanceColor: "#f59e0b"
    # Local LLM — see §10. MANUAL Helm only: `memex-local` users leave these two out — it sets the
    # endpoint in its tracked defaults and picks the model per Mac (llm.auto.yaml), and a model pinned
    # here would win over that choice even on a Mac too small for it.
    OpenAICompatible__Endpoint: "http://ollama.memex.svc.cluster.local:11434/v1"   # never the short name `ollama`: Ollama answers it 403
    OpenAICompatible__Models__0: "qwen3.8:27b-mlx"   # Apple Silicon with >= 32 GiB beside the VM; otherwise qwen3:8b (§2 Host sizing)
    OpenAICompatible__ApiKey: "ollama"

ingress:
  enabled: true                  # see §6
  host: "memex.localhost"
  tlsSecret: "memex-portal-tls"
  clusterIssuer: "none"          # the Secret is minted by mkcert (§5), not issued by cert-manager

ollama:
  external:
    enabled: true                # see §11
    host: "192.168.5.2"          # Colima's host-gateway IP
    port: 11434

Install (or upgrade) the release into the memex namespace, layering the local overlay on top of the chart defaults:

helm upgrade --install memex deploy/helm \
  -f deploy/helm/values.yaml \
  -f ~/.memex-local/values.local.yaml \
  -n memex --create-namespace

The chart wires up everything the portal needs to come up cleanly:

• Postgres runs as a StatefulSet (pgvector/pgvector:pg17) with a 10 Gi PVC (volumeClaimTemplates), so your data survives pod restarts and reboots.
• A wait-for-postgres initContainer on the portal pod (busybox + nc -z memex-postgres-service 5432) gates portal startup on Postgres TCP readiness — fixing the portal-vs-Postgres startup race on a fresh install.
• The migration job runs the schema migrations; the portal then boots against the migrated DB.

ConfigMap changes don't restart pods. After a helm upgrade that only changed config values, run kubectl rollout restart deploy/memex-portal-deployment -n memex for the new env to take effect. Secret/image changes that the chart templates as pod-spec changes do trigger a rollout on their own.

5. HTTPS via ingress-nginx + mkcert

k3s here has no Traefik, so install ingress-nginx once:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
  -n ingress-nginx --create-namespace \
  --set controller.service.type=ClusterIP \
  --set controller.admissionWebhooks.enabled=false

This registers the nginx IngressClass. The chart's deploy/helm/templates/memex-portal/ingress.yaml (gated on ingress.enabled) then creates a standard Ingress that terminates TLS for ingress.host using ingress.tlsSecret and proxies HTTP to memex-portal-service. The proxy-read-timeout / proxy-send-timeout annotations (3600 s) keep the Blazor Server SignalR WebSocket circuit alive. nginx sets X-Forwarded-Proto: https / X-Forwarded-Host, and the portal clears KnownProxies/KnownIPNetworks so it trusts them and builds the correct https://memex.localhost:8443/signin-microsoft redirect URI.

Mint a locally-trusted certificate with mkcert and load it as the TLS secret the ingress references:

mkcert -install                                    # one-time: trust the mkcert CA in the system keychain
mkdir -p ~/.memex-local
mkcert -cert-file ~/.memex-local/memex.localhost.pem \
       -key-file  ~/.memex-local/memex.localhost-key.pem \
       "memex.localhost" "*.memex.localhost"

# (Re)create the secret in the memex namespace — idempotent apply:
kubectl create secret tls memex-portal-tls -n memex \
  --cert="$HOME/.memex-local/memex.localhost.pem" \
  --key="$HOME/.memex-local/memex.localhost-key.pem" \
  --dry-run=client -o yaml | kubectl apply -f -

mkcert -install adds the mkcert root CA to the macOS System keychain, so Safari/Chrome show a trusted padlock with no warning. Re-run the create secret … | kubectl apply command any time the certificate changes.

6. Hostname & access

The default hostname is memex.localhost, accessed at https://memex.localhost:8443.

• *.localhost auto-resolves to loopback on macOS — dscacheutil -q host -a name memex.localhost returns 127.0.0.1, so no /etc/hosts entry is needed. This is why memex.localhost is the default.
• A custom single-label name (e.g. just memex) does not auto-resolve and needs an explicit hosts entry:

  echo "127.0.0.1 memex" | sudo tee -a /etc/hosts
  

• The mkcert certificate above covers memex.localhost (and *.memex.localhost), not plain localhost — so prefer https://memex.localhost for a green padlock.

Why 8443 and not 443: port 443 is privileged on macOS, so a kubectl port-forward … 443:443 needs sudo — which breaks the no-sudo launchd auto-start (§8). 8443 is unprivileged, so the durable login agent can bring it up automatically. (See §9 for an optional clean :443 setup.)

7. Durable access (survives reboot)

Because 8443 is unprivileged, a launchd login agent can keep https://memex.localhost:8443 available with no sudo, automatically after every reboot.

~/.memex-local/port-forward.sh does three things in order:

#!/bin/bash
# 1. Start Colima if it isn't running (brings the k3s cluster + portal back).
colima status >/dev/null 2>&1 || colima start
# 2. Wait until the ingress-nginx namespace/controller is ready.
until kubectl get ns ingress-nginx >/dev/null 2>&1; do sleep 2; done
# 3. Forward 8443 on the host to the ingress controller's :443.
exec kubectl port-forward -n ingress-nginx svc/ingress-nginx-controller 8443:443

~/Library/LaunchAgents/com.memex.local.plist runs that script with RunAtLoad + KeepAlive (it restarts the forward if it ever drops), logging to ~/.memex-local/port-forward.log. Manage it with:

launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.memex.local.plist   # load + start
launchctl bootout   gui/$(id -u) ~/Library/LaunchAgents/com.memex.local.plist   # stop + unload

After a reboot, logging in starts Colima (the portal replicas=1 + the Postgres PVC persist on the VM disk) and re-establishes the port-forward — so https://memex.localhost:8443 comes back on its own.

8. Optional: a clean :443

If you want the URL without the :8443 suffix, forward the privileged port 443. Because that needs root, use a root LaunchDaemon in /Library/LaunchDaemons/ (not a user LaunchAgent) running essentially the same kubectl port-forward -n ingress-nginx svc/ingress-nginx-controller 443:443. Installing a LaunchDaemon requires sudo (it runs as root at boot, before login). This does not require restarting Colima. Prefer https://memex.localhost (no port) here — both the mkcert cert and the Entra redirect URI are valid for it.

This is optional convenience only. The 8443 path (§7) is the recommended default because it needs no sudo.

9. Auth — Microsoft Entra OAuth

The Distributed portal authenticates via Microsoft Entra OAuth (the callback path is /signin-microsoft, set in MeshWeaver.Plugins/src/MeshWeaver.Blazor.Portal/Authentication/AuthenticationBuilderExtensions.cs — the portal GUI moved there with #2293). The sign-in page itself is Memex.Portal.Gui/Pages/Login.razor in the same repo, and it is compiled into the image: curl -sk -o /dev/null -w '%{http_code}' https://memex.localhost:8443/login answering 404 means the image predates the move, not that a package is missing. Create a dedicated app registration for local dev (so its redirect URIs don't collide with the cloud apps):

1. Azure Portal → App registrations → New registration (or reuse a dedicated local-dev app).

2. Under Authentication → Web → Redirect URIs, register the local callbacks. Register both the no-port (443) and :8443 forms so either port works:

   Redirect URI
   https://memex.localhost:8443/signin-microsoft
   https://memex.localhost/signin-microsoft

3. Note the Application (client) ID and Directory (tenant) ID from the Overview page → put them in your overlay (Authentication__Microsoft__ClientId / __TenantId).

4. Under Certificates & secrets, create a client secret → put it in the overlay's secrets.memex_portal.Authentication__Microsoft__ClientSecret.

The SameSite=None → Secure cookie fix

OAuth over http://localhost originally failed with /login?error=auth_failed and a portal log line AuthenticationFailureException: Correlation failed (.AspNetCore.Correlation.* cookie not found). Root cause: the OIDC correlation + nonce cookies are SameSite=None (the Microsoft callback is a cross-site form_post), and browsers drop a SameSite=None cookie that isn't also Secure. The handler's default SecurePolicy = SameAsRequest left them non-Secure over plain HTTP, so they were never stored.

The fix in MeshWeaver.Plugins/src/MeshWeaver.Blazor.Portal/Authentication/AuthenticationBuilderExtensions.cs (AddMicrosoftAuthentication) forces them Secure:

options.CorrelationCookie.SecurePolicy = CookieSecurePolicy.Always;
options.NonceCookie.SecurePolicy = CookieSecurePolicy.Always;

Browsers make a localhost exception (they accept Secure cookies over http://localhost), so login works even without TLS, and it is a no-op in prod (already HTTPS). This is the security-correct setting everywhere.

The owner is the instance's global administrator (the onboarding rule, part of §9)

Every onboarding collects the OWNER, and the owner is the instance's global administrator (maintainer, 2026-11-08). memex-local up, fleet join, the .pkg's unattended daemon and first-attach.sh all ask for it, or read it from --owner/--owner-email, from MEMEX_OWNER/MEMEX_OWNER_EMAIL, or from the managed preferences OwnerId/OwnerEmail. It is the MeshWeaver user id the owner signs in as, not the e-mail, plus their e-mail. Both go into this install's own overlay and nowhere else:

• Auth__GlobalAdmins__0, from which the portal seeds the Admin grant, so the owner can install packages such as ClaudeCode from Admin;
• Instance__Owner__Id and Instance__Owner__Email.

An unattended install with no owner fails rather than come up without an administrator. Change the owner with memex-local owner --id <user-id> --email <e-mail>, never by editing the YAML. The same owner answers the fleet opt-in questions.

10. Local LLM — Qwen via Ollama

The local LLM is wired through the chart's OpenAICompatible provider, and it is the default chat model on every memex-local install — no key, no command. The wrinkle on macOS is the GPU: Colima's arm64 VM has no Metal passthrough, so a containerized Ollama would run CPU-only. Therefore Ollama runs on the host (keeping the Metal GPU), bound to the Mac's loopback (it has no authentication), and the cluster reaches it through the VM's gateway, which Colima maps to the Mac's loopback. memex-local up proves that reach from inside the VM. Only when the VM cannot reach a loopback listener does memex-local move its own Ollama agent to every interface, and then it says so loudly. MEMEX_OLLAMA_ALLOW_LAN defaults to 1, which permits that fallback, so an ordinary install is not protected from it by default. The reason is availability: on a Mac whose VM cannot reach the loopback (a networking set-up Colima's default does not have), refusing would leave the portal with no model at all. So the default is to announce it and keep working. That is a different thing from the warning below, which is about choosing a LAN-wide bind to silence the 403. Set it to 0 on a Mac whose network is not trusted: the fallback and an Ollama someone else already started LAN-wide are then refused outright, and the portal has no local model rather than an exposed one.

The chart's deploy/helm/templates/ollama/service.yaml (gated on ollama.external.enabled) creates a selector-less Service ollama + a manual Endpoints object pointing at the VM's gateway (ollama.external.host). memex-local resolves that gateway per install from inside the VM (Colima's is usually 192.168.5.2, but it can differ). What the portal is configured with:

OpenAICompatible__Endpoint                       = http://ollama.memex.svc.cluster.local:11434/v1
OpenAICompatible__Models__0                      = qwen3.8:27b-mlx      (this Mac's model)
AI__Tiers__chat                                  = Provider/OpenAICompatible/qwen3.8:27b-mlx
OpenAICompatible__ContextWindowTokens__0__Model  = qwen3.8:27b-mlx
OpenAICompatible__ContextWindowTokens__0__Tokens = 262144               (what Ollama serves it with here)
OpenAICompatible__ApiKey                         = ollama               (any non-empty token)

🚨 The endpoint is the Service's cluster FQDN, never its short name. Ollama bound to loopback checks the Host header of every request and answers 403 unless that host is an IP that is loopback or private, localhost, the machine's own hostname, or a name ending in .localhost, .local or .internal. Until 2026-11-08 every install called http://ollama:11434/v1, so every chat round was refused — the portal logged [OpenAIWire] HTTP 403 … endpoint=ollama and closed the provider (PROVIDER_KEY_FAILING … Forbidden). The same Service as ollama.memex.svc.cluster.local ends in .local and is admitted, with nothing about Ollama changed. Measured on a Mac Studio, Ollama 0.40.14: Host: ollama:11434 → 403, Host: ollama.memex.svc.cluster.local:11434 → 200, and from inside the portal pod the same split. Making Ollama listen LAN-wide (OLLAMA_HOST=0.0.0.0) would also silence the 403 — and expose an unauthenticated model server to every machine on the network. Do not.

Where each value comes from (so an upgrade changes existing installs too):

An overlay generated before 2026-11-08 carries OpenAICompatible__Endpoint: "http://ollama:11434/v1", OpenAICompatible__Models__0: "qwen3.6-code" and host: "192.168.5.2". Because the overlay wins, those lines would keep the 403 forever, so the next update removes exactly those three stale template lines and keeps a copy (values.local.yaml.bak-llm). A value the owner wrote differently is left alone and still wins, and the chat tier and context window follow the owner's model. On such an install the old qwen3.6-code stays available as Models__1.

The model. qwen3.8:27b-mlx (~18 GB of weights, Ollama's MLX engine, tools and vision, trained for a 262,144-token window) on Apple Silicon when at least 32 GiB of unified memory remain beside the VM — a 48 GB Mac with the default 16 GiB VM, or a 64 GB Mac with the mac-studio profile's 32 GiB. Otherwise, or on an Intel Mac, qwen3:8b (~5 GB). The MLX build needs Ollama ≥ 0.32.12; an older Ollama is upgraded through Homebrew, or the host falls back. MEMEX_OLLAMA_MODEL pins another model, none pulls none. update pulls the model once on an existing install.

The context window is the one Ollama SERVES, not the trained one. Ollama loads a model at its own default context, which it sizes from the GPU memory it sees (on a 128 GB Mac Studio its log reads vram-based default context total_vram="107.5 GiB" default_num_ctx=262144; a smaller Mac gets less) or from OLLAMA_CONTEXT_LENGTH. memex-local loads the model and reads the loaded context back from Ollama's /api/ps, and declares that. The portal's context-window guard is then armed at what the server accepts. A model Ollama cannot load gets no window; it stays undeclared and logged rather than guessed.

Ollama itself. memex-local starts it as Homebrew's service (brew services start ollama), which binds 127.0.0.1:11434 and sets OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0 (an 8-bit KV cache: half the memory per token of context). Where that service is not available, memex-local's own launchd agent (com.memex.local.ollama) carries the same settings. An Ollama already listening beyond loopback is handled according to whoever runs it. If it is memex-local's own agent, memex-local rebinds it to loopback. If it is someone else's, memex-local reports it and goes on; stop it (brew services stop ollama, or end the process), then re-run memex-local up --skip-image. Under MEMEX_OLLAMA_ALLOW_LAN=0, a run that finds such a listener still running refuses to continue.

Without memex-local (the manual Helm route of §4). Do by hand what memex-local does:

brew services start ollama                     # loopback 127.0.0.1:11434, flash attention + q8_0 KV cache
ollama pull qwen3.8:27b-mlx                    # Apple Silicon, >= 32 GiB beside the VM; else: ollama pull qwen3:8b
curl -s localhost:11434/api/generate -d '{"model":"qwen3.8:27b-mlx","prompt":""}' >/dev/null   # load it
curl -s localhost:11434/api/ps                 # its "context_length" is the window Ollama serves it with

Put the endpoint and model in your values (the §4 example), set ollama.external.enabled: true, and set ollama.external.host to the VM's gateway (colima ssh -- ip route → default via <ip>). Declare the window you just read as OpenAICompatible__ContextWindowTokens__0__Model / __Tokens, and the default chat model as AI__Tiers__chat: "Provider/OpenAICompatible/<model>". Then prove it from inside the pod: kubectl -n memex exec deploy/memex-portal-deployment -c memex-portal -- curl -s -o /dev/null -w '%{http_code}' -H 'Content-Type: application/json' -d '{"model":"<model>","messages":[{"role":"user","content":"OK?"}]}' http://ollama.memex.svc.cluster.local:11434/v1/chat/completions must print 200.

"Never 403" is checked, not hoped.

• up, update and fleet join end with a real chat completion from inside the portal pod, using the endpoint, model and key the pod itself was given (the key never leaves the pod). On a 403 or any other non-200 the command fails and names the cause: 403 is the Host check, 404 a model Ollama does not have, 000 an Ollama that is down or unreachable. MEMEX_LLM_SMOKE=0 skips the check, loudly.
• memex-local status runs the same check.
• memex-local doctor runs it too, and first re-measures Ollama's Host rule on this Mac: the configured host must be answered 200, and ollama:11434 must be answered 403. The second probe is the negative control that shows the check can fail.

Apple Intelligence is not an alternative from inside the cluster. MeshWeaver.AI.AppleIntelligence reaches Apple's Foundation Models through a helper process (applefm-bridge) that it starts on the same machine, and it registers only where OperatingSystem.IsMacOS() is true. The portal runs on Linux inside the Colima VM, so the provider never registers there, and nothing bridges the pod to the Mac's Foundation Models today. It serves a portal that runs natively on macOS, not this one.

Pick a tool-capable model. The agent round sends tool/function definitions. A pure roleplay/completion GGUF (e.g. hf.co/TheBloke/Mythalion-13B-GGUF:Q4_K_M) returns HTTP 400 "does not support tools". The portal probes Ollama's /api/show capabilities and stamps SupportsTools=false on such a model, so the round sends it a plain, tool-free chat request instead of erroring. For agent use, prefer a model whose capabilities include tools; both defaults do.

k3s v1.35 logs a deprecation warning for Endpoints (in favor of EndpointSlice), but the mirroring controller auto-creates the slice, so routing works.

10.1 Content indexing / embeddings (semantic search + agent document discovery)

§10 wires the local chat model. Content indexing — the pipeline behind vector/semantic search, RAG, and agent document discovery — is a separate provider (Embedding__*) and stays inert until you set it. Leave it unset and uploads are stored and readable by path but produce no _Documents index node: nodeType:Document returns 0 everywhere and the Doc space isn't searchable, so an agent that finds its input via search correctly refuses to proceed. This reads like an agent/plugin bug but is missing portal config (the path is ContentIndexingObserver → ContentIndexingService.IndexFile → EmbeddingOptions; empty Embedding__* = full-text ILIKE search only).

Reuse the same host Ollama already serving chat — just pull an embedding model:

ollama pull bge-m3                                 # 1024-dim embeddings; no API key for Ollama

Add to the local overlay ~/.memex-local/values.local.yaml under config: → memex_portal: (these keys are in the chart's ConfigMap template, so the overlay is their persistent home — no kubectl set env needed):

    Embedding__Provider: "Ollama"                  # → OllamaEmbeddingProvider ( /v1/embeddings )
    Embedding__Endpoint: "http://ollama.memex.svc.cluster.local:11434/v1"  # same host-safe name as the chat provider (§10)
    Embedding__Model: "bge-m3"                      # no API key needed for Ollama

Then memex-local up to apply, and re-upload your documents — indexing fires on the upload event, so files stored before embeddings were enabled are not retro-indexed.

Dimensions are fixed per portal. The pgvector column + HNSW index are sized to the model (bge-m3 = 1024, embed-v-4-0 = 1536; a model not in the built-in lookup defaults to 1536). Changing Embedding__Model to a different-dimension model re-migrates the column + HNSW index on the next startup — expected, but not free on a large store.

10.2 Ingestion — OCR, descriptions, icons, tables (on by default)

Once §10 and §10.1 are in place, every file the instance ingests is also enriched — on by default, on every instance, inside the instance and on these local models (MeshWeaver.Plugins Indexing/DocumentEnrichment, Hosting/MacStudioOnboarding → Ingestion):

• a PDF is taken apart into section, table and figure nodes; tables become typed rows and columns shown in a grid; scanned pages are read by OCR first and their tables taken from the transcription;
• a picture is read (OCR) and described;
• the document's Description gets a summary (a picture: what it shows and the gist of its text), and it gets an icon (a picture: a simplified thumbnail; a document: a sanitised SVG the model draws) — never over a description or icon a person set;
• every part is searchable, full text and semantic.

Nothing to configure when the chat model can see. memex-local declares the OCR engine itself: when the §10 chat model (or the second model it names) reports vision in Ollama's capabilities — qwen3.6 and qwen3.8 do; check with curl -s localhost:11434/api/show -d '{"name":"qwen3.6"}' — every up/update writes these keys into ~/.memex-local/llm.auto.yaml, and memex-local doctor reports whether it did (MeshWeaver.Deploy, MEMEX_OCR=none turns it off):

config:
  memex_portal:
    Ocr__Engine: "OllamaVision"
    Ocr__OllamaVision__Endpoint: "http://ollama.memex.svc.cluster.local:11434/v1"   # the host-safe name; a host Ollama answers the short name 403
    Ocr__OllamaVision__Model: "qwen3.6"

Write them yourself only to read with a DIFFERENT model — the chat model has no vision, or you want a dedicated OCR model. Put the same block, with your model, in the overlay ~/.memex-local/values.local.yaml (under config: → memex_portal:, like the embedding keys of §10.1). The overlay is applied after llm.auto.yaml, so its value wins. With a chat model that has no vision, pictures are still read (OCR) but not described, and the document summaries and icons still come from the chat model.

Table and section detection, figure extraction and the picture icons run in-process — nothing more to install. To turn enrichment off for one instance: ContentIndexing__Enrichment__Enabled: "false" in the overlay.

Verify: upload a photo and a PDF with a table, open the file's page under …/_Documents/: the Enrichment panel goes Running (step) → Done with per-step timings, the page shows the summary, the recognised text and the icon, and the table is a Table-001 child rendered as a grid.

11. Observability

Install the Grafana Loki stack into a monitoring namespace (Grafana + Loki + Promtail + Prometheus) — the same stack deploy/aks/scripts/install-observability.sh uses:

helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade --install loki grafana/loki-stack \
  -n monitoring --create-namespace \
  --set grafana.enabled=true,prometheus.enabled=true,promtail.enabled=true

Port-forward Grafana to view dashboards/logs; the admin password is in ~/.memex-local/grafana-password.txt.

12. Instance identity (tab title + favicon badge)

So a "Local" tab is unmistakable next to Test/Prod, the portal supports two optional config keys, set in the local overlay and allow-listed in the chart's config.yaml:

These are implemented in MeshWeaver.Plugins/src/Memex.Portal.Gui/App.razor (the portal GUI moved there with #2293): when Portal:InstanceName is set it emits an inline-SVG data-URI favicon (a rounded-square badge with the initial) and a small MutationObserver that pins the tab title to the instance name. Unset → the standard favicon.ico and Memex Portal title. Set Portal__InstanceName: "Local" in your overlay so your local tab is obvious at a glance.

🚨 On a node page the NODE's icon wins over the badge, by design. ApplicationPage publishes the node's own icon into the head from inside HeadOutlet, which App.razor places after the badge — and a later icon of equal rank outranks an earlier one, the same ordering SeoHead relies on. The environment stays unmistakable through the title, which the MutationObserver pins to the instance name on every page including node pages; the icon is what tells your open tabs apart. The badge still shows on every non-node page (login, search, onboarding, welcome). If an instance ever needs the badge to win everywhere, move that <link rel="icon"> below <SeoHead /> — precedence here is document order, nothing else.

13. Optional (advanced): home-wide access with a real cheap domain

This is an optional, advanced recipe — skip it unless you want every device on your home network (phones, other laptops) to reach the portal over trusted HTTPS with no per-device CA install. The mkcert approach (§5) only trusts the Mac that ran mkcert -install; a real Let's Encrypt certificate is trusted everywhere automatically.

The shape (a plan, not exact secrets):

1. Buy a cheap domain and create an A record memex.<yourdomain> → your Mac's LAN IP. Pin that IP with a DHCP reservation on your router so it doesn't change.
2. Install cert-manager into k3s (helm upgrade --install cert-manager jetstack/cert-manager --set crds.enabled=true).
3. Issue a real wildcard cert via DNS-01. Create a ClusterIssuer (Let's Encrypt ACME) with a DNS-01 solver for your DNS provider (e.g. Cloudflare API token in a secret), then a Certificate requesting *.<yourdomain> / memex.<yourdomain>. DNS-01 only needs cert-manager to write a TXT record — it works without ever exposing the cluster to the internet.
4. Point the ingress at the issued secret — set ingress.host to memex.<yourdomain> and ingress.tlsSecret to the Certificate's secret. Keep ingress.clusterIssuer: "none" (memex-local's tracked defaults layer already sets it). The explicit Certificate from step 3 owns that Secret. A real issuer name would also annotate the Ingress, and cert-manager's ingress-shim would then try to manage the same Secret through a second Certificate. Name the issuer in ingress.clusterIssuer only if you skip step 3's Certificate and want ingress-shim to create it instead. The chart refuses to render an ingress that names a tlsSecret and no issuer at all. Bind the ingress port-forward / service to the LAN interface so other devices can reach it.

The result: every device trusts the cert out of the box (no mkcert CA install), and the portal is reachable on the home network by a real name — all without any inbound internet exposure. Treat the provider tokens and the ClusterIssuer email as secrets kept outside the repo.

14. Troubleshooting

Verify end-to-end that the portal is usable, not merely answering:

memex-local verify
# Asserts three things and exits non-zero, naming a remedy, if any fails:
#   • the portal SERVES        — an HTTP status in the serving range (a 503 is not "reachable")
#   • /login is ROUTED         — there is a way to sign in
#   • the view packs are installed and loaded — Views + Graph + EntityViews

The on-disk probe checks both landed bundles under Modules__Root/modules and image modules under /app/modules, since Modules__Root=/data does not move modules already shipped in the image. The boot [ModuleLoad] report then confirms that the running portal actually loaded all three packs. A loaded image pack must not be reported as missing just because /data/modules does not contain its DLL (#6025).

🚨 up and update run this for you and refuse to report success without it. The check it replaced curl'd / and printed "Portal reachable" whenever curl exited 0 — so a 503, a 404 and a portal rendering every control as its ToString() all produced the same green line (#2367). If you want the raw TLS/routing signal on its own:

curl --cacert "$(mkcert -CAROOT)/rootCA.pem" \
  -sS -o /dev/null -w 'http=%{http_code} ssl_verify=%{ssl_verify_result}\n' \
  https://memex.localhost:8443/
# Expect: http=200  (ssl_verify_result 0 once `mkcert -install` has trusted the CA)

15. Playwright E2E test env (memex-local e2e)

Browser E2E (Playwright) needs a portal it can log into without Entra (DevLogin) that also has a real language model. The dev Monolith has DevLogin but no model; this memex stack has the model but uses Entra OAuth and holds your real data. memex-local e2e stands up a throwaway, DevLogin portal built from the current working tree, in the same namespace, reusing this stack's Postgres, host Ollama and ingress/TLS — but against its own database (memex_e2e) with DevLogin on, behind the ingress (reverse proxy) at https://e2e.memex.localhost:8444. It is additive: it never touches the memex release, DB, or config.

memex-local up                              # the base stack (PG, Ollama, ingress, TLS) — once
memex-local e2e up                          # build working tree → create memex_e2e → migrate → deploy → reverse-proxy
memex-local e2e test HomeChatExecuteTest    # run the Playwright E2E (E2E_BASE_URL + DevLogin preset)
memex-local e2e down                        # delete the e2e objects + drop the e2e DB (--keep-db to keep it)

What e2e up does, and why:

PortalFixture authenticates via POST /dev/signin?personId=Roland and sets IgnoreHTTPSErrors=true, so the self-signed cert is fine. The repeatable flow is captured as the /playwright skill. Always deploy on Colima and drive THAT — never run a model E2E against the Monolith (no model) or the memex portal (Entra + real data).

16. Plugin registry + provisioning a new instance (memex-local instance)

This stack is also a plugin registry, and it can ramp a second portal that registers itself with it and installs its plugins unattended — the local rehearsal of provisioning a real deployment, with nothing copied by hand.

The local portal serves plugins from your CHECKOUT

A registry normally reads its plugin repos from GitHub through the App identity. A local stack has no such credential, so it reads them straight off disk: helm_deploy mounts each checkout read-only (pluginCatalog.localRepoMounts → a hostPath; Colima runs the node on this very machine) and points a source at the in-container path.

Discovery is automatic, and by CONTENT. Every directory beside your MEMEX_REPO that declares a package root — a <Folder>/index.json whose nodeType is Space, Store/Plugin or Store/Catalog, the registry's own predicate — is mounted and served. Clone a node repo next to the platform checkout and it is picked up on the next memex-local update; nothing to configure.

The platform checkout and its worktrees exclude themselves: they carry no root manifest at that depth, so the rule needs no list of names to skip, and none can go stale.

MEMEX_PLUGIN_REPOS (colon-separated absolute paths) still overrides the discovered set, for serving fewer repos than you have checked out:

# Each folder's basename (minus a "MeshWeaver." prefix) becomes the SOURCE NAME, and every new
# instance is granted that source.
MEMEX_PLUGIN_REPOS=/Users/me/code/MeshWeaver.Plugins \
  memex-local update --build

🚨 It is an environment variable, so it applies to that one invocation. A later memex-local update typed without it serves the discovered set again — which is why discovery, not the variable, is the default.

Verify what actually landed — the values file is not the evidence, the pod is:

kubectl -n memex get cm memex-portal-config -o json | tr ',' '\n' | grep PluginCatalog
kubectl -n memex exec deploy/memex-portal-deployment -- ls /plugin-repos/Plugins   # the mount

🚨 Local/dev only. hostPath means this machine. On a real cluster the sources are URLs and the registry holds the GitHub App identity — see PluginRegistry.

Ramp an instance

memex-local instance up --id acme        # mint key → own DB → migrate → deploy → verify
memex-local instance status --id acme    # what it REGISTERED and what it INSTALLED
memex-local instance down --id acme      # delete objects, drop DB, release the id on the registry

What instance up does, and why:

Expected on a healthy run — the instance registers, is granted Plugins/* by pluginCatalog.defaultGrants, and installs them all in dependency order:

 ok  registered as instance 'acme'
 ok  installed packages: Agent, BusinessRules, Chess, ClaudeCode, Collaboration, Copilot,
     DataModelling, DoublePendulum, Edu, Essentials, Feedback, FractalStars, Manufacturing,
     Publish, RolePlay, Skill, Store, ThreeBody, Training, Video

Adding another plugin repo

1. Clone it next to your checkout — nothing else to do, it is discovered. (Somewhere else? Then list every repo you want in MEMEX_PLUGIN_REPOS, which replaces the discovered set.)
2. memex-local update --build — the chart mounts it and adds it as a source and grants it to every new instance (defaultGrants: <name>/*).
3. Whether new instances install it as well is separate: pluginCatalog.installByDefault (default ["Plugins/*"]) is deliberately source-scoped, so a repo you are merely granted is browsable but not auto-installed. Add <name>/* there to auto-install it too.

🚨 The grant/install split is the security property, not a formality: an instance is routinely entitled to paid course content, and "install everything I'm entitled to" would auto-install it.

🚨 What a self-registry install can NEVER have: a module binary. A checkout holds source, not assemblies, and a registry serves a module only from its own modules/<name>/ — so every module-declaring package installs its content and silently skips its DLL (MeshWeaver#2417; the self-sealing "up to date" gates were fixed in #2659, the lane itself stays closed by construction). No Radzen charts, Analysis, EntityViews, GoogleMaps or Speech, and values.local.self-registry.yaml blanks those five Modules__Required entries so readiness does not stall on them. The way to a complete local portal is not a local build lane; it is the next section.

17. Registry mode — consume the public plugin registry (memex-local registry)

The other way round from §16: this install is a consumer of a remote plugin registry, exactly like every cloud instance is a consumer of the plugin registry instance. It runs the CI-built multi-arch image from ACR (the native arm64 member — no source checkout, no .NET SDK, az login once), registers itself at the registry on first boot, installs the packages it is granted and lands their compiled modules from the registry's bundles into /data. Module bundles are platform-floor-gated IL, so the amd64-built bundles the registry serves land on an arm64 install unchanged. (Prebuilt NodeType bakes are identity-gated and amd64-only; a local install compiles those on first access, which is the PreWarm__DynamicTypes: "false" behaviour it already has.)

memex-local registry https://registry.example.com --id my-mac    # no key: an OPEN registration → the free tier
memex-local up            # a fresh install …
memex-local update        # … or an existing self-registry one: pulls the ACR image, re-renders the chart
memex-local registry status

No key is the default. An un-keyed registration is an open one: the registry instance accepts it and enrols the install into its default plan, the free tier — every package the free plan covers, plus the platform baseline. Raising it is a platform admin's edit of the instance's grant on the registry (Instance grants ▸ Plan); an install never asks for a plan itself. A registration key (--key mwr_…, minted by a platform admin under Settings ▸ Administration ▸ Instance grants ▸ Registration keys, for a plan) is the way an install lands on a higher plan from its first boot — and the only way in on a registry that accepts no open registrations (PluginRegistry.md → Plans).

What changes under the hood, and why:

The registry file is always 0600: it may carry a bootstrap key, and a bootstrap key is a secret — it registers instances under the minting admin's identity — so the mode does not vary by whether one was given; registry status never prints it. What this install may pull is decided on the registry, per instance, in Admin/_PluginGrant/{instanceId} — registering is identity, not entitlement (PluginRegistry.md). A first boot that installs nothing has one of two causes, and memex-local verify names both: the registration was refused (401 — with a key: revoked or expired, mint a new one and re-run registry; without one: that registry accepts no open registrations, ask its platform admin for a key), or the registry advertises 0 package(s) to this instance (no grant yet — a platform admin adds one for it).

memex-local registry off deletes the file and the next update serves a checkout again; the instance stays registered on the registry until an admin revokes it there.

Related

• PluginRegistry.md — the registry model this stack implements locally: instance keys, grants, default installs.
• Deployment.md — the deploy-route index (AKS vs Container Apps) and shared Azure AD / secrets setup.
• DeploymentAKS.md — the cloud counterpart that uses the same deploy/helm chart shape.
• LocalDevWorkflow.md — the faster Aspire/Monolith inner loop for everyday code iteration.
• ControlledIoPooling.md — why all the portal's I/O (including Postgres) goes through the I/O pool, never Observable.FromAsync.