Running memex locally on Colima k3s (Mac)

This page is a step-by-step guide for standing up a prod-like memex portal on a Mac, on a real Kubernetes cluster (k3s) inside Colima. It exercises the same Helm chart, the same Postgres + ingress + OAuth path that the cloud deployments use — but entirely on your laptop, with a local LLM and trusted local TLS, no cloud dependency.

When to use this (vs. the other routes)

You want Use Doc
Fastest inner loop — edit code, hit a browser, no Docker/k8s Monolith (dotnet run) or Aspire local mode Deployment.md → Running Locally · LocalDevWorkflow.md
A prod-like stack on your Mac — real k8s, Helm chart, ingress/TLS, Postgres PVC, OAuth, local LLM This page (Colima k3s)
Ship a code update to the shared memex portal AKS DeploymentAKS.md
Deploy an Aspire test/prod environment Azure Container Apps DeploymentContainerApps.md
Understand how an install updates itself (policy-driven) Self-update ReleaseStrategy.md

Because this is the same Helm chart as AKS, the in-pod self-updater applies here too: a Continuous install patches its own deployment to a newer ACR tag (see ReleaseStrategy.md). The ACR images are now multi-arch (linux/amd64 + linux/arm64) — built that way by CI (see §3) — so on this arm64 VM the self-updater pulls the native arm64 variant of each new tag and it Just Works. A pure local-build loop (images built straight into Colima's Docker store, never pushed to ACR) has nothing for the in-pod poll to list. For that loop, run memex-local autoroll up — a host-side launchd watcher that re-runs the migration (one-shot Job) then rollout restarts the portal whenever a fresh local portal/migration image is built. It's the local stand-in for the in-pod self-updater (same outcome: latest build goes live automatically), documented in deploy/homebrew/README.md. Needs nothing extra installed — launchd is built into macOS, docker/kubectl are already §1 prerequisites.

The Colima k3s route is the closest thing to "prod on your laptop": it runs the deploy/helm chart (the same chart the AKS environments use), terminates TLS at an ingress controller, authenticates through Microsoft Entra, persists Postgres on a PVC, and survives reboots. The trade-off is build time and a one-time setup. For everyday code iteration, prefer the Monolith / Aspire workflow — reach for Colima k3s when you need to validate the deployment shape, ingress/TLS, OAuth redirects, or the self-hosted-LLM path.

Everything below was set up and verified on an arm64 Mac (Apple Silicon). The defaults — hostname memex.localhost, port 8443, a host-native Ollama — are chosen so the whole thing runs without sudo and survives a reboot.


1. Prerequisites

Install the toolchain via Homebrew:

brew install colima kubectl helm mkcert
brew install ollama                     # local LLM runtime (runs on the host — see §11)
# socket_vmnet enables Colima's vmnet networking (host-gateway reachability):
brew install socket_vmnet

You also need the .NET SDK (10.0) to build the portal image — install from dotnet.microsoft.com or brew install --cask dotnet-sdk.

Prefer one command? brew tap systemorph/memex && brew trust systemorph/memex && brew install memex-local — the memex-local CLI automates every step on this page, idempotently: up / down / status / logs / update. The tap is published by this repository's CI on every merge to main (brew upgrade memex-local follows main). Point it at the cloud plugin registry first (§17, memex-local registry https://memex.meshweaver.cloud — no key needed, that is the free tier) and it needs neither a source checkout nor the .NET SDK. See deploy/homebrew/README.md. The rest of this page is the manual reference the CLI follows 1:1.

The work splits across three areas, which the rest of this page walks through in order:

Browser → https://memex.localhost:8443 launchd port-forward 8443 → ingress :443 ingress-nginx TLS (mkcert) · k3s memex-portal pod arm64 image · Helm Postgres (PVC) pgvector · persists Ollama (host) Metal GPU · qwen3.6 Microsoft Entra OAuth · /signin-microsoft All in-cluster components run on Colima's k3s VM (arm64, 8 CPU / 16 GiB) Ollama runs on the host so it keeps the Metal GPU (the VM has no GPU passthrough)

2. Start Colima with k3s

Start Colima with the Docker runtime and Kubernetes (k3s) enabled, sized for the portal + Postgres + observability stack:

colima start --kubernetes --cpu 8 --memory 16

This brings up a single arm64 VM running both a Docker daemon and a k3s cluster. The key reason to enable k3s with the Docker runtime (rather than containerd) is that k3s then shares Colima's Docker image store — so an image you build and docker tag locally is immediately visible to the cluster's IfNotPresent pull policy, with no registry push. That is what makes the build loop in §4 fast.

colima start writes a kubeconfig context; verify:

kubectl config current-context     # → colima
kubectl get nodes                  # → one Ready node

k3s here ships without Traefik (we install ingress-nginx instead — §6). If you ever recreate the profile, that's expected.


3. Get the portal image (arm64)

You have two ways to get an arm64 image onto the VM. CI now publishes multi-arch images, so for an unmodified portal you can just pull from ACR; build locally only when you're iterating on un-pushed source changes.

Option A — pull the multi-arch image from ACR (no local build)

Both pipelines build multi-arch manifest lists (linux/amd64 + linux/arm64) via the .NET SDK's ContainerRuntimeIdentifiers="linux-x64;linux-arm64" (an OCI image index — supported since SDK 8.0.405, and we build on .NET 10). They do not build the same set:

🚨 This only holds for genuinely multi-arch tags. Tags built before the multi-arch CI change (and any tag hand-built single-arch) are amd64-only; run emulated on the arm64 VM they make .NET's ConfigurationBinder throw a spurious NullReferenceException (InvokeStub_GraphStorageConfig.get_ConnectionString) that crashes the portal on startup. Verify a tag is multi-arch before relying on it:

docker manifest inspect meshweaver.azurecr.io/memex-portal-ai:latest \
  | grep -A1 '"architecture"'        # expect both "amd64" and "arm64"

One-time operator step: the very first multi-arch roll needs the base rebuilt multi-arch before the app build (the app's arm64 leg has no base layer otherwise). gh workflow run base-image-acr.yml --ref main does it; by hand into ACR: az acr build --registry meshweaver --image memex-portal-ai-base:latest --platform linux/amd64 --platform linux/arm64 deploy/base-images/portal-ai (or docker buildx ... --push).

Option B — build natively (fast inner loop for un-pushed edits)

When you've changed source that isn't in any pushed tag, build straight into Colima's Docker store. The verified rebuild loop (a few minutes on an M-series Mac):

🚨 The portal host lives in MeshWeaver.Plugins, not here. The GUI extraction (#2169 / #2293) moved Memex.Portal.Distributed, Memex.Portal.Monolith, Memex.AppHost and the login pages (Memex.Portal.Gui/Pages/{Login,DevLogin,DevLoginConfirm}.razor) into the plugins repo. The commands below publish from a checkout of it beside this one; memex-local resolves the same path and fails loudly if it is missing (override with MEMEX_PLUGINS_REPO). The migration image stays here — it is not GUI.

Two consequences worth stating, because each presented as something else when they were missed (#2367): the login UI is compiled INTO the image (it is shell, not a pack — a portal serving 404 on /login is running an image built before the move, and the fix is to rebuild), while the view packs are NOTDefaultViews/GraphViews left the image in #2169 Phase B2 and arrive only as registry bundles, which the boot reconcile installs because both declare preInstalled. A portal missing them renders every control as its ToString(), and the Plugin Catalog that would repair it is itself one of the missing views. memex-local up/update verify both and refuse to report success otherwise; memex-local verify asks the same question of a running install at any time.

# 1. Publish a native arm64 container image straight into Colima's Docker store.
#    ../MeshWeaver.Plugins is the sibling checkout; $(MeshWeaverRoot) defaults to this repo.
dotnet publish ../MeshWeaver.Plugins/src/Memex.Portal.Distributed/Memex.Portal.Distributed.csproj -c Release \
  -t:PublishContainer -p:ContainerRepository=memex-portal-ai-local

# 2. Tag it to the name the chart expects (IfNotPresent then finds it locally).
docker tag memex-portal-ai-local:latest ghcr.io/systemorph/memex-portal-ai:latest

# 3. Roll the deployment so the pod picks up the new image.
kubectl rollout restart deploy/memex-portal-deployment -n memex

# 4. Restart the port-forward (a rollout invalidates the old pod binding — see §8/Troubleshooting).

Do the same for the migration image if you changed schema/migrations:

dotnet publish <MeshWeaver.Plugins>/src/Memex.Database.Migration/Memex.Database.Migration.csproj -c Release -p:MeshWeaverRoot=<this checkout> \
  -t:PublishContainer -p:ContainerRepository=memex-migration-local
docker tag memex-migration-local:latest ghcr.io/systemorph/memex-migration:latest

Notes:


4. Deploy via Helm

The chart lives at deploy/helm. It is neutral by default — every self-host-only feature (ingress, external Ollama, instance identity) is off unless an overlay turns it on. Keep your machine-specific secrets and toggles in a local overlay outside the repo so nothing sensitive is ever committed:

mkdir -p ~/.memex-local
# Generate strong secrets once and write them into the overlay (do this with your
# own values — examples shown as placeholders).

A minimal ~/.memex-local/values.local.yaml looks like this (placeholders — fill in your own generated secrets and OAuth values):

secrets:
  memex_postgres:
    memex_postgres_password: "<generated-pg-password>"
  memex_migration:
    ConnectionStrings__memex: "Host=memex-postgres-service;Port=5432;Database=memex;Username=postgres;Password=<generated-pg-password>"
    memex_postgres_password: "<generated-pg-password>"
  memex_portal:
    ConnectionStrings__memex: "Host=memex-postgres-service;Port=5432;Database=memex;Username=postgres;Password=<generated-pg-password>"
    memex_postgres_password: "<generated-pg-password>"
    Ai__KeyProtection__MasterKey: "<generated-ai-master-key>"
    Authentication__Microsoft__ClientSecret: "<entra-client-secret>"   # see §10

config:
  memex_portal:
    Authentication__Provider: "Microsoft"
    Authentication__Microsoft__ClientId: "<entra-client-id>"
    Authentication__Microsoft__TenantId: "<entra-tenant-id>"
    Portal__InstanceName: "Local"           # tab title + favicon badge — see §13
    Portal__InstanceColor: "#f59e0b"
    OpenAICompatible__Endpoint: "http://ollama:11434/v1"   # local LLM — see §11
    OpenAICompatible__Models__0: "qwen3.6-code"
    OpenAICompatible__ApiKey: "ollama"

ingress:
  enabled: true                  # see §6
  host: "memex.localhost"
  tlsSecret: "memex-portal-tls"

ollama:
  external:
    enabled: true                # see §11
    host: "192.168.5.2"          # Colima's host-gateway IP
    port: 11434

Install (or upgrade) the release into the memex namespace, layering the local overlay on top of the chart defaults:

helm upgrade --install memex deploy/helm \
  -f deploy/helm/values.yaml \
  -f ~/.memex-local/values.local.yaml \
  -n memex --create-namespace

The chart wires up everything the portal needs to come up cleanly:

ConfigMap changes don't restart pods. After a helm upgrade that only changed config values, run kubectl rollout restart deploy/memex-portal-deployment -n memex for the new env to take effect. Secret/image changes that the chart templates as pod-spec changes do trigger a rollout on their own.


5. HTTPS via ingress-nginx + mkcert

k3s here has no Traefik, so install ingress-nginx once:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
  -n ingress-nginx --create-namespace \
  --set controller.service.type=ClusterIP \
  --set controller.admissionWebhooks.enabled=false

This registers the nginx IngressClass. The chart's deploy/helm/templates/memex-portal/ingress.yaml (gated on ingress.enabled) then creates a standard Ingress that terminates TLS for ingress.host using ingress.tlsSecret and proxies HTTP to memex-portal-service. The proxy-read-timeout / proxy-send-timeout annotations (3600 s) keep the Blazor Server SignalR WebSocket circuit alive. nginx sets X-Forwarded-Proto: https / X-Forwarded-Host, and the portal clears KnownProxies/KnownIPNetworks so it trusts them and builds the correct https://memex.localhost:8443/signin-microsoft redirect URI.

Mint a locally-trusted certificate with mkcert and load it as the TLS secret the ingress references:

mkcert -install                                    # one-time: trust the mkcert CA in the system keychain
mkdir -p ~/.memex-local
mkcert -cert-file ~/.memex-local/memex.localhost.pem \
       -key-file  ~/.memex-local/memex.localhost-key.pem \
       "memex.localhost" "*.memex.localhost"

# (Re)create the secret in the memex namespace — idempotent apply:
kubectl create secret tls memex-portal-tls -n memex \
  --cert="$HOME/.memex-local/memex.localhost.pem" \
  --key="$HOME/.memex-local/memex.localhost-key.pem" \
  --dry-run=client -o yaml | kubectl apply -f -

mkcert -install adds the mkcert root CA to the macOS System keychain, so Safari/Chrome show a trusted padlock with no warning. Re-run the create secret … | kubectl apply command any time the certificate changes.


6. Hostname & access

The default hostname is memex.localhost, accessed at https://memex.localhost:8443.

Why 8443 and not 443: port 443 is privileged on macOS, so a kubectl port-forward … 443:443 needs sudo — which breaks the no-sudo launchd auto-start (§8). 8443 is unprivileged, so the durable login agent can bring it up automatically. (See §9 for an optional clean :443 setup.)


7. Durable access (survives reboot)

Because 8443 is unprivileged, a launchd login agent can keep https://memex.localhost:8443 available with no sudo, automatically after every reboot.

~/.memex-local/port-forward.sh does three things in order:

#!/bin/bash
# 1. Start Colima if it isn't running (brings the k3s cluster + portal back).
colima status >/dev/null 2>&1 || colima start
# 2. Wait until the ingress-nginx namespace/controller is ready.
until kubectl get ns ingress-nginx >/dev/null 2>&1; do sleep 2; done
# 3. Forward 8443 on the host to the ingress controller's :443.
exec kubectl port-forward -n ingress-nginx svc/ingress-nginx-controller 8443:443

~/Library/LaunchAgents/com.memex.local.plist runs that script with RunAtLoad + KeepAlive (it restarts the forward if it ever drops), logging to ~/.memex-local/port-forward.log. Manage it with:

launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.memex.local.plist   # load + start
launchctl bootout   gui/$(id -u) ~/Library/LaunchAgents/com.memex.local.plist   # stop + unload

After a reboot, logging in starts Colima (the portal replicas=1 + the Postgres PVC persist on the VM disk) and re-establishes the port-forward — so https://memex.localhost:8443 comes back on its own.


8. Optional: a clean :443

If you want the URL without the :8443 suffix, forward the privileged port 443. Because that needs root, use a root LaunchDaemon in /Library/LaunchDaemons/ (not a user LaunchAgent) running essentially the same kubectl port-forward -n ingress-nginx svc/ingress-nginx-controller 443:443. Installing a LaunchDaemon requires sudo (it runs as root at boot, before login). This does not require restarting Colima. Prefer https://memex.localhost (no port) here — both the mkcert cert and the Entra redirect URI are valid for it.

This is optional convenience only. The 8443 path (§7) is the recommended default because it needs no sudo.


9. Auth — Microsoft Entra OAuth

The Distributed portal authenticates via Microsoft Entra OAuth (the callback path is /signin-microsoft, set in MeshWeaver.Plugins/src/MeshWeaver.Blazor.Portal/Authentication/AuthenticationBuilderExtensions.cs — the portal GUI moved there with #2293). The sign-in page itself is Memex.Portal.Gui/Pages/Login.razor in the same repo, and it is compiled into the image: curl -sk -o /dev/null -w '%{http_code}' https://memex.localhost:8443/login answering 404 means the image predates the move, not that a package is missing. Create a dedicated app registration for local dev (so its redirect URIs don't collide with the cloud apps):

  1. Azure Portal → App registrationsNew registration (or reuse a dedicated local-dev app).

  2. Under Authentication → Web → Redirect URIs, register the local callbacks. Register both the no-port (443) and :8443 forms so either port works:

    Redirect URI
    https://memex.localhost:8443/signin-microsoft
    https://memex.localhost/signin-microsoft
  3. Note the Application (client) ID and Directory (tenant) ID from the Overview page → put them in your overlay (Authentication__Microsoft__ClientId / __TenantId).

  4. Under Certificates & secrets, create a client secret → put it in the overlay's secrets.memex_portal.Authentication__Microsoft__ClientSecret.

OAuth over http://localhost originally failed with /login?error=auth_failed and a portal log line AuthenticationFailureException: Correlation failed (.AspNetCore.Correlation.* cookie not found). Root cause: the OIDC correlation + nonce cookies are SameSite=None (the Microsoft callback is a cross-site form_post), and browsers drop a SameSite=None cookie that isn't also Secure. The handler's default SecurePolicy = SameAsRequest left them non-Secure over plain HTTP, so they were never stored.

The fix in MeshWeaver.Plugins/src/MeshWeaver.Blazor.Portal/Authentication/AuthenticationBuilderExtensions.cs (AddMicrosoftAuthentication) forces them Secure:

options.CorrelationCookie.SecurePolicy = CookieSecurePolicy.Always;
options.NonceCookie.SecurePolicy = CookieSecurePolicy.Always;

Browsers make a localhost exception (they accept Secure cookies over http://localhost), so login works even without TLS, and it is a no-op in prod (already HTTPS). This is the security-correct setting everywhere.


10. Local LLM — Qwen via Ollama

The local LLM is wired through the chart's OpenAICompatible provider. The wrinkle on macOS is the GPU: Colima's arm64 VM has no Metal passthrough, so a containerized Ollama would run CPU-only. Therefore Ollama runs on the host (keeping the Metal GPU) and is exposed to the cluster by a stable in-cluster name.

The chart's deploy/helm/templates/ollama/service.yaml (gated on ollama.external.enabled) creates a selector-less Service ollama + a manual Endpoints object pointing at the host gateway (ollama.external.host, e.g. Colima's 192.168.5.2:11434). The portal then addresses Ollama by stable name instead of a brittle hardcoded IP:

OpenAICompatible__Endpoint = http://ollama:11434/v1
OpenAICompatible__Models__0 = qwen3.6-code
OpenAICompatible__ApiKey    = ollama

Start Ollama on the host, bound to all interfaces so the VM's host-gateway can reach it, and alias the model to the id the config expects:

OLLAMA_CONTEXT_LENGTH=16384 OLLAMA_HOST=0.0.0.0:11434 ollama serve   # 0.0.0.0 so the cluster can reach it; 16k ctx (see below)
ollama pull qwen3.6                                # the Ollama-library model
ollama cp qwen3.6 qwen3.6-code                     # alias to the id used by OpenAICompatible__Models__0

🚨 Set OLLAMA_CONTEXT_LENGTH. Ollama loads every model at a default context of 4096 tokens regardless of the model's real maximum. The agent system prompt alone is now larger than that, so a fresh thread fails with request (…tokens) exceeds the available context size (4096 tokens) … exceed_context_size_error and the model "can't run." Ollama's OpenAI-compatible /v1 endpoint does not accept a per-request num_ctx, so the host is the only lever: start ollama serve with OLLAMA_CONTEXT_LENGTH=16384 (or higher — bounded by the model's trained max and your VRAM). On the Ollama macOS app, set it once with launchctl setenv OLLAMA_CONTEXT_LENGTH 16384 and restart the app. Per-model alternative: a Modelfile with PARAMETER num_ctx 16384 + ollama create.

Pick a tool-capable model. The agent round sends tool/function definitions. A pure roleplay/completion GGUF (e.g. hf.co/TheBloke/Mythalion-13B-GGUF:Q4_K_M) returns HTTP 400 "does not support tools". The portal now probes Ollama's /api/show capabilities and stamps SupportsTools=false on such a model (so the round sends it a plain, tool-free chat request instead of erroring) — but for agent use, prefer a model whose capabilities include tools (qwen3.6 does).

k3s v1.35 logs a deprecation warning for Endpoints (in favor of EndpointSlice), but the mirroring controller auto-creates the slice, so routing works.

10.1 Content indexing / embeddings (semantic search + agent document discovery)

§10 wires the local chat model. Content indexing — the pipeline behind vector/semantic search, RAG, and agent document discovery — is a separate provider (Embedding__*) and stays inert until you set it. Leave it unset and uploads are stored and readable by path but produce no _Documents index node: nodeType:Document returns 0 everywhere and the Doc space isn't searchable, so an agent that finds its input via search correctly refuses to proceed. This reads like an agent/plugin bug but is missing portal config (the path is ContentIndexingObserverContentIndexingService.IndexFileEmbeddingOptions; empty Embedding__* = full-text ILIKE search only).

Reuse the same host Ollama already serving chat — just pull an embedding model:

ollama pull bge-m3                                 # 1024-dim embeddings; no API key for Ollama

Add to the local overlay ~/.memex-local/values.local.yaml under config: → memex_portal: (these keys are in the chart's ConfigMap template, so the overlay is their persistent home — no kubectl set env needed):

    Embedding__Provider: "Ollama"                  # → OllamaEmbeddingProvider ( /v1/embeddings )
    Embedding__Endpoint: "http://ollama:11434/v1"  # same in-cluster name as the chat provider (§10)
    Embedding__Model: "bge-m3"                      # no API key needed for Ollama

Then memex-local up to apply, and re-upload your documents — indexing fires on the upload event, so files stored before embeddings were enabled are not retro-indexed.

Dimensions are fixed per portal. The pgvector column + HNSW index are sized to the model (bge-m3 = 1024, embed-v-4-0 = 1536; a model not in the built-in lookup defaults to 1536). Changing Embedding__Model to a different-dimension model re-migrates the column + HNSW index on the next startup — expected, but not free on a large store.


11. Observability

Install the Grafana Loki stack into a monitoring namespace (Grafana + Loki + Promtail + Prometheus) — the same stack deploy/aks/scripts/install-observability.sh uses:

helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade --install loki grafana/loki-stack \
  -n monitoring --create-namespace \
  --set grafana.enabled=true,prometheus.enabled=true,promtail.enabled=true

Port-forward Grafana to view dashboards/logs; the admin password is in ~/.memex-local/grafana-password.txt.


12. Instance identity (tab title + favicon badge)

So a "Local" tab is unmistakable next to Test/Prod, the portal supports two optional config keys, set in the local overlay and allow-listed in the chart's config.yaml:

Key Effect
Portal__InstanceName Browser-tab title becomes this name, and the favicon becomes a distinct colored badge showing the name's initial. Empty (prod) = default Memex branding.
Portal__InstanceColor Badge fill color (hex, e.g. #f59e0b). Empty = amber default.

These are implemented in MeshWeaver.Plugins/src/Memex.Portal.Gui/App.razor (the portal GUI moved there with #2293): when Portal:InstanceName is set it emits an inline-SVG data-URI favicon (a rounded-square badge with the initial) and a small MutationObserver that pins the tab title to the instance name. Unset → the standard favicon.ico and Memex Portal title. Set Portal__InstanceName: "Local" in your overlay so your local tab is obvious at a glance.

🚨 On a node page the NODE's icon wins over the badge, by design. ApplicationPage publishes the node's own icon into the head from inside HeadOutlet, which App.razor places after the badge — and a later icon of equal rank outranks an earlier one, the same ordering SeoHead relies on. The environment stays unmistakable through the title, which the MutationObserver pins to the instance name on every page including node pages; the icon is what tells your open tabs apart. The badge still shows on every non-node page (login, search, onboarding, welcome). If an instance ever needs the badge to win everywhere, move that <link rel="icon"> below <SeoHead /> — precedence here is document order, nothing else.


13. Optional (advanced): home-wide access with a real cheap domain

This is an optional, advanced recipe — skip it unless you want every device on your home network (phones, other laptops) to reach the portal over trusted HTTPS with no per-device CA install. The mkcert approach (§5) only trusts the Mac that ran mkcert -install; a real Let's Encrypt certificate is trusted everywhere automatically.

The shape (a plan, not exact secrets):

  1. Buy a cheap domain and create an A record memex.<yourdomain> → your Mac's LAN IP. Pin that IP with a DHCP reservation on your router so it doesn't change.
  2. Install cert-manager into k3s (helm upgrade --install cert-manager jetstack/cert-manager --set crds.enabled=true).
  3. Issue a real wildcard cert via DNS-01. Create a ClusterIssuer (Let's Encrypt ACME) with a DNS-01 solver for your DNS provider (e.g. Cloudflare API token in a secret), then a Certificate requesting *.<yourdomain> / memex.<yourdomain>. DNS-01 only needs cert-manager to write a TXT record — it works without ever exposing the cluster to the internet.
  4. Point the ingress at the issued secret — set ingress.host to memex.<yourdomain> and ingress.tlsSecret to the Certificate's secret. Bind the ingress port-forward / service to the LAN interface so other devices can reach it.

The result: every device trusts the cert out of the box (no mkcert CA install), and the portal is reachable on the home network by a real name — all without any inbound internet exposure. Treat the provider tokens and the ClusterIssuer email as secrets kept outside the repo.


14. Troubleshooting

Symptom Cause & fix
Portal crashes on startup with NullReferenceException in ConfigurationBinder / get_ConnectionString You're running an amd64-only image emulated on the arm64 VM — i.e. a pre-multi-arch (or hand-built single-arch) tag. Confirm with docker manifest inspect … (§3 Option A) that the tag carries an arm64 entry; if not, pull a multi-arch tag or build natively (§3 Option B).
/login?error=auth_failed; log shows Correlation failed / correlation cookie not found The SameSite=None correlation/nonce cookies were dropped because they weren't Secure. Ensure you're on the build with the SecurePolicy = Always fix (§9). Verify: curl -i http://localhost:8080/auth/login?provider=Microsoft shows secure; samesite=none on both Set-Cookie lines.
Page returns empty reply / HTTP 000 after a portal rollout The kubectl port-forward binds one pod; a rollout restart replaces it and the old forward goes stale. Restart the port-forward (or let the launchd agent's KeepAlive do it).
A route 404s for ~1 second right after a Helm upgrade or ingress patch ingress-nginx reload lag — the controller is reloading its config. Retry; it clears within a second.
OAuth redirect URI mismatch The redirect URI the portal built must exactly match one registered on the app (§9). Check you're hitting the host/port whose /signin-microsoft is registered.
Ollama unreachable from the portal Ollama must be started with OLLAMA_HOST=0.0.0.0:11434 (not the default loopback bind), and ollama.external.host must be Colima's host-gateway IP (192.168.5.2).
Local model errors exceed_context_size_error / exceeds the available context size (4096 tokens) Ollama loads at a 4096-token default context, smaller than the agent prompt. Restart ollama serve with OLLAMA_CONTEXT_LENGTH=16384 (macOS app: launchctl setenv OLLAMA_CONTEXT_LENGTH 16384 + restart). The /v1 endpoint can't set num_ctx per request — it's a host setting. See §10.
Local model errors HTTP 400 does not support tools The selected model is a completion-only/roleplay GGUF. The portal auto-stamps SupportsTools=false (round then omits tools), but for agent work pick a model whose Ollama capabilities include tools (e.g. qwen3.6). See §10.
Uploaded documents aren't searchable / nodeType:Document returns 0 / an agent can't find its input Content indexing needs an embedding provider, which is separate from the chat model and off by default. Set Embedding__Provider/Embedding__Endpoint/Embedding__Model (§10.1) and re-upload — indexing fires on the upload event, so pre-existing files aren't retro-indexed.
memex-local targets the wrong clusterhelm fails kubernetes cluster unreachable: …privatelink…azmk8s.io, or (worse) a reachable cloud cluster gets rolled 🚨 memex-local uses the ambient kubectl context, not a pinned one. If your current context is an AKS cluster, up/update deploy there. Check with kubectl config current-context and switch: kubectl config use-context colima. It failed loudly here only because that cluster is private and unroutable — a reachable one would have been rolled silently.
https://memex.localhost:8443 refuses the connection although the pods are healthy Same root cause as above: the launchd port-forward agent runs kubectl port-forward with no context, so an AKS current-context breaks it. Fix the context, then memex-local port-forward.
Plugin Catalog is empty / a new instance installs nothing The local portal is the registry and serves plugins from your checkout (§16). No checkout found ⇒ nothing to serve. Confirm the config landed: kubectl -n memex get cm memex-portal-config -o json \| grep PluginCatalog__Sources and that the mount exists: kubectl -n memex exec deploy/memex-portal-deployment -- ls /plugin-repos/Plugins. Node repos beside MEMEX_REPO are discovered by content (a <Folder>/index.json with a root nodeType), so a repo that declares no package root is silently not one; point at it explicitly with MEMEX_PLUGIN_REPOS=/path/to/repo.
instance up fails 409 — Instance id '<id>' is already registered An instance id is claimed globally on the registry, and dropping the instance's database does not release it. Use memex-local instance down --id <id> (which releases the claim and restarts the registry) before re-running, or pick a new id.
A plugin install fails with NodeType(s) not registered: <Other>/<Type> The package depends on another that is not installed yet. The default install orders by the manifest's requires; a package that depends on something outside the granted set cannot be ordered against and will fail. Grant the dependency too (§16).
Instance pod OOMs mid-install (OutOfMemoryException, often surfacing inside Npgsql) and the remaining packages never install A first boot compiles every default-installed plugin's node types back to back, and each compile retains its collectible ALC. memex-local sizes the instance pod 3cpu/6Gi for this; a smaller pod dies partway. Installing fewer packages by default (a narrower InstallByDefault) is the other lever.
Every control renders as debug textNamedAreaControl { Id = , Style = , … } instead of UI The view packs are not installed. DefaultViews/GraphViews left the image in #2169 Phase B2 and arrive only as registry bundles; without them nothing has a view and the control falls back to its ToString(). 🚨 The documented repair — Settings ▸ Administration ▸ Plugin Catalog — is itself rendered by the missing views, so the portal cannot repair itself through its own UI. Diagnose from outside: memex-local verify. Both packs declare preInstalled, so the boot reconcile installs them from your checkout on every boot; if it did not, memex-local logs --no-follow --tail 400 \| grep DefaultInstall says why (a source name that matches no grant, an unreadable repo, a platform floor). (#2367)
/login 404s — no way to sign in at all The login pages are Blazor pages compiled into the image (MeshWeaver.Plugins/src/Memex.Portal.Gui/Pages/), so this is an image built before the GUI move (#2293), not a missing package. Rebuild: memex-local update --build, with the plugins checkout on main. (#2367)

Verify end-to-end that the portal is usable, not merely answering:

memex-local verify
# Asserts three things and exits non-zero, naming a remedy, if any fails:
#   • the portal SERVES        — an HTTP status in the serving range (a 503 is not "reachable")
#   • /login is ROUTED         — there is a way to sign in
#   • the view packs are there — MeshWeaver.Blazor.Views + MeshWeaver.Blazor.Graph

🚨 up and update run this for you and refuse to report success without it. The check it replaced curl'd / and printed "Portal reachable" whenever curl exited 0 — so a 503, a 404 and a portal rendering every control as its ToString() all produced the same green line (#2367). If you want the raw TLS/routing signal on its own:

curl --cacert "$(mkcert -CAROOT)/rootCA.pem" \
  -sS -o /dev/null -w 'http=%{http_code} ssl_verify=%{ssl_verify_result}\n' \
  https://memex.localhost:8443/
# Expect: http=200  (ssl_verify_result 0 once `mkcert -install` has trusted the CA)

15. Playwright E2E test env (memex-local e2e)

Browser E2E (Playwright) needs a portal it can log into without Entra (DevLogin) that also has a real language model. The dev Monolith has DevLogin but no model; this memex stack has the model but uses Entra OAuth and holds your real data. memex-local e2e stands up a throwaway, DevLogin portal built from the current working tree, in the same namespace, reusing this stack's Postgres, host Ollama and ingress/TLS — but against its own database (memex_e2e) with DevLogin on, behind the ingress (reverse proxy) at https://e2e.memex.localhost:8444. It is additive: it never touches the memex release, DB, or config.

memex-local up                              # the base stack (PG, Ollama, ingress, TLS) — once
memex-local e2e up                          # build working tree → create memex_e2e → migrate → deploy → reverse-proxy
memex-local e2e test HomeChatExecuteTest    # run the Playwright E2E (E2E_BASE_URL + DevLogin preset)
memex-local e2e down                        # delete the e2e objects + drop the e2e DB (--keep-db to keep it)

What e2e up does, and why:

Step Why
Build portal + migration image (native arm64) from the working tree Test the code you're holding, not a stale image. --skip-build reuses the last build.
CREATE DATABASE memex_e2e in the existing memex-postgres Your own data — never the memex DB.
Run the migration Job against it The portal's DbVersionGate refuses to start against an un-migrated DB.
Deploy memex-e2e-portal (Deployment + Service + Ingress) reusing memex-portal-config/-secrets via envFrom Proven config; override only ConnectionStrings__memexmemex_e2e and Authentication__EnableDevLogin=true. Clustering stays Localhost (own in-process silo). /data is an ephemeral emptyDir (mesh data is in PG).
Ingress for e2e.memex.localhost (covered by the *.memex.localhost mkcert cert) + a :443→:8444 port-forward The reverse proxy Playwright drives.

PortalFixture authenticates via POST /dev/signin?personId=Roland and sets IgnoreHTTPSErrors=true, so the self-signed cert is fine. The repeatable flow is captured as the /playwright skill. Always deploy on Colima and drive THAT — never run a model E2E against the Monolith (no model) or the memex portal (Entra + real data).


16. Plugin registry + provisioning a new instance (memex-local instance)

This stack is also a plugin registry, and it can ramp a second portal that registers itself with it and installs its plugins unattended — the local rehearsal of provisioning a real deployment, with nothing copied by hand.

The local portal serves plugins from your CHECKOUT

A registry normally reads its plugin repos from GitHub through the App identity. A local stack has no such credential, so it reads them straight off disk: helm_deploy mounts each checkout read-only (pluginCatalog.localRepoMounts → a hostPath; Colima runs the node on this very machine) and points a source at the in-container path.

Discovery is automatic, and by CONTENT. Every directory beside your MEMEX_REPO that declares a package root — a <Folder>/index.json whose nodeType is Space, Store/Plugin or Store/Catalog, the registry's own predicate — is mounted and served. Clone a node repo next to the platform checkout and it is picked up on the next memex-local update; nothing to configure.

The platform checkout and its worktrees exclude themselves: they carry no root manifest at that depth, so the rule needs no list of names to skip, and none can go stale.

MEMEX_PLUGIN_REPOS (colon-separated absolute paths) still overrides the discovered set, for serving fewer repos than you have checked out:

# Each folder's basename (minus a "MeshWeaver." prefix) becomes the SOURCE NAME, and every new
# instance is granted that source.
MEMEX_PLUGIN_REPOS=/Users/me/code/MeshWeaver.Plugins \
  memex-local update --build

🚨 It is an environment variable, so it applies to that one invocation. A later memex-local update typed without it serves the discovered set again — which is why discovery, not the variable, is the default.

Verify what actually landed — the values file is not the evidence, the pod is:

kubectl -n memex get cm memex-portal-config -o json | tr ',' '\n' | grep PluginCatalog
kubectl -n memex exec deploy/memex-portal-deployment -- ls /plugin-repos/Plugins   # the mount

🚨 Local/dev only. hostPath means this machine. On a real cluster the sources are URLs and the registry holds the GitHub App identity — see PluginRegistry.

Ramp an instance

memex-local instance up --id acme        # mint key → own DB → migrate → deploy → verify
memex-local instance status --id acme    # what it REGISTERED and what it INSTALLED
memex-local instance down --id acme      # delete objects, drop DB, release the id on the registry

What instance up does, and why:

Step Why
Mint an mwr_ registration key on the registry (/bootstrap/registration-key, secret-gated) A scripted ramp-up has no UI to click. The key is owned by MEMEX_INSTANCE_OWNER, so instances land in that user's partition — never more than that user could self-register.
CREATE DATABASE memex_<id> + migration Job Its own data; the portal's DbVersionGate refuses an un-migrated DB.
Deploy the instance portal with PluginCatalog__BootstrapKey + __InstanceId and no registry token This is the thing under test: it presents the bootstrap key once, receives its own mwi_ key, and stores it encrypted (Admin/PluginRegistryCredential/…).
Wait for the installed set to stop growing Packages install sequentially; the first lands in seconds and the rest take minutes. Reporting after the first one announces "installed: Agent" for a run that installs twenty.
Report from the databases A Running pod proves neither registration nor installation.

Expected on a healthy run — the instance registers, is granted Plugins/* by pluginCatalog.defaultGrants, and installs them all in dependency order:

 ok  registered as instance 'acme'
 ok  installed packages: Agent, BusinessRules, Chess, ClaudeCode, Collaboration, Copilot,
     DataModelling, DoublePendulum, Edu, Essentials, Feedback, FractalStars, Manufacturing,
     Publish, RolePlay, Skill, Store, ThreeBody, Training, Video

Adding another plugin repo

  1. Clone it next to your checkout — nothing else to do, it is discovered. (Somewhere else? Then list every repo you want in MEMEX_PLUGIN_REPOS, which replaces the discovered set.)
  2. memex-local update --build — the chart mounts it and adds it as a source and grants it to every new instance (defaultGrants: <name>/*).
  3. Whether new instances install it as well is separate: pluginCatalog.installByDefault (default ["Plugins/*"]) is deliberately source-scoped, so a repo you are merely granted is browsable but not auto-installed. Add <name>/* there to auto-install it too.

🚨 The grant/install split is the security property, not a formality: an instance is routinely entitled to paid course content, and "install everything I'm entitled to" would auto-install it.

🚨 What a self-registry install can NEVER have: a module binary. A checkout holds source, not assemblies, and a registry serves a module only from its own modules/<name>/ — so every module-declaring package installs its content and silently skips its DLL (MeshWeaver#2417; the self-sealing "up to date" gates were fixed in #2659, the lane itself stays closed by construction). No Radzen charts, Analysis, EntityViews, GoogleMaps or Speech, and values.local.self-registry.yaml blanks those five Modules__Required entries so readiness does not stall on them. The way to a complete local portal is not a local build lane; it is the next section.


17. Registry mode — consume memex.meshweaver.cloud (memex-local registry)

The other way round from §16: this install is a consumer of a remote plugin registry, exactly like every cloud instance is a consumer of memex.meshweaver.cloud. It runs the CI-built multi-arch image from ACR (the native arm64 member — no source checkout, no .NET SDK, az login once), registers itself at the registry on first boot, installs the packages it is granted and lands their compiled modules from the registry's bundles into /data. Module bundles are platform-floor-gated IL, so the amd64-built bundles the registry serves land on an arm64 install unchanged. (Prebuilt NodeType bakes are identity-gated and amd64-only; a local install compiles those on first access, which is the PreWarm__DynamicTypes: "false" behaviour it already has.)

memex-local registry https://memex.meshweaver.cloud --id my-mac    # no key: an OPEN registration → the free tier
memex-local up            # a fresh install …
memex-local update        # … or an existing self-registry one: pulls the ACR image, re-renders the chart
memex-local registry status

No key is the default. An un-keyed registration is an open one: memex.meshweaver.cloud accepts it and enrols the install into its default plan, the free tier — every package the free plan covers, plus the platform baseline. Raising it is a platform admin's edit of the instance's grant on the registry (Instance grants ▸ Plan); an install never asks for a plan itself. A registration key (--key mwr_…, minted by a platform admin under Settings ▸ Administration ▸ Instance grants ▸ Registration keys, for a plan) is the way an install lands on a higher plan from its first boot — and the only way in on a registry that accepts no open registrations (PluginRegistry.mdPlans).

What changes under the hood, and why:

Self-registry (§16) Registry mode (§17)
values layers values.local.defaults.yaml + values.local.self-registry.yaml + overlay values.local.defaults.yaml + ~/.memex-local/registry.yaml + overlay
pluginCatalog sources + localRepoMounts off the checkout, defaultGrants registryUrl, instanceId, homeUrl; the chart's installByDefault: ["Plugins/*"]
bootstrap key minted on the local portal (instance up) none by default (open registration → the registry's default plan); with --key, presented once (PluginCatalog__BootstrapKey). Either way the issued mwi_ key is stored encrypted in this install's database and survives every update
Modules__Required__0..4 blanked (nothing can land them) the image baseline stands — a store-delivered required module reads ExpectedLater, never a rollout blocker, and verify --repair performs the one activation restart
image built from source (Option B) ACR main (--build still works — up/update only change their default)

The registry file is always 0600: it may carry a bootstrap key, and a bootstrap key is a secret — it registers instances under the minting admin's identity — so the mode does not vary by whether one was given; registry status never prints it. What this install may pull is decided on the registry, per instance, in Admin/_PluginGrant/{instanceId} — registering is identity, not entitlement (PluginRegistry.md). A first boot that installs nothing has one of two causes, and memex-local verify names both: the registration was refused (401 — with a key: revoked or expired, mint a new one and re-run registry; without one: that registry accepts no open registrations, ask its platform admin for a key), or the registry advertises 0 package(s) to this instance (no grant yet — a platform admin adds one for it).

memex-local registry off deletes the file and the next update serves a checkout again; the instance stays registered on the registry until an admin revokes it there.


Reconnecting…
The server was updated. Reloading the page to pick up the latest version.