Running memex locally on Colima k3s (Mac)
This page is a step-by-step guide for standing up a prod-like memex portal on a Mac, on a real Kubernetes cluster (k3s) inside Colima. It exercises the same Helm chart, the same Postgres + ingress + OAuth path that the cloud deployments use — but entirely on your laptop, with a local LLM and trusted local TLS, no cloud dependency.
When to use this (vs. the other routes)
| You want | Use | Doc |
|---|---|---|
| Fastest inner loop — edit code, hit a browser, no Docker/k8s | Monolith (dotnet run) or Aspire local mode |
Deployment.md → Running Locally · LocalDevWorkflow.md |
| A prod-like stack on your Mac — real k8s, Helm chart, ingress/TLS, Postgres PVC, OAuth, local LLM | This page (Colima k3s) | — |
Ship a code update to the shared memex portal |
AKS | DeploymentAKS.md |
Deploy an Aspire test/prod environment |
Azure Container Apps | DeploymentContainerApps.md |
| Understand how an install updates itself (policy-driven) | Self-update | ReleaseStrategy.md |
Because this is the same Helm chart as AKS, the in-pod self-updater applies here too: a
Continuousinstall patches its own deployment to a newer ACR tag (see ReleaseStrategy.md). The ACR images are now multi-arch (linux/amd64 + linux/arm64) — built that way by CI (see §3) — so on this arm64 VM the self-updater pulls the native arm64 variant of each new tag and it Just Works. A pure local-build loop (images built straight into Colima's Docker store, never pushed to ACR) has nothing for the in-pod poll to list. For that loop, runmemex-local autoroll up— a host-side launchd watcher that re-runs the migration (one-shot Job) thenrollout restarts the portal whenever a fresh local portal/migration image is built. It's the local stand-in for the in-pod self-updater (same outcome: latest build goes live automatically), documented indeploy/homebrew/README.md. Needs nothing extra installed — launchd is built into macOS,docker/kubectlare already §1 prerequisites.
The Colima k3s route is the closest thing to "prod on your laptop": it runs the deploy/helm chart (the same chart the AKS environments use), terminates TLS at an ingress controller, authenticates through Microsoft Entra, persists Postgres on a PVC, and survives reboots. The trade-off is build time and a one-time setup. For everyday code iteration, prefer the Monolith / Aspire workflow — reach for Colima k3s when you need to validate the deployment shape, ingress/TLS, OAuth redirects, or the self-hosted-LLM path.
Everything below was set up and verified on an arm64 Mac (Apple Silicon). The defaults — hostname
memex.localhost, port8443, a host-native Ollama — are chosen so the whole thing runs without sudo and survives a reboot.
1. Prerequisites
Install the toolchain via Homebrew:
brew install colima kubectl helm mkcert
brew install ollama # local LLM runtime (runs on the host — see §11)
# socket_vmnet enables Colima's vmnet networking (host-gateway reachability):
brew install socket_vmnet
You also need the .NET SDK (10.0) to build the portal image — install from dotnet.microsoft.com or brew install --cask dotnet-sdk.
Prefer one command?
brew tap systemorph/memex && brew trust systemorph/memex && brew install memex-local— thememex-localCLI automates every step on this page, idempotently:up/down/status/logs/update. The tap is published by this repository's CI on every merge tomain(brew upgrade memex-localfollows main). Point it at the cloud plugin registry first (§17,memex-local registry https://memex.meshweaver.cloud— no key needed, that is the free tier) and it needs neither a source checkout nor the .NET SDK. Seedeploy/homebrew/README.md. The rest of this page is the manual reference the CLI follows 1:1.
The work splits across three areas, which the rest of this page walks through in order:
2. Start Colima with k3s
Start Colima with the Docker runtime and Kubernetes (k3s) enabled, sized for the portal + Postgres + observability stack:
colima start --kubernetes --cpu 8 --memory 16
This brings up a single arm64 VM running both a Docker daemon and a k3s cluster. The key reason to enable k3s with the Docker runtime (rather than containerd) is that k3s then shares Colima's Docker image store — so an image you build and docker tag locally is immediately visible to the cluster's IfNotPresent pull policy, with no registry push. That is what makes the build loop in §4 fast.
colima start writes a kubeconfig context; verify:
kubectl config current-context # → colima
kubectl get nodes # → one Ready node
k3s here ships without Traefik (we install ingress-nginx instead — §6). If you ever recreate the profile, that's expected.
3. Get the portal image (arm64)
You have two ways to get an arm64 image onto the VM. CI now publishes multi-arch images, so for an unmodified portal you can just pull from ACR; build locally only when you're iterating on un-pushed source changes.
Option A — pull the multi-arch image from ACR (no local build)
Both pipelines build multi-arch manifest lists (linux/amd64 + linux/arm64) via the .NET SDK's ContainerRuntimeIdentifiers="linux-x64;linux-arm64" (an OCI image index — supported since SDK 8.0.405, and we build on .NET 10). They do not build the same set:
- CONTINUOUS (
main-cd.yml, every green merge tomain→ ACR) publishes the image setmemex-portal-ai,memex-migration,mw-plugin-test, all-or-nothing via itspromotejob. This is the channel the self-updater watches. - RELEASE (
release.yml, on av*.*.*tag) publishes nothing new: it promotes the sealed continuous set of the tagged commit — the same three images retagged with the clean version in ACR and mirrored to GHCR. There is no leanmemex-portalany more (it existed only in the retired rebuild lane). The hand-authored basememex-portal-ai-baseis built multi-arch bybase-image-acr.ymlwithdocker buildx --platform linux/amd64,linux/arm64. So on this arm64 VM, Docker/k3s pulls the native arm64 variant automatically — no emulation, and the in-pod self-updater (the blockquote in the intro) can roll forward to new ACR tags on its own.
🚨 This only holds for genuinely multi-arch tags. Tags built before the multi-arch CI change (and any tag hand-built single-arch) are amd64-only; run emulated on the arm64 VM they make .NET's
ConfigurationBinderthrow a spuriousNullReferenceException(InvokeStub_GraphStorageConfig.get_ConnectionString) that crashes the portal on startup. Verify a tag is multi-arch before relying on it:docker manifest inspect meshweaver.azurecr.io/memex-portal-ai:latest \ | grep -A1 '"architecture"' # expect both "amd64" and "arm64"One-time operator step: the very first multi-arch roll needs the base rebuilt multi-arch before the app build (the app's arm64 leg has no base layer otherwise).
gh workflow run base-image-acr.yml --ref maindoes it; by hand into ACR:az acr build --registry meshweaver --image memex-portal-ai-base:latest --platform linux/amd64 --platform linux/arm64 deploy/base-images/portal-ai(ordocker buildx ... --push).
Option B — build natively (fast inner loop for un-pushed edits)
When you've changed source that isn't in any pushed tag, build straight into Colima's Docker store. The verified rebuild loop (a few minutes on an M-series Mac):
🚨 The portal host lives in
MeshWeaver.Plugins, not here. The GUI extraction (#2169 / #2293) movedMemex.Portal.Distributed,Memex.Portal.Monolith,Memex.AppHostand the login pages (Memex.Portal.Gui/Pages/{Login,DevLogin,DevLoginConfirm}.razor) into the plugins repo. The commands below publish from a checkout of it beside this one;memex-localresolves the same path and fails loudly if it is missing (override withMEMEX_PLUGINS_REPO). The migration image stays here — it is not GUI.Two consequences worth stating, because each presented as something else when they were missed (#2367): the login UI is compiled INTO the image (it is shell, not a pack — a portal serving 404 on
/loginis running an image built before the move, and the fix is to rebuild), while the view packs are NOT —DefaultViews/GraphViewsleft the image in #2169 Phase B2 and arrive only as registry bundles, which the boot reconcile installs because both declarepreInstalled. A portal missing them renders every control as itsToString(), and the Plugin Catalog that would repair it is itself one of the missing views.memex-local up/updateverify both and refuse to report success otherwise;memex-local verifyasks the same question of a running install at any time.
# 1. Publish a native arm64 container image straight into Colima's Docker store.
# ../MeshWeaver.Plugins is the sibling checkout; $(MeshWeaverRoot) defaults to this repo.
dotnet publish ../MeshWeaver.Plugins/src/Memex.Portal.Distributed/Memex.Portal.Distributed.csproj -c Release \
-t:PublishContainer -p:ContainerRepository=memex-portal-ai-local
# 2. Tag it to the name the chart expects (IfNotPresent then finds it locally).
docker tag memex-portal-ai-local:latest ghcr.io/systemorph/memex-portal-ai:latest
# 3. Roll the deployment so the pod picks up the new image.
kubectl rollout restart deploy/memex-portal-deployment -n memex
# 4. Restart the port-forward (a rollout invalidates the old pod binding — see §8/Troubleshooting).
Do the same for the migration image if you changed schema/migrations:
dotnet publish <MeshWeaver.Plugins>/src/Memex.Database.Migration/Memex.Database.Migration.csproj -c Release -p:MeshWeaverRoot=<this checkout> \
-t:PublishContainer -p:ContainerRepository=memex-migration-local
docker tag memex-migration-local:latest ghcr.io/systemorph/memex-migration:latest
Notes:
- The local build uses the default
mcr.microsoft.com/dotnet/aspnet:10.0base image, not the custommemex-portal-ai-base(that base bundles Node / Claude Code / Copilot, is now multi-arch in ACR, and is unneeded for the local inner loop). - Because k3s runs the Docker runtime (
docker://), the retag is visible to the cluster immediately — there is nodocker push/ registry step. - This is the only step that is slow. Once the image exists, config-only changes (§5) don't need a rebuild.
4. Deploy via Helm
The chart lives at deploy/helm. It is neutral by default — every self-host-only feature (ingress, external Ollama, instance identity) is off unless an overlay turns it on. Keep your machine-specific secrets and toggles in a local overlay outside the repo so nothing sensitive is ever committed:
mkdir -p ~/.memex-local
# Generate strong secrets once and write them into the overlay (do this with your
# own values — examples shown as placeholders).
A minimal ~/.memex-local/values.local.yaml looks like this (placeholders — fill in your own generated secrets and OAuth values):
secrets:
memex_postgres:
memex_postgres_password: "<generated-pg-password>"
memex_migration:
ConnectionStrings__memex: "Host=memex-postgres-service;Port=5432;Database=memex;Username=postgres;Password=<generated-pg-password>"
memex_postgres_password: "<generated-pg-password>"
memex_portal:
ConnectionStrings__memex: "Host=memex-postgres-service;Port=5432;Database=memex;Username=postgres;Password=<generated-pg-password>"
memex_postgres_password: "<generated-pg-password>"
Ai__KeyProtection__MasterKey: "<generated-ai-master-key>"
Authentication__Microsoft__ClientSecret: "<entra-client-secret>" # see §10
config:
memex_portal:
Authentication__Provider: "Microsoft"
Authentication__Microsoft__ClientId: "<entra-client-id>"
Authentication__Microsoft__TenantId: "<entra-tenant-id>"
Portal__InstanceName: "Local" # tab title + favicon badge — see §13
Portal__InstanceColor: "#f59e0b"
OpenAICompatible__Endpoint: "http://ollama:11434/v1" # local LLM — see §11
OpenAICompatible__Models__0: "qwen3.6-code"
OpenAICompatible__ApiKey: "ollama"
ingress:
enabled: true # see §6
host: "memex.localhost"
tlsSecret: "memex-portal-tls"
ollama:
external:
enabled: true # see §11
host: "192.168.5.2" # Colima's host-gateway IP
port: 11434
Install (or upgrade) the release into the memex namespace, layering the local overlay on top of the chart defaults:
helm upgrade --install memex deploy/helm \
-f deploy/helm/values.yaml \
-f ~/.memex-local/values.local.yaml \
-n memex --create-namespace
The chart wires up everything the portal needs to come up cleanly:
- Postgres runs as a StatefulSet (
pgvector/pgvector:pg17) with a 10 Gi PVC (volumeClaimTemplates), so your data survives pod restarts and reboots. - A
wait-for-postgresinitContainer on the portal pod (busybox+nc -z memex-postgres-service 5432) gates portal startup on Postgres TCP readiness — fixing the portal-vs-Postgres startup race on a fresh install. - The migration job runs the schema migrations; the portal then boots against the migrated DB.
ConfigMap changes don't restart pods. After a
helm upgradethat only changedconfigvalues, runkubectl rollout restart deploy/memex-portal-deployment -n memexfor the new env to take effect. Secret/image changes that the chart templates as pod-spec changes do trigger a rollout on their own.
5. HTTPS via ingress-nginx + mkcert
k3s here has no Traefik, so install ingress-nginx once:
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
-n ingress-nginx --create-namespace \
--set controller.service.type=ClusterIP \
--set controller.admissionWebhooks.enabled=false
This registers the nginx IngressClass. The chart's deploy/helm/templates/memex-portal/ingress.yaml (gated on ingress.enabled) then creates a standard Ingress that terminates TLS for ingress.host using ingress.tlsSecret and proxies HTTP to memex-portal-service. The proxy-read-timeout / proxy-send-timeout annotations (3600 s) keep the Blazor Server SignalR WebSocket circuit alive. nginx sets X-Forwarded-Proto: https / X-Forwarded-Host, and the portal clears KnownProxies/KnownIPNetworks so it trusts them and builds the correct https://memex.localhost:8443/signin-microsoft redirect URI.
Mint a locally-trusted certificate with mkcert and load it as the TLS secret the ingress references:
mkcert -install # one-time: trust the mkcert CA in the system keychain
mkdir -p ~/.memex-local
mkcert -cert-file ~/.memex-local/memex.localhost.pem \
-key-file ~/.memex-local/memex.localhost-key.pem \
"memex.localhost" "*.memex.localhost"
# (Re)create the secret in the memex namespace — idempotent apply:
kubectl create secret tls memex-portal-tls -n memex \
--cert="$HOME/.memex-local/memex.localhost.pem" \
--key="$HOME/.memex-local/memex.localhost-key.pem" \
--dry-run=client -o yaml | kubectl apply -f -
mkcert -install adds the mkcert root CA to the macOS System keychain, so Safari/Chrome show a trusted padlock with no warning. Re-run the create secret … | kubectl apply command any time the certificate changes.
6. Hostname & access
The default hostname is memex.localhost, accessed at https://memex.localhost:8443.
*.localhostauto-resolves to loopback on macOS —dscacheutil -q host -a name memex.localhostreturns127.0.0.1, so no/etc/hostsentry is needed. This is whymemex.localhostis the default.- A custom single-label name (e.g. just
memex) does not auto-resolve and needs an explicit hosts entry:echo "127.0.0.1 memex" | sudo tee -a /etc/hosts - The mkcert certificate above covers
memex.localhost(and*.memex.localhost), not plainlocalhost— so preferhttps://memex.localhostfor a green padlock.
Why 8443 and not 443: port 443 is privileged on macOS, so a kubectl port-forward … 443:443 needs sudo — which breaks the no-sudo launchd auto-start (§8). 8443 is unprivileged, so the durable login agent can bring it up automatically. (See §9 for an optional clean :443 setup.)
7. Durable access (survives reboot)
Because 8443 is unprivileged, a launchd login agent can keep https://memex.localhost:8443 available with no sudo, automatically after every reboot.
~/.memex-local/port-forward.sh does three things in order:
#!/bin/bash
# 1. Start Colima if it isn't running (brings the k3s cluster + portal back).
colima status >/dev/null 2>&1 || colima start
# 2. Wait until the ingress-nginx namespace/controller is ready.
until kubectl get ns ingress-nginx >/dev/null 2>&1; do sleep 2; done
# 3. Forward 8443 on the host to the ingress controller's :443.
exec kubectl port-forward -n ingress-nginx svc/ingress-nginx-controller 8443:443
~/Library/LaunchAgents/com.memex.local.plist runs that script with RunAtLoad + KeepAlive (it restarts the forward if it ever drops), logging to ~/.memex-local/port-forward.log. Manage it with:
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.memex.local.plist # load + start
launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/com.memex.local.plist # stop + unload
After a reboot, logging in starts Colima (the portal replicas=1 + the Postgres PVC persist on the VM disk) and re-establishes the port-forward — so https://memex.localhost:8443 comes back on its own.
8. Optional: a clean :443
If you want the URL without the :8443 suffix, forward the privileged port 443. Because that needs root, use a root LaunchDaemon in /Library/LaunchDaemons/ (not a user LaunchAgent) running essentially the same kubectl port-forward -n ingress-nginx svc/ingress-nginx-controller 443:443. Installing a LaunchDaemon requires sudo (it runs as root at boot, before login). This does not require restarting Colima. Prefer https://memex.localhost (no port) here — both the mkcert cert and the Entra redirect URI are valid for it.
This is optional convenience only. The 8443 path (§7) is the recommended default because it needs no sudo.
9. Auth — Microsoft Entra OAuth
The Distributed portal authenticates via Microsoft Entra OAuth (the callback path is /signin-microsoft, set in MeshWeaver.Plugins/src/MeshWeaver.Blazor.Portal/Authentication/AuthenticationBuilderExtensions.cs — the portal GUI moved there with #2293). The sign-in page itself is Memex.Portal.Gui/Pages/Login.razor in the same repo, and it is compiled into the image: curl -sk -o /dev/null -w '%{http_code}' https://memex.localhost:8443/login answering 404 means the image predates the move, not that a package is missing. Create a dedicated app registration for local dev (so its redirect URIs don't collide with the cloud apps):
Azure Portal → App registrations → New registration (or reuse a dedicated local-dev app).
Under Authentication → Web → Redirect URIs, register the local callbacks. Register both the no-port (443) and
:8443forms so either port works:Redirect URI https://memex.localhost:8443/signin-microsofthttps://memex.localhost/signin-microsoftNote the Application (client) ID and Directory (tenant) ID from the Overview page → put them in your overlay (
Authentication__Microsoft__ClientId/__TenantId).Under Certificates & secrets, create a client secret → put it in the overlay's
secrets.memex_portal.Authentication__Microsoft__ClientSecret.
The SameSite=None → Secure cookie fix
OAuth over http://localhost originally failed with /login?error=auth_failed and a portal log line AuthenticationFailureException: Correlation failed (.AspNetCore.Correlation.* cookie not found). Root cause: the OIDC correlation + nonce cookies are SameSite=None (the Microsoft callback is a cross-site form_post), and browsers drop a SameSite=None cookie that isn't also Secure. The handler's default SecurePolicy = SameAsRequest left them non-Secure over plain HTTP, so they were never stored.
The fix in MeshWeaver.Plugins/src/MeshWeaver.Blazor.Portal/Authentication/AuthenticationBuilderExtensions.cs (AddMicrosoftAuthentication) forces them Secure:
options.CorrelationCookie.SecurePolicy = CookieSecurePolicy.Always;
options.NonceCookie.SecurePolicy = CookieSecurePolicy.Always;
Browsers make a localhost exception (they accept Secure cookies over http://localhost), so login works even without TLS, and it is a no-op in prod (already HTTPS). This is the security-correct setting everywhere.
10. Local LLM — Qwen via Ollama
The local LLM is wired through the chart's OpenAICompatible provider. The wrinkle on macOS is the GPU: Colima's arm64 VM has no Metal passthrough, so a containerized Ollama would run CPU-only. Therefore Ollama runs on the host (keeping the Metal GPU) and is exposed to the cluster by a stable in-cluster name.
The chart's deploy/helm/templates/ollama/service.yaml (gated on ollama.external.enabled) creates a selector-less Service ollama + a manual Endpoints object pointing at the host gateway (ollama.external.host, e.g. Colima's 192.168.5.2:11434). The portal then addresses Ollama by stable name instead of a brittle hardcoded IP:
OpenAICompatible__Endpoint = http://ollama:11434/v1
OpenAICompatible__Models__0 = qwen3.6-code
OpenAICompatible__ApiKey = ollama
Start Ollama on the host, bound to all interfaces so the VM's host-gateway can reach it, and alias the model to the id the config expects:
OLLAMA_CONTEXT_LENGTH=16384 OLLAMA_HOST=0.0.0.0:11434 ollama serve # 0.0.0.0 so the cluster can reach it; 16k ctx (see below)
ollama pull qwen3.6 # the Ollama-library model
ollama cp qwen3.6 qwen3.6-code # alias to the id used by OpenAICompatible__Models__0
🚨 Set
OLLAMA_CONTEXT_LENGTH. Ollama loads every model at a default context of 4096 tokens regardless of the model's real maximum. The agent system prompt alone is now larger than that, so a fresh thread fails withrequest (…tokens) exceeds the available context size (4096 tokens) … exceed_context_size_errorand the model "can't run." Ollama's OpenAI-compatible/v1endpoint does not accept a per-requestnum_ctx, so the host is the only lever: startollama servewithOLLAMA_CONTEXT_LENGTH=16384(or higher — bounded by the model's trained max and your VRAM). On the Ollama macOS app, set it once withlaunchctl setenv OLLAMA_CONTEXT_LENGTH 16384and restart the app. Per-model alternative: a Modelfile withPARAMETER num_ctx 16384+ollama create.Pick a tool-capable model. The agent round sends tool/function definitions. A pure roleplay/completion GGUF (e.g.
hf.co/TheBloke/Mythalion-13B-GGUF:Q4_K_M) returns HTTP 400 "does not support tools". The portal now probes Ollama's/api/showcapabilities and stampsSupportsTools=falseon such a model (so the round sends it a plain, tool-free chat request instead of erroring) — but for agent use, prefer a model whosecapabilitiesincludetools(qwen3.6 does).
k3s v1.35 logs a deprecation warning for
Endpoints(in favor ofEndpointSlice), but the mirroring controller auto-creates the slice, so routing works.
10.1 Content indexing / embeddings (semantic search + agent document discovery)
§10 wires the local chat model. Content indexing — the pipeline behind vector/semantic search, RAG, and agent document discovery — is a separate provider (Embedding__*) and stays inert until you set it. Leave it unset and uploads are stored and readable by path but produce no _Documents index node: nodeType:Document returns 0 everywhere and the Doc space isn't searchable, so an agent that finds its input via search correctly refuses to proceed. This reads like an agent/plugin bug but is missing portal config (the path is ContentIndexingObserver → ContentIndexingService.IndexFile → EmbeddingOptions; empty Embedding__* = full-text ILIKE search only).
Reuse the same host Ollama already serving chat — just pull an embedding model:
ollama pull bge-m3 # 1024-dim embeddings; no API key for Ollama
Add to the local overlay ~/.memex-local/values.local.yaml under config: → memex_portal: (these keys are in the chart's ConfigMap template, so the overlay is their persistent home — no kubectl set env needed):
Embedding__Provider: "Ollama" # → OllamaEmbeddingProvider ( /v1/embeddings )
Embedding__Endpoint: "http://ollama:11434/v1" # same in-cluster name as the chat provider (§10)
Embedding__Model: "bge-m3" # no API key needed for Ollama
Then memex-local up to apply, and re-upload your documents — indexing fires on the upload event, so files stored before embeddings were enabled are not retro-indexed.
Dimensions are fixed per portal. The pgvector column + HNSW index are sized to the model (
bge-m3= 1024,embed-v-4-0= 1536; a model not in the built-in lookup defaults to 1536). ChangingEmbedding__Modelto a different-dimension model re-migrates the column + HNSW index on the next startup — expected, but not free on a large store.
11. Observability
Install the Grafana Loki stack into a monitoring namespace (Grafana + Loki + Promtail + Prometheus) — the same stack deploy/aks/scripts/install-observability.sh uses:
helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade --install loki grafana/loki-stack \
-n monitoring --create-namespace \
--set grafana.enabled=true,prometheus.enabled=true,promtail.enabled=true
Port-forward Grafana to view dashboards/logs; the admin password is in ~/.memex-local/grafana-password.txt.
12. Instance identity (tab title + favicon badge)
So a "Local" tab is unmistakable next to Test/Prod, the portal supports two optional config keys, set in the local overlay and allow-listed in the chart's config.yaml:
| Key | Effect |
|---|---|
Portal__InstanceName |
Browser-tab title becomes this name, and the favicon becomes a distinct colored badge showing the name's initial. Empty (prod) = default Memex branding. |
Portal__InstanceColor |
Badge fill color (hex, e.g. #f59e0b). Empty = amber default. |
These are implemented in MeshWeaver.Plugins/src/Memex.Portal.Gui/App.razor (the portal GUI moved there with #2293): when Portal:InstanceName is set it emits an inline-SVG data-URI favicon (a rounded-square badge with the initial) and a small MutationObserver that pins the tab title to the instance name. Unset → the standard favicon.ico and Memex Portal title. Set Portal__InstanceName: "Local" in your overlay so your local tab is obvious at a glance.
🚨 On a node page the NODE's icon wins over the badge, by design. ApplicationPage publishes the
node's own icon into the head from inside HeadOutlet, which App.razor places after the badge —
and a later icon of equal rank outranks an earlier one, the same ordering SeoHead relies on. The
environment stays unmistakable through the title, which the MutationObserver pins to the
instance name on every page including node pages; the icon is what tells your open tabs apart. The
badge still shows on every non-node page (login, search, onboarding, welcome). If an instance ever
needs the badge to win everywhere, move that <link rel="icon"> below <SeoHead /> — precedence
here is document order, nothing else.
13. Optional (advanced): home-wide access with a real cheap domain
This is an optional, advanced recipe — skip it unless you want every device on your home network (phones, other laptops) to reach the portal over trusted HTTPS with no per-device CA install. The mkcert approach (§5) only trusts the Mac that ran mkcert -install; a real Let's Encrypt certificate is trusted everywhere automatically.
The shape (a plan, not exact secrets):
- Buy a cheap domain and create an
Arecordmemex.<yourdomain>→ your Mac's LAN IP. Pin that IP with a DHCP reservation on your router so it doesn't change. - Install cert-manager into k3s (
helm upgrade --install cert-manager jetstack/cert-manager --set crds.enabled=true). - Issue a real wildcard cert via DNS-01. Create a
ClusterIssuer(Let's Encrypt ACME) with a DNS-01 solver for your DNS provider (e.g. Cloudflare API token in a secret), then aCertificaterequesting*.<yourdomain>/memex.<yourdomain>. DNS-01 only needs cert-manager to write a TXT record — it works without ever exposing the cluster to the internet. - Point the ingress at the issued secret — set
ingress.hosttomemex.<yourdomain>andingress.tlsSecretto the Certificate's secret. Bind the ingress port-forward / service to the LAN interface so other devices can reach it.
The result: every device trusts the cert out of the box (no mkcert CA install), and the portal is reachable on the home network by a real name — all without any inbound internet exposure. Treat the provider tokens and the ClusterIssuer email as secrets kept outside the repo.
14. Troubleshooting
| Symptom | Cause & fix |
|---|---|
Portal crashes on startup with NullReferenceException in ConfigurationBinder / get_ConnectionString |
You're running an amd64-only image emulated on the arm64 VM — i.e. a pre-multi-arch (or hand-built single-arch) tag. Confirm with docker manifest inspect … (§3 Option A) that the tag carries an arm64 entry; if not, pull a multi-arch tag or build natively (§3 Option B). |
/login?error=auth_failed; log shows Correlation failed / correlation cookie not found |
The SameSite=None correlation/nonce cookies were dropped because they weren't Secure. Ensure you're on the build with the SecurePolicy = Always fix (§9). Verify: curl -i http://localhost:8080/auth/login?provider=Microsoft shows secure; samesite=none on both Set-Cookie lines. |
| Page returns empty reply / HTTP 000 after a portal rollout | The kubectl port-forward binds one pod; a rollout restart replaces it and the old forward goes stale. Restart the port-forward (or let the launchd agent's KeepAlive do it). |
| A route 404s for ~1 second right after a Helm upgrade or ingress patch | ingress-nginx reload lag — the controller is reloading its config. Retry; it clears within a second. |
| OAuth redirect URI mismatch | The redirect URI the portal built must exactly match one registered on the app (§9). Check you're hitting the host/port whose /signin-microsoft is registered. |
| Ollama unreachable from the portal | Ollama must be started with OLLAMA_HOST=0.0.0.0:11434 (not the default loopback bind), and ollama.external.host must be Colima's host-gateway IP (192.168.5.2). |
Local model errors exceed_context_size_error / exceeds the available context size (4096 tokens) |
Ollama loads at a 4096-token default context, smaller than the agent prompt. Restart ollama serve with OLLAMA_CONTEXT_LENGTH=16384 (macOS app: launchctl setenv OLLAMA_CONTEXT_LENGTH 16384 + restart). The /v1 endpoint can't set num_ctx per request — it's a host setting. See §10. |
Local model errors HTTP 400 does not support tools |
The selected model is a completion-only/roleplay GGUF. The portal auto-stamps SupportsTools=false (round then omits tools), but for agent work pick a model whose Ollama capabilities include tools (e.g. qwen3.6). See §10. |
Uploaded documents aren't searchable / nodeType:Document returns 0 / an agent can't find its input |
Content indexing needs an embedding provider, which is separate from the chat model and off by default. Set Embedding__Provider/Embedding__Endpoint/Embedding__Model (§10.1) and re-upload — indexing fires on the upload event, so pre-existing files aren't retro-indexed. |
memex-local targets the wrong cluster — helm fails kubernetes cluster unreachable: …privatelink…azmk8s.io, or (worse) a reachable cloud cluster gets rolled |
🚨 memex-local uses the ambient kubectl context, not a pinned one. If your current context is an AKS cluster, up/update deploy there. Check with kubectl config current-context and switch: kubectl config use-context colima. It failed loudly here only because that cluster is private and unroutable — a reachable one would have been rolled silently. |
https://memex.localhost:8443 refuses the connection although the pods are healthy |
Same root cause as above: the launchd port-forward agent runs kubectl port-forward with no context, so an AKS current-context breaks it. Fix the context, then memex-local port-forward. |
| Plugin Catalog is empty / a new instance installs nothing | The local portal is the registry and serves plugins from your checkout (§16). No checkout found ⇒ nothing to serve. Confirm the config landed: kubectl -n memex get cm memex-portal-config -o json \| grep PluginCatalog__Sources and that the mount exists: kubectl -n memex exec deploy/memex-portal-deployment -- ls /plugin-repos/Plugins. Node repos beside MEMEX_REPO are discovered by content (a <Folder>/index.json with a root nodeType), so a repo that declares no package root is silently not one; point at it explicitly with MEMEX_PLUGIN_REPOS=/path/to/repo. |
instance up fails 409 — Instance id '<id>' is already registered |
An instance id is claimed globally on the registry, and dropping the instance's database does not release it. Use memex-local instance down --id <id> (which releases the claim and restarts the registry) before re-running, or pick a new id. |
A plugin install fails with NodeType(s) not registered: <Other>/<Type> |
The package depends on another that is not installed yet. The default install orders by the manifest's requires; a package that depends on something outside the granted set cannot be ordered against and will fail. Grant the dependency too (§16). |
Instance pod OOMs mid-install (OutOfMemoryException, often surfacing inside Npgsql) and the remaining packages never install |
A first boot compiles every default-installed plugin's node types back to back, and each compile retains its collectible ALC. memex-local sizes the instance pod 3cpu/6Gi for this; a smaller pod dies partway. Installing fewer packages by default (a narrower InstallByDefault) is the other lever. |
Every control renders as debug text — NamedAreaControl { Id = , Style = , … } instead of UI |
The view packs are not installed. DefaultViews/GraphViews left the image in #2169 Phase B2 and arrive only as registry bundles; without them nothing has a view and the control falls back to its ToString(). 🚨 The documented repair — Settings ▸ Administration ▸ Plugin Catalog — is itself rendered by the missing views, so the portal cannot repair itself through its own UI. Diagnose from outside: memex-local verify. Both packs declare preInstalled, so the boot reconcile installs them from your checkout on every boot; if it did not, memex-local logs --no-follow --tail 400 \| grep DefaultInstall says why (a source name that matches no grant, an unreadable repo, a platform floor). (#2367) |
/login 404s — no way to sign in at all |
The login pages are Blazor pages compiled into the image (MeshWeaver.Plugins/src/Memex.Portal.Gui/Pages/), so this is an image built before the GUI move (#2293), not a missing package. Rebuild: memex-local update --build, with the plugins checkout on main. (#2367) |
Verify end-to-end that the portal is usable, not merely answering:
memex-local verify
# Asserts three things and exits non-zero, naming a remedy, if any fails:
# • the portal SERVES — an HTTP status in the serving range (a 503 is not "reachable")
# • /login is ROUTED — there is a way to sign in
# • the view packs are there — MeshWeaver.Blazor.Views + MeshWeaver.Blazor.Graph
🚨 up and update run this for you and refuse to report success without it. The check it replaced
curl'd / and printed "Portal reachable" whenever curl exited 0 — so a 503, a 404 and a portal
rendering every control as its ToString() all produced the same green line (#2367). If you want the
raw TLS/routing signal on its own:
curl --cacert "$(mkcert -CAROOT)/rootCA.pem" \
-sS -o /dev/null -w 'http=%{http_code} ssl_verify=%{ssl_verify_result}\n' \
https://memex.localhost:8443/
# Expect: http=200 (ssl_verify_result 0 once `mkcert -install` has trusted the CA)
15. Playwright E2E test env (memex-local e2e)
Browser E2E (Playwright) needs a portal it can log into without Entra (DevLogin) that also has a real language model. The dev Monolith has DevLogin but no model; this memex stack has the model but uses Entra OAuth and holds your real data. memex-local e2e stands up a throwaway, DevLogin portal built from the current working tree, in the same namespace, reusing this stack's Postgres, host Ollama and ingress/TLS — but against its own database (memex_e2e) with DevLogin on, behind the ingress (reverse proxy) at https://e2e.memex.localhost:8444. It is additive: it never touches the memex release, DB, or config.
memex-local up # the base stack (PG, Ollama, ingress, TLS) — once
memex-local e2e up # build working tree → create memex_e2e → migrate → deploy → reverse-proxy
memex-local e2e test HomeChatExecuteTest # run the Playwright E2E (E2E_BASE_URL + DevLogin preset)
memex-local e2e down # delete the e2e objects + drop the e2e DB (--keep-db to keep it)
What e2e up does, and why:
| Step | Why |
|---|---|
| Build portal + migration image (native arm64) from the working tree | Test the code you're holding, not a stale image. --skip-build reuses the last build. |
CREATE DATABASE memex_e2e in the existing memex-postgres |
Your own data — never the memex DB. |
Run the migration Job against it |
The portal's DbVersionGate refuses to start against an un-migrated DB. |
Deploy memex-e2e-portal (Deployment + Service + Ingress) reusing memex-portal-config/-secrets via envFrom |
Proven config; override only ConnectionStrings__memex → memex_e2e and Authentication__EnableDevLogin=true. Clustering stays Localhost (own in-process silo). /data is an ephemeral emptyDir (mesh data is in PG). |
Ingress for e2e.memex.localhost (covered by the *.memex.localhost mkcert cert) + a :443→:8444 port-forward |
The reverse proxy Playwright drives. |
PortalFixture authenticates via POST /dev/signin?personId=Roland and sets IgnoreHTTPSErrors=true, so the self-signed cert is fine. The repeatable flow is captured as the /playwright skill. Always deploy on Colima and drive THAT — never run a model E2E against the Monolith (no model) or the memex portal (Entra + real data).
16. Plugin registry + provisioning a new instance (memex-local instance)
This stack is also a plugin registry, and it can ramp a second portal that registers itself with it and installs its plugins unattended — the local rehearsal of provisioning a real deployment, with nothing copied by hand.
The local portal serves plugins from your CHECKOUT
A registry normally reads its plugin repos from GitHub through the App identity. A local stack has
no such credential, so it reads them straight off disk: helm_deploy mounts each checkout
read-only (pluginCatalog.localRepoMounts → a hostPath; Colima runs the node on this very
machine) and points a source at the in-container path.
Discovery is automatic, and by CONTENT. Every directory beside your MEMEX_REPO that declares
a package root — a <Folder>/index.json whose nodeType is Space, Store/Plugin or
Store/Catalog, the registry's own predicate — is mounted and served. Clone a node repo next to the
platform checkout and it is picked up on the next memex-local update; nothing to configure.
The platform checkout and its worktrees exclude themselves: they carry no root manifest at that depth, so the rule needs no list of names to skip, and none can go stale.
MEMEX_PLUGIN_REPOS (colon-separated absolute paths) still overrides the discovered set, for
serving fewer repos than you have checked out:
# Each folder's basename (minus a "MeshWeaver." prefix) becomes the SOURCE NAME, and every new
# instance is granted that source.
MEMEX_PLUGIN_REPOS=/Users/me/code/MeshWeaver.Plugins \
memex-local update --build
🚨 It is an environment variable, so it applies to that one invocation. A later
memex-local update typed without it serves the discovered set again — which is why discovery,
not the variable, is the default.
Verify what actually landed — the values file is not the evidence, the pod is:
kubectl -n memex get cm memex-portal-config -o json | tr ',' '\n' | grep PluginCatalog
kubectl -n memex exec deploy/memex-portal-deployment -- ls /plugin-repos/Plugins # the mount
🚨 Local/dev only.
hostPathmeans this machine. On a real cluster the sources are URLs and the registry holds the GitHub App identity — see PluginRegistry.
Ramp an instance
memex-local instance up --id acme # mint key → own DB → migrate → deploy → verify
memex-local instance status --id acme # what it REGISTERED and what it INSTALLED
memex-local instance down --id acme # delete objects, drop DB, release the id on the registry
What instance up does, and why:
| Step | Why |
|---|---|
Mint an mwr_ registration key on the registry (/bootstrap/registration-key, secret-gated) |
A scripted ramp-up has no UI to click. The key is owned by MEMEX_INSTANCE_OWNER, so instances land in that user's partition — never more than that user could self-register. |
CREATE DATABASE memex_<id> + migration Job |
Its own data; the portal's DbVersionGate refuses an un-migrated DB. |
Deploy the instance portal with PluginCatalog__BootstrapKey + __InstanceId and no registry token |
This is the thing under test: it presents the bootstrap key once, receives its own mwi_ key, and stores it encrypted (Admin/PluginRegistryCredential/…). |
| Wait for the installed set to stop growing | Packages install sequentially; the first lands in seconds and the rest take minutes. Reporting after the first one announces "installed: Agent" for a run that installs twenty. |
| Report from the databases | A Running pod proves neither registration nor installation. |
Expected on a healthy run — the instance registers, is granted Plugins/* by
pluginCatalog.defaultGrants, and installs them all in dependency order:
ok registered as instance 'acme'
ok installed packages: Agent, BusinessRules, Chess, ClaudeCode, Collaboration, Copilot,
DataModelling, DoublePendulum, Edu, Essentials, Feedback, FractalStars, Manufacturing,
Publish, RolePlay, Skill, Store, ThreeBody, Training, Video
Adding another plugin repo
- Clone it next to your checkout — nothing else to do, it is discovered. (Somewhere else? Then
list every repo you want in
MEMEX_PLUGIN_REPOS, which replaces the discovered set.) memex-local update --build— the chart mounts it and adds it as a source and grants it to every new instance (defaultGrants: <name>/*).- Whether new instances install it as well is separate:
pluginCatalog.installByDefault(default["Plugins/*"]) is deliberately source-scoped, so a repo you are merely granted is browsable but not auto-installed. Add<name>/*there to auto-install it too.
🚨 The grant/install split is the security property, not a formality: an instance is routinely entitled to paid course content, and "install everything I'm entitled to" would auto-install it.
🚨 What a self-registry install can NEVER have: a module binary. A checkout holds source, not assemblies, and a registry serves a module only from its own
modules/<name>/— so every module-declaring package installs its content and silently skips its DLL (MeshWeaver#2417; the self-sealing "up to date" gates were fixed in #2659, the lane itself stays closed by construction). No Radzen charts, Analysis, EntityViews, GoogleMaps or Speech, andvalues.local.self-registry.yamlblanks those fiveModules__Requiredentries so readiness does not stall on them. The way to a complete local portal is not a local build lane; it is the next section.
17. Registry mode — consume memex.meshweaver.cloud (memex-local registry)
The other way round from §16: this install is a consumer of a remote plugin registry, exactly
like every cloud instance is a consumer of memex.meshweaver.cloud. It runs the CI-built
multi-arch image from ACR (the native arm64 member — no source checkout, no .NET SDK, az login
once), registers itself at the registry on first boot, installs the packages it is granted and
lands their compiled modules from the registry's bundles into /data. Module bundles are
platform-floor-gated IL, so the amd64-built bundles the registry serves land on an arm64 install
unchanged. (Prebuilt NodeType bakes are identity-gated and amd64-only; a local install compiles
those on first access, which is the PreWarm__DynamicTypes: "false" behaviour it already has.)
memex-local registry https://memex.meshweaver.cloud --id my-mac # no key: an OPEN registration → the free tier
memex-local up # a fresh install …
memex-local update # … or an existing self-registry one: pulls the ACR image, re-renders the chart
memex-local registry status
No key is the default. An un-keyed registration is an open one: memex.meshweaver.cloud
accepts it and enrols the install into its default plan, the free tier — every package the
free plan covers, plus the platform baseline. Raising it is a platform admin's edit of the
instance's grant on the registry (Instance grants ▸ Plan); an install never asks for a plan
itself. A registration key (--key mwr_…, minted by a platform admin under Settings ▸
Administration ▸ Instance grants ▸ Registration keys, for a plan) is the way an install lands
on a higher plan from its first boot — and the only way in on a registry that accepts no open
registrations (PluginRegistry.md → Plans).
What changes under the hood, and why:
| Self-registry (§16) | Registry mode (§17) | |
|---|---|---|
| values layers | values.local.defaults.yaml + values.local.self-registry.yaml + overlay |
values.local.defaults.yaml + ~/.memex-local/registry.yaml + overlay |
pluginCatalog |
sources + localRepoMounts off the checkout, defaultGrants |
registryUrl, instanceId, homeUrl; the chart's installByDefault: ["Plugins/*"] |
| bootstrap key | minted on the local portal (instance up) |
none by default (open registration → the registry's default plan); with --key, presented once (PluginCatalog__BootstrapKey). Either way the issued mwi_ key is stored encrypted in this install's database and survives every update |
Modules__Required__0..4 |
blanked (nothing can land them) | the image baseline stands — a store-delivered required module reads ExpectedLater, never a rollout blocker, and verify --repair performs the one activation restart |
| image | built from source (Option B) | ACR main (--build still works — up/update only change their default) |
The registry file is always 0600: it may carry a bootstrap key, and a bootstrap key is a secret
— it registers instances under the minting admin's identity — so the mode does not vary by whether
one was given; registry status never prints it. What this install may pull is decided on the
registry, per instance, in Admin/_PluginGrant/{instanceId} — registering is identity, not
entitlement (PluginRegistry.md). A first boot that installs
nothing has one of two causes, and memex-local verify names both: the registration was refused
(401 — with a key: revoked or expired, mint a new one and re-run registry; without one: that
registry accepts no open registrations, ask its platform admin for a key), or the registry
advertises 0 package(s) to this instance (no grant yet — a platform admin adds one for it).
memex-local registry off deletes the file and the next update serves a checkout again; the
instance stays registered on the registry until an admin revokes it there.
Related
- PluginRegistry.md — the registry model this stack implements locally: instance keys, grants, default installs.
- Deployment.md — the deploy-route index (AKS vs Container Apps) and shared Azure AD / secrets setup.
- DeploymentAKS.md — the cloud counterpart that uses the same
deploy/helmchart shape. - LocalDevWorkflow.md — the faster Aspire/Monolith inner loop for everyday code iteration.
- ControlledIoPooling.md — why all the portal's I/O (including Postgres) goes through the I/O pool, never
Observable.FromAsync.