Voice — talk to your mesh

The Voice package ships a spoken-answer agent tuned for text-to-speech surfaces: at most two speakable sentences, chat model tier, mesh lookups (Get/Search) and web search only when the question needs them, long work redirected to the portal. Installing the package (Get) registers the agent for you; every voice surface then reaches it by agent name Voice.

Surfaces that speak to it

• A voice satellite (e.g. FutureProofHomes Satellite1): the memex voice gateway (clients/voice-gateway in the MeshWeaver.Plugins repo) connects the device on your LAN to any portal — wake word, streaming replies, conversation mode, spoken brain switching, and fire-and-forget delegation to the standard agents ("Frag den Researcher …").
• The mobile app: the microphone in the chat records and transcribes through the portal's POST /api/speech/transcribe and submits to your thread.
• The portal: the chat composer's mic uses the same centralized speech service.

In the setup wizard

The setup wizard (/{you}/GettingStarted) has a Your voice assistant panel in its home chapter for anyone who holds this package. It hands off to the Memex app on an iPhone or a Mac (You → Voice assistant), where the assistant is named and the family records its wake phrase: both talk to the voice gateway on the home network, which the portal cannot reach. It needs both halves up to date: a Memex app build that has You → Voice assistant (it ships through the app store / TestFlight, not with the portal), and a voice gateway restarted on a version that serves /wakeword/* — an older gateway answers the app with "has no wake-word routes".

Speech-to-text

Speech is transcribed by the centralized Whisper container (Doc/Architecture/CentralizedSpeech). language: de transcribes Swiss German OUT as Standard German. Instance operators deploy the container and set Speech__Endpoint / Speech__Enabled. The standard model — Swiss German, German and English in one Apache-2.0 Whisper fine-tune, with per-deployment builds, artifact locations and license rules — is documented in Speech models.

Your assistant's name and wake word

Setting up a voice satellite at home is a short flow in the Memex app on an iPhone or on a Mac — You → Voice assistant — in this order (one device can record the whole family). The app first needs the voice gateway's address: the gateway prints it when it starts (voice assistant setup: … enter http://<computer>.local:8200). In the Mac app on the computer that runs the gateway it connects to http://localhost:8200 by itself. A browser page served over https (a cloud portal) cannot do this — it may not call the gateway's plain http on your network — and says so.

It takes about five minutes per person. It needs a current Memex app and an up-to-date voice gateway: if Voice assistant does not show under You, update the app first, then restart the gateway on its latest version.

1. Name your assistant. The household gives it a name ("Memex" until you choose, or e.g. "Juniper"); the wake phrase follows from it — "Hey Juniper" — and can be edited ("Okay Juniper"). The app checks the phrase as you type: aim for 3–4 syllables, crisp sounds (k, t, p, x, z, sch), and not an everyday word or a name you say in normal conversation — a phrase that is too short or too common wakes the speaker by accident. The choice is saved on the voice gateway (GET/PUT /wakeword/phrase), and the name is what the assistant calls itself (the gateway adds it to the voice prompt at start).
2. Each family member records the wake phrase in their own voice: added by first name, each says it 25 times the ways it is really said (normally, quietly, from across the room, quickly, slowly, while busy, over noise) and records 10 short sentences it must NOT wake on (near-misses of the name, everyday speech). Clips go to the voice gateway on the home network, which keeps them per phrase; the app shows each person's progress and lets a clip be recorded again.
3. Teach the speaker. The gateway retrains the on-device model for the phrase — synthetic voicings plus the family's recordings, weighted so real voices count — and reports the run. The new model then goes onto the speaker with a firmware build and an over-the-air flash; the steps are in the gateway's wakeword/README.md.

Real family voices are the point: the shipped model was trained on synthetic voices only and misses about one "Hey Memex" in nine at its default sensitivity.

Choosing a name changes what the household records and trains for; the speaker keeps listening for its current model (the gateway's WAKE_WORD) until a model for the new phrase is trained and flashed. The app says so while the two differ.

The gateway repository carries a trained "Hey Memex" microWakeWord model with its full, reproducible training pipeline (clients/voice-gateway/wakeword/). It is licensed for personal, non-commercial use (mixed-license training data).

Pricing

Voice is planned for the top subscription tier once subscriptions ship in the Store; until then the package installs without a price.