notes/ LLM memories doesn't go in git

This commit is contained in:
James Babcock committed 2026-10-25 14:20:39 -07:00
1 parent 53812609ed
commit 22f869d31c
7 files changed
-859

No files matched your search

-87
View File
@@ -1,87 +0,0 @@
# Even-Analogous Apps
Ideally we'd like full feature parity with the stock Even app (or at least,
close enough that not-using the stock app doesn't feel like a sacrifice).
A rough list of things the stock app has that we currently don't:
## Quicklist
Quicklist is Even's version of a todo app, in the form of a dashboard card
that you add to and remove from with assistant voice commands. It's pretty
simple to build once a general voice assistant is in place, but I don't
actually like their design: a todo list really wants to be synchronized
with something external like Roam or Notion, so I think it'd be better to
build one of those integrations instead.
## Stocks, News
Even has a dashboard card which is a stock ticker. I think this is probably
misguided; while there do exist people who would want and benefit from a stock
tracker on their glasses, I do'm not sure there exist people who would want and
benefit from a stock tracker that isn't a professional-tier tool connected
to carefully chosen low-latency data sources, and neither I (nor Even) can
build that tool.
The News dashboard card is more broadly useful, but has a similar issue: a
generic news ticker isn't really useful without a much higher degree of
customization. Probably the right thing to build instead is more like an RSS
reader.
## Ring Connection Management
The stock Even app is responsible for pairing the glasses with the ring (ie,
sending the glasses a configuration messages telling it to connect to the
ring's MAC address). If you pair them in the stock app and then switch to a
custom app, they remain paired indefinitely (this configuration is stored in
flash), and our custom app can make use of the fact that they're paired.
We don't yet have support for connecting or disconnecting the glasses and ring
to/from each other. For latency reasons, it would be nice if we could unpair
the glasses from the ring and connect the phone to it directly; we don't yet
support that either.
## Ring Health Metrics
The R1 ring contains an oximeter-style LED sensor and an accelerometer, which
can be used for a step count, heart rate measurement, etc. I'm not sure
whether the sensor interpretation is done within the firmware (in which case
it would make sense to display them in the UI) or in the Even phone app (in
which case we'd be reimplementing all the interpretation algorithms, which
isn't worth it.
## Navigate
Even has a Navigate app, which is basically Google Maps for the specific use
case of "name a destination, get turn by turn directions to there". I believe
this is using Mapbox, which takes care of most of the heavy lifting (if you
have an API key). I don't think Even's version is very good (they ran into
trouble with image-upload performance preventing a good map display size), so
it's likely not too hard to do better. One caveat is the location permission;
Android normally only wants to prompt for location permission "while using the
app", but Android's idea of in use (phone screen unlocked and window focused)
doesn't match our idea of in use (G2 glasses on, phone could be locked). This
app also gets a lot better if we figure out how to use the device's
magnetometer (if it indeed has a working magnetometer).
## Distance/Viewport Configuration
During glasses onboarding, the stock firmware has you select a vertical offset
for the screen image. This accounts for glasses resting at different positions
on your face (the optics don't let you see the whole screen area), but cuts
the usable screen height from 480px to 288px. There is also a "distance"
config option in the stock Android app, which adjusts the horizontal position
of the image on the two lenses separately to create the impression of the
image being closer or farther; this cuts the usable screen width from 640px to
576px. Unlike the vertical size reduction, this 64px reduction is not
optically necessary.
## Even AI
## Translate and Conversate
## Other Configuration
Brightness, head-up trigger
# Other Apps
-41
View File
@@ -1,41 +0,0 @@
# Hermes agent as the assistant
The "Hermes agent" assistant model sends what you say to a
[Hermes Agent](https://github.com/NousResearch/hermes-agent) gateway through its
OpenAI-compatible API server and streams the answer back to the glasses. Hermes
runs its own tools and keeps the conversation: each Faceclaw assistant
conversation maps to one Hermes session (`X-Hermes-Session-Id`), and only the
latest message is sent. Faceclaw's own glasses tools are not offered to Hermes.
The session ID is saved with the conversation, so a conversation restored after
an app restart continues the same Hermes session.
A Hermes turn may run for up to 20 minutes (other models get 2), because Hermes
runs whole multi-tool tasks in one request and closing the stream interrupts the
run on the server. A run that fails or is interrupted on the server shows as an
error on the glasses, with any partial answer kept.
## Enable the API server in Hermes
In the Hermes config, enable the `api_server` platform and bind it where the
phone can reach it, then set a key in Hermes's `.env` and restart the gateway:
hermes config set platforms.api_server.enabled true
hermes config set platforms.api_server.extra.host <address the phone can reach>
hermes config set platforms.api_server.extra.port 8642
echo "API_SERVER_KEY=$(openssl rand -hex 24)" >> ~/.hermes/.env
Check it with `curl -H "Authorization: Bearer <key>" http://<host>:8642/v1/models`.
## Settings (Settings > Assistant)
- **Assistant model**: Hermes agent. Disabled until the URL is set.
- **Hermes URL**: e.g. `http://host:8642/v1` (a bare host gets `/v1` added).
- **Hermes key**: the `API_SERVER_KEY` value.
On iOS, App Transport Security blocks plain `http://` to IP addresses unless
`App_Resources/iOS/Info.plist` has an `NSExceptionDomains` entry for that host
(`NSExceptionAllowsInsecureHTTPLoads`); otherwise serve the API over https, for
example with `tailscale serve`.
Actions that Hermes holds for approval (`approvals.mode manual`) cannot be
approved from the glasses; approve them from Hermes itself.
-66
View File
@@ -1,66 +0,0 @@
# iOS protocol synchronization — 2026-09-17
Merged `main` at `1727474` into `ios-port`. The iOS session now uses the
same transport contract as Android's `CfwTransport`, `CfwMessageWindow`,
`OutboundMessage`, and `BleImageOptimizer`.
- The EvenHub page contains only the input text container. Display commands
and cleanup use SID `0xf0`, without image-container or image-update headers.
- Mode 6 full frames and mode 3 bounding boxes contain uncompressed RLE.
Persistent zlib with SYNC_FLUSH runs in wire order at the transport layer.
Record lengths and checksums describe the same data as Android; packet
sizes respect CoreBluetooth's maximum write value length (ATT MTU minus 3).
- Commands larger than the uint16 record limit become independent 64-row
mode 3 bands. Frame metrics advance only when all bands complete.
- The three-command window requires ACKs for both processing lenses on the
ingress BLE link, with matching stream ID, ordinal, decoded size, and CRC.
Primary and redundant ACKs are validated before application; completion
remains ordered even when replies arrive out of order.
- NACKs or 500 ms ACK timeouts replay the entire unresolved window, using
fresh stream IDs and reset compression contexts. Three retries are allowed
before reconnecting and invalidating the delta base. Deadlines begin after
writes finish. Old queued attempts and late ACKs cannot complete replays.
- Firmware acceptance uses shared numeric revision checks (currently
Faceclaw/13 or newer), replacing the old EVENCFW feature-token check.
Legacy or stock firmware is rejected before display startup.
Merge conflicts retained iOS voice restrictions while preserving main's
asynchronous transcription finalization. The iOS voice-state bridge supplies
the new listening/detail fields. Test harnesses account for platform globals
and the shared SVG rasterizer.
Validation: TypeScript checking and signed physical-device iOS build pass.
The simulator build launches and renders the main screen and shared launcher.
The final signed app is installed on the development iPhone. The user renewed
developer-certificate trust after iOS initially blocked launch. A subsequent
NativeScript deployment launched successfully, connected to the glasses, and
received both-lens ACKs for a display frame.
The transport parity test compiles Android's production Java encoder and
compares complete packets, including persistent history, reset, MTU limits,
and raw fallback for incompressible maximum-size records. Session tests cover
ACK identity, two-lens completion, ordered pipelining, band reconstruction,
NACK replay, timeout/retry limits, microphone/input responsiveness, and cleanup.
The full suite has 420 passing tests and 21 failures. All 21 reproduce on a
clean archive of main at 1727474: compass, pinball/touch, and terminal test
harness/behavior checks. There are no additional failing tests from this merge.
Further hardware acceptance: scroll and animate the display, visually check
both lenses, exercise mic and ring input, lock the phone, and disconnect/reconnect.
The initial successful frame does not establish sustained radio reliability.
## Development deployment and stale LiveSync code
Installing the IPA with `devicectl` alone left an old September 8 JavaScript
bundle in `Library/Application Support/LiveSync/app` in the app's data container.
NativeScript debug startup used that bundle instead of the newly installed
one, causing the obsolete `img640, fbguard, wearnotify` firmware rejection.
Refreshing through NativeScript fixed the issue without clearing app settings:
```sh
npm run run:ios -- --device 00008150-000C383921A3C01C --no-hmr --no-watch
```
Use this deployment path when updating an existing development installation.
The refreshed on-device `bundle.mjs` was copied back and verified against the
prepared bundle; it contains the shared revision-13 check and SID `0xf0`.
-13
View File
@@ -1,13 +0,0 @@
# Music: "Resume" action via media-key dispatch
Follow-up to the MediaBrowserService browsing support. When no media session
is active, `AudioManager.dispatchMediaKeyEvent(KEYCODE_MEDIA_PLAY)` (down+up
pair) is routed by the system to the *last* media app's manifest-declared
`MEDIA_BUTTON` receiver, even if that app's process is dead — the same
mechanism a Bluetooth headset play button uses. Most players respond by
resuming the last queue.
Idea: add a "Resume last" row to the Music app's no-active-session screen
(next to the browse entry). ~5 lines in `FaceclawMediaController` plus one UI
row; no per-app negotiation needed. Caveat: it's fire-and-forget — no feedback
if nothing handles it, so the UI should just wait for a session to appear.
-133
View File
@@ -1,133 +0,0 @@
# MentraOS app compatibility for Faceclaw — feasibility assessment (2026-07-23)
MentraOS: https://github.com/Mentra-Community/MentraOS (MIT), docs at https://docs.mentraglass.com.
## TL;DR
Technically feasible with moderate effort, but the timing is awkward: MentraOS is
**deprecating its entire current app API**. MentraOS 3.0 ships **Aug 3, 2026** and
cloud-SDK apps stop working on it (a "MentraOS Legacy" mode preserves them through
**October 2026**). The replacement — a **Miniapp SDK** where apps are phone-local JS
bundles running in a WebView — gets a public spec in **September 2026**, and that model
is a *much* better fit for Faceclaw than the cloud model. Recommendation: wait for the
miniapp spec and target that; only build a minimal cloud-protocol emulator if access to
the existing app corpus is wanted in the interim.
## MentraOS app model (current, "Cloud SDK" / TPA era)
- Apps are developer-hosted server processes (Node/Bun, `@mentra/sdk`). Chain:
glasses → phone (BLE) → MentraOS Cloud (WS) → app server (WS).
- Start flow: cloud POSTs a `session_request` webhook `{sessionId, userId, websocketUrl}`
to the app's registered URL; the app dials back to `websocketUrl` and sends
`tpa_connection_init {packageName, sessionId, apiKey}`; cloud acks with settings +
device capabilities. **The webhook carries the WS URL, so an unmodified `@mentra/sdk`
app can be pointed at any server** — that is the compat hook.
- Protocol: JSON over WebSocket (binary frames for audio). 28 app→cloud + 28 cloud→app
message types, defined in `cloud/packages/sdk/src/types/` (`message-types.ts`,
`streams.ts`, `layouts.ts`, `webhooks.ts`, `capabilities.ts`). The TS types are the spec.
- Subscriptions: `subscription_update` with a full list of stream types, some
parameterized (`transcription:en-US`, `touch_event:forward_swipe`). Events arrive in a
`data_stream` envelope.
- Display: 7 layout types — `text_wall`, `double_text_wall`, `dashboard_card`,
`reference_card`, `bitmap_view` (base64), `bitmap_animation`, `clear_view` — sent as
`display_event {view: 'main'|'dashboard', layout, durationMs?, forceDisplay?}`.
Cloud throttles displays to 1 per 300 ms. Dashboard = shared look-up surface.
- Streams: `transcription` (`{text, isFinal, ...}`), `translation`, `VAD`, `audio_chunk`
(PCM), `button_press`, `touch_event`, `head_position` (up/down), `glasses_battery_update`,
`phone_battery_update`, `location_update`, `phone_notification`(+dismissed),
`calendar_event`, plus camera/RTMP/LED families irrelevant to G2.
- Other: ElevenLabs TTS (`audio.speak`), per-app settings (schema in dev console, 9
setting types, live `settings_update`), webview UI with signed-token auth, cloud KV
storage, app-to-app messaging.
- Hardware abstraction: `Capabilities` flags per device. **Even Realities G2 is a
first-class device** (`capabilities/even-realities-g2.ts`): display 640×200 (their
rendering canvas, not the 576×288 EvenHub raster), maxTextLines 5, bitmap-capable,
capacitive swipe bar with TAP/DOUBLE_TAP/TRIPLE_TAP/PRESS_HOLD/SWIPE_UP/SWIPE_DOWN,
IMU, mic, no camera/speaker/LED.
- Licensing: MIT throughout; cloud is self-hostable in principle (Bun + MongoDB + S3 +
third-party STT/TTS keys) but heavy.
## Coming model (Miniapp SDK, MentraOS 3.0)
Miniapps are JS bundles (`index.html` + `miniapp.json`) running **on the phone** inside
the MentraOS host app's WebView ("Mentra Runtime"), packages `@mentra/miniapp` +
`@mentra/miniapp-cli`, sideloaded via QR. API is analogous but smaller:
`session.transcription`, `session.display.showTextWall`, `session.storage`. Example:
https://github.com/Mentra-Community/LiveCaptionsLocalSdkMiniapp. Public spec Sept 2026.
## Fit with Faceclaw
Faceclaw's worker-per-app architecture is unusually favorable: apps are already isolated
contexts speaking a small typed JSON vocabulary (`WorkerAppMessage`/`WorkerAppReply` in
`app/ui/shell/worker-window.ts`), with pixels out-of-band. The compat layer would be one
compiled-in "MentraOS host" app (a peer of `WorkerAppHost`) that presents each external
app as a `ShellWindow` and:
- **Rasterizes layouts** → trivial. All 7 layout types map onto `GrayImage` +
`drawTextWrapped`/`bitBlt` into the 540×260 `APP_VIEWPORT`. Bitmaps need scaling from
their 640×200/576×136 assumptions; monochrome already matches.
- **Maps input** → good. `DashboardInputEvent` (`click`/`double-click`/`long-press`/
`scroll-up`/`scroll-down`) covers their G2 button-event list almost 1:1 (TRIPLE_TAP
missing). Work item: input is currently routed only to the focused window; MentraOS
wants multi-subscriber fan-out (background apps), so a broadcast layer is needed.
- **Transcription** → the load-bearing stream, and Faceclaw already has it:
`voiceControlBridge.onTranscript` (`{text, isFinal}`, replace semantics — matches
MentraOS's `TranscriptionData` model well). Onboard Moonshine is English-only;
parameterized languages would fall back to ElevenLabs/Whisper cloud providers.
Needs multi-consumer fan-out and a policy for when the mic is open (MentraOS keeps it
effectively continuous; Faceclaw is PTT/wakeword-oriented — battery implications).
- **VAD** → sherpa endpointing exists. **audio_chunk** → PCM already surfaced on the
ElevenLabs path (`onPcm`).
- **Battery / phone notifications / calendar** → data already flows in Faceclaw
(top-bar battery, notification listener, calendar app); just needs re-emission as
streams.
- **head_position** → gap. Faceclaw exposes raw IMU only (no gesture classification);
the look-up dashboard gesture would need a small accelerometer classifier (cmd 19
stream exists, see imu-accelerometer memory).
- **TTS / audio out** → G2 has no speaker; MentraOS plays through the phone. Faceclaw
would need phone-side audio playback (new but simple Android work).
- **Settings** → MentraOS per-app settings map onto the shared `FaceclawSettings` store
with a `mentra.<packageName>.*` prefix; the 9 setting types map onto
`ConfigSetting` subclasses, mostly.
- **Not applicable on G2**: camera/photo/RTMP/LiveKit, LED, WiFi setup — can be
rejected via capabilities, which the protocol explicitly supports.
## The hard parts
1. **Who plays "cloud"?** Cloud-SDK apps dial out to a WS URL given in a webhook, so
Faceclaw must (a) run a WS **server** — none exists anywhere in the codebase today,
all networking is outbound okhttp client (`FaceclawWebSocket`, g2mirror pattern) —
reachable from wherever the app server runs (fine on LAN; NAT pain otherwise), and
(b) send the session webhook itself (simple outbound HTTP POST). A phone-hosted
mini-cloud with Java-WebSocket/NanoWSD listening on LAN is the sane MVP; self-hosting
their actual cloud (Bun+Mongo+Docker) is overkill.
2. **Session lifecycle**: connection init/ack, `subscription_update` bookkeeping,
`reconnect`/`reconnect_ack/rejected/deferred`, `app_stopped`, per-app permission
gating. This is bookkeeping-heavy but not deep.
3. **Display arbitration**: foreground vs `background`+`forceDisplay` apps, `durationMs`
auto-revert, the shared dashboard view. Faceclaw's shell has focus/foreground but no
equivalent of background-app display preemption or a look-up dashboard.
4. **App discovery/registration UX**: no dev console; needs a simple "add app by URL +
API key" settings flow (API key check can be a no-op or shared secret).
## Effort estimate
MVP covering the bulk of display-glasses apps (live captions, teleprompters, dashboards,
notification relays): ~10–15 message types + ~8 stream types + webhook sender + LAN WS
server + layout rasterizer + settings mapping. Roughly a few weeks of focused work, with
transcription-stream policy and the WS server being the riskiest pieces. Full-surface
emulation (webview auth, app messaging rooms, TTS, location tiers) is substantially more
and mostly not worth it.
## Recommendation
- **Don't invest in the cloud-TPA protocol as the long-term target** — the vendor is
killing it within ~3 months of now, and the ecosystem's apps will migrate to miniapps.
- **Target the Miniapp SDK when its spec lands (Sept 2026).** It's strictly easier for
Faceclaw: JS bundle in an Android WebView with an injected `session` bridge — no
server, no webhooks, no reconnect protocol — and Faceclaw already has every underlying
facility the bridge needs. The rasterizer/input/transcription plumbing built for it
would be shared with any interim TPA emulator anyway.
- If interim access to existing cloud-SDK apps matters, build the minimal LAN emulator
described above, scoped to text layouts + transcription + button streams, and treat it
as disposable.
-466
View File
@@ -1,466 +0,0 @@
# Voice Assistant Architecture
Status: design draft, 2026-07-19; revised 2026-07-24 (tool availability
ontology, misc fixes); revised 2026-08-01 (external bridge landed).
**Phases 1 and 2 are implemented**, and **phase 3 + the OpenClaw part of
phase 4 landed 2026-08-01** with one architecture change: the bridge server
is not a standalone package — it *is* the OpenClaw plugin
(`~/repositories/faceclaw-agent-bridge`, installed on the OpenClaw host).
The plugin hosts the websocket endpoint the phone dials into, runs
utterances as embedded OpenClaw agent turns (`runEmbeddedAgent` with a
dedicated session, default key `faceclaw:glasses`), and exposes the
glasses' MCP-served tools to the agent via two fixed meta-tools
(`glasses_list_tools` / `glasses_call` — fixed names because OpenClaw's
manifest `contracts.tools` wants static declarations, while the glasses
toolset is dynamic). The generic "MCP face for any agent" from the original
design is unbuilt; the wire protocol is unchanged, so a standalone bridge
for other agents can be added later without phone changes.
## Goal
A default voice assistant reachable from anywhere in the shell: hold the
button, speak, and the utterance goes to an agent that can answer in text on
the lenses and act on the glasses through tools. The agent is pluggable:
- **Direct mode**: the phone runs the agent loop itself against the Anthropic
Messages API (tool-use loop, conversation state on-phone).
- **External mode**: the utterance is routed to the user's own long-running
agent (OpenClaw, Hermes, or anything similar running on their machine), and
that agent gets access to the glasses' tools while it works.
Tools come from two sources, with four availability tiers (see "Tool
availability ontology" below):
- **Glasses/system tools**: provided by the shell, always available (show
text, notifications, screen state, open app, ...).
- **App tools**: contributed by apps. Each app tool declares how much of the
app needs to exist for the tool to be live: merely **installed** ("start a
5 minute timer" must not require the Timer app to be open), **open**
("next track" needs a running Music window, foreground or not), or
**foreground** ("scroll down" only makes sense aimed at the foreground app).
## What we already have
- **PTT capture UI**: `VoiceInputLayer` (app/ui/apps/voice-input.ts) handles
push-to-talk, live transcripts, and a Send/Continue/Discard menu; `Shell`
opens it on long-press and delivers Send to the foreground window's
`receiveTextInput` (app/ui/shell/shell.ts).
- **Wakeword**: On-device classifier listens for the wakeword "Hey Even",
which sends a message to the phone which wakes the screen and starts voice
input.
- **STT**: `voiceControlBridge` with onboard Moonshine, Whisper, or ElevenLabs
cloud.
- **LLM client**: `streamAnthropicMessage` (app/native/anthropic.ts) — raw
SSE streaming over `FaceclawSseRequest`. Text-only today; no tool use.
- **Worker app protocol**: `WorkerAppMessage`/`WorkerAppReply`
(app/ui/shell/worker-window.ts) — small-JSON postMessage protocol between
shell and app workers; the natural place to add tool declarations.
- **Dial-out websocket precedent**: `G2MirrorClient` — JSON over a websocket
to a server on the user's machine (tailscale host/port/token settings).
The external-agent bridge should copy this shape.
## Architecture overview
```
main isolate (shell)
┌───────────────────────────────────────────────────────────┐
│ Shell ──(PTT)──> VoiceInputLayer ──> AssistantSession │
│ │ │
│ AssistantLayer (overlay UI) <──streaming──┤ │
│ │ │
│ AssistantBackend │
│ ┌────────┴─────────┐ │
│ │ Direct │ External│ │
│ └────┬────┴────┬────┘ │
│ │ │ │
│ ToolRegistry <────────────────────────┴─────────┘ │
│ ├─ system tools (in-process handlers) │
│ ├─ app "installed" tools (shell-side or launch-on-call) │
│ └─ app "open"/"foreground" tools (proxied to app worker) │
└──────────┬──────────────────────────────────────┬─────────┘
│ postMessage │
app worker (e.g. terminal) Anthropic API (SSE)
set-tools / tool-call / or agent bridge (WS)
tool-result on the user's machine
```
Everything assistant-side lives on the main isolate. Rationale: tool handlers
need shell state (window list, focus, notifications, settings), the agent
loop is I/O-bound not CPU-bound, and the overlay UI is a shell layer anyway.
If conversation rendering ever gets expensive it can move to a worker later
without changing the tool protocol, which is already async message passing.
### Core pieces
1. **`ToolRegistry`** (new, `app/assistant/tool-registry.ts`)
- Holds `ToolSpec`s and handlers; answers `listTools(scopeFilter)` and
`callTool(name, args) -> Promise<ToolResult>`.
- Emits `onToolsChanged` (needed for MCP `tools/list_changed` in external
mode, and consulted at each turn boundary in direct mode).
2. **`AssistantBackend`** (interface) with two implementations:
- `DirectAnthropicBackend`: tool-use loop over the Messages API.
- `ExternalAgentBackend`: websocket bridge to the user's agent.
3. **`AssistantSession`**: one conversation. Owns history/turn state,
exposes `sendUtterance(text, ctx)`, `cancel()`, and streaming callbacks
(`onTextDelta`, `onToolActivity`, `onTurnDone`, `onError`). Both backends
implement the same session surface so the UI is backend-agnostic.
4. **`AssistantLayer`**: shell overlay showing streamed reply text, a status
line for tool activity ("→ calendar.list_events"), and a small menu
(Follow-up / Done). Double-click cancels the in-flight turn.
## Entry points and routing
- **Wakeword** Saying "Hey Even" wakes the screen if necessary and opens the
dictation dialog. For one-off voice commands, this will be the main entry
point.
- **Context menu** The sidebar context menu and app context menu already have
a "Voice input" option, which opens the dictation dialog whose Send goes to
the foreground window. We'll split the Send option in this dialog into two
options, "Send to Assistant" and "Type Into App". If the entry point was via
context menu, "Type Into App" is highlighted by default; if the entry point
was the wakeword, "Send to Assistant" is highlighted by default instead.
(The capture UI is the existing `VoiceInputLayer` flow, with Send handing off
to `AssistantSession` and morphing into `AssistantLayer`).
## Tool model
```ts
type ToolAvailability =
| "always" // system tools; no app involved
| "installed" // app tool, live whenever the app is installed —
// the app need not have any window open
| "open" // app tool, live while the app has a window open,
// foreground or backgrounded
| "foreground"; // app tool, live only while the app owns the
// foreground window
type ToolSpec = {
name: string; // namespaced: "glasses.show_notification",
// "app.terminal.send_input"
description: string;
inputSchema: object; // JSON Schema
availability: ToolAvailability;
/** May be invoked by the external agent outside an active voice turn. */
proactive?: boolean;
timeoutMs?: number; // default 10s; registry enforces
};
type ToolResult =
| { ok: true; content: string } // text payload back to the model
| { ok: false; error: string };
```
### Tool availability ontology
Every tool sits in exactly one tier, and the tier answers two questions: *when
does the tool appear in `listTools()`*, and *who handles the call*.
| Tier | Live when | Handler | Examples |
|---|---|---|---|
| `always` | always | shell, in-process | `glasses.get_state`, `notifications.list` |
| `installed` | app installed (launcher entry exists) | shell-side handler registered with the app entry, or launch-the-app-then-proxy | `timer.set` ("start a 5 minute timer" must work with no Timer window), `music.play` (cold-start playback) |
| `open` | app has ≥1 open window, foreground or not | app worker (or in-process window) via tool-call proxy | `music.next_track`, `terminal.send_input` |
| `foreground` | app owns the foreground window | app worker via tool-call proxy | `teleprompt.scroll_down`, anything meaning "act on what I'm looking at" |
Mechanics per tier:
- **`installed`** tools cannot be declared by the worker (it may not be
running), so they are declared statically alongside the app's launcher
registration. Two handler flavors: (a) a shell-side handler function
registered with the entry — right for tools that don't actually need the
app UI (timers, starting playback); (b) *launch-on-call*: the shell opens
the app (as if launched from the launcher), waits for its `set-tools`
declaration, then proxies the call. Start with (a) only; (b) adds a
launch-latency/timeout dance that no phase-1/2 tool needs.
- **`open`** and **`foreground`** tools are declared by the running app via
`set-tools` (worker apps) or direct registration (in-process windows).
The registry filters `foreground` tools by whether the declaring app owns
the current foreground window at list time and again at call time —
a `foreground` tool called after a foreground swap fails with a normal tool
error rather than acting on a window the user is no longer looking at.
- A tier is a *floor*, not a routing rule: an `installed` tool stays listed
while the app is open or foreground. If an app wants different behavior when
its window exists (e.g. `timer.set` also shows a countdown in the open
Timer window), the shell-side handler checks for the window itself.
> **Flag:** with multiple windows per app (the worker protocol is
> per-window), tool names are namespaced per *app*, so two windows of one
> app declaring the same tool name would collide. Rule: `set-tools`
> replaces the app's toolset per window, and a call routes to the declaring
> window; if two windows declare the same name, last declaration wins and
> earlier ones are dropped with a logged warning. Revisit only if a real
> app needs per-window tools with the same name (none of the planned ones
> do).
### System tools (initial set)
| Tool | Notes |
|---|---|
| `glasses.show_alert` | short text popup on the lenses; proactive-capable |
| `glasses.list_windows` / `glasses.focus_window` / `glasses.open_app` | shell control |
| `glasses.get_state` | screen on/off, foreground app, battery levels, time |
| `glasses.read_screen` | text/description of the current foreground surface (start with window title + app-reported summary; pixel OCR is out of scope) |
| `calendar.list_events` | wraps FaceclawCalendarProvider |
| `media.play_pause` / `media.next` / `media.now_playing` | wraps FaceclawMediaController |
| `notifications.list` / `notifications.dismiss` | wraps notification listener |
| `timer.set` / `timer.cancel` | new, but trivial and high-value for an assistant |
Keep the first release to ~8–12 tools; every tool costs prompt tokens and
model attention in direct mode.
Note that `calendar.*`, `media.*`, and `timer.*` are conceptually
`installed`-tier app tools that happen to be implemented by in-process shell
code; listing them here just means the shell registers their handlers
directly. If any of them later becomes a real app, its tools move to the
app's launcher registration without the model-visible surface changing.
### App tools
Running apps declare `open`/`foreground` tools per window over the existing
worker protocol (`installed` tools are declared statically with the launcher
entry, per the ontology above):
```ts
// WorkerAppReply additions (worker -> shell)
| { type: "set-tools"; windowId: string; tools: ToolSpec[] } // replaces the set
| { type: "tool-result"; callId: string; result: ToolResult }
// WorkerAppMessage additions (shell -> worker)
| { type: "tool-call"; callId: string; windowId: string; name: string; args: unknown }
```
Registry rules:
- Liveness follows the availability tier: `open` tools are removed when the
app's last window closes, `foreground` tools additionally drop out of
`listTools()` when the app loses the foreground. Either transition fires
`onToolsChanged`; in-flight calls to a closed window resolve as errors,
and calls to a `foreground` tool are re-checked at dispatch time.
- Names are auto-prefixed `app.<appId>.` by the registry so apps can't
shadow system tools or each other.
- The shell enforces `timeoutMs` on the postMessage round-trip; a hung
worker yields a tool error, not a hung turn (same philosophy as input
handling — the shell must survive a stuck app).
- In-process shell apps (calendar, music, ...) register handlers directly
with the registry; the worker protocol is just the remote flavor of the
same registration.
- **Mid-turn availability changes** (foreground swap, window close): direct
mode re-lists tools at each loop iteration (tool set is per-API-call
anyway); external mode sends MCP `notifications/tools/list_changed`. A
call to a tool that just vanished returns a normal tool error the model
can react to.
First app to wire up: the terminal —
`app.terminal.send_input` (type into the attached session; `open`,
proactive: false — "rerun the build" should work while the terminal is
backgrounded), `app.terminal.read_screen` (current grid contents; `open`),
`app.terminal.list_sessions` (`open`). This immediately enables "tell the
terminal to rerun the build" style commands and doubles as the reference
implementation — deliberately all `open`-tier; the first `foreground` tool
should come from an app where focus genuinely matters (teleprompt scrolling).
## Direct mode: agent loop on the phone
Extend `app/native/anthropic.ts`:
1. **Streaming tool-use support**: handle `content_block_start` with
`tool_use`, accumulate `input_json_delta`, surface complete tool_use
blocks alongside text deltas. (Today the client only reads `text_delta`.)
2. **`runAgentTurn` loop** (new, `app/assistant/direct-backend.ts`):
messages + `tools` from the registry → stream → if `stop_reason ==
"tool_use"`, execute each call via the registry, append `tool_result`
blocks, continue; until `end_turn`/`refusal` or a safety cap
(max 8 loop iterations, max ~2 min per turn).
3. **Session state**: history kept in memory per `AssistantSession`,
trimmed from the head past a token budget. Sessions idle-expire
(~10 min) so a new PTT starts fresh; "Follow-up" continues the session.
4. **System prompt**: glasses context — output renders on a 576×288
monochrome HUD, so answer in 1–3 short sentences, no markdown, no lists
unless asked; prefer acting via tools over describing; current time,
foreground app, and screen state are injected per turn.
5. Model `claude-sonnet-5`, `effort: "low"` (same latency posture as the
dictation refiner). Model/effort become settings later if needed.
## External mode: bridge to the user's agent
### Topology
The phone **dials out** (g2mirror pattern — phones don't accept inbound
connections reliably; tailscale host + port + token settings) to a small
**bridge server** co-located with the user's agent. One websocket, JSON
frames, three multiplexed channels:
```
{ v: 1, chan: "ctl" | "chat" | "mcp", ... }
ctl: hello {deviceName, token, capabilities}, hello-ack, ping/pong, error
chat: utterance {turnId, text, ctx} (phone -> agent)
text-delta {turnId, text} (agent -> phone)
tool-activity {turnId, label} (agent -> phone, optional status)
turn-done {turnId, stopReason} / turn-error {turnId, message}
cancel {turnId} (phone -> agent)
mcp: { msg: <raw MCP JSON-RPC frame> } (bidirectional)
```
On the `mcp` channel the **phone is an MCP server** (MCP roles come from the
initialize handshake, not from who dialed). It serves `tools/list` from the
ToolRegistry, `tools/call` into it, and emits `tools/list_changed` on
foreground changes. This means the agent-side integration needs zero
Faceclaw-specific tool code — the glasses appear as a normal MCP server.
### Agent-side adapters
The bridge server is a small standalone package (`faceclaw-agent-bridge`,
sibling of g2mirror) with two faces:
1. **MCP face**: re-exposes the phone's MCP channel as a localhost
streamable-HTTP MCP endpoint. The user registers
`http://localhost:<port>/mcp` with their agent (OpenClaw, Hermes, Claude
Code, anything MCP-capable) exactly like any other MCP server. When the
phone is disconnected, `tools/list` returns empty rather than erroring.
2. **Chat face**: pluggable per-agent adapter that forwards `utterance`
frames into the agent and streams replies back:
- **OpenClaw**: a plugin that injects the utterance as an inbound message
on a dedicated "glasses" channel/session and relays the streamed reply.
- **Hermes**: equivalent module against its session API.
- **Fallback adapter**: spawn a configured CLI (e.g. `claude -p --resume`)
per turn — lowest-fidelity but makes the bridge useful with no plugin.
Utterance `ctx` carries `{foregroundApp, screenOn, localTime}` so the remote
agent has the same situational grounding the direct loop gets in its system
prompt.
### Proactive agent actions
Because the MCP channel is live whenever the bridge is connected, the
external agent can call glasses tools **outside a voice turn** — e.g. push a
notification to the lenses when a long job finishes. Gating:
- Only tools marked `proactive: true` are callable outside an active turn
(registry-enforced; enforced on the phone, never trusted to the bridge).
Availability tiers apply to proactive calls unchanged: a proactive
`installed` tool (e.g. `timer.set`) works anytime, while a proactive call
to an `open`/`foreground` tool whose app isn't in the right state gets the
same tool error a mid-turn call would.
- Master setting `assistant.allowProactive` (default on — it's a marquee
feature — but visible and easy to turn off).
- Proactive display actions are rate-limited (e.g. 6/min) and never turn
the screen on unless the user enabled that specifically.
## Settings
New "Assistant" section in dashboard settings:
- `assistant.backend`: `anthropic` | `external` (default `anthropic` if an
API key is set, else `external` if a bridge host is set).
- `assistant.bridgeHost` / `assistant.bridgePort` / `assistant.bridgeToken`
(mirrors the terminal.* trio).
- `assistant.allowProactive`: boolean.
- `voice.skipConfirmationAfterWakeword`: Wakeword-triggered input sends to the
assistant immediately at the end of an utterance, rather than waiting for a
menu selection.
- Reuses existing `llm.anthropicApiKey` for direct mode.
## Security notes
- Bridge auth: bearer token in the `hello` frame, TLS optional because the
expected transport is tailscale (same stance as g2mirror); document that
plainly.
- The phone enforces all tool gating (proactive flags, rate limits,
timeouts). The bridge and agent are treated as honest-but-fallible.
- Tool results may contain private data (calendar, notifications); external
mode ships them to the user's own machine only — no third-party hop. In
direct mode they go to the Anthropic API like any other request content.
- App tools execute inside the app's worker with the app's existing
privileges; the registry adds no new capability beyond what the app could
already do.
## Implementation plan
**Phase 1 — direct assistant, system tools only.** ✅ Landed 2026-07-24
(builds + typechecks; not yet hardware-tested).
ToolRegistry (`app/assistant/tool-registry.ts`) with the full availability-tier
model but only `always` tools wired; 8 system tools (`glasses.get_state`,
`glasses.show_alert`, `calendar.list_events`, `media.now_playing`/`play_pause`/
`next`, `notifications.list`/`dismiss`) in `app/assistant/system-tools.ts`;
tool-use streaming in `anthropic.ts` (content-block assembly, `tools` param);
`DirectAnthropicBackend` + `AssistantSession`; `AssistantLayer` overlay
(`app/ui/apps/assistant.ts`); `VoiceInputLayer` reworked to route via a
`sendTargets` list ("Send to Assistant" / "Type Into App") with entry-point
default highlight and wakeword skip-confirmation auto-send;
`shell.showAlert` popup; `assistant.skipConfirmationAfterWakeword` setting.
*Deliverable: speak wakeword, ask "what's on my calendar", get an answer
on-lens.*
**Phase 2 — app tools.** ✅ Landed 2026-07-25 (builds + typechecks; not yet
hardware-tested).
Worker protocol extended with `set-tools` / `tool-result` (WorkerAppReply) and
`tool-call` (WorkerAppMessage). `ToolRegistry.setAppTools`/`removeAppTools`
key tools by windowId, prefix names `app.<appId>.`, and gate `foreground`
tools on an `isForeground()` predicate (checked at list and dispatch time).
`WorkerAppHost` proxies calls to its worker (callId round-trip, 15s host
timeout backstop on top of the registry's per-tool timeout), registers a
window's tools on `set-tools`, and withdraws them + fails in-flight calls on
window close. Direct mode now re-lists tools each loop iteration.
Terminal is the reference: the hub window declares `send_input` /
`read_screen` / `list_sessions` (all `open`-tier); send_input/read_screen act
on the active view (foregrounded → last-active → sole view). *Deliverable:
"run the build again" typed into the terminal even while it's backgrounded.*
*Not yet done from the Phase 2 design:* `installed`-tier registration on
launcher entries (no `installed` app tool exists yet — timer stays deferred);
in-process-window tool registration (only worker apps wired); the per-window
same-name collision handling (the terminal sidesteps it by declaring all tools
on the single hub window).
**Phase 3 — external bridge.** ✅ Landed 2026-08-01 (see the status note at
the top for the architecture change: bridge server == OpenClaw plugin).
Phone side: `app/assistant/bridge-client.ts` (`assistantBridge` singleton —
dial-out FaceclawWebSocket, hello/token auth, auto-reconnect with 1s→60s
backoff, started at boot by the dashboard controller when configured),
`app/assistant/mcp-server.ts` (MCP over the `mcp` channel: initialize /
tools/list / tools/call from the ToolRegistry, list_changed on
onToolsChanged, proactive gating + 6/min rate limit enforced on-phone),
external mode in `AssistantSession` (`AssistantBackendConfig` union selects
direct vs external; external history lives on the agent's machine), settings
(`assistant.backend`, `assistant.bridgeHost/Port/Token`,
`assistant.allowProactive`) in a new Assistant settings section.
Server side: `~/repositories/faceclaw-agent-bridge` (own repo/package).
*Verified against the live OpenClaw on serac with a simulated phone
(`test/fake-phone.js`): auth, streamed turns, and both tool directions
work; a real model turn needs the OpenClaw instance to have provider auth.*
**Phase 4 — OpenClaw/Hermes adapters + proactive polish.** Partially landed
2026-08-01: the OpenClaw adapter is the plugin itself (no separate chat
adapter needed); proactive gating + rate limits and `tools/list_changed`
are done. Remaining: Hermes (or generic-agent) support — likely a
standalone bridge speaking the same wire protocol with an MCP face +
CLI chat adapter; session persistence questions across reconnects
(currently: OpenClaw session is persistent, phone session is UI-only).
**Hardware-test TODO (external mode):** end-to-end on the real phone +
glasses against serac — connect status, a streamed turn, cancel, follow-up,
proactive `glasses.show_alert` from another OpenClaw channel, and behavior
across phone network transitions (the reconnect backoff is untested on
device).
## Open questions (deferred, not blockers):
- Continuous conversation: after a turn ends, listen for a follow-up
utterance without requiring the wakeword/PTT again? There are some UI
design questions here; maybe multi-turn conversations take the form of an
Assistant app.
- Launch-on-call for `installed`-tier tools that genuinely need the app UI
(open the app, await `set-tools`, proxy the call) — deferred until a tool
needs it.
- `glasses.read_screen` fidelity beyond app-reported summaries? We probably
don't want to do OCR, but we might do something like extending the draw-text
draw call with an on-by-default option that also appends the drawn text to
a string that gets submitted along with the frame.
- Whether assistant conversations should be reviewable in the phone UI
(transcript log) — probably yes, cheap once sessions are objects. The
natural UI home for this would be inside an Assistant app window.
-53
View File
@@ -1,53 +0,0 @@
# Self-hosted Whisper transcription
The "Whisper server (self-hosted)" transcription provider sends voice audio to
any OpenAI-compatible Whisper server instead of a cloud service: vLLM, speaches
(faster-whisper-server), LocalAI and similar. It uses the batch endpoint
`POST {url}/audio/transcriptions` with a 16 kHz mono WAV. Push-to-talk
utterances are one request each; in Transcribe, each span ended by a pause is
one request. While a span is open its audio so far is re-sent every 1.5 s, so
text appears while you are still talking. A span that runs 25 s without a pause
(nonstop speech, music) is cut at its quietest moment in the last 5 s, so no
request outgrows Whisper's 30 s window; the pieces still arrive as one final.
On Android the WAV goes out through `FaceclawBinaryPost` (okhttp), since
NativeScript's http module copies a binary body into Java a byte at a time.
## Server
Any server exposing `/v1/models` and `/v1/audio/transcriptions` works, e.g.
vllm serve openai/whisper-large-v3 --port 8004
## Settings (Settings > Voice)
- **Transcription Provider**: Whisper server (self-hosted). Disabled until the
URL is set.
- **Whisper server URL**: the API root, e.g. `http://host:8004/v1`. A bare
`http://host:8004` gets `/v1` added.
- **Whisper server model**: empty uses the first model the server lists.
- **Whisper language**: an ISO code such as `en`, or empty to auto-detect.
A wrong fixed language makes Whisper translate rather than transcribe.
- **Whisper server key**: optional bearer token.
## iOS: plain http
iOS App Transport Security blocks plain `http://` to IP addresses (including
Tailscale's 100.x addresses) unless the app's Info.plist has an exception for
that host. Either serve the API over https (for example with `tailscale serve`)
or add your server to `NSExceptionDomains` in `App_Resources/iOS/Info.plist`
before building:
<key>192.0.2.10</key>
<dict><key>NSExceptionAllowsInsecureHTTPLoads</key><true/></dict>
## Transcribe on iOS
Transcribe needs a cloud or Whisper-server provider on iOS (Apple's recognizer
only handles short utterances). Its capture follows the glasses: a disconnect,
a glasses lock or backgrounding (for phone-mic capture) transcribes what was
already heard, and listening resumes once the glasses reconnect, unlock or the
app returns to the foreground. Without a glasses session the phone mic stands in
while Faceclaw is in the foreground. With "Save Transcribe audio" on (Voice
settings), each session's audio is kept as `transcribe-<time>.wav` in Downloads,
next to `transcript-<time>.txt` when the text is saved; a resumed capture keeps
appending to the same file.