mirror of
https://git.undefined-dev.fr/Blendman974/faceclaw.git
synced 2026-11-19 21:03:09 +00:00
notes/ LLM memories doesn't go in git
This commit is contained in:
1 parent
53812609ed
commit
22f869d31c
7 files changed
-859
No files matched your search
@@ -1,87 +0,0 @@
|
||||
# Even-Analogous Apps
|
||||
|
||||
Ideally we'd like full feature parity with the stock Even app (or at least,
|
||||
close enough that not-using the stock app doesn't feel like a sacrifice).
|
||||
A rough list of things the stock app has that we currently don't:
|
||||
|
||||
## Quicklist
|
||||
|
||||
Quicklist is Even's version of a todo app, in the form of a dashboard card
|
||||
that you add to and remove from with assistant voice commands. It's pretty
|
||||
simple to build once a general voice assistant is in place, but I don't
|
||||
actually like their design: a todo list really wants to be synchronized
|
||||
with something external like Roam or Notion, so I think it'd be better to
|
||||
build one of those integrations instead.
|
||||
|
||||
## Stocks, News
|
||||
|
||||
Even has a dashboard card which is a stock ticker. I think this is probably
|
||||
misguided; while there do exist people who would want and benefit from a stock
|
||||
tracker on their glasses, I do'm not sure there exist people who would want and
|
||||
benefit from a stock tracker that isn't a professional-tier tool connected
|
||||
to carefully chosen low-latency data sources, and neither I (nor Even) can
|
||||
build that tool.
|
||||
|
||||
The News dashboard card is more broadly useful, but has a similar issue: a
|
||||
generic news ticker isn't really useful without a much higher degree of
|
||||
customization. Probably the right thing to build instead is more like an RSS
|
||||
reader.
|
||||
|
||||
## Ring Connection Management
|
||||
|
||||
The stock Even app is responsible for pairing the glasses with the ring (ie,
|
||||
sending the glasses a configuration messages telling it to connect to the
|
||||
ring's MAC address). If you pair them in the stock app and then switch to a
|
||||
custom app, they remain paired indefinitely (this configuration is stored in
|
||||
flash), and our custom app can make use of the fact that they're paired.
|
||||
|
||||
We don't yet have support for connecting or disconnecting the glasses and ring
|
||||
to/from each other. For latency reasons, it would be nice if we could unpair
|
||||
the glasses from the ring and connect the phone to it directly; we don't yet
|
||||
support that either.
|
||||
|
||||
## Ring Health Metrics
|
||||
|
||||
The R1 ring contains an oximeter-style LED sensor and an accelerometer, which
|
||||
can be used for a step count, heart rate measurement, etc. I'm not sure
|
||||
whether the sensor interpretation is done within the firmware (in which case
|
||||
it would make sense to display them in the UI) or in the Even phone app (in
|
||||
which case we'd be reimplementing all the interpretation algorithms, which
|
||||
isn't worth it.
|
||||
|
||||
## Navigate
|
||||
|
||||
Even has a Navigate app, which is basically Google Maps for the specific use
|
||||
case of "name a destination, get turn by turn directions to there". I believe
|
||||
this is using Mapbox, which takes care of most of the heavy lifting (if you
|
||||
have an API key). I don't think Even's version is very good (they ran into
|
||||
trouble with image-upload performance preventing a good map display size), so
|
||||
it's likely not too hard to do better. One caveat is the location permission;
|
||||
Android normally only wants to prompt for location permission "while using the
|
||||
app", but Android's idea of in use (phone screen unlocked and window focused)
|
||||
doesn't match our idea of in use (G2 glasses on, phone could be locked). This
|
||||
app also gets a lot better if we figure out how to use the device's
|
||||
magnetometer (if it indeed has a working magnetometer).
|
||||
|
||||
## Distance/Viewport Configuration
|
||||
|
||||
During glasses onboarding, the stock firmware has you select a vertical offset
|
||||
for the screen image. This accounts for glasses resting at different positions
|
||||
on your face (the optics don't let you see the whole screen area), but cuts
|
||||
the usable screen height from 480px to 288px. There is also a "distance"
|
||||
config option in the stock Android app, which adjusts the horizontal position
|
||||
of the image on the two lenses separately to create the impression of the
|
||||
image being closer or farther; this cuts the usable screen width from 640px to
|
||||
576px. Unlike the vertical size reduction, this 64px reduction is not
|
||||
optically necessary.
|
||||
|
||||
## Even AI
|
||||
## Translate and Conversate
|
||||
|
||||
## Other Configuration
|
||||
|
||||
Brightness, head-up trigger
|
||||
|
||||
|
||||
# Other Apps
|
||||
|
||||
@@ -1,41 +0,0 @@
|
||||
# Hermes agent as the assistant
|
||||
|
||||
The "Hermes agent" assistant model sends what you say to a
|
||||
[Hermes Agent](https://github.com/NousResearch/hermes-agent) gateway through its
|
||||
OpenAI-compatible API server and streams the answer back to the glasses. Hermes
|
||||
runs its own tools and keeps the conversation: each Faceclaw assistant
|
||||
conversation maps to one Hermes session (`X-Hermes-Session-Id`), and only the
|
||||
latest message is sent. Faceclaw's own glasses tools are not offered to Hermes.
|
||||
The session ID is saved with the conversation, so a conversation restored after
|
||||
an app restart continues the same Hermes session.
|
||||
|
||||
A Hermes turn may run for up to 20 minutes (other models get 2), because Hermes
|
||||
runs whole multi-tool tasks in one request and closing the stream interrupts the
|
||||
run on the server. A run that fails or is interrupted on the server shows as an
|
||||
error on the glasses, with any partial answer kept.
|
||||
|
||||
## Enable the API server in Hermes
|
||||
|
||||
In the Hermes config, enable the `api_server` platform and bind it where the
|
||||
phone can reach it, then set a key in Hermes's `.env` and restart the gateway:
|
||||
|
||||
hermes config set platforms.api_server.enabled true
|
||||
hermes config set platforms.api_server.extra.host <address the phone can reach>
|
||||
hermes config set platforms.api_server.extra.port 8642
|
||||
echo "API_SERVER_KEY=$(openssl rand -hex 24)" >> ~/.hermes/.env
|
||||
|
||||
Check it with `curl -H "Authorization: Bearer <key>" http://<host>:8642/v1/models`.
|
||||
|
||||
## Settings (Settings > Assistant)
|
||||
|
||||
- **Assistant model**: Hermes agent. Disabled until the URL is set.
|
||||
- **Hermes URL**: e.g. `http://host:8642/v1` (a bare host gets `/v1` added).
|
||||
- **Hermes key**: the `API_SERVER_KEY` value.
|
||||
|
||||
On iOS, App Transport Security blocks plain `http://` to IP addresses unless
|
||||
`App_Resources/iOS/Info.plist` has an `NSExceptionDomains` entry for that host
|
||||
(`NSExceptionAllowsInsecureHTTPLoads`); otherwise serve the API over https, for
|
||||
example with `tailscale serve`.
|
||||
|
||||
Actions that Hermes holds for approval (`approvals.mode manual`) cannot be
|
||||
approved from the glasses; approve them from Hermes itself.
|
||||
@@ -1,66 +0,0 @@
|
||||
# iOS protocol synchronization — 2026-09-17
|
||||
|
||||
Merged `main` at `1727474` into `ios-port`. The iOS session now uses the
|
||||
same transport contract as Android's `CfwTransport`, `CfwMessageWindow`,
|
||||
`OutboundMessage`, and `BleImageOptimizer`.
|
||||
|
||||
- The EvenHub page contains only the input text container. Display commands
|
||||
and cleanup use SID `0xf0`, without image-container or image-update headers.
|
||||
- Mode 6 full frames and mode 3 bounding boxes contain uncompressed RLE.
|
||||
Persistent zlib with SYNC_FLUSH runs in wire order at the transport layer.
|
||||
Record lengths and checksums describe the same data as Android; packet
|
||||
sizes respect CoreBluetooth's maximum write value length (ATT MTU minus 3).
|
||||
- Commands larger than the uint16 record limit become independent 64-row
|
||||
mode 3 bands. Frame metrics advance only when all bands complete.
|
||||
- The three-command window requires ACKs for both processing lenses on the
|
||||
ingress BLE link, with matching stream ID, ordinal, decoded size, and CRC.
|
||||
Primary and redundant ACKs are validated before application; completion
|
||||
remains ordered even when replies arrive out of order.
|
||||
- NACKs or 500 ms ACK timeouts replay the entire unresolved window, using
|
||||
fresh stream IDs and reset compression contexts. Three retries are allowed
|
||||
before reconnecting and invalidating the delta base. Deadlines begin after
|
||||
writes finish. Old queued attempts and late ACKs cannot complete replays.
|
||||
- Firmware acceptance uses shared numeric revision checks (currently
|
||||
Faceclaw/13 or newer), replacing the old EVENCFW feature-token check.
|
||||
Legacy or stock firmware is rejected before display startup.
|
||||
|
||||
Merge conflicts retained iOS voice restrictions while preserving main's
|
||||
asynchronous transcription finalization. The iOS voice-state bridge supplies
|
||||
the new listening/detail fields. Test harnesses account for platform globals
|
||||
and the shared SVG rasterizer.
|
||||
|
||||
Validation: TypeScript checking and signed physical-device iOS build pass.
|
||||
The simulator build launches and renders the main screen and shared launcher.
|
||||
The final signed app is installed on the development iPhone. The user renewed
|
||||
developer-certificate trust after iOS initially blocked launch. A subsequent
|
||||
NativeScript deployment launched successfully, connected to the glasses, and
|
||||
received both-lens ACKs for a display frame.
|
||||
The transport parity test compiles Android's production Java encoder and
|
||||
compares complete packets, including persistent history, reset, MTU limits,
|
||||
and raw fallback for incompressible maximum-size records. Session tests cover
|
||||
ACK identity, two-lens completion, ordered pipelining, band reconstruction,
|
||||
NACK replay, timeout/retry limits, microphone/input responsiveness, and cleanup.
|
||||
|
||||
The full suite has 420 passing tests and 21 failures. All 21 reproduce on a
|
||||
clean archive of main at 1727474: compass, pinball/touch, and terminal test
|
||||
harness/behavior checks. There are no additional failing tests from this merge.
|
||||
|
||||
Further hardware acceptance: scroll and animate the display, visually check
|
||||
both lenses, exercise mic and ring input, lock the phone, and disconnect/reconnect.
|
||||
The initial successful frame does not establish sustained radio reliability.
|
||||
|
||||
## Development deployment and stale LiveSync code
|
||||
|
||||
Installing the IPA with `devicectl` alone left an old September 8 JavaScript
|
||||
bundle in `Library/Application Support/LiveSync/app` in the app's data container.
|
||||
NativeScript debug startup used that bundle instead of the newly installed
|
||||
one, causing the obsolete `img640, fbguard, wearnotify` firmware rejection.
|
||||
Refreshing through NativeScript fixed the issue without clearing app settings:
|
||||
|
||||
```sh
|
||||
npm run run:ios -- --device 00008150-000C383921A3C01C --no-hmr --no-watch
|
||||
```
|
||||
|
||||
Use this deployment path when updating an existing development installation.
|
||||
The refreshed on-device `bundle.mjs` was copied back and verified against the
|
||||
prepared bundle; it contains the shared revision-13 check and SID `0xf0`.
|
||||
@@ -1,13 +0,0 @@
|
||||
# Music: "Resume" action via media-key dispatch
|
||||
|
||||
Follow-up to the MediaBrowserService browsing support. When no media session
|
||||
is active, `AudioManager.dispatchMediaKeyEvent(KEYCODE_MEDIA_PLAY)` (down+up
|
||||
pair) is routed by the system to the *last* media app's manifest-declared
|
||||
`MEDIA_BUTTON` receiver, even if that app's process is dead — the same
|
||||
mechanism a Bluetooth headset play button uses. Most players respond by
|
||||
resuming the last queue.
|
||||
|
||||
Idea: add a "Resume last" row to the Music app's no-active-session screen
|
||||
(next to the browse entry). ~5 lines in `FaceclawMediaController` plus one UI
|
||||
row; no per-app negotiation needed. Caveat: it's fire-and-forget — no feedback
|
||||
if nothing handles it, so the UI should just wait for a session to appear.
|
||||
@@ -1,133 +0,0 @@
|
||||
# MentraOS app compatibility for Faceclaw — feasibility assessment (2026-07-23)
|
||||
|
||||
MentraOS: https://github.com/Mentra-Community/MentraOS (MIT), docs at https://docs.mentraglass.com.
|
||||
|
||||
## TL;DR
|
||||
|
||||
Technically feasible with moderate effort, but the timing is awkward: MentraOS is
|
||||
**deprecating its entire current app API**. MentraOS 3.0 ships **Aug 3, 2026** and
|
||||
cloud-SDK apps stop working on it (a "MentraOS Legacy" mode preserves them through
|
||||
**October 2026**). The replacement — a **Miniapp SDK** where apps are phone-local JS
|
||||
bundles running in a WebView — gets a public spec in **September 2026**, and that model
|
||||
is a *much* better fit for Faceclaw than the cloud model. Recommendation: wait for the
|
||||
miniapp spec and target that; only build a minimal cloud-protocol emulator if access to
|
||||
the existing app corpus is wanted in the interim.
|
||||
|
||||
## MentraOS app model (current, "Cloud SDK" / TPA era)
|
||||
|
||||
- Apps are developer-hosted server processes (Node/Bun, `@mentra/sdk`). Chain:
|
||||
glasses → phone (BLE) → MentraOS Cloud (WS) → app server (WS).
|
||||
- Start flow: cloud POSTs a `session_request` webhook `{sessionId, userId, websocketUrl}`
|
||||
to the app's registered URL; the app dials back to `websocketUrl` and sends
|
||||
`tpa_connection_init {packageName, sessionId, apiKey}`; cloud acks with settings +
|
||||
device capabilities. **The webhook carries the WS URL, so an unmodified `@mentra/sdk`
|
||||
app can be pointed at any server** — that is the compat hook.
|
||||
- Protocol: JSON over WebSocket (binary frames for audio). 28 app→cloud + 28 cloud→app
|
||||
message types, defined in `cloud/packages/sdk/src/types/` (`message-types.ts`,
|
||||
`streams.ts`, `layouts.ts`, `webhooks.ts`, `capabilities.ts`). The TS types are the spec.
|
||||
- Subscriptions: `subscription_update` with a full list of stream types, some
|
||||
parameterized (`transcription:en-US`, `touch_event:forward_swipe`). Events arrive in a
|
||||
`data_stream` envelope.
|
||||
- Display: 7 layout types — `text_wall`, `double_text_wall`, `dashboard_card`,
|
||||
`reference_card`, `bitmap_view` (base64), `bitmap_animation`, `clear_view` — sent as
|
||||
`display_event {view: 'main'|'dashboard', layout, durationMs?, forceDisplay?}`.
|
||||
Cloud throttles displays to 1 per 300 ms. Dashboard = shared look-up surface.
|
||||
- Streams: `transcription` (`{text, isFinal, ...}`), `translation`, `VAD`, `audio_chunk`
|
||||
(PCM), `button_press`, `touch_event`, `head_position` (up/down), `glasses_battery_update`,
|
||||
`phone_battery_update`, `location_update`, `phone_notification`(+dismissed),
|
||||
`calendar_event`, plus camera/RTMP/LED families irrelevant to G2.
|
||||
- Other: ElevenLabs TTS (`audio.speak`), per-app settings (schema in dev console, 9
|
||||
setting types, live `settings_update`), webview UI with signed-token auth, cloud KV
|
||||
storage, app-to-app messaging.
|
||||
- Hardware abstraction: `Capabilities` flags per device. **Even Realities G2 is a
|
||||
first-class device** (`capabilities/even-realities-g2.ts`): display 640×200 (their
|
||||
rendering canvas, not the 576×288 EvenHub raster), maxTextLines 5, bitmap-capable,
|
||||
capacitive swipe bar with TAP/DOUBLE_TAP/TRIPLE_TAP/PRESS_HOLD/SWIPE_UP/SWIPE_DOWN,
|
||||
IMU, mic, no camera/speaker/LED.
|
||||
- Licensing: MIT throughout; cloud is self-hostable in principle (Bun + MongoDB + S3 +
|
||||
third-party STT/TTS keys) but heavy.
|
||||
|
||||
## Coming model (Miniapp SDK, MentraOS 3.0)
|
||||
|
||||
Miniapps are JS bundles (`index.html` + `miniapp.json`) running **on the phone** inside
|
||||
the MentraOS host app's WebView ("Mentra Runtime"), packages `@mentra/miniapp` +
|
||||
`@mentra/miniapp-cli`, sideloaded via QR. API is analogous but smaller:
|
||||
`session.transcription`, `session.display.showTextWall`, `session.storage`. Example:
|
||||
https://github.com/Mentra-Community/LiveCaptionsLocalSdkMiniapp. Public spec Sept 2026.
|
||||
|
||||
## Fit with Faceclaw
|
||||
|
||||
Faceclaw's worker-per-app architecture is unusually favorable: apps are already isolated
|
||||
contexts speaking a small typed JSON vocabulary (`WorkerAppMessage`/`WorkerAppReply` in
|
||||
`app/ui/shell/worker-window.ts`), with pixels out-of-band. The compat layer would be one
|
||||
compiled-in "MentraOS host" app (a peer of `WorkerAppHost`) that presents each external
|
||||
app as a `ShellWindow` and:
|
||||
|
||||
- **Rasterizes layouts** → trivial. All 7 layout types map onto `GrayImage` +
|
||||
`drawTextWrapped`/`bitBlt` into the 540×260 `APP_VIEWPORT`. Bitmaps need scaling from
|
||||
their 640×200/576×136 assumptions; monochrome already matches.
|
||||
- **Maps input** → good. `DashboardInputEvent` (`click`/`double-click`/`long-press`/
|
||||
`scroll-up`/`scroll-down`) covers their G2 button-event list almost 1:1 (TRIPLE_TAP
|
||||
missing). Work item: input is currently routed only to the focused window; MentraOS
|
||||
wants multi-subscriber fan-out (background apps), so a broadcast layer is needed.
|
||||
- **Transcription** → the load-bearing stream, and Faceclaw already has it:
|
||||
`voiceControlBridge.onTranscript` (`{text, isFinal}`, replace semantics — matches
|
||||
MentraOS's `TranscriptionData` model well). Onboard Moonshine is English-only;
|
||||
parameterized languages would fall back to ElevenLabs/Whisper cloud providers.
|
||||
Needs multi-consumer fan-out and a policy for when the mic is open (MentraOS keeps it
|
||||
effectively continuous; Faceclaw is PTT/wakeword-oriented — battery implications).
|
||||
- **VAD** → sherpa endpointing exists. **audio_chunk** → PCM already surfaced on the
|
||||
ElevenLabs path (`onPcm`).
|
||||
- **Battery / phone notifications / calendar** → data already flows in Faceclaw
|
||||
(top-bar battery, notification listener, calendar app); just needs re-emission as
|
||||
streams.
|
||||
- **head_position** → gap. Faceclaw exposes raw IMU only (no gesture classification);
|
||||
the look-up dashboard gesture would need a small accelerometer classifier (cmd 19
|
||||
stream exists, see imu-accelerometer memory).
|
||||
- **TTS / audio out** → G2 has no speaker; MentraOS plays through the phone. Faceclaw
|
||||
would need phone-side audio playback (new but simple Android work).
|
||||
- **Settings** → MentraOS per-app settings map onto the shared `FaceclawSettings` store
|
||||
with a `mentra.<packageName>.*` prefix; the 9 setting types map onto
|
||||
`ConfigSetting` subclasses, mostly.
|
||||
- **Not applicable on G2**: camera/photo/RTMP/LiveKit, LED, WiFi setup — can be
|
||||
rejected via capabilities, which the protocol explicitly supports.
|
||||
|
||||
## The hard parts
|
||||
|
||||
1. **Who plays "cloud"?** Cloud-SDK apps dial out to a WS URL given in a webhook, so
|
||||
Faceclaw must (a) run a WS **server** — none exists anywhere in the codebase today,
|
||||
all networking is outbound okhttp client (`FaceclawWebSocket`, g2mirror pattern) —
|
||||
reachable from wherever the app server runs (fine on LAN; NAT pain otherwise), and
|
||||
(b) send the session webhook itself (simple outbound HTTP POST). A phone-hosted
|
||||
mini-cloud with Java-WebSocket/NanoWSD listening on LAN is the sane MVP; self-hosting
|
||||
their actual cloud (Bun+Mongo+Docker) is overkill.
|
||||
2. **Session lifecycle**: connection init/ack, `subscription_update` bookkeeping,
|
||||
`reconnect`/`reconnect_ack/rejected/deferred`, `app_stopped`, per-app permission
|
||||
gating. This is bookkeeping-heavy but not deep.
|
||||
3. **Display arbitration**: foreground vs `background`+`forceDisplay` apps, `durationMs`
|
||||
auto-revert, the shared dashboard view. Faceclaw's shell has focus/foreground but no
|
||||
equivalent of background-app display preemption or a look-up dashboard.
|
||||
4. **App discovery/registration UX**: no dev console; needs a simple "add app by URL +
|
||||
API key" settings flow (API key check can be a no-op or shared secret).
|
||||
|
||||
## Effort estimate
|
||||
|
||||
MVP covering the bulk of display-glasses apps (live captions, teleprompters, dashboards,
|
||||
notification relays): ~10–15 message types + ~8 stream types + webhook sender + LAN WS
|
||||
server + layout rasterizer + settings mapping. Roughly a few weeks of focused work, with
|
||||
transcription-stream policy and the WS server being the riskiest pieces. Full-surface
|
||||
emulation (webview auth, app messaging rooms, TTS, location tiers) is substantially more
|
||||
and mostly not worth it.
|
||||
|
||||
## Recommendation
|
||||
|
||||
- **Don't invest in the cloud-TPA protocol as the long-term target** — the vendor is
|
||||
killing it within ~3 months of now, and the ecosystem's apps will migrate to miniapps.
|
||||
- **Target the Miniapp SDK when its spec lands (Sept 2026).** It's strictly easier for
|
||||
Faceclaw: JS bundle in an Android WebView with an injected `session` bridge — no
|
||||
server, no webhooks, no reconnect protocol — and Faceclaw already has every underlying
|
||||
facility the bridge needs. The rasterizer/input/transcription plumbing built for it
|
||||
would be shared with any interim TPA emulator anyway.
|
||||
- If interim access to existing cloud-SDK apps matters, build the minimal LAN emulator
|
||||
described above, scoped to text layouts + transcription + button streams, and treat it
|
||||
as disposable.
|
||||
@@ -1,466 +0,0 @@
|
||||
# Voice Assistant Architecture
|
||||
|
||||
Status: design draft, 2026-07-19; revised 2026-07-24 (tool availability
|
||||
ontology, misc fixes); revised 2026-08-01 (external bridge landed).
|
||||
**Phases 1 and 2 are implemented**, and **phase 3 + the OpenClaw part of
|
||||
phase 4 landed 2026-08-01** with one architecture change: the bridge server
|
||||
is not a standalone package — it *is* the OpenClaw plugin
|
||||
(`~/repositories/faceclaw-agent-bridge`, installed on the OpenClaw host).
|
||||
The plugin hosts the websocket endpoint the phone dials into, runs
|
||||
utterances as embedded OpenClaw agent turns (`runEmbeddedAgent` with a
|
||||
dedicated session, default key `faceclaw:glasses`), and exposes the
|
||||
glasses' MCP-served tools to the agent via two fixed meta-tools
|
||||
(`glasses_list_tools` / `glasses_call` — fixed names because OpenClaw's
|
||||
manifest `contracts.tools` wants static declarations, while the glasses
|
||||
toolset is dynamic). The generic "MCP face for any agent" from the original
|
||||
design is unbuilt; the wire protocol is unchanged, so a standalone bridge
|
||||
for other agents can be added later without phone changes.
|
||||
|
||||
## Goal
|
||||
|
||||
A default voice assistant reachable from anywhere in the shell: hold the
|
||||
button, speak, and the utterance goes to an agent that can answer in text on
|
||||
the lenses and act on the glasses through tools. The agent is pluggable:
|
||||
|
||||
- **Direct mode**: the phone runs the agent loop itself against the Anthropic
|
||||
Messages API (tool-use loop, conversation state on-phone).
|
||||
- **External mode**: the utterance is routed to the user's own long-running
|
||||
agent (OpenClaw, Hermes, or anything similar running on their machine), and
|
||||
that agent gets access to the glasses' tools while it works.
|
||||
|
||||
Tools come from two sources, with four availability tiers (see "Tool
|
||||
availability ontology" below):
|
||||
|
||||
- **Glasses/system tools**: provided by the shell, always available (show
|
||||
text, notifications, screen state, open app, ...).
|
||||
- **App tools**: contributed by apps. Each app tool declares how much of the
|
||||
app needs to exist for the tool to be live: merely **installed** ("start a
|
||||
5 minute timer" must not require the Timer app to be open), **open**
|
||||
("next track" needs a running Music window, foreground or not), or
|
||||
**foreground** ("scroll down" only makes sense aimed at the foreground app).
|
||||
|
||||
## What we already have
|
||||
|
||||
- **PTT capture UI**: `VoiceInputLayer` (app/ui/apps/voice-input.ts) handles
|
||||
push-to-talk, live transcripts, and a Send/Continue/Discard menu; `Shell`
|
||||
opens it on long-press and delivers Send to the foreground window's
|
||||
`receiveTextInput` (app/ui/shell/shell.ts).
|
||||
- **Wakeword**: On-device classifier listens for the wakeword "Hey Even",
|
||||
which sends a message to the phone which wakes the screen and starts voice
|
||||
input.
|
||||
- **STT**: `voiceControlBridge` with onboard Moonshine, Whisper, or ElevenLabs
|
||||
cloud.
|
||||
- **LLM client**: `streamAnthropicMessage` (app/native/anthropic.ts) — raw
|
||||
SSE streaming over `FaceclawSseRequest`. Text-only today; no tool use.
|
||||
- **Worker app protocol**: `WorkerAppMessage`/`WorkerAppReply`
|
||||
(app/ui/shell/worker-window.ts) — small-JSON postMessage protocol between
|
||||
shell and app workers; the natural place to add tool declarations.
|
||||
- **Dial-out websocket precedent**: `G2MirrorClient` — JSON over a websocket
|
||||
to a server on the user's machine (tailscale host/port/token settings).
|
||||
The external-agent bridge should copy this shape.
|
||||
|
||||
## Architecture overview
|
||||
|
||||
```
|
||||
main isolate (shell)
|
||||
┌───────────────────────────────────────────────────────────┐
|
||||
│ Shell ──(PTT)──> VoiceInputLayer ──> AssistantSession │
|
||||
│ │ │
|
||||
│ AssistantLayer (overlay UI) <──streaming──┤ │
|
||||
│ │ │
|
||||
│ AssistantBackend │
|
||||
│ ┌────────┴─────────┐ │
|
||||
│ │ Direct │ External│ │
|
||||
│ └────┬────┴────┬────┘ │
|
||||
│ │ │ │
|
||||
│ ToolRegistry <────────────────────────┴─────────┘ │
|
||||
│ ├─ system tools (in-process handlers) │
|
||||
│ ├─ app "installed" tools (shell-side or launch-on-call) │
|
||||
│ └─ app "open"/"foreground" tools (proxied to app worker) │
|
||||
└──────────┬──────────────────────────────────────┬─────────┘
|
||||
│ postMessage │
|
||||
app worker (e.g. terminal) Anthropic API (SSE)
|
||||
set-tools / tool-call / or agent bridge (WS)
|
||||
tool-result on the user's machine
|
||||
```
|
||||
|
||||
Everything assistant-side lives on the main isolate. Rationale: tool handlers
|
||||
need shell state (window list, focus, notifications, settings), the agent
|
||||
loop is I/O-bound not CPU-bound, and the overlay UI is a shell layer anyway.
|
||||
If conversation rendering ever gets expensive it can move to a worker later
|
||||
without changing the tool protocol, which is already async message passing.
|
||||
|
||||
### Core pieces
|
||||
|
||||
1. **`ToolRegistry`** (new, `app/assistant/tool-registry.ts`)
|
||||
- Holds `ToolSpec`s and handlers; answers `listTools(scopeFilter)` and
|
||||
`callTool(name, args) -> Promise<ToolResult>`.
|
||||
- Emits `onToolsChanged` (needed for MCP `tools/list_changed` in external
|
||||
mode, and consulted at each turn boundary in direct mode).
|
||||
|
||||
2. **`AssistantBackend`** (interface) with two implementations:
|
||||
- `DirectAnthropicBackend`: tool-use loop over the Messages API.
|
||||
- `ExternalAgentBackend`: websocket bridge to the user's agent.
|
||||
|
||||
3. **`AssistantSession`**: one conversation. Owns history/turn state,
|
||||
exposes `sendUtterance(text, ctx)`, `cancel()`, and streaming callbacks
|
||||
(`onTextDelta`, `onToolActivity`, `onTurnDone`, `onError`). Both backends
|
||||
implement the same session surface so the UI is backend-agnostic.
|
||||
|
||||
4. **`AssistantLayer`**: shell overlay showing streamed reply text, a status
|
||||
line for tool activity ("→ calendar.list_events"), and a small menu
|
||||
(Follow-up / Done). Double-click cancels the in-flight turn.
|
||||
|
||||
## Entry points and routing
|
||||
|
||||
- **Wakeword** Saying "Hey Even" wakes the screen if necessary and opens the
|
||||
dictation dialog. For one-off voice commands, this will be the main entry
|
||||
point.
|
||||
- **Context menu** The sidebar context menu and app context menu already have
|
||||
a "Voice input" option, which opens the dictation dialog whose Send goes to
|
||||
the foreground window. We'll split the Send option in this dialog into two
|
||||
options, "Send to Assistant" and "Type Into App". If the entry point was via
|
||||
context menu, "Type Into App" is highlighted by default; if the entry point
|
||||
was the wakeword, "Send to Assistant" is highlighted by default instead.
|
||||
(The capture UI is the existing `VoiceInputLayer` flow, with Send handing off
|
||||
to `AssistantSession` and morphing into `AssistantLayer`).
|
||||
|
||||
## Tool model
|
||||
|
||||
```ts
|
||||
type ToolAvailability =
|
||||
| "always" // system tools; no app involved
|
||||
| "installed" // app tool, live whenever the app is installed —
|
||||
// the app need not have any window open
|
||||
| "open" // app tool, live while the app has a window open,
|
||||
// foreground or backgrounded
|
||||
| "foreground"; // app tool, live only while the app owns the
|
||||
// foreground window
|
||||
|
||||
type ToolSpec = {
|
||||
name: string; // namespaced: "glasses.show_notification",
|
||||
// "app.terminal.send_input"
|
||||
description: string;
|
||||
inputSchema: object; // JSON Schema
|
||||
availability: ToolAvailability;
|
||||
/** May be invoked by the external agent outside an active voice turn. */
|
||||
proactive?: boolean;
|
||||
timeoutMs?: number; // default 10s; registry enforces
|
||||
};
|
||||
|
||||
type ToolResult =
|
||||
| { ok: true; content: string } // text payload back to the model
|
||||
| { ok: false; error: string };
|
||||
```
|
||||
|
||||
### Tool availability ontology
|
||||
|
||||
Every tool sits in exactly one tier, and the tier answers two questions: *when
|
||||
does the tool appear in `listTools()`*, and *who handles the call*.
|
||||
|
||||
| Tier | Live when | Handler | Examples |
|
||||
|---|---|---|---|
|
||||
| `always` | always | shell, in-process | `glasses.get_state`, `notifications.list` |
|
||||
| `installed` | app installed (launcher entry exists) | shell-side handler registered with the app entry, or launch-the-app-then-proxy | `timer.set` ("start a 5 minute timer" must work with no Timer window), `music.play` (cold-start playback) |
|
||||
| `open` | app has ≥1 open window, foreground or not | app worker (or in-process window) via tool-call proxy | `music.next_track`, `terminal.send_input` |
|
||||
| `foreground` | app owns the foreground window | app worker via tool-call proxy | `teleprompt.scroll_down`, anything meaning "act on what I'm looking at" |
|
||||
|
||||
Mechanics per tier:
|
||||
|
||||
- **`installed`** tools cannot be declared by the worker (it may not be
|
||||
running), so they are declared statically alongside the app's launcher
|
||||
registration. Two handler flavors: (a) a shell-side handler function
|
||||
registered with the entry — right for tools that don't actually need the
|
||||
app UI (timers, starting playback); (b) *launch-on-call*: the shell opens
|
||||
the app (as if launched from the launcher), waits for its `set-tools`
|
||||
declaration, then proxies the call. Start with (a) only; (b) adds a
|
||||
launch-latency/timeout dance that no phase-1/2 tool needs.
|
||||
- **`open`** and **`foreground`** tools are declared by the running app via
|
||||
`set-tools` (worker apps) or direct registration (in-process windows).
|
||||
The registry filters `foreground` tools by whether the declaring app owns
|
||||
the current foreground window at list time and again at call time —
|
||||
a `foreground` tool called after a foreground swap fails with a normal tool
|
||||
error rather than acting on a window the user is no longer looking at.
|
||||
- A tier is a *floor*, not a routing rule: an `installed` tool stays listed
|
||||
while the app is open or foreground. If an app wants different behavior when
|
||||
its window exists (e.g. `timer.set` also shows a countdown in the open
|
||||
Timer window), the shell-side handler checks for the window itself.
|
||||
|
||||
> **Flag:** with multiple windows per app (the worker protocol is
|
||||
> per-window), tool names are namespaced per *app*, so two windows of one
|
||||
> app declaring the same tool name would collide. Rule: `set-tools`
|
||||
> replaces the app's toolset per window, and a call routes to the declaring
|
||||
> window; if two windows declare the same name, last declaration wins and
|
||||
> earlier ones are dropped with a logged warning. Revisit only if a real
|
||||
> app needs per-window tools with the same name (none of the planned ones
|
||||
> do).
|
||||
|
||||
### System tools (initial set)
|
||||
|
||||
| Tool | Notes |
|
||||
|---|---|
|
||||
| `glasses.show_alert` | short text popup on the lenses; proactive-capable |
|
||||
| `glasses.list_windows` / `glasses.focus_window` / `glasses.open_app` | shell control |
|
||||
| `glasses.get_state` | screen on/off, foreground app, battery levels, time |
|
||||
| `glasses.read_screen` | text/description of the current foreground surface (start with window title + app-reported summary; pixel OCR is out of scope) |
|
||||
| `calendar.list_events` | wraps FaceclawCalendarProvider |
|
||||
| `media.play_pause` / `media.next` / `media.now_playing` | wraps FaceclawMediaController |
|
||||
| `notifications.list` / `notifications.dismiss` | wraps notification listener |
|
||||
| `timer.set` / `timer.cancel` | new, but trivial and high-value for an assistant |
|
||||
|
||||
Keep the first release to ~8–12 tools; every tool costs prompt tokens and
|
||||
model attention in direct mode.
|
||||
|
||||
Note that `calendar.*`, `media.*`, and `timer.*` are conceptually
|
||||
`installed`-tier app tools that happen to be implemented by in-process shell
|
||||
code; listing them here just means the shell registers their handlers
|
||||
directly. If any of them later becomes a real app, its tools move to the
|
||||
app's launcher registration without the model-visible surface changing.
|
||||
|
||||
### App tools
|
||||
|
||||
Running apps declare `open`/`foreground` tools per window over the existing
|
||||
worker protocol (`installed` tools are declared statically with the launcher
|
||||
entry, per the ontology above):
|
||||
|
||||
```ts
|
||||
// WorkerAppReply additions (worker -> shell)
|
||||
| { type: "set-tools"; windowId: string; tools: ToolSpec[] } // replaces the set
|
||||
| { type: "tool-result"; callId: string; result: ToolResult }
|
||||
|
||||
// WorkerAppMessage additions (shell -> worker)
|
||||
| { type: "tool-call"; callId: string; windowId: string; name: string; args: unknown }
|
||||
```
|
||||
|
||||
Registry rules:
|
||||
|
||||
- Liveness follows the availability tier: `open` tools are removed when the
|
||||
app's last window closes, `foreground` tools additionally drop out of
|
||||
`listTools()` when the app loses the foreground. Either transition fires
|
||||
`onToolsChanged`; in-flight calls to a closed window resolve as errors,
|
||||
and calls to a `foreground` tool are re-checked at dispatch time.
|
||||
- Names are auto-prefixed `app.<appId>.` by the registry so apps can't
|
||||
shadow system tools or each other.
|
||||
- The shell enforces `timeoutMs` on the postMessage round-trip; a hung
|
||||
worker yields a tool error, not a hung turn (same philosophy as input
|
||||
handling — the shell must survive a stuck app).
|
||||
- In-process shell apps (calendar, music, ...) register handlers directly
|
||||
with the registry; the worker protocol is just the remote flavor of the
|
||||
same registration.
|
||||
- **Mid-turn availability changes** (foreground swap, window close): direct
|
||||
mode re-lists tools at each loop iteration (tool set is per-API-call
|
||||
anyway); external mode sends MCP `notifications/tools/list_changed`. A
|
||||
call to a tool that just vanished returns a normal tool error the model
|
||||
can react to.
|
||||
|
||||
First app to wire up: the terminal —
|
||||
`app.terminal.send_input` (type into the attached session; `open`,
|
||||
proactive: false — "rerun the build" should work while the terminal is
|
||||
backgrounded), `app.terminal.read_screen` (current grid contents; `open`),
|
||||
`app.terminal.list_sessions` (`open`). This immediately enables "tell the
|
||||
terminal to rerun the build" style commands and doubles as the reference
|
||||
implementation — deliberately all `open`-tier; the first `foreground` tool
|
||||
should come from an app where focus genuinely matters (teleprompt scrolling).
|
||||
|
||||
## Direct mode: agent loop on the phone
|
||||
|
||||
Extend `app/native/anthropic.ts`:
|
||||
|
||||
1. **Streaming tool-use support**: handle `content_block_start` with
|
||||
`tool_use`, accumulate `input_json_delta`, surface complete tool_use
|
||||
blocks alongside text deltas. (Today the client only reads `text_delta`.)
|
||||
2. **`runAgentTurn` loop** (new, `app/assistant/direct-backend.ts`):
|
||||
messages + `tools` from the registry → stream → if `stop_reason ==
|
||||
"tool_use"`, execute each call via the registry, append `tool_result`
|
||||
blocks, continue; until `end_turn`/`refusal` or a safety cap
|
||||
(max 8 loop iterations, max ~2 min per turn).
|
||||
3. **Session state**: history kept in memory per `AssistantSession`,
|
||||
trimmed from the head past a token budget. Sessions idle-expire
|
||||
(~10 min) so a new PTT starts fresh; "Follow-up" continues the session.
|
||||
4. **System prompt**: glasses context — output renders on a 576×288
|
||||
monochrome HUD, so answer in 1–3 short sentences, no markdown, no lists
|
||||
unless asked; prefer acting via tools over describing; current time,
|
||||
foreground app, and screen state are injected per turn.
|
||||
5. Model `claude-sonnet-5`, `effort: "low"` (same latency posture as the
|
||||
dictation refiner). Model/effort become settings later if needed.
|
||||
|
||||
## External mode: bridge to the user's agent
|
||||
|
||||
### Topology
|
||||
|
||||
The phone **dials out** (g2mirror pattern — phones don't accept inbound
|
||||
connections reliably; tailscale host + port + token settings) to a small
|
||||
**bridge server** co-located with the user's agent. One websocket, JSON
|
||||
frames, three multiplexed channels:
|
||||
|
||||
```
|
||||
{ v: 1, chan: "ctl" | "chat" | "mcp", ... }
|
||||
|
||||
ctl: hello {deviceName, token, capabilities}, hello-ack, ping/pong, error
|
||||
chat: utterance {turnId, text, ctx} (phone -> agent)
|
||||
text-delta {turnId, text} (agent -> phone)
|
||||
tool-activity {turnId, label} (agent -> phone, optional status)
|
||||
turn-done {turnId, stopReason} / turn-error {turnId, message}
|
||||
cancel {turnId} (phone -> agent)
|
||||
mcp: { msg: <raw MCP JSON-RPC frame> } (bidirectional)
|
||||
```
|
||||
|
||||
On the `mcp` channel the **phone is an MCP server** (MCP roles come from the
|
||||
initialize handshake, not from who dialed). It serves `tools/list` from the
|
||||
ToolRegistry, `tools/call` into it, and emits `tools/list_changed` on
|
||||
foreground changes. This means the agent-side integration needs zero
|
||||
Faceclaw-specific tool code — the glasses appear as a normal MCP server.
|
||||
|
||||
### Agent-side adapters
|
||||
|
||||
The bridge server is a small standalone package (`faceclaw-agent-bridge`,
|
||||
sibling of g2mirror) with two faces:
|
||||
|
||||
1. **MCP face**: re-exposes the phone's MCP channel as a localhost
|
||||
streamable-HTTP MCP endpoint. The user registers
|
||||
`http://localhost:<port>/mcp` with their agent (OpenClaw, Hermes, Claude
|
||||
Code, anything MCP-capable) exactly like any other MCP server. When the
|
||||
phone is disconnected, `tools/list` returns empty rather than erroring.
|
||||
2. **Chat face**: pluggable per-agent adapter that forwards `utterance`
|
||||
frames into the agent and streams replies back:
|
||||
- **OpenClaw**: a plugin that injects the utterance as an inbound message
|
||||
on a dedicated "glasses" channel/session and relays the streamed reply.
|
||||
- **Hermes**: equivalent module against its session API.
|
||||
- **Fallback adapter**: spawn a configured CLI (e.g. `claude -p --resume`)
|
||||
per turn — lowest-fidelity but makes the bridge useful with no plugin.
|
||||
|
||||
Utterance `ctx` carries `{foregroundApp, screenOn, localTime}` so the remote
|
||||
agent has the same situational grounding the direct loop gets in its system
|
||||
prompt.
|
||||
|
||||
### Proactive agent actions
|
||||
|
||||
Because the MCP channel is live whenever the bridge is connected, the
|
||||
external agent can call glasses tools **outside a voice turn** — e.g. push a
|
||||
notification to the lenses when a long job finishes. Gating:
|
||||
|
||||
- Only tools marked `proactive: true` are callable outside an active turn
|
||||
(registry-enforced; enforced on the phone, never trusted to the bridge).
|
||||
Availability tiers apply to proactive calls unchanged: a proactive
|
||||
`installed` tool (e.g. `timer.set`) works anytime, while a proactive call
|
||||
to an `open`/`foreground` tool whose app isn't in the right state gets the
|
||||
same tool error a mid-turn call would.
|
||||
- Master setting `assistant.allowProactive` (default on — it's a marquee
|
||||
feature — but visible and easy to turn off).
|
||||
- Proactive display actions are rate-limited (e.g. 6/min) and never turn
|
||||
the screen on unless the user enabled that specifically.
|
||||
|
||||
## Settings
|
||||
|
||||
New "Assistant" section in dashboard settings:
|
||||
|
||||
- `assistant.backend`: `anthropic` | `external` (default `anthropic` if an
|
||||
API key is set, else `external` if a bridge host is set).
|
||||
- `assistant.bridgeHost` / `assistant.bridgePort` / `assistant.bridgeToken`
|
||||
(mirrors the terminal.* trio).
|
||||
- `assistant.allowProactive`: boolean.
|
||||
- `voice.skipConfirmationAfterWakeword`: Wakeword-triggered input sends to the
|
||||
assistant immediately at the end of an utterance, rather than waiting for a
|
||||
menu selection.
|
||||
- Reuses existing `llm.anthropicApiKey` for direct mode.
|
||||
|
||||
## Security notes
|
||||
|
||||
- Bridge auth: bearer token in the `hello` frame, TLS optional because the
|
||||
expected transport is tailscale (same stance as g2mirror); document that
|
||||
plainly.
|
||||
- The phone enforces all tool gating (proactive flags, rate limits,
|
||||
timeouts). The bridge and agent are treated as honest-but-fallible.
|
||||
- Tool results may contain private data (calendar, notifications); external
|
||||
mode ships them to the user's own machine only — no third-party hop. In
|
||||
direct mode they go to the Anthropic API like any other request content.
|
||||
- App tools execute inside the app's worker with the app's existing
|
||||
privileges; the registry adds no new capability beyond what the app could
|
||||
already do.
|
||||
|
||||
## Implementation plan
|
||||
|
||||
**Phase 1 — direct assistant, system tools only.** ✅ Landed 2026-07-24
|
||||
(builds + typechecks; not yet hardware-tested).
|
||||
ToolRegistry (`app/assistant/tool-registry.ts`) with the full availability-tier
|
||||
model but only `always` tools wired; 8 system tools (`glasses.get_state`,
|
||||
`glasses.show_alert`, `calendar.list_events`, `media.now_playing`/`play_pause`/
|
||||
`next`, `notifications.list`/`dismiss`) in `app/assistant/system-tools.ts`;
|
||||
tool-use streaming in `anthropic.ts` (content-block assembly, `tools` param);
|
||||
`DirectAnthropicBackend` + `AssistantSession`; `AssistantLayer` overlay
|
||||
(`app/ui/apps/assistant.ts`); `VoiceInputLayer` reworked to route via a
|
||||
`sendTargets` list ("Send to Assistant" / "Type Into App") with entry-point
|
||||
default highlight and wakeword skip-confirmation auto-send;
|
||||
`shell.showAlert` popup; `assistant.skipConfirmationAfterWakeword` setting.
|
||||
*Deliverable: speak wakeword, ask "what's on my calendar", get an answer
|
||||
on-lens.*
|
||||
|
||||
**Phase 2 — app tools.** ✅ Landed 2026-07-25 (builds + typechecks; not yet
|
||||
hardware-tested).
|
||||
Worker protocol extended with `set-tools` / `tool-result` (WorkerAppReply) and
|
||||
`tool-call` (WorkerAppMessage). `ToolRegistry.setAppTools`/`removeAppTools`
|
||||
key tools by windowId, prefix names `app.<appId>.`, and gate `foreground`
|
||||
tools on an `isForeground()` predicate (checked at list and dispatch time).
|
||||
`WorkerAppHost` proxies calls to its worker (callId round-trip, 15s host
|
||||
timeout backstop on top of the registry's per-tool timeout), registers a
|
||||
window's tools on `set-tools`, and withdraws them + fails in-flight calls on
|
||||
window close. Direct mode now re-lists tools each loop iteration.
|
||||
Terminal is the reference: the hub window declares `send_input` /
|
||||
`read_screen` / `list_sessions` (all `open`-tier); send_input/read_screen act
|
||||
on the active view (foregrounded → last-active → sole view). *Deliverable:
|
||||
"run the build again" typed into the terminal even while it's backgrounded.*
|
||||
|
||||
*Not yet done from the Phase 2 design:* `installed`-tier registration on
|
||||
launcher entries (no `installed` app tool exists yet — timer stays deferred);
|
||||
in-process-window tool registration (only worker apps wired); the per-window
|
||||
same-name collision handling (the terminal sidesteps it by declaring all tools
|
||||
on the single hub window).
|
||||
|
||||
**Phase 3 — external bridge.** ✅ Landed 2026-08-01 (see the status note at
|
||||
the top for the architecture change: bridge server == OpenClaw plugin).
|
||||
Phone side: `app/assistant/bridge-client.ts` (`assistantBridge` singleton —
|
||||
dial-out FaceclawWebSocket, hello/token auth, auto-reconnect with 1s→60s
|
||||
backoff, started at boot by the dashboard controller when configured),
|
||||
`app/assistant/mcp-server.ts` (MCP over the `mcp` channel: initialize /
|
||||
tools/list / tools/call from the ToolRegistry, list_changed on
|
||||
onToolsChanged, proactive gating + 6/min rate limit enforced on-phone),
|
||||
external mode in `AssistantSession` (`AssistantBackendConfig` union selects
|
||||
direct vs external; external history lives on the agent's machine), settings
|
||||
(`assistant.backend`, `assistant.bridgeHost/Port/Token`,
|
||||
`assistant.allowProactive`) in a new Assistant settings section.
|
||||
Server side: `~/repositories/faceclaw-agent-bridge` (own repo/package).
|
||||
*Verified against the live OpenClaw on serac with a simulated phone
|
||||
(`test/fake-phone.js`): auth, streamed turns, and both tool directions
|
||||
work; a real model turn needs the OpenClaw instance to have provider auth.*
|
||||
|
||||
**Phase 4 — OpenClaw/Hermes adapters + proactive polish.** Partially landed
|
||||
2026-08-01: the OpenClaw adapter is the plugin itself (no separate chat
|
||||
adapter needed); proactive gating + rate limits and `tools/list_changed`
|
||||
are done. Remaining: Hermes (or generic-agent) support — likely a
|
||||
standalone bridge speaking the same wire protocol with an MCP face +
|
||||
CLI chat adapter; session persistence questions across reconnects
|
||||
(currently: OpenClaw session is persistent, phone session is UI-only).
|
||||
|
||||
**Hardware-test TODO (external mode):** end-to-end on the real phone +
|
||||
glasses against serac — connect status, a streamed turn, cancel, follow-up,
|
||||
proactive `glasses.show_alert` from another OpenClaw channel, and behavior
|
||||
across phone network transitions (the reconnect backoff is untested on
|
||||
device).
|
||||
|
||||
## Open questions (deferred, not blockers):
|
||||
|
||||
- Continuous conversation: after a turn ends, listen for a follow-up
|
||||
utterance without requiring the wakeword/PTT again? There are some UI
|
||||
design questions here; maybe multi-turn conversations take the form of an
|
||||
Assistant app.
|
||||
- Launch-on-call for `installed`-tier tools that genuinely need the app UI
|
||||
(open the app, await `set-tools`, proxy the call) — deferred until a tool
|
||||
needs it.
|
||||
- `glasses.read_screen` fidelity beyond app-reported summaries? We probably
|
||||
don't want to do OCR, but we might do something like extending the draw-text
|
||||
draw call with an on-by-default option that also appends the drawn text to
|
||||
a string that gets submitted along with the frame.
|
||||
- Whether assistant conversations should be reviewable in the phone UI
|
||||
(transcript log) — probably yes, cheap once sessions are objects. The
|
||||
natural UI home for this would be inside an Assistant app window.
|
||||
|
||||
@@ -1,53 +0,0 @@
|
||||
# Self-hosted Whisper transcription
|
||||
|
||||
The "Whisper server (self-hosted)" transcription provider sends voice audio to
|
||||
any OpenAI-compatible Whisper server instead of a cloud service: vLLM, speaches
|
||||
(faster-whisper-server), LocalAI and similar. It uses the batch endpoint
|
||||
`POST {url}/audio/transcriptions` with a 16 kHz mono WAV. Push-to-talk
|
||||
utterances are one request each; in Transcribe, each span ended by a pause is
|
||||
one request. While a span is open its audio so far is re-sent every 1.5 s, so
|
||||
text appears while you are still talking. A span that runs 25 s without a pause
|
||||
(nonstop speech, music) is cut at its quietest moment in the last 5 s, so no
|
||||
request outgrows Whisper's 30 s window; the pieces still arrive as one final.
|
||||
On Android the WAV goes out through `FaceclawBinaryPost` (okhttp), since
|
||||
NativeScript's http module copies a binary body into Java a byte at a time.
|
||||
|
||||
## Server
|
||||
|
||||
Any server exposing `/v1/models` and `/v1/audio/transcriptions` works, e.g.
|
||||
|
||||
vllm serve openai/whisper-large-v3 --port 8004
|
||||
|
||||
## Settings (Settings > Voice)
|
||||
|
||||
- **Transcription Provider**: Whisper server (self-hosted). Disabled until the
|
||||
URL is set.
|
||||
- **Whisper server URL**: the API root, e.g. `http://host:8004/v1`. A bare
|
||||
`http://host:8004` gets `/v1` added.
|
||||
- **Whisper server model**: empty uses the first model the server lists.
|
||||
- **Whisper language**: an ISO code such as `en`, or empty to auto-detect.
|
||||
A wrong fixed language makes Whisper translate rather than transcribe.
|
||||
- **Whisper server key**: optional bearer token.
|
||||
|
||||
## iOS: plain http
|
||||
|
||||
iOS App Transport Security blocks plain `http://` to IP addresses (including
|
||||
Tailscale's 100.x addresses) unless the app's Info.plist has an exception for
|
||||
that host. Either serve the API over https (for example with `tailscale serve`)
|
||||
or add your server to `NSExceptionDomains` in `App_Resources/iOS/Info.plist`
|
||||
before building:
|
||||
|
||||
<key>192.0.2.10</key>
|
||||
<dict><key>NSExceptionAllowsInsecureHTTPLoads</key><true/></dict>
|
||||
|
||||
## Transcribe on iOS
|
||||
|
||||
Transcribe needs a cloud or Whisper-server provider on iOS (Apple's recognizer
|
||||
only handles short utterances). Its capture follows the glasses: a disconnect,
|
||||
a glasses lock or backgrounding (for phone-mic capture) transcribes what was
|
||||
already heard, and listening resumes once the glasses reconnect, unlock or the
|
||||
app returns to the foreground. Without a glasses session the phone mic stands in
|
||||
while Faceclaw is in the foreground. With "Save Transcribe audio" on (Voice
|
||||
settings), each session's audio is kept as `transcribe-<time>.wav` in Downloads,
|
||||
next to `transcript-<time>.txt` when the text is saved; a resumed capture keeps
|
||||
appending to the same file.
|
||||
Reference in new issue
Block a user