mirror of
https://git.undefined-dev.fr/Blendman974/faceclaw.git
synced 2026-11-19 21:03:09 +00:00
Add OpenAI-compatible assistant provider
This commit is contained in:
1 parent
a5eda29368
commit
77e8a86196
19 files changed
+807
-23
No files matched your search
@@ -116,12 +116,22 @@ provider; Faceclaw does not manage accounts or process billing information.
|
||||
|
||||
## AI Assistant Transcripts
|
||||
|
||||
Faceclaw has a builtin AI voice assistant. Using the voice assistant requires
|
||||
providing an API key for a model provider such as Anthropic or OpenAI. Prompts
|
||||
and responses are subject to that model provider's privacy policy:
|
||||
Faceclaw has a builtin AI voice assistant. Using a cloud model requires
|
||||
providing an API key for a model provider such as Anthropic, OpenAI, or Venice,
|
||||
or configuring an OpenAI-compatible endpoint such as a private Open WebUI
|
||||
server. Prompts and responses are subject to the selected provider or server
|
||||
operator's privacy policy:
|
||||
|
||||
OpenAI: https://openai.com/policies/row-privacy-policy/
|
||||
Anthropic: https://www.anthropic.com/legal/privacy
|
||||
Venice: https://venice.ai/privacy
|
||||
|
||||
With an OpenAI-compatible provider, the configured server receives the
|
||||
assistant system prompt, conversation history, available tool definitions, and
|
||||
tool results needed to answer. Its URL, model ID, and optional API key are
|
||||
stored on the phone. Faceclaw does not operate that server and cannot make
|
||||
representations about its logging or retention; for a private Open WebUI
|
||||
server, the server's operator controls those policies.
|
||||
|
||||
AI conversation histories and memory files may be stored on your phone.
|
||||
Faceclaw does not transmit voice-assistant prompts or responses to any party
|
||||
|
||||
@@ -64,8 +64,10 @@ import it into the new version.
|
||||
* **A voice assistant** that wakes up when you say "Hey Even", transcribes
|
||||
text with an on-device model, OpenAI Whisper, ElevenLabs, or Soniox API,
|
||||
and responds to queries and commands using an onboard model (Qwen3 4B,
|
||||
slow) or with an Anthropic or OpenAI model (requires an API key), or using
|
||||
your own long-running OpenClaw agent.
|
||||
slow) or with an Anthropic, OpenAI, or OpenAI-compatible Chat Completions
|
||||
model (including Venice and private Open WebUI servers; model ID and
|
||||
usually an API key required), or using your own long-running OpenClaw
|
||||
agent.
|
||||
* **Multitasking**, with an app-switcher sidebar and app launcher.
|
||||
* **Glanceboard**, a dashboard with widgets that you can quickly view with a
|
||||
* **Mostly-compatible with EvenHub.** Install and runapps from the Even app
|
||||
@@ -95,6 +97,38 @@ import it into the new version.
|
||||
the glasses, and you can use the phone touchscreen as an input device..
|
||||
<!-- END GENERATED: features -->
|
||||
|
||||
## OpenAI-compatible assistant providers (Venice / Open WebUI)
|
||||
|
||||
The assistant can call an OpenAI-compatible **Chat Completions** endpoint directly
|
||||
from the phone. This uses the same conversation history and glasses tool loop as
|
||||
the built-in cloud providers; it does not require different glasses firmware.
|
||||
|
||||
In **Settings > Assistant**, set:
|
||||
|
||||
1. **Compatible API URL** — the explicit API root.
|
||||
2. **Compatible model ID** — the exact model identifier accepted by the server.
|
||||
3. **Compatible API key** — optional; when set, it is sent only to the configured
|
||||
compatible endpoint as a Bearer token, not to OpenAI's Whisper service.
|
||||
4. **Assistant model** — choose **OpenAI-compatible**. Auto never selects it.
|
||||
|
||||
Provider examples:
|
||||
|
||||
| Provider | Compatible API URL | Notes |
|
||||
| --- | --- | --- |
|
||||
| Venice | `https://api.venice.ai/api/v1` | Use a current Venice model ID. Choose one with function-calling support if you want glasses tools. |
|
||||
| Open WebUI | `https://your-openwebui-host/api` | This produces `/api/chat/completions`. Use the model ID exposed by your server and an Open WebUI API key if required. |
|
||||
|
||||
You may paste the full `/chat/completions` endpoint instead; Faceclaw removes
|
||||
that suffix. It otherwise preserves the path you enter and does not guess `/v1`
|
||||
or `/api`. There is no automatic model discovery in this version.
|
||||
|
||||
The same provider can supply Calculator's rewrite fallback and Translate.
|
||||
Actual tool calling depends on the selected model and server. Use HTTPS where
|
||||
possible; mobile cleartext policies may block plain HTTP. For a private server,
|
||||
make it reachable from the phone (for example through your tailnet) without
|
||||
exposing it publicly. Exported settings contain API keys and must be treated as
|
||||
secrets.
|
||||
|
||||
## Watch remote (Wear OS)
|
||||
|
||||
The `wear/` directory holds a companion app for Wear OS 3+ watches that turns
|
||||
|
||||
@@ -4,7 +4,7 @@ import { truncateText, wrapText } from "../../graphics/textwrap";
|
||||
import { ASSISTANT_MODEL_CHOICES, assistantModelLabel, resolveAssistantModel } from "../../assistant/models";
|
||||
import type { AssistantTranscriptEntry } from "../../assistant/session";
|
||||
import type { AssistantConversations, ReasoningLevel } from "../../assistant/conversations";
|
||||
import { anthropicApiKeySetting, assistantBackendSetting, openAiApiKeySetting } from "../../ui/dashboard-settings";
|
||||
import { anthropicApiKeySetting, assistantBackendSetting, assistantOpenAiCompatibleKeySetting, assistantOpenAiCompatibleModelSetting, assistantOpenAiCompatibleUrlSetting, openAiApiKeySetting } from "../../ui/dashboard-settings";
|
||||
import { GESTURE_CLICK, GESTURE_DOUBLE_CLICK, GESTURE_LONG_PRESS, gestureHints, type InputEvent } from "../../ui/gestures";
|
||||
import type { Layer, LayerContext } from "../../ui/layers";
|
||||
import type { MenuItem } from "../../ui/menu";
|
||||
@@ -102,7 +102,7 @@ class AiChatLayer implements Layer {
|
||||
}
|
||||
case "long-press":
|
||||
if (!this.conversations.current().session?.isTurnActive()) {
|
||||
if (!shell.isAssistantAvailable()) { shell.showAlert(global.isIOS ? "Set an OpenAI or Anthropic API key in Settings." : "Set an API key or download the on-phone model in Settings."); return; }
|
||||
if (!shell.isAssistantAvailable()) { shell.showAlert(global.isIOS ? "Configure an assistant model in Settings." : "Set an API key or download the on-phone model in Settings."); return; }
|
||||
void this.draft.start();
|
||||
}
|
||||
return;
|
||||
@@ -133,7 +133,12 @@ class AiChatLayer implements Layer {
|
||||
onSelect: (pickCtx) => { if (this.conversations.configure(model, record.reasoning)) pickCtx.stack.clearToBase(); },
|
||||
}))) },
|
||||
{ label: `Reasoning: ${record.reasoning}`, disabled: () => {
|
||||
const llm = resolveAssistantModel(record.model, { anthropic: anthropicApiKeySetting.get(), openai: openAiApiKeySetting.get() });
|
||||
const llm = resolveAssistantModel(record.model, {
|
||||
anthropic: anthropicApiKeySetting.get(), openai: openAiApiKeySetting.get(),
|
||||
openAiCompatibleUrl: assistantOpenAiCompatibleUrlSetting.get(),
|
||||
openAiCompatibleKey: assistantOpenAiCompatibleKeySetting.get(),
|
||||
openAiCompatibleModel: assistantOpenAiCompatibleModelSetting.get(),
|
||||
});
|
||||
return external || busy() || !llm?.effort;
|
||||
}, onSelect: (ctx) => submenu(ctx, "Reasoning", (["default", "low", "medium", "high"] as ReasoningLevel[]).map((reasoning) => ({
|
||||
label: `${reasoning === record.reasoning ? "✓ " : ""}${reasoning === "default" ? "Model default" : reasoning}`,
|
||||
|
||||
@@ -9,11 +9,15 @@
|
||||
import { streamAnthropicMessage } from "../../native/anthropic";
|
||||
import { streamLocalQwen } from "../../native/llama";
|
||||
import { streamOpenAiResponse } from "../../native/openai";
|
||||
import { streamOpenAiCompatibleChat } from "../../native/openai-chat";
|
||||
import { resolveAssistantModel } from "../../assistant/models";
|
||||
import { type LlmStreamOptions } from "../../assistant/llm-protocol";
|
||||
import {
|
||||
anthropicApiKeySetting,
|
||||
assistantModelSetting,
|
||||
assistantOpenAiCompatibleKeySetting,
|
||||
assistantOpenAiCompatibleModelSetting,
|
||||
assistantOpenAiCompatibleUrlSetting,
|
||||
openAiApiKeySetting,
|
||||
} from "../../ui/dashboard-settings";
|
||||
import { CALCULATOR_REWRITE_SYSTEM_PROMPT, buildCalculatorRewriteUserMessage } from "../../prompts";
|
||||
@@ -22,6 +26,9 @@ export function rewriteSpokenProblem(spoken: string): Promise<string | null> {
|
||||
const resolved = resolveAssistantModel(assistantModelSetting.get(), {
|
||||
anthropic: anthropicApiKeySetting.get(),
|
||||
openai: openAiApiKeySetting.get(),
|
||||
openAiCompatibleUrl: assistantOpenAiCompatibleUrlSetting.get(),
|
||||
openAiCompatibleKey: assistantOpenAiCompatibleKeySetting.get(),
|
||||
openAiCompatibleModel: assistantOpenAiCompatibleModelSetting.get(),
|
||||
});
|
||||
if (!resolved) return Promise.resolve(null);
|
||||
|
||||
@@ -40,6 +47,9 @@ export function rewriteSpokenProblem(spoken: string): Promise<string | null> {
|
||||
case "openai":
|
||||
streamOpenAiResponse(options);
|
||||
break;
|
||||
case "openai-compatible":
|
||||
streamOpenAiCompatibleChat({ ...options, baseUrl: resolved.baseUrl ?? "" });
|
||||
break;
|
||||
case "local":
|
||||
streamLocalQwen(options);
|
||||
break;
|
||||
|
||||
@@ -28,8 +28,8 @@ const TRAY_ICON_ID = "translate";
|
||||
const NO_PROVIDER_PROBLEM =
|
||||
"Translate needs a transcription provider that understands any language: add an ElevenLabs, OpenAI, or Soniox API key (Settings > API Keys), or a Whisper server URL (Settings > Voice).";
|
||||
const NO_MODEL_PROBLEM = global.isIOS
|
||||
? "Translate needs a language model: add an Anthropic or OpenAI API key (Settings > API Keys)."
|
||||
: "Translate needs a language model: add an Anthropic or OpenAI API key (Settings > API Keys), or download the on-phone model (Settings > Assistant).";
|
||||
? "Translate needs a language model: configure a cloud or OpenAI-compatible model in Settings > Assistant."
|
||||
: "Translate needs a language model: configure one in Settings > Assistant, or download the on-phone model.";
|
||||
|
||||
export type TranslateAppOptions = InProcessAppOptions & {
|
||||
/** Begin (or retarget) continuous mic capture, kept on while the window is open. */
|
||||
|
||||
@@ -6,9 +6,13 @@
|
||||
import { streamAnthropicMessage } from "../../native/anthropic";
|
||||
import { streamLocalQwen } from "../../native/llama";
|
||||
import { streamOpenAiResponse } from "../../native/openai";
|
||||
import { streamOpenAiCompatibleChat } from "../../native/openai-chat";
|
||||
import { resolveAssistantModel, type ResolvedAssistantModel } from "../../assistant/models";
|
||||
import { type LlmStreamHandle, type LlmStreamOptions } from "../../assistant/llm-protocol";
|
||||
import { anthropicApiKeySetting, assistantModelSetting, openAiApiKeySetting } from "../../ui/dashboard-settings";
|
||||
import {
|
||||
anthropicApiKeySetting, assistantModelSetting, assistantOpenAiCompatibleKeySetting,
|
||||
assistantOpenAiCompatibleModelSetting, assistantOpenAiCompatibleUrlSetting, openAiApiKeySetting,
|
||||
} from "../../ui/dashboard-settings";
|
||||
import { TRANSLATE_SYSTEM_PROMPT, buildTranslateUserMessage } from "../../prompts";
|
||||
import { type TranslationTurn } from "./translation-model";
|
||||
|
||||
@@ -23,6 +27,9 @@ export function resolveTranslationModel(): ResolvedAssistantModel | null {
|
||||
return resolveAssistantModel(selection === "hermes" ? "auto" : selection, {
|
||||
anthropic: anthropicApiKeySetting.get(),
|
||||
openai: openAiApiKeySetting.get(),
|
||||
openAiCompatibleUrl: assistantOpenAiCompatibleUrlSetting.get(),
|
||||
openAiCompatibleKey: assistantOpenAiCompatibleKeySetting.get(),
|
||||
openAiCompatibleModel: assistantOpenAiCompatibleModelSetting.get(),
|
||||
});
|
||||
}
|
||||
|
||||
@@ -54,6 +61,8 @@ export function streamTranslation(model: ResolvedAssistantModel, request: Transl
|
||||
switch (model.provider) {
|
||||
case "openai":
|
||||
return streamOpenAiResponse(options);
|
||||
case "openai-compatible":
|
||||
return streamOpenAiCompatibleChat({ ...options, baseUrl: model.baseUrl ?? "" });
|
||||
case "local":
|
||||
return streamLocalQwen(options);
|
||||
default:
|
||||
|
||||
@@ -2,6 +2,7 @@ import { streamAnthropicMessage } from "../native/anthropic";
|
||||
import { streamHermesChat } from "../native/hermes";
|
||||
import { streamLocalQwen } from "../native/llama";
|
||||
import { streamOpenAiResponse } from "../native/openai";
|
||||
import { streamOpenAiCompatibleChat } from "../native/openai-chat";
|
||||
import type {
|
||||
LlmContentBlock,
|
||||
LlmMessage,
|
||||
@@ -143,6 +144,8 @@ export class DirectAssistantBackend {
|
||||
streamHandle =
|
||||
options.provider === "hermes"
|
||||
? streamHermesChat({ ...streamOptions, baseUrl: options.baseUrl ?? "", sessionId: options.sessionId ?? "" })
|
||||
: options.provider === "openai-compatible"
|
||||
? streamOpenAiCompatibleChat({ ...streamOptions, baseUrl: options.baseUrl ?? "" })
|
||||
: options.provider === "openai"
|
||||
? streamOpenAiResponse(streamOptions)
|
||||
: options.provider === "local"
|
||||
|
||||
@@ -15,7 +15,7 @@ export type LlmContentBlock =
|
||||
}
|
||||
| { type: "tool_result"; tool_use_id: string; content: string; is_error?: boolean }
|
||||
/** Opaque state required to continue a provider's stateless conversation. */
|
||||
| { type: "provider_item"; provider: "openai"; item: any };
|
||||
| { type: "provider_item"; provider: "openai" | "openai-compatible"; item: any };
|
||||
|
||||
export type LlmMessage = {
|
||||
role: "user" | "assistant";
|
||||
|
||||
+19
-2
@@ -11,6 +11,7 @@ export const ASSISTANT_MODEL_VALUES = [
|
||||
"sol",
|
||||
"qwen",
|
||||
"hermes",
|
||||
"openai-compatible",
|
||||
] as const;
|
||||
|
||||
export type AssistantModel = (typeof ASSISTANT_MODEL_VALUES)[number];
|
||||
@@ -21,7 +22,7 @@ export function supportedAssistantModel(value: AssistantModel): AssistantModel {
|
||||
return global.isIOS && value === "qwen" ? "auto" : value;
|
||||
}
|
||||
|
||||
export type AssistantProvider = "anthropic" | "openai" | "local" | "hermes";
|
||||
export type AssistantProvider = "anthropic" | "openai" | "local" | "hermes" | "openai-compatible";
|
||||
|
||||
export type AssistantApiKeys = {
|
||||
anthropic: string;
|
||||
@@ -29,6 +30,9 @@ export type AssistantApiKeys = {
|
||||
/** A Hermes Agent API server; only the voice assistant passes these. */
|
||||
hermesUrl?: string;
|
||||
hermesKey?: string;
|
||||
openAiCompatibleUrl?: string;
|
||||
openAiCompatibleKey?: string;
|
||||
openAiCompatibleModel?: string;
|
||||
};
|
||||
|
||||
export type ResolvedAssistantModel = {
|
||||
@@ -37,7 +41,7 @@ export type ResolvedAssistantModel = {
|
||||
model: string;
|
||||
apiKey: string;
|
||||
effort?: "low" | "medium" | "high";
|
||||
/** API root for providers that are not fixed services (Hermes). */
|
||||
/** API root for Hermes or an OpenAI-compatible provider. */
|
||||
baseUrl?: string;
|
||||
};
|
||||
|
||||
@@ -63,6 +67,7 @@ const MODEL_DEFINITIONS: Record<Exclude<AssistantModel, "auto">, ModelDefinition
|
||||
qwen: { label: `${LOCAL_MODEL.label} (on-phone)`, provider: "local", model: LOCAL_MODEL.id },
|
||||
// Your own Hermes Agent (its API server), with its own tools and memory.
|
||||
hermes: { label: "Hermes agent", provider: "hermes", model: "hermes-agent" },
|
||||
"openai-compatible": { label: "OpenAI-compatible", provider: "openai-compatible", model: "custom" },
|
||||
};
|
||||
|
||||
export function assistantModelLabel(value: AssistantModel): string {
|
||||
@@ -106,6 +111,18 @@ export function resolveAssistantModel(
|
||||
if (!baseUrl) return null;
|
||||
return { selection, provider: "hermes", model: definition.model, apiKey: keys.hermesKey?.trim() ?? "", baseUrl };
|
||||
}
|
||||
if (definition.provider === "openai-compatible") {
|
||||
const baseUrl = keys.openAiCompatibleUrl?.trim() ?? "";
|
||||
const model = keys.openAiCompatibleModel?.trim() ?? "";
|
||||
if (!baseUrl || !model) return null;
|
||||
return {
|
||||
selection,
|
||||
provider: definition.provider,
|
||||
model,
|
||||
apiKey: keys.openAiCompatibleKey?.trim() ?? "",
|
||||
baseUrl,
|
||||
};
|
||||
}
|
||||
if (definition.provider === "local") {
|
||||
if (!isLocalModelReady()) return null;
|
||||
return {
|
||||
|
||||
@@ -95,14 +95,20 @@ export class AssistantSession {
|
||||
// original model. Preserve visible conversation when switching engines.
|
||||
if (this.config.kind !== config.kind ||
|
||||
(this.config.kind === "direct" && config.kind === "direct" &&
|
||||
(this.config.llm.model !== config.llm.model || this.config.llm.provider !== config.llm.provider))) {
|
||||
(this.config.llm.model !== config.llm.model || this.config.llm.provider !== config.llm.provider ||
|
||||
(this.config.llm.baseUrl ?? "") !== (config.llm.baseUrl ?? "")))) {
|
||||
this.rebuildTextHistory();
|
||||
}
|
||||
this.config = config;
|
||||
}
|
||||
|
||||
private engine(): string {
|
||||
return this.config.kind === "direct" ? `${this.config.llm.provider}:${this.config.llm.model}` : "external";
|
||||
if (this.config.kind !== "direct") return "external";
|
||||
const llm = this.config.llm;
|
||||
// Keep the legacy engine ID for built-in providers so upgrading does not
|
||||
// discard their opaque history. Custom endpoints need endpoint isolation.
|
||||
const endpoint = llm.provider === "openai-compatible" ? `:${llm.baseUrl ?? ""}` : "";
|
||||
return `${llm.provider}:${llm.model}${endpoint}`;
|
||||
}
|
||||
|
||||
private rebuildTextHistory(): void {
|
||||
|
||||
@@ -0,0 +1,272 @@
|
||||
import type {
|
||||
LlmContentBlock,
|
||||
LlmStreamHandle,
|
||||
LlmStreamOptions,
|
||||
} from "../assistant/llm-protocol";
|
||||
import { openSseRequest, type SseListener } from "./sse";
|
||||
|
||||
/** Streaming client for OpenAI-compatible Chat Completions APIs. */
|
||||
export type OpenAiCompatibleStreamOptions = LlmStreamOptions & {
|
||||
/** Explicit API root, for example https://api.venice.ai/api/v1 or https://host/api. */
|
||||
baseUrl: string;
|
||||
};
|
||||
|
||||
type PendingToolCall = {
|
||||
id: string;
|
||||
name: string;
|
||||
inputJson: string;
|
||||
};
|
||||
|
||||
/** Remove an optional endpoint suffix, but never guess a provider's /v1 or /api prefix. */
|
||||
export function normalizeOpenAiCompatibleUrl(input: string): string {
|
||||
let url = input.trim().replace(/\/+$/, "");
|
||||
url = url.replace(/\/chat\/completions$/i, "").replace(/\/+$/, "");
|
||||
return /^https?:\/\/[^/\s?#@\\]+(?:\/[^\s?#\\]*)?$/i.test(url) ? url : "";
|
||||
}
|
||||
|
||||
export function streamOpenAiCompatibleChat(options: OpenAiCompatibleStreamOptions): LlmStreamHandle {
|
||||
const apiKey = options.apiKey.trim();
|
||||
const baseUrl = normalizeOpenAiCompatibleUrl(options.baseUrl);
|
||||
let request: { cancel(): void } | null = null;
|
||||
let settled = false;
|
||||
let text = "";
|
||||
let finishReason: string | null = null;
|
||||
const toolCalls = new Map<number, PendingToolCall>();
|
||||
const providerMetadata: Record<string, any> = {};
|
||||
|
||||
const fail = (message: string) => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
request?.cancel();
|
||||
options.onError(message);
|
||||
};
|
||||
|
||||
const done = () => {
|
||||
if (settled) return;
|
||||
const hasToolCalls = toolCalls.size > 0;
|
||||
// Never execute calls from a truncated stream or a length-limited response.
|
||||
if (hasToolCalls && finishReason !== "tool_calls" && finishReason !== "stop") {
|
||||
fail("OpenAI-compatible stream ended before its tool calls completed");
|
||||
return;
|
||||
}
|
||||
if (!text && !hasToolCalls) {
|
||||
fail("OpenAI-compatible provider ended the turn without a reply");
|
||||
return;
|
||||
}
|
||||
|
||||
const content: LlmContentBlock[] = [];
|
||||
if (text) content.push({ type: "text", text });
|
||||
if (Object.keys(providerMetadata).length) {
|
||||
content.push({ type: "provider_item", provider: "openai-compatible", item: providerMetadata });
|
||||
}
|
||||
for (const index of Array.from(toolCalls.keys()).sort((a, b) => a - b)) {
|
||||
const call = toolCalls.get(index)!;
|
||||
if (!call.name) {
|
||||
fail("OpenAI-compatible provider returned a tool call without a name");
|
||||
return;
|
||||
}
|
||||
let input: any = {};
|
||||
if (call.inputJson.trim()) {
|
||||
try {
|
||||
input = JSON.parse(call.inputJson);
|
||||
} catch {
|
||||
fail("OpenAI-compatible provider returned malformed tool arguments");
|
||||
return;
|
||||
}
|
||||
}
|
||||
content.push({
|
||||
type: "tool_use",
|
||||
id: call.id || `call_${index + 1}`,
|
||||
name: call.name,
|
||||
input,
|
||||
});
|
||||
}
|
||||
|
||||
settled = true;
|
||||
request?.cancel();
|
||||
options.onDone({
|
||||
text,
|
||||
content,
|
||||
stopReason: finishReason === "content_filter" ? "refusal"
|
||||
: hasToolCalls ? "tool_use"
|
||||
: finishReason === "length" ? "max_tokens" : "end_turn",
|
||||
});
|
||||
};
|
||||
|
||||
if (!baseUrl || !options.model.trim()) {
|
||||
setTimeout(() => fail(!baseUrl
|
||||
? "Set a valid OpenAI-compatible API URL"
|
||||
: "Set an OpenAI-compatible model ID"), 0);
|
||||
return { cancel: () => { settled = true; } };
|
||||
}
|
||||
|
||||
const body: any = {
|
||||
model: options.model.trim(),
|
||||
messages: buildChatMessages(options),
|
||||
stream: true,
|
||||
};
|
||||
// Let the provider choose its default unless a one-shot feature supplies a cap.
|
||||
if (options.maxTokens !== undefined) body.max_tokens = options.maxTokens;
|
||||
if (options.tools?.length) {
|
||||
body.tools = options.tools.map((tool) => ({
|
||||
type: "function",
|
||||
function: {
|
||||
name: tool.name,
|
||||
description: tool.description,
|
||||
parameters: tool.input_schema,
|
||||
},
|
||||
}));
|
||||
}
|
||||
// Do not send provider-specific reasoning fields by default. Compatible
|
||||
// servers vary widely; advanced reasoning controls can be added separately.
|
||||
|
||||
const headers: Record<string, string> = { "Content-Type": "application/json" };
|
||||
if (apiKey) headers.Authorization = `Bearer ${apiKey}`;
|
||||
|
||||
const listener: SseListener = {
|
||||
onLine: (line: string) => {
|
||||
if (settled || !line.startsWith("data:")) return;
|
||||
const payload = line.slice(5).trim();
|
||||
if (!payload) return;
|
||||
if (payload === "[DONE]") {
|
||||
if (finishReason || (!toolCalls.size && text)) done();
|
||||
else fail("OpenAI-compatible stream ended before the response completed");
|
||||
return;
|
||||
}
|
||||
let event: any;
|
||||
try {
|
||||
event = JSON.parse(payload);
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
if (event?.error) {
|
||||
fail(`OpenAI-compatible: ${String(event.error.message ?? event.error)}`);
|
||||
return;
|
||||
}
|
||||
const choice = event?.choices?.[0];
|
||||
if (!choice) return; // Usage and provider-metadata chunks may have no choices.
|
||||
const delta = choice.delta ?? {};
|
||||
if (typeof delta.content === "string" && delta.content) {
|
||||
text += delta.content;
|
||||
options.onTextDelta?.(delta.content, text);
|
||||
}
|
||||
// Reasoning is not displayed. Preserve provider metadata for replay in
|
||||
// multi-turn tool loops when the service supplies it.
|
||||
if (delta.reasoning_details !== undefined) providerMetadata.reasoning_details = delta.reasoning_details;
|
||||
if (delta.thought_signature !== undefined) providerMetadata.thought_signature = delta.thought_signature;
|
||||
if (Array.isArray(delta.tool_calls)) {
|
||||
delta.tool_calls.forEach((call: any, position: number) => {
|
||||
const index = Number.isInteger(call?.index) ? Number(call.index) : position;
|
||||
const pending = toolCalls.get(index) ?? { id: "", name: "", inputJson: "" };
|
||||
if (call?.id) pending.id = String(call.id);
|
||||
if (call?.function?.name) pending.name = String(call.function.name);
|
||||
if (typeof call?.function?.arguments === "string") pending.inputJson += call.function.arguments;
|
||||
toolCalls.set(index, pending);
|
||||
});
|
||||
}
|
||||
if (choice.finish_reason) finishReason = String(choice.finish_reason);
|
||||
},
|
||||
onHttpError: (code: number, errorBody: string) => fail(describeHttpError(Number(code), String(errorBody))),
|
||||
onComplete: () => {
|
||||
// Some compatible servers close after the finish chunk without [DONE].
|
||||
if (finishReason) done();
|
||||
else fail("OpenAI-compatible stream ended before the response completed");
|
||||
},
|
||||
onFailure: (message: string) => fail(`OpenAI-compatible connection failed: ${String(message)}`),
|
||||
};
|
||||
|
||||
try {
|
||||
request = openSseRequest(`${baseUrl}/chat/completions`, JSON.stringify(body), headers, listener);
|
||||
} catch (error) {
|
||||
setTimeout(() => fail(`OpenAI-compatible request failed: ${String((error as Error)?.message ?? error)}`), 0);
|
||||
return { cancel: () => { settled = true; } };
|
||||
}
|
||||
|
||||
return {
|
||||
cancel: () => {
|
||||
settled = true;
|
||||
try { request?.cancel(); } catch { /* ignore */ }
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/** Convert neutral conversation history to Chat Completions messages. */
|
||||
function buildChatMessages(options: OpenAiCompatibleStreamOptions): any[] {
|
||||
const messages: any[] = [];
|
||||
if (options.system) messages.push({ role: "system", content: options.system });
|
||||
|
||||
for (const message of options.messages) {
|
||||
if (typeof message.content === "string") {
|
||||
messages.push({ role: message.role, content: message.content });
|
||||
continue;
|
||||
}
|
||||
|
||||
if (message.role === "assistant") {
|
||||
let text = "";
|
||||
const toolCalls: any[] = [];
|
||||
let providerMetadata: Record<string, any> = {};
|
||||
for (const block of message.content) {
|
||||
if (block.type === "text") text += block.text;
|
||||
else if (block.type === "tool_use") {
|
||||
toolCalls.push({
|
||||
id: block.id,
|
||||
type: "function",
|
||||
function: { name: block.name, arguments: JSON.stringify(block.input ?? {}) },
|
||||
});
|
||||
} else if (block.type === "provider_item" && block.provider === "openai-compatible") {
|
||||
providerMetadata = { ...providerMetadata, ...block.item };
|
||||
}
|
||||
}
|
||||
if (text || toolCalls.length) {
|
||||
messages.push({
|
||||
role: "assistant",
|
||||
content: text || (toolCalls.length ? null : ""),
|
||||
...(toolCalls.length ? { tool_calls: toolCalls } : {}),
|
||||
...providerMetadata,
|
||||
});
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
let pendingText = "";
|
||||
const flushText = () => {
|
||||
if (!pendingText) return;
|
||||
messages.push({ role: "user", content: pendingText });
|
||||
pendingText = "";
|
||||
};
|
||||
for (const block of message.content) {
|
||||
if (block.type === "text") pendingText += block.text;
|
||||
else if (block.type === "tool_result") {
|
||||
flushText();
|
||||
messages.push({
|
||||
role: "tool",
|
||||
tool_call_id: block.tool_use_id,
|
||||
content: block.is_error ? `Error: ${block.content}` : block.content,
|
||||
});
|
||||
}
|
||||
}
|
||||
flushText();
|
||||
}
|
||||
return messages;
|
||||
}
|
||||
|
||||
function describeHttpError(code: number, body: string): string {
|
||||
let detail = "";
|
||||
try {
|
||||
const parsed = JSON.parse(body);
|
||||
detail = String(parsed?.error?.message ?? parsed?.error ?? parsed?.message ?? "");
|
||||
} catch {
|
||||
// Non-JSON error body.
|
||||
}
|
||||
switch (code) {
|
||||
case 401: return "OpenAI-compatible: invalid API key";
|
||||
case 403: return "OpenAI-compatible: API key lacks permission";
|
||||
case 429: return "OpenAI-compatible: rate limited, try again shortly";
|
||||
case 500:
|
||||
case 502:
|
||||
case 503:
|
||||
case 504:
|
||||
return "OpenAI-compatible: service unavailable, try again shortly";
|
||||
default: return `OpenAI-compatible: HTTP ${code}${detail ? ` (${detail})` : ""}`;
|
||||
}
|
||||
}
|
||||
@@ -18,6 +18,7 @@ import {
|
||||
} from "~/assistant/models";
|
||||
import { isLocalModelReady } from "../native/llama";
|
||||
import { normalizeWhisperServerUrl } from "../native/whisper-server-stt";
|
||||
import { normalizeOpenAiCompatibleUrl } from "../native/openai-chat";
|
||||
import { isValidSpotifyClientId, SPOTIFY_CLIENT_ID_KEY } from "../native/spotify-auth";
|
||||
import { drawRightValueMenuItem, drawToggleMenuItem, MenuItem, openModalMenu } from "./menu";
|
||||
import { ANIMATION_SPEEDS, animationSpeedLabel, scaleAnimationDuration, type AnimationSpeed } from "./animation-speed";
|
||||
@@ -942,7 +943,7 @@ export const assistantBackendSetting = new ConfigSettingEnum<AssistantBackendKin
|
||||
values: global.isIOS ? ["direct"] : ["direct", "external"],
|
||||
formatValue: (value) => assistantBackendLabels[value] ?? value,
|
||||
description:
|
||||
global.isIOS ? "Cloud models called with your OpenAI or Anthropic API key. Add your key in Settings > API Keys." : "Who answers assistant queries: an LLM called from the phone (a cloud API with your key, or the downloaded on-phone model), or your own long-running agent (e.g. OpenClaw) reached through the faceclaw-agent-bridge plugin.",
|
||||
global.isIOS ? "Cloud models called through OpenAI, Anthropic, or your configured OpenAI-compatible API." : "Who answers assistant queries: an LLM called from the phone (a cloud API with your key, or the downloaded on-phone model), or your own long-running agent (e.g. OpenClaw) reached through the faceclaw-agent-bridge plugin.",
|
||||
});
|
||||
|
||||
export const assistantHermesUrlSetting = new ConfigSettingString({
|
||||
@@ -1008,6 +1009,40 @@ export const assistantAllowProactiveSetting = new ConfigSettingBoolean({
|
||||
"Let the external agent use glasses tools outside of a conversation, e.g. showing an alert when a long-running job finishes. Rate-limited; only tools marked proactive-safe are allowed.",
|
||||
});
|
||||
|
||||
export const assistantOpenAiCompatibleUrlSetting = new ConfigSettingString({
|
||||
id: "assistant-openai-compatible-url",
|
||||
label: "Compatible API URL",
|
||||
storageKey: "assistant.openAiCompatibleUrl",
|
||||
defaultValue: "",
|
||||
editorTitle: "OpenAI-compatible API root",
|
||||
glassesEditTitle: "Edit compatible API URL",
|
||||
formatValue: (value) => value || "(not set)",
|
||||
validate: (value) => (!value.trim() || normalizeOpenAiCompatibleUrl(value) ? null : "Enter an http:// or https:// API root."),
|
||||
description: "Explicit API root for the OpenAI-compatible Chat Completions assistant. Venice: https://api.venice.ai/api/v1. Open WebUI: https://your-host/api. The provider's path is not guessed.",
|
||||
});
|
||||
|
||||
export const assistantOpenAiCompatibleKeySetting = new ConfigSettingString({
|
||||
id: "assistant-openai-compatible-key",
|
||||
label: "Compatible API key",
|
||||
storageKey: "assistant.openAiCompatibleKey",
|
||||
defaultValue: "",
|
||||
editorTitle: "OpenAI-compatible API key (optional)",
|
||||
glassesEditTitle: "Edit compatible API key",
|
||||
formatValue: (value) => (value ? `${value.slice(0, 6)}...` : "(none)"),
|
||||
description: "Bearer key sent only to the compatible assistant URL above. Optional for private servers that do not require authentication.",
|
||||
});
|
||||
|
||||
export const assistantOpenAiCompatibleModelSetting = new ConfigSettingString({
|
||||
id: "assistant-openai-compatible-model",
|
||||
label: "Compatible model ID",
|
||||
storageKey: "assistant.openAiCompatibleModel",
|
||||
defaultValue: "",
|
||||
editorTitle: "OpenAI-compatible model ID",
|
||||
glassesEditTitle: "Edit compatible model ID",
|
||||
formatValue: (value) => value || "(not set)",
|
||||
description: "Exact model ID sent to the compatible provider. For glasses tools, choose a model that supports function calling.",
|
||||
});
|
||||
|
||||
export const elevenLabsApiKeySetting = new ConfigSettingString({
|
||||
id: "elevenlabs-api-key",
|
||||
label: "ElevenLabs key",
|
||||
@@ -1112,11 +1147,12 @@ export const assistantModelSetting = new ConfigSettingEnum<AssistantModel>({
|
||||
if (provider === "anthropic") return anthropicApiKeySetting.get().trim().length === 0;
|
||||
if (provider === "openai") return openAiApiKeySetting.get().trim().length === 0;
|
||||
if (provider === "local") return !isLocalModelReady();
|
||||
if (provider === "openai-compatible") return !assistantOpenAiCompatibleUrlSetting.get().trim() || !assistantOpenAiCompatibleModelSetting.get().trim();
|
||||
if (provider === "hermes") return assistantHermesUrlSetting.get().trim().length === 0;
|
||||
return false;
|
||||
},
|
||||
description:
|
||||
global.isIOS ? "Model used by the voice assistant. Auto prefers Terra with an OpenAI key, then Sonnet with an Anthropic key." : "Model used by the voice assistant. Auto prefers Terra when an OpenAI key is set, then Sonnet when an Anthropic key is set, then the downloaded on-phone model.",
|
||||
global.isIOS ? "Model used by the assistant. OpenAI-compatible uses the URL, model ID, and optional key below. Auto only considers the built-in providers." : "Model used by the assistant. OpenAI-compatible uses the URL, model ID, and optional key below. Auto only considers the built-in providers.",
|
||||
});
|
||||
|
||||
export const mapboxApiKeySetting = new ConfigSettingString({
|
||||
|
||||
@@ -42,6 +42,9 @@ import {
|
||||
assistantBridgePortSetting,
|
||||
assistantBridgeTokenSetting,
|
||||
assistantModelSetting,
|
||||
assistantOpenAiCompatibleKeySetting,
|
||||
assistantOpenAiCompatibleModelSetting,
|
||||
assistantOpenAiCompatibleUrlSetting,
|
||||
assistantSkipConfirmationSetting,
|
||||
batteryDisplayModeSetting,
|
||||
brightnessSetting,
|
||||
@@ -148,6 +151,9 @@ function settingsSections(): SettingsSection[] {
|
||||
enumSettingMenuItem(assistantModelSetting),
|
||||
textSettingMenuItem(assistantHermesUrlSetting),
|
||||
textSettingMenuItem(assistantHermesKeySetting),
|
||||
textSettingMenuItem(assistantOpenAiCompatibleUrlSetting),
|
||||
textSettingMenuItem(assistantOpenAiCompatibleModelSetting),
|
||||
textSettingMenuItem(assistantOpenAiCompatibleKeySetting),
|
||||
...(!global.isIOS ? [localModelMenuItem()] : []),
|
||||
toggleSettingMenuItem(assistantSkipConfirmationSetting),
|
||||
// iOS has no local LLM or agent bridge yet.
|
||||
|
||||
@@ -38,6 +38,9 @@ import {
|
||||
assistantBridgePortSetting,
|
||||
assistantBridgeTokenSetting,
|
||||
assistantModelSetting,
|
||||
assistantOpenAiCompatibleKeySetting,
|
||||
assistantOpenAiCompatibleModelSetting,
|
||||
assistantOpenAiCompatibleUrlSetting,
|
||||
assistantSkipConfirmationSetting,
|
||||
batteryIndicatorSettingsKey,
|
||||
brightnessSetting,
|
||||
@@ -1720,6 +1723,9 @@ class Shell {
|
||||
openai: openAiApiKeySetting.get(),
|
||||
hermesUrl: assistantHermesUrlSetting.get(),
|
||||
hermesKey: assistantHermesKeySetting.get(),
|
||||
openAiCompatibleUrl: assistantOpenAiCompatibleUrlSetting.get(),
|
||||
openAiCompatibleKey: assistantOpenAiCompatibleKeySetting.get(),
|
||||
openAiCompatibleModel: assistantOpenAiCompatibleModelSetting.get(),
|
||||
});
|
||||
if (llm && reasoning !== "default" && llm.effort) llm.effort = reasoning;
|
||||
return llm ? { kind: "direct", llm } : null;
|
||||
@@ -1749,7 +1755,7 @@ class Shell {
|
||||
this.showAlert(
|
||||
assistantBackendSetting.get() === "external"
|
||||
? "Configure the agent bridge host and token in Settings."
|
||||
: global.isIOS ? "Set an OpenAI or Anthropic API key in Settings." : "Set an API key or download the on-phone model in Settings.",
|
||||
: global.isIOS ? "Configure an assistant model in Settings." : "Set an API key or download the on-phone model in Settings.",
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,275 @@
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const { loader } = require('./helpers/load-typescript.cjs');
|
||||
const flush = () => new Promise(resolve => setImmediate(resolve));
|
||||
|
||||
function assistant({ apiKey = 'fixture-key', baseUrl = 'https://provider.example/api/v1' } = {}) {
|
||||
const requests = [], calls = [], deltas = [], errors = [], done = [];
|
||||
const registry = {
|
||||
listTools: () => [
|
||||
{ name: 'timer.set', description: 'Set a timer', inputSchema: { type: 'object', properties: { seconds: { type: 'number' } } } },
|
||||
{ name: 'glasses.show_alert', description: 'Show an alert', inputSchema: { type: 'object', properties: { text: { type: 'string' } } } },
|
||||
],
|
||||
callTool: async (name, args) => {
|
||||
calls.push({ name, args });
|
||||
return { ok: true, content: `${name} completed` };
|
||||
},
|
||||
};
|
||||
const load = loader({ global: { isIOS: false, isAndroid: true }, setTimeout, clearTimeout }, {
|
||||
'./sse': { openSseRequest: (url, body, headers, listener) => {
|
||||
const request = { url, body: JSON.parse(body), headers, listener, cancelled: false };
|
||||
requests.push(request);
|
||||
return { cancel: () => { request.cancelled = true; } };
|
||||
} },
|
||||
'../native/llama': { streamLocalQwen: () => assert.fail('unexpected local model'), isLocalModelReady: () => false, LOCAL_MODEL: { id: 'qwen', label: 'Qwen' } },
|
||||
'./bridge-client': { assistantBridge: {} },
|
||||
'./tool-registry': { toolRegistry: registry },
|
||||
'../prompts': {
|
||||
ASSISTANT_SYSTEM_PROMPT_BASE: 'test',
|
||||
buildAssistantSystemPrompt: () => 'glasses system',
|
||||
describeAssistantContext: () => '',
|
||||
},
|
||||
});
|
||||
const { AssistantSession } = load('app/assistant/session.ts');
|
||||
const session = new AssistantSession({ kind: 'direct', llm: {
|
||||
selection: 'openai-compatible', provider: 'openai-compatible', model: 'fixture-model', apiKey, baseUrl,
|
||||
} }, registry);
|
||||
const send = text => session.sendUtterance(text, {}, {
|
||||
onTextDelta: (_delta, full) => deltas.push(full),
|
||||
onToolActivity() {},
|
||||
onError: error => errors.push(error),
|
||||
onTurnDone: result => done.push(result),
|
||||
});
|
||||
const event = (request, value) => request.listener.onLine('data: ' + JSON.stringify(value));
|
||||
const chunk = (request, delta, finishReason = null) => event(request, { choices: [{ index: 0, delta, finish_reason: finishReason }] });
|
||||
const finish = (request, reason = 'stop', close = 'done') => {
|
||||
chunk(request, {}, reason);
|
||||
if (close === 'done') request.listener.onLine('data: [DONE]');
|
||||
else request.listener.onComplete();
|
||||
};
|
||||
return { load, session, send, event, chunk, finish, requests, calls, deltas, errors, done };
|
||||
}
|
||||
|
||||
test('compatible provider sends full history, nested tools, the exact API root and optional auth', () => {
|
||||
const h = assistant();
|
||||
h.send('Remember the blue door');
|
||||
const first = h.requests[0];
|
||||
assert.equal(first.url, 'https://provider.example/api/v1/chat/completions');
|
||||
assert.equal(first.headers.Authorization, 'Bearer fixture-key');
|
||||
assert.equal(first.headers['Content-Type'], 'application/json');
|
||||
assert.equal(first.body.model, 'fixture-model');
|
||||
assert.equal(first.body.stream, true);
|
||||
assert.equal('max_tokens' in first.body, false, 'provider default output limit is preserved');
|
||||
assert.equal('reasoning_effort' in first.body, false, 'custom models default to provider reasoning');
|
||||
assert.deepEqual(first.body.messages.map(m => [m.role, m.content]), [
|
||||
['system', 'glasses system'], ['user', 'Remember the blue door'],
|
||||
]);
|
||||
assert.equal(first.body.tools[0].type, 'function');
|
||||
assert.equal(first.body.tools[0].function.name, 'timer_set');
|
||||
assert.equal(first.body.tools[0].function.parameters.type, 'object');
|
||||
h.chunk(first, { content: 'Remembered.' }); h.finish(first);
|
||||
h.send('What color?');
|
||||
assert.deepEqual(h.requests[1].body.messages.slice(-3).map(m => [m.role, m.content]), [
|
||||
['user', 'Remember the blue door'], ['assistant', 'Remembered.'], ['user', 'What color?'],
|
||||
]);
|
||||
h.session.cancel();
|
||||
|
||||
const noKey = assistant({ apiKey: '' });
|
||||
noKey.send('Private request');
|
||||
assert.equal('Authorization' in noKey.requests[0].headers, false);
|
||||
noKey.session.cancel();
|
||||
});
|
||||
|
||||
test('Venice and Open WebUI roots are preserved; only a complete endpoint suffix is removed', () => {
|
||||
const { normalizeOpenAiCompatibleUrl } = assistant().load('app/native/openai-chat.ts');
|
||||
assert.equal(normalizeOpenAiCompatibleUrl(' https://api.venice.ai/api/v1/ '), 'https://api.venice.ai/api/v1');
|
||||
assert.equal(normalizeOpenAiCompatibleUrl('https://api.venice.ai/api/v1/chat/completions'), 'https://api.venice.ai/api/v1');
|
||||
assert.equal(normalizeOpenAiCompatibleUrl('http://100.64.0.1:3000/api'), 'http://100.64.0.1:3000/api');
|
||||
assert.equal(normalizeOpenAiCompatibleUrl('https://openwebui.example/api/chat/completions/'), 'https://openwebui.example/api');
|
||||
assert.equal(normalizeOpenAiCompatibleUrl('https://provider.example/custom/root'), 'https://provider.example/custom/root');
|
||||
assert.equal(normalizeOpenAiCompatibleUrl('provider.example/v1'), '');
|
||||
assert.equal(normalizeOpenAiCompatibleUrl('https://provider.example/v1?key=secret'), '');
|
||||
assert.equal(normalizeOpenAiCompatibleUrl('https://provider.example/v1#fragment'), '');
|
||||
assert.equal(normalizeOpenAiCompatibleUrl('https://user:secret@provider.example/v1'), '', 'credentials never belong in saved URLs');
|
||||
});
|
||||
|
||||
test('compatible model resolves only with URL and model, and Auto remains unchanged', () => {
|
||||
const { ASSISTANT_MODEL_CHOICES, resolveAssistantModel } = assistant().load('app/assistant/models.ts');
|
||||
assert.equal(ASSISTANT_MODEL_CHOICES.includes('openai-compatible'), true);
|
||||
assert.equal(resolveAssistantModel('openai-compatible', { anthropic: '', openai: '' }), null);
|
||||
assert.equal(resolveAssistantModel('openai-compatible', {
|
||||
anthropic: '', openai: '', openAiCompatibleUrl: 'https://provider.example/api/v1', openAiCompatibleModel: '',
|
||||
}), null);
|
||||
const resolved = resolveAssistantModel('openai-compatible', {
|
||||
anthropic: '', openai: '',
|
||||
openAiCompatibleUrl: ' https://provider.example/api/v1 ',
|
||||
openAiCompatibleKey: ' secret ', openAiCompatibleModel: ' model-id ',
|
||||
});
|
||||
assert.equal(resolved.provider, 'openai-compatible');
|
||||
assert.equal(resolved.model, 'model-id');
|
||||
assert.equal(resolved.apiKey, 'secret');
|
||||
assert.equal(resolved.baseUrl, 'https://provider.example/api/v1');
|
||||
assert.equal(resolved.effort, undefined);
|
||||
assert.equal(resolveAssistantModel('auto', {
|
||||
anthropic: '', openai: '', openAiCompatibleUrl: 'https://provider.example/api/v1', openAiCompatibleModel: 'model-id',
|
||||
}), null, 'Auto never selects a custom provider');
|
||||
});
|
||||
|
||||
test('streamed tool calls execute through Faceclaw, results continue the turn, and history survives follow-up', async () => {
|
||||
const h = assistant();
|
||||
h.send('Set a one-minute timer');
|
||||
const first = h.requests[0];
|
||||
h.chunk(first, { content: 'Setting it now.' });
|
||||
h.chunk(first, {
|
||||
reasoning_content: 'hidden reasoning',
|
||||
reasoning_details: [{ type: 'reasoning.encrypted', data: 'opaque-fixture' }],
|
||||
tool_calls: [{ index: 0, id: 'call-1', type: 'function', function: { name: 'timer_set', arguments: '{"seconds":' } }],
|
||||
});
|
||||
h.chunk(first, { tool_calls: [{ index: 0, function: { arguments: '60}' } }] });
|
||||
h.finish(first, 'tool_calls');
|
||||
await flush();
|
||||
assert.equal(h.calls.length, 1);
|
||||
assert.equal(h.calls[0].name, 'timer.set');
|
||||
assert.equal(h.calls[0].args.seconds, 60);
|
||||
assert.equal(h.requests.length, 2);
|
||||
const second = h.requests[1];
|
||||
const assistantMessage = second.body.messages.find(m => m.role === 'assistant' && m.tool_calls);
|
||||
assert.equal(assistantMessage.content, 'Setting it now.');
|
||||
assert.equal(assistantMessage.tool_calls[0].id, 'call-1');
|
||||
assert.equal(assistantMessage.tool_calls[0].function.name, 'timer_set');
|
||||
assert.equal(assistantMessage.tool_calls[0].function.arguments, '{"seconds":60}');
|
||||
assert.equal(assistantMessage.reasoning_details[0].data, 'opaque-fixture');
|
||||
const toolMessage = second.body.messages.find(m => m.role === 'tool');
|
||||
assert.equal(toolMessage.tool_call_id, 'call-1');
|
||||
assert.equal(toolMessage.content, 'timer.set completed');
|
||||
assert.equal(JSON.stringify(second.body).includes('hidden reasoning'), false);
|
||||
h.chunk(second, { content: 'Your timer is running.' });
|
||||
h.finish(second, 'stop', 'close');
|
||||
assert.equal(h.session.isTurnActive(), false);
|
||||
assert.equal(h.done.length, 1);
|
||||
assert.deepEqual(h.errors, []);
|
||||
assert.equal(h.deltas.includes('hidden reasoning'), false);
|
||||
h.send('What did you set?');
|
||||
assert.match(JSON.stringify(h.requests[2].body), /Your timer is running/);
|
||||
h.session.cancel();
|
||||
assert.equal(h.requests[2].cancelled, true);
|
||||
});
|
||||
|
||||
test('changing the API endpoint preserves visible history but drops provider-specific state', () => {
|
||||
const h = assistant();
|
||||
const original = h.session.config;
|
||||
const config = (baseUrl) => ({ kind: 'direct', llm: {
|
||||
selection: 'openai-compatible', provider: 'openai-compatible', model: 'fixture-model', apiKey: 'fixture-key', baseUrl,
|
||||
} });
|
||||
h.session.sendUtterance('Remember the blue door', {}, {
|
||||
onTextDelta() {}, onToolActivity() {}, onError: error => assert.fail(error), onTurnDone() {},
|
||||
});
|
||||
const first = h.requests[0];
|
||||
h.chunk(first, { reasoning_details: [{ type: 'opaque', data: 'server-one-only' }], content: 'Remembered.' });
|
||||
h.finish(first);
|
||||
const history = JSON.parse(JSON.stringify(h.session.history()));
|
||||
assert.equal(JSON.stringify(history).includes('server-one-only'), true);
|
||||
h.session.configure(config('https://provider-two.example/api/v1'));
|
||||
h.send('What color?');
|
||||
const second = h.requests[1];
|
||||
assert.equal(second.url, 'https://provider-two.example/api/v1/chat/completions');
|
||||
assert.equal(JSON.stringify(second.body).includes('server-one-only'), false);
|
||||
assert.equal(JSON.stringify(second.body).includes('Remember the blue door'), true);
|
||||
h.session.cancel();
|
||||
const restored = new h.session.constructor(config('https://provider-two.example/api/v1'), undefined, history);
|
||||
restored.sendUtterance('What color?', {}, {
|
||||
onTextDelta() {}, onToolActivity() {}, onError: error => assert.fail(error), onTurnDone() {},
|
||||
});
|
||||
assert.equal(JSON.stringify(h.requests.at(-1).body).includes('server-one-only'), false);
|
||||
restored.cancel();
|
||||
});
|
||||
|
||||
test('multiple interleaved tool calls are reassembled independently and all results are returned', async () => {
|
||||
const h = assistant();
|
||||
h.send('Set a timer and show an alert');
|
||||
const first = h.requests[0];
|
||||
h.chunk(first, { tool_calls: [
|
||||
{ index: 0, id: 'timer-id', function: { name: 'timer_set', arguments: '{"sec' } },
|
||||
{ index: 1, id: 'alert-id', function: { name: 'glasses_show_alert', arguments: '{"text":' } },
|
||||
] });
|
||||
h.chunk(first, { tool_calls: [
|
||||
{ index: 0, function: { arguments: 'onds":30}' } },
|
||||
{ index: 1, function: { arguments: '"Done"}' } },
|
||||
] });
|
||||
h.finish(first, 'tool_calls');
|
||||
await flush();
|
||||
// Values produced inside the VM have a different Object prototype.
|
||||
assert.equal(JSON.stringify(h.calls.map(c => [c.name, c.args])), JSON.stringify([
|
||||
['timer.set', { seconds: 30 }], ['glasses.show_alert', { text: 'Done' }],
|
||||
]));
|
||||
const second = h.requests[1];
|
||||
const tools = second.body.messages.filter(m => m.role === 'tool');
|
||||
assert.equal(JSON.stringify(tools.map(m => [m.tool_call_id, m.content])), JSON.stringify([
|
||||
['timer-id', 'timer.set completed'], ['alert-id', 'glasses.show_alert completed'],
|
||||
]));
|
||||
h.chunk(second, { content: 'Both actions completed.' });
|
||||
h.finish(second);
|
||||
assert.equal(h.session.isTurnActive(), false);
|
||||
assert.deepEqual(h.errors, []);
|
||||
});
|
||||
|
||||
test('content-only streams work with [DONE], while empty and in-band error streams fail clearly', () => {
|
||||
const h = assistant();
|
||||
h.send('Hello');
|
||||
h.chunk(h.requests[0], { role: 'assistant' });
|
||||
h.chunk(h.requests[0], { reasoning_content: 'not for the lenses' });
|
||||
h.chunk(h.requests[0], { content: 'Hello ' });
|
||||
h.chunk(h.requests[0], { content: 'there' });
|
||||
h.requests[0].listener.onLine('data: [DONE]');
|
||||
assert.equal(h.done[0].stopReason, 'end_turn');
|
||||
assert.equal(h.session.transcript.at(-1).text, 'Hello there');
|
||||
|
||||
h.send('Empty'); h.finish(h.requests[1]);
|
||||
assert.match(h.errors.at(-1), /without a reply/);
|
||||
h.send('Failure');
|
||||
h.event(h.requests[2], { error: { message: 'model overloaded' } });
|
||||
assert.match(h.errors.at(-1), /model overloaded/);
|
||||
});
|
||||
|
||||
test('HTTP errors, malformed tool JSON and premature EOF never execute incomplete tools', async () => {
|
||||
const h = assistant();
|
||||
h.send('Unauthorized');
|
||||
h.requests[0].listener.onHttpError(401, '{"error":{"message":"bad key"}}');
|
||||
assert.match(h.errors.at(-1), /invalid API key/);
|
||||
assert.equal(h.session.isTurnActive(), false);
|
||||
|
||||
h.send('Malformed');
|
||||
h.chunk(h.requests[1], { tool_calls: [{ index: 0, id: 'bad', function: { name: 'timer_set', arguments: '{broken' } }] });
|
||||
h.finish(h.requests[1], 'tool_calls');
|
||||
await flush();
|
||||
assert.match(h.errors.at(-1), /malformed tool arguments/);
|
||||
assert.equal(h.calls.length, 0);
|
||||
|
||||
h.send('Truncated');
|
||||
h.chunk(h.requests[2], { tool_calls: [{ index: 0, id: 'partial', function: { name: 'timer_set', arguments: '{"seconds":60}' } }] });
|
||||
h.requests[2].listener.onComplete();
|
||||
await flush();
|
||||
assert.match(h.errors.at(-1), /before the response completed/);
|
||||
assert.equal(h.calls.length, 0);
|
||||
|
||||
h.send('Length limited');
|
||||
h.chunk(h.requests[3], { tool_calls: [{ index: 0, id: 'partial', function: { name: 'timer_set', arguments: '{"seconds":60}' } }] });
|
||||
h.finish(h.requests[3], 'length');
|
||||
await flush();
|
||||
assert.match(h.errors.at(-1), /before its tool calls completed/);
|
||||
assert.equal(h.calls.length, 0);
|
||||
});
|
||||
|
||||
test('cancellation suppresses late deltas and tool side effects', async () => {
|
||||
const h = assistant();
|
||||
h.send('Cancel this');
|
||||
const request = h.requests[0];
|
||||
h.session.cancel();
|
||||
h.chunk(request, { content: 'late text', tool_calls: [{ index: 0, id: 'late', function: { name: 'timer_set', arguments: '{}' } }] });
|
||||
h.finish(request, 'tool_calls');
|
||||
await flush();
|
||||
assert.equal(h.calls.length, 0);
|
||||
assert.equal(h.deltas.length, 0);
|
||||
assert.equal(h.done.length, 0);
|
||||
assert.equal(request.cancelled, true);
|
||||
});
|
||||
@@ -0,0 +1,85 @@
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const { loader } = require('./helpers/load-typescript.cjs');
|
||||
|
||||
function sharedEnv() {
|
||||
const compatible = [], unexpected = [];
|
||||
const llama = {
|
||||
streamLocalQwen: () => unexpected.push('local'),
|
||||
isLocalModelReady: () => false,
|
||||
LOCAL_MODEL: { id: 'qwen', label: 'Qwen' },
|
||||
};
|
||||
const settings = {
|
||||
anthropicApiKeySetting: { get: () => '' },
|
||||
assistantModelSetting: { get: () => 'openai-compatible' },
|
||||
assistantOpenAiCompatibleUrlSetting: { get: () => 'https://provider.example/api/v1' },
|
||||
assistantOpenAiCompatibleKeySetting: { get: () => 'compatible-key' },
|
||||
assistantOpenAiCompatibleModelSetting: { get: () => 'compatible-model' },
|
||||
openAiApiKeySetting: { get: () => '' },
|
||||
};
|
||||
const fail = provider => () => { unexpected.push(provider); };
|
||||
const load = loader({ global: { isIOS: false, isAndroid: true } }, {
|
||||
'../../native/anthropic': { streamAnthropicMessage: fail('anthropic') },
|
||||
'../../native/openai': { streamOpenAiResponse: fail('openai') },
|
||||
'../../native/llama': llama,
|
||||
'../native/llama': llama,
|
||||
'../../native/openai-chat': {
|
||||
streamOpenAiCompatibleChat: options => {
|
||||
compatible.push(options);
|
||||
return { cancel() {} };
|
||||
},
|
||||
},
|
||||
'../../ui/dashboard-settings': settings,
|
||||
'../../prompts': {
|
||||
CALCULATOR_REWRITE_SYSTEM_PROMPT: 'calculator system',
|
||||
buildCalculatorRewriteUserMessage: spoken => `rewrite: ${spoken}`,
|
||||
TRANSLATE_SYSTEM_PROMPT: 'translate system',
|
||||
buildTranslateUserMessage: speech => `translate: ${speech}`,
|
||||
},
|
||||
});
|
||||
return { load, compatible, unexpected };
|
||||
}
|
||||
|
||||
test('Calculator uses the configured compatible provider and preserves its small output cap', async () => {
|
||||
const env = sharedEnv();
|
||||
const { rewriteSpokenProblem } = env.load('app/apps/calculator/calculator-llm.ts');
|
||||
const result = rewriteSpokenProblem('one plus one');
|
||||
assert.equal(env.compatible.length, 1);
|
||||
assert.deepEqual(env.unexpected, []);
|
||||
const options = env.compatible[0];
|
||||
assert.equal(options.baseUrl, 'https://provider.example/api/v1');
|
||||
assert.equal(options.apiKey, 'compatible-key');
|
||||
assert.equal(options.model, 'compatible-model');
|
||||
assert.equal(options.maxTokens, 200);
|
||||
assert.equal(options.messages[0].content, 'rewrite: one plus one');
|
||||
options.onDone({ text: '1 + 1', content: [], stopReason: 'end_turn' });
|
||||
assert.equal(await result, '1 + 1');
|
||||
});
|
||||
|
||||
test('Translate resolves and streams through the compatible provider without changing its cancellation contract', () => {
|
||||
const env = sharedEnv();
|
||||
const { resolveTranslationModel, streamTranslation } = env.load('app/apps/translate/translate-llm.ts');
|
||||
const model = resolveTranslationModel();
|
||||
assert.equal(model.provider, 'openai-compatible');
|
||||
assert.equal(model.baseUrl, 'https://provider.example/api/v1');
|
||||
const progress = [], answers = [], errors = [];
|
||||
const handle = streamTranslation(model, {
|
||||
speech: 'Hola', context: [], language: 'Spanish',
|
||||
onProgress: text => progress.push(text),
|
||||
onDone: text => answers.push(text),
|
||||
onError: error => errors.push(error),
|
||||
});
|
||||
assert.equal(typeof handle.cancel, 'function');
|
||||
assert.equal(env.compatible.length, 1);
|
||||
assert.deepEqual(env.unexpected, []);
|
||||
const options = env.compatible[0];
|
||||
assert.equal(options.baseUrl, 'https://provider.example/api/v1');
|
||||
assert.equal(options.model, 'compatible-model');
|
||||
assert.equal(options.maxTokens, 1024);
|
||||
assert.equal(options.messages[0].content, 'translate: Hola');
|
||||
options.onTextDelta('Hello', '<english>Hello</english>');
|
||||
options.onDone({ text: '<english>Hello</english>', content: [], stopReason: 'end_turn' });
|
||||
assert.deepEqual(progress, ['Hello']);
|
||||
assert.deepEqual(answers, ['Hello']);
|
||||
assert.deepEqual(errors, []);
|
||||
});
|
||||
@@ -6,7 +6,7 @@
|
||||
Inline markdown supported by the sync script: **bold**, [text](url), `code`.
|
||||
-->
|
||||
|
||||
- **A voice assistant** that wakes up when you say "Hey Even", transcribes text with an on-device model, OpenAI Whisper, ElevenLabs, or Soniox API, and responds to queries and commands using an onboard model (Qwen3 4B, slow) or with an Anthropic or OpenAI model (requires an API key), or using your own long-running OpenClaw agent.
|
||||
- **A voice assistant** that wakes up when you say "Hey Even", transcribes text with an on-device model, OpenAI Whisper, ElevenLabs, or Soniox API, and responds to queries and commands using an onboard model (Qwen3 4B, slow) or with an Anthropic, OpenAI, or OpenAI-compatible Chat Completions model (including Venice and private Open WebUI servers; model ID and usually an API key required), or using your own long-running OpenClaw agent.
|
||||
- **Multitasking**, with an app-switcher sidebar and app launcher.
|
||||
- **Glanceboard**, a dashboard with widgets that you can quickly view with a
|
||||
- **Mostly-compatible with EvenHub.** Install and runapps from the Even app store.
|
||||
|
||||
+1
-1
@@ -89,7 +89,7 @@
|
||||
<h2>Features</h2>
|
||||
<!-- BEGIN GENERATED: features -->
|
||||
<ul class="features">
|
||||
<li><strong>A voice assistant</strong> that wakes up when you say "Hey Even", transcribes text with an on-device model, OpenAI Whisper, ElevenLabs, or Soniox API, and responds to queries and commands using an onboard model (Qwen3 4B, slow) or with an Anthropic or OpenAI model (requires an API key), or using your own long-running OpenClaw agent.</li>
|
||||
<li><strong>A voice assistant</strong> that wakes up when you say "Hey Even", transcribes text with an on-device model, OpenAI Whisper, ElevenLabs, or Soniox API, and responds to queries and commands using an onboard model (Qwen3 4B, slow) or with an Anthropic, OpenAI, or OpenAI-compatible Chat Completions model (including Venice and private Open WebUI servers; model ID and usually an API key required), or using your own long-running OpenClaw agent.</li>
|
||||
<li><strong>Multitasking</strong>, with an app-switcher sidebar and app launcher.</li>
|
||||
<li><strong>Glanceboard</strong>, a dashboard with widgets that you can quickly view with a</li>
|
||||
<li><strong>Mostly-compatible with EvenHub.</strong> Install and runapps from the Even app store.</li>
|
||||
|
||||
+12
-2
@@ -70,11 +70,13 @@
|
||||
</ul>
|
||||
<p>Third party data retention and privacy policies may differ based on the API key provided. API keys are part of a relationship between the user and the API provider; Faceclaw does not manage accounts or process billing information.</p>
|
||||
<h2>AI Assistant Transcripts</h2>
|
||||
<p>Faceclaw has a builtin AI voice assistant. Using the voice assistant requires providing an API key for a model provider such as Anthropic or OpenAI. Prompts and responses are subject to that model provider's privacy policy:</p>
|
||||
<p>Faceclaw has a builtin AI voice assistant. Using a cloud model requires providing an API key for a model provider such as Anthropic, OpenAI, or Venice, or configuring an OpenAI-compatible endpoint such as a private Open WebUI server. Prompts and responses are subject to the selected provider or server operator's privacy policy:</p>
|
||||
<ul class="links">
|
||||
<li>OpenAI: <a href="https://openai.com/policies/row-privacy-policy/">https://openai.com/policies/row-privacy-policy/</a></li>
|
||||
<li>Anthropic: <a href="https://www.anthropic.com/legal/privacy">https://www.anthropic.com/legal/privacy</a></li>
|
||||
<li>Venice: <a href="https://venice.ai/privacy">https://venice.ai/privacy</a></li>
|
||||
</ul>
|
||||
<p>With an OpenAI-compatible provider, the configured server receives the assistant system prompt, conversation history, available tool definitions, and tool results needed to answer. Its URL, model ID, and optional API key are stored on the phone. Faceclaw does not operate that server and cannot make representations about its logging or retention; for a private Open WebUI server, the server's operator controls those policies.</p>
|
||||
<p>AI conversation histories and memory files may be stored on your phone. Faceclaw does not transmit voice-assistant prompts or responses to any party other than a user-selected AI model provider. The voice assistant is given tools with which it is able to access most things on the user's phone, including all content displayed on the glasses, location, notifications, and content accessible through the Terminal app.</p>
|
||||
<p>The AI assistant can take actions on the user's behalf, including sharing any information it has access to. The user is responsible for any instructions they give it.</p>
|
||||
<h2>Location</h2>
|
||||
@@ -82,7 +84,15 @@
|
||||
<ul class="links">
|
||||
<li>Mapbox: <a href="https://www.mapbox.com/legal/privacy">https://www.mapbox.com/legal/privacy</a></li>
|
||||
</ul>
|
||||
<p>The Weather app will retrieve the user's coarse location, and will use it to query a third-party weather information provider: the US National Weather Service, then <a href="https://open-meteo.com/en/terms">Open-Meteo</a> if the location is outside the US. Outside the US, it also asks the phone's built-in geocoder (Google's on most Android phones, Apple's on iOS) for the name of the place.</p>
|
||||
<p>The Weather app will retrieve the user's coarse location, and will use it to query a third-party weather information provider: the US National Weather Service, then Open-Meteo if the location is outside the US. Outside the US, it also asks the phone's built-in geocoder (Google's on most Android phones, Apple's on iOS) for the name of the place.</p>
|
||||
<ul class="links">
|
||||
<li>Open-Meteo: <a href="https://open-meteo.com/en/terms">https://open-meteo.com/en/terms</a></li>
|
||||
</ul>
|
||||
<h2>Spotify</h2>
|
||||
<p>On iOS, the Music app, Glanceboard and the voice assistant can show and control Spotify through Spotify's Web API, once the user has created their own Spotify developer app and signed in. Signing in happens on Spotify's website in the system's sign-in sheet; Faceclaw never sees the password, and stores the resulting sign-in tokens on the phone. While Spotify is shown or used, Faceclaw requests the user's playback state, queue, library and search results from Spotify and sends it playback commands, directly from the phone. This is subject to Spotify's privacy policy.</p>
|
||||
<ul class="links">
|
||||
<li>Spotify: <a href="https://www.spotify.com/legal/privacy-policy/">https://www.spotify.com/legal/privacy-policy/</a></li>
|
||||
</ul>
|
||||
<h2>EvenHub</h2>
|
||||
<p>The EvenHub app store is subject to Even Realities' privacy policy: <a href="https://support.evenrealities.com/hc/en-us/articles/14127464826511-Privacy-Policy">https://support.evenrealities.com/hc/en-us/articles/14127464826511-Privacy-Policy</a>. Apps downloaded from the EvenHub app store come with their own privacy policies.</p>
|
||||
<h2>Wear OS Watch</h2>
|
||||
|
||||
Reference in new issue
Block a user