mirror of
https://git.undefined-dev.fr/Blendman974/faceclaw.git
synced 2026-11-19 21:03:09 +00:00
<details> <summary>Prompts Used</summary> User: Review the Microphones app, focusing on error handling and reliability. Opus 5.5: > I've finished reading the Microphones code and its native pieces. The 2026-10-14 commit `b50898e` fixed a lot, and the main problem left is the same in most places: the app reports "Listening" or "captions on" but never checks that audio is actually arriving. When something fails partway, the session usually stops working with no visible sign. > > ## High: silent loss of function or data > > **1. In stock mode, other mic users can take the mic away for good.** > - `stopRawCapture()` tears down the single shared audio tap no matter who started it (`app/native/voice-control.ts:257`). > - The EvenHub mic router calls it whenever a mic-using EvenHub app goes to the background or closes (`app/apps/evenhub/mic-router.ts:88`). That kills Microphones' audio. > - Transcribe takes the mic without setting `voiceActivity` (`voice-control.ts:269`), so Microphones is never told to get it back. > - If capture starts while the session isn't ready, the native controller only records the request, so no audio ever flows. > - `checkExtendedCapture` (`mic-session.ts:456`) only covers 4-mic mode. Its comment says the stock path "is resumed by the voice-control bridge", but that only covers the session-dropped-while-active case. > - In 4-mic mode the problem runs the other way: Transcribe takes over the shared Kotlin listener (`GlassesSessionCore.kt:470`) while the glasses keep sending 4-mic frames, so Transcribe gets audio in the wrong format. > - **Fix:** track who holds the raw tap (like `captureHolders`), add a stock-mode branch to the 3 s check, and have Transcribe set the same "mic in use" signal as the assistant. > > **2. One bad utterance ends captions for the rest of the session.** > - The caption worker has no per-utterance error handling. Any exception from `recognize()` or `embed()` stops the worker for good (`CaptionEngineCore.kt:334`). > - `started` stays true, so TypeScript still thinks captions are running and recording carries on. The only sign is a "Captions failed:" status string. > - **Fix:** catch errors per utterance in `onUtterance` (`:367`). Report a fatal worker exit through its own callback so the session can restart the engine or mark captions inactive. > > **3. A crash or low-memory kill during recording leaves files that are never shown or deleted.** > - The WAV header is written with a data size of 0 (`FaceclawWavRecorder.kt:26`) and only corrected in `finish()`. > - The file is only linked to its conversation in `stopRecording` (`mic-session.ts:1200`), and the conversation row is never closed. > - Nothing cleans up `mic-recordings/`, so after a crash you get the same never-expired orphan files the last commit fixed for the captions-off case. > - **Fix:** link the file to its conversation when recording starts. In the boot-time cleanup: close conversations left open (`ended_at IS NULL`), repair the WAV header from the file length, and delete files no conversation points to. > > **4. When the transcoder's success callback fails, it deletes both copies of the recording.** > - `FaceclawAudioTranscoder.kt:66-83` deletes the WAV before calling `onDone`. If `onDone` throws, the catch block deletes the M4A too, and the database still points at the deleted WAV. > - Separately, the JS listener is referenced only from Java (`mic-session.ts:1208`). Under `markingMode: 'none'` that's the situation `mic-control.ts:74` warns about. > - I haven't confirmed the garbage-collection risk. `asr-model.ts` does the same thing and its downloads seem to work. But the delete order is wrong either way. > - `rediarize.ts` and `mic-models.ts` hold their listeners the same way. > > ## Medium > > 5. **The last utterance is lost when you close the app or turn captions off.** When the caption engine stops, it flushes the utterance in progress and posts it to the main thread. By the time it arrives, `stop()` has already run `finishSessionRow()` (`mic-session.ts:328`), so the line is never saved and is missing from the session insights. If decoding takes longer than the 2 s wait, the line can land in the *next* session with wrong times. Also, `stop()` can block the UI for up to 2 s. > 6. **The beam filter can mute 4-mic captions permanently.** The direction and signal strength only update on frames with a non-zero angle or strength (`:685`), and they never go stale. If you turn off "Direction detection", the last direction stays frozen. If it was outside the beam, `deliverExtendedAudio` drops all audio from then on (`:751-757`). Stock mode avoids this because it updates the strength on every frame. **Fix:** timestamp the fused direction and treat it as unknown after `TEMPLE_STALE_MS`. > 7. **The 4-mic health check looks at a phone-side flag, not at whether audio is arriving.** > - The check only re-arms when `isAudioCaptureActive()` is false. > - A configure command can be skipped while the session isn't ready even though `send()` returns true (`mic-control.ts:155`, `GlassesSessionCore.kt:890`). > - A temple can reboot on its own, or `apply()` can refuse to arm when the developer setting is turned off mid-session. > - Each of these leaves forwarding "active" with no frames arriving. > - **Fix:** you already track `templeFrameAtMs`. Re-send the config when an enabled temple has gone quiet, and show that in the status instead of "Listening (4-mic array)". > 8. **A failed start is permanent.** `start()` returns before installing any timers (`:280`). A failed mode switch mid-session (`:376`) leaves `forwardingProxy` null, so the health check exits at `:457` every time. **Fix:** stay in a "waiting for glasses" state and let the timer retry. > 9. **The caption feed can flip between left and right on every frame.** The choice at `:744` doesn't take the current side into account, and `strongerChanged` is just a `!==` check. Near the 1.5× threshold, each flip splices about 25 ms from the other temple (which runs on its own clock) into captions and the recording, either repeating or skipping audio. > > ## Lower > - **Unguarded speaker call (`:1034`):** the speaker rename for a pending introduction runs before the line is added. A database error drops the line and leaves the introduction pending, so the next 2 minutes of lines from unnamed speakers fail the same way. Add the line first, then do the extra steps each with its own error handling. > - **Speaker list grows forever:** every unknown voice adds a permanent "Speaker N" with a voiceprint. Retention never prunes them. `assign()` reloads and decodes every profile on every utterance, on the main thread. > - **Translation:** there's no timeout, so lines can wait indefinitely while a model download is stuck offline. The detected language is only saved after translation finishes (`:1132`). The "hysteresis" comment at `:1120` describes logic that doesn't exist: `conversationLang` is written but never read. > - **Frame fields ignored:** the sequence number (but `TempleAudioPipeline` assumes consecutive frames), the truncated flag, and `sampleRateHz`. The UI offers 8–48 kHz while the code assumes 16 kHz. > - **Duplicated model path:** `moonshineModelDir()` (`:1337`) copies `asr-model.ts`'s directory name. A model version bump there would quietly drop captions to "no ASR model". > - **Silent recording gaps:** `WavRecorder.append` swallows each write error, so a full disk means log spam and silent gaps in the recording. > > The startup order with cleanup on failure, the 90 s lease that recovers on its own from a lost STOP, and keeping the last good speaker cache all hold up well. > > I worked from the code only; nothing was tested on hardware. #6 relies on the protocol comment saying the angle is 0 when direction detection is off. > > I'd fix #1, #2, #5 and #6 first, since those are the ones you'd hit in normal use. Want me to start on them? <hr> User: Thanks. Let's fix all of the high and medium issues you found now. Opus 5.5: > All nine high and medium issues are fixed. TS typecheck, lint, 813 TS tests, the Kotlin host tests (4 new), the iOS Kotlin compile and `build.sh` all pass, and I confirmed the new code is in the APK. Nothing has been tested on the glasses yet, and nothing is committed. > > **High** > 1. **Mic taken by other apps:** > - The shared audio tap now tracks who's using it, so an EvenHub app backgrounding no longer cuts off Microphones. > - Transcribe now marks the mic as in use, like the assistant does, so Microphones and EvenHub apps step aside while it runs. > - A check every 3 s now covers both modes. It restarts a dead stock tap and switches back to stock mode if the developer setting is turned off mid-session. > 2. **One bad utterance ending captions:** a decode error now skips that utterance instead of stopping the caption engine (`CaptionEngineCore.kt`). If the engine dies anyway, it's restarted on the same conversation up to 3 times. After that, the Captions toggle shows off and the view says why; turning it on retries. > 3. **Recordings orphaned by a crash:** the recording is linked to its conversation as soon as it starts. At boot, the app closes conversations a crash left open, repairs their WAV headers (`FaceclawWavRecorder.repairHeader`), transcodes them, and deletes files no conversation points to. That cleanup matches by file name, so moving the app to other storage can't delete live recordings. > 4. **Transcoder deleting both copies:** a failure while recording the new file no longer deletes both files. The app now deletes the WAV itself, only after the conversation points at the M4A. The transcoder, re-diarize and model-download callbacks are now held by the app until they fire. > > **Medium** > 5. **Last utterance lost:** stopping captions no longer blocks the UI. The engine finishes the audio still queued, then sends a new `onStopped` callback, and the conversation stays open until that arrives (20 s fallback). A late line from a previous session won't show up in a newly reopened one. > 6. **Beam filter muting captions:** the direction of arrival only filters audio while it's under 1 s old, and it resets when "Direction detection" changes. > 7. **4-mic health check:** a temple that sends no audio for 5 s gets its config re-sent, to that temple only. Mic-control sends now report when they weren't actually queued. The status line says when no audio is arriving, but not after routine restarts like the voice dialog closing. > 8. **Start failures being permanent:** the session now waits ("Waiting for the glasses microphone...") and the 3 s check retries. Captions start only once the mic is up, so waiting doesn't create empty conversations. > 9. **Left/right flapping:** the caption feed only switches temples when the other one is clearly louder for 500 ms. > > Behaviour changes you'll notice: > - The watch shows "Listening…" during Transcribe. > - EvenHub mic apps get no audio while Transcribe runs. > - In 4-mic mode, an EvenHub mic app still takes over the single native audio listener. Fixing that needs a change on the Kotlin side, so I left it. > > On hardware, I'd test: > - The assistant, Transcribe and an EvenHub mic app taking the mic while Microphones is open, then giving it back. > - Reconnecting in both stock and 4-mic modes. > - Killing the app mid-recording, then checking the conversation is recovered at the next start. > > I saved these notes to memory as `microphones-reliability.md`. </details>
223 lines
6.8 KiB
TypeScript
223 lines
6.8 KiB
TypeScript
import { Utils } from "@nativescript/core";
|
|
|
|
declare const com: any;
|
|
declare const java: any;
|
|
|
|
/**
|
|
* Download management for the Microphones app's on-device models, mirroring
|
|
* the Moonshine flow in app/native/asr-model.ts (FaceclawModelDownloader,
|
|
* resume + pinned sha256 per file):
|
|
*
|
|
* - Speaker embedding (WeSpeaker CAM++ English, 512-dim): voice-prints for
|
|
* caption attribution, wearer recognition, and re-diarization.
|
|
* - Pyannote segmentation 3.0: offline re-diarization of saved recordings.
|
|
*
|
|
* The caption ASR model itself is the shared Moonshine download managed by
|
|
* asr-model.ts; this module only adds the speaker models.
|
|
*/
|
|
|
|
export type MicModelId = "speaker-embedding" | "diarization-segmentation";
|
|
|
|
type MicModelFile = {
|
|
/** Local file name under the model directory. */
|
|
name: string;
|
|
/** Source file name in the mirror repo (may differ from the local name). */
|
|
remoteName: string;
|
|
sha256: string;
|
|
sizeBytes: number;
|
|
};
|
|
|
|
type MicModel = {
|
|
id: MicModelId;
|
|
label: string;
|
|
baseUrl: string;
|
|
files: MicModelFile[];
|
|
totalBytes: number;
|
|
};
|
|
|
|
// Hugging Face mirrors of the sherpa-onnx release assets (served per-file,
|
|
// sha256 pinned from the repos' LFS pointers).
|
|
export const MIC_MODELS: MicModel[] = [
|
|
{
|
|
id: "speaker-embedding",
|
|
label: "Speaker voices (WeSpeaker English)",
|
|
baseUrl: "https://huggingface.co/csukuangfj/speaker-embedding-models/resolve/main/",
|
|
files: [
|
|
{
|
|
name: "speaker-embedding.onnx",
|
|
remoteName: "wespeaker_en_voxceleb_CAM%2B%2B.onnx",
|
|
sha256: "c46fad10b5f81e1aa4a60c162714208577093655076c5450f8c469e522ec54ef",
|
|
sizeBytes: 29292684,
|
|
},
|
|
],
|
|
totalBytes: 29292684,
|
|
},
|
|
{
|
|
id: "diarization-segmentation",
|
|
label: "Speaker turns (pyannote segmentation)",
|
|
baseUrl: "https://huggingface.co/csukuangfj/sherpa-onnx-pyannote-segmentation-3-0/resolve/main/",
|
|
files: [
|
|
{
|
|
name: "segmentation.onnx",
|
|
remoteName: "model.onnx",
|
|
sha256: "220ad67ca923bef2fa91f2390c786097bf305bceb5e261d4af67b38e938e1079",
|
|
sizeBytes: 5992913,
|
|
},
|
|
],
|
|
totalBytes: 5992913,
|
|
},
|
|
];
|
|
|
|
export type MicModelState = {
|
|
status: "absent" | "downloading" | "ready";
|
|
bytesDownloaded: number;
|
|
totalBytes: number;
|
|
};
|
|
|
|
const downloaders = new Map<MicModelId, any>();
|
|
// Each download's listener is referenced only from Java while it runs; hold
|
|
// it here so it isn't GC'd before its callbacks.
|
|
const downloadListeners = new Map<MicModelId, any>();
|
|
const progress = new Map<MicModelId, number>();
|
|
const stateListeners = new Set<(id: MicModelId, state: MicModelState) => void>();
|
|
|
|
function model(id: MicModelId): MicModel {
|
|
const found = MIC_MODELS.find((entry) => entry.id === id);
|
|
if (!found) throw new Error(`unknown mic model ${id}`);
|
|
return found;
|
|
}
|
|
|
|
function modelDirPath(id: MicModelId): string {
|
|
const context = Utils.android.getApplicationContext();
|
|
return `${context.getFilesDir().getAbsolutePath()}/faceclaw-mic-models/${id}`;
|
|
}
|
|
|
|
function filePath(id: MicModelId, file: MicModelFile): string {
|
|
return `${modelDirPath(id)}/${file.name}`;
|
|
}
|
|
|
|
function isFilePresent(id: MicModelId, file: MicModelFile): boolean {
|
|
try {
|
|
const javaFile = new java.io.File(filePath(id, file));
|
|
return javaFile.exists() && javaFile.length() > 0;
|
|
} catch {
|
|
return false;
|
|
}
|
|
}
|
|
|
|
export function isMicModelReady(id: MicModelId): boolean {
|
|
if (!global.isAndroid) return false;
|
|
return model(id).files.every((file) => isFilePresent(id, file));
|
|
}
|
|
|
|
/** Absolute path of a ready model's primary file, or null when absent. */
|
|
export function micModelPath(id: MicModelId): string | null {
|
|
if (!isMicModelReady(id)) return null;
|
|
return filePath(id, model(id).files[0]!);
|
|
}
|
|
|
|
export function micModelState(id: MicModelId): MicModelState {
|
|
if (downloaders.has(id)) {
|
|
return {
|
|
status: "downloading",
|
|
bytesDownloaded: progress.get(id) ?? 0,
|
|
totalBytes: model(id).totalBytes,
|
|
};
|
|
}
|
|
return {
|
|
status: isMicModelReady(id) ? "ready" : "absent",
|
|
bytesDownloaded: 0,
|
|
totalBytes: model(id).totalBytes,
|
|
};
|
|
}
|
|
|
|
export function onMicModelStateChanged(
|
|
listener: (id: MicModelId, state: MicModelState) => void,
|
|
): () => void {
|
|
stateListeners.add(listener);
|
|
return () => stateListeners.delete(listener);
|
|
}
|
|
|
|
function notifyStateChanged(id: MicModelId): void {
|
|
const state = micModelState(id);
|
|
stateListeners.forEach((listener) => listener(id, state));
|
|
}
|
|
|
|
export function startMicModelDownload(id: MicModelId): void {
|
|
if (!global.isAndroid || downloaders.has(id) || isMicModelReady(id)) return;
|
|
progress.delete(id);
|
|
downloadNextFile(id);
|
|
notifyStateChanged(id);
|
|
}
|
|
|
|
function downloadNextFile(id: MicModelId): void {
|
|
const definition = model(id);
|
|
const nextFile = definition.files.find((file) => !isFilePresent(id, file));
|
|
if (!nextFile) {
|
|
downloaders.delete(id);
|
|
downloadListeners.delete(id);
|
|
notifyStateChanged(id);
|
|
return;
|
|
}
|
|
const completedBytes = definition.files
|
|
.filter((file) => isFilePresent(id, file))
|
|
.reduce((sum, file) => sum + file.sizeBytes, 0);
|
|
const listener = new com.faceclaw.app.FaceclawModelDownloaderListener({
|
|
onProgress: (bytes: number, _total: number) => {
|
|
progress.set(id, completedBytes + Number(bytes));
|
|
notifyStateChanged(id);
|
|
},
|
|
onDone: () => {
|
|
downloadNextFile(id);
|
|
},
|
|
onError: (message: string) => {
|
|
console.error(`Mic model download failed (${id}/${nextFile.name}): ${message}`);
|
|
downloaders.delete(id);
|
|
downloadListeners.delete(id);
|
|
notifyStateChanged(id);
|
|
},
|
|
});
|
|
downloadListeners.set(id, listener);
|
|
try {
|
|
const downloader = new com.faceclaw.app.FaceclawModelDownloader(
|
|
`${definition.baseUrl}${nextFile.remoteName}`,
|
|
filePath(id, nextFile),
|
|
nextFile.sha256,
|
|
nextFile.sizeBytes,
|
|
listener,
|
|
);
|
|
downloaders.set(id, downloader);
|
|
downloader.start();
|
|
} catch (error) {
|
|
// Without this, a dead entry would pin the model at "downloading".
|
|
console.error(`Mic model download failed to start (${id}/${nextFile.name}): ${String(error)}`);
|
|
downloaders.delete(id);
|
|
downloadListeners.delete(id);
|
|
notifyStateChanged(id);
|
|
}
|
|
}
|
|
|
|
export function cancelMicModelDownload(id: MicModelId): void {
|
|
const downloader = downloaders.get(id);
|
|
if (!downloader) return;
|
|
downloader.cancel();
|
|
downloaders.delete(id);
|
|
downloadListeners.delete(id);
|
|
notifyStateChanged(id);
|
|
}
|
|
|
|
export function deleteMicModel(id: MicModelId): void {
|
|
if (!global.isAndroid) return;
|
|
cancelMicModelDownload(id);
|
|
try {
|
|
for (const file of model(id).files) {
|
|
new java.io.File(filePath(id, file)).delete();
|
|
new java.io.File(`${filePath(id, file)}.part`).delete();
|
|
}
|
|
new java.io.File(modelDirPath(id)).delete();
|
|
} catch (error) {
|
|
console.error(`Mic model delete failed: ${String(error)}`);
|
|
}
|
|
notifyStateChanged(id);
|
|
}
|