Files
faceclaw/app/apps/microphones/rediarize.ts
T
James Babcock 60c87521b0 Fix a bunch of microphones app reliability issues
<details>
<summary>Prompts Used</summary>
User: Review the Microphones app, focusing on error handling and reliability.

Opus 5.5:
> I've finished reading the Microphones code and its native pieces. The 2026-10-14 commit `b50898e` fixed a lot, and the main problem left is the same in most places: the app reports "Listening" or "captions on" but never checks that audio is actually arriving. When something fails partway, the session usually stops working with no visible sign.
>
> ## High: silent loss of function or data
>
> **1. In stock mode, other mic users can take the mic away for good.**
> - `stopRawCapture()` tears down the single shared audio tap no matter who started it (`app/native/voice-control.ts:257`).
> - The EvenHub mic router calls it whenever a mic-using EvenHub app goes to the background or closes (`app/apps/evenhub/mic-router.ts:88`). That kills Microphones' audio.
> - Transcribe takes the mic without setting `voiceActivity` (`voice-control.ts:269`), so Microphones is never told to get it back.
> - If capture starts while the session isn't ready, the native controller only records the request, so no audio ever flows.
> - `checkExtendedCapture` (`mic-session.ts:456`) only covers 4-mic mode. Its comment says the stock path "is resumed by the voice-control bridge", but that only covers the session-dropped-while-active case.
> - In 4-mic mode the problem runs the other way: Transcribe takes over the shared Kotlin listener (`GlassesSessionCore.kt:470`) while the glasses keep sending 4-mic frames, so Transcribe gets audio in the wrong format.
> - **Fix:** track who holds the raw tap (like `captureHolders`), add a stock-mode branch to the 3 s check, and have Transcribe set the same "mic in use" signal as the assistant.
>
> **2. One bad utterance ends captions for the rest of the session.**
> - The caption worker has no per-utterance error handling. Any exception from `recognize()` or `embed()` stops the worker for good (`CaptionEngineCore.kt:334`).
> - `started` stays true, so TypeScript still thinks captions are running and recording carries on. The only sign is a "Captions failed:" status string.
> - **Fix:** catch errors per utterance in `onUtterance` (`:367`). Report a fatal worker exit through its own callback so the session can restart the engine or mark captions inactive.
>
> **3. A crash or low-memory kill during recording leaves files that are never shown or deleted.**
> - The WAV header is written with a data size of 0 (`FaceclawWavRecorder.kt:26`) and only corrected in `finish()`.
> - The file is only linked to its conversation in `stopRecording` (`mic-session.ts:1200`), and the conversation row is never closed.
> - Nothing cleans up `mic-recordings/`, so after a crash you get the same never-expired orphan files the last commit fixed for the captions-off case.
> - **Fix:** link the file to its conversation when recording starts. In the boot-time cleanup: close conversations left open (`ended_at IS NULL`), repair the WAV header from the file length, and delete files no conversation points to.
>
> **4. When the transcoder's success callback fails, it deletes both copies of the recording.**
> - `FaceclawAudioTranscoder.kt:66-83` deletes the WAV before calling `onDone`. If `onDone` throws, the catch block deletes the M4A too, and the database still points at the deleted WAV.
> - Separately, the JS listener is referenced only from Java (`mic-session.ts:1208`). Under `markingMode: 'none'` that's the situation `mic-control.ts:74` warns about.
> - I haven't confirmed the garbage-collection risk. `asr-model.ts` does the same thing and its downloads seem to work. But the delete order is wrong either way.
> - `rediarize.ts` and `mic-models.ts` hold their listeners the same way.
>
> ## Medium
>
> 5. **The last utterance is lost when you close the app or turn captions off.** When the caption engine stops, it flushes the utterance in progress and posts it to the main thread. By the time it arrives, `stop()` has already run `finishSessionRow()` (`mic-session.ts:328`), so the line is never saved and is missing from the session insights. If decoding takes longer than the 2 s wait, the line can land in the *next* session with wrong times. Also, `stop()` can block the UI for up to 2 s.
> 6. **The beam filter can mute 4-mic captions permanently.** The direction and signal strength only update on frames with a non-zero angle or strength (`:685`), and they never go stale. If you turn off "Direction detection", the last direction stays frozen. If it was outside the beam, `deliverExtendedAudio` drops all audio from then on (`:751-757`). Stock mode avoids this because it updates the strength on every frame. **Fix:** timestamp the fused direction and treat it as unknown after `TEMPLE_STALE_MS`.
> 7. **The 4-mic health check looks at a phone-side flag, not at whether audio is arriving.**
>    - The check only re-arms when `isAudioCaptureActive()` is false.
>    - A configure command can be skipped while the session isn't ready even though `send()` returns true (`mic-control.ts:155`, `GlassesSessionCore.kt:890`).
>    - A temple can reboot on its own, or `apply()` can refuse to arm when the developer setting is turned off mid-session.
>    - Each of these leaves forwarding "active" with no frames arriving.
>    - **Fix:** you already track `templeFrameAtMs`. Re-send the config when an enabled temple has gone quiet, and show that in the status instead of "Listening (4-mic array)".
> 8. **A failed start is permanent.** `start()` returns before installing any timers (`:280`). A failed mode switch mid-session (`:376`) leaves `forwardingProxy` null, so the health check exits at `:457` every time. **Fix:** stay in a "waiting for glasses" state and let the timer retry.
> 9. **The caption feed can flip between left and right on every frame.** The choice at `:744` doesn't take the current side into account, and `strongerChanged` is just a `!==` check. Near the 1.5× threshold, each flip splices about 25 ms from the other temple (which runs on its own clock) into captions and the recording, either repeating or skipping audio.
>
> ## Lower
> - **Unguarded speaker call (`:1034`):** the speaker rename for a pending introduction runs before the line is added. A database error drops the line and leaves the introduction pending, so the next 2 minutes of lines from unnamed speakers fail the same way. Add the line first, then do the extra steps each with its own error handling.
> - **Speaker list grows forever:** every unknown voice adds a permanent "Speaker N" with a voiceprint. Retention never prunes them. `assign()` reloads and decodes every profile on every utterance, on the main thread.
> - **Translation:** there's no timeout, so lines can wait indefinitely while a model download is stuck offline. The detected language is only saved after translation finishes (`:1132`). The "hysteresis" comment at `:1120` describes logic that doesn't exist: `conversationLang` is written but never read.
> - **Frame fields ignored:** the sequence number (but `TempleAudioPipeline` assumes consecutive frames), the truncated flag, and `sampleRateHz`. The UI offers 8–48 kHz while the code assumes 16 kHz.
> - **Duplicated model path:** `moonshineModelDir()` (`:1337`) copies `asr-model.ts`'s directory name. A model version bump there would quietly drop captions to "no ASR model".
> - **Silent recording gaps:** `WavRecorder.append` swallows each write error, so a full disk means log spam and silent gaps in the recording.
>
> The startup order with cleanup on failure, the 90 s lease that recovers on its own from a lost STOP, and keeping the last good speaker cache all hold up well.
>
> I worked from the code only; nothing was tested on hardware. #6 relies on the protocol comment saying the angle is 0 when direction detection is off.
>
> I'd fix #1, #2, #5 and #6 first, since those are the ones you'd hit in normal use. Want me to start on them?

<hr>

User: Thanks. Let's fix all of the high and medium issues you found now.

Opus 5.5:
> All nine high and medium issues are fixed. TS typecheck, lint, 813 TS tests, the Kotlin host tests (4 new), the iOS Kotlin compile and `build.sh` all pass, and I confirmed the new code is in the APK. Nothing has been tested on the glasses yet, and nothing is committed.
>
> **High**
> 1. **Mic taken by other apps:**
>    - The shared audio tap now tracks who's using it, so an EvenHub app backgrounding no longer cuts off Microphones.
>    - Transcribe now marks the mic as in use, like the assistant does, so Microphones and EvenHub apps step aside while it runs.
>    - A check every 3 s now covers both modes. It restarts a dead stock tap and switches back to stock mode if the developer setting is turned off mid-session.
> 2. **One bad utterance ending captions:** a decode error now skips that utterance instead of stopping the caption engine (`CaptionEngineCore.kt`). If the engine dies anyway, it's restarted on the same conversation up to 3 times. After that, the Captions toggle shows off and the view says why; turning it on retries.
> 3. **Recordings orphaned by a crash:** the recording is linked to its conversation as soon as it starts. At boot, the app closes conversations a crash left open, repairs their WAV headers (`FaceclawWavRecorder.repairHeader`), transcodes them, and deletes files no conversation points to. That cleanup matches by file name, so moving the app to other storage can't delete live recordings.
> 4. **Transcoder deleting both copies:** a failure while recording the new file no longer deletes both files. The app now deletes the WAV itself, only after the conversation points at the M4A. The transcoder, re-diarize and model-download callbacks are now held by the app until they fire.
>
> **Medium**
> 5. **Last utterance lost:** stopping captions no longer blocks the UI. The engine finishes the audio still queued, then sends a new `onStopped` callback, and the conversation stays open until that arrives (20 s fallback). A late line from a previous session won't show up in a newly reopened one.
> 6. **Beam filter muting captions:** the direction of arrival only filters audio while it's under 1 s old, and it resets when "Direction detection" changes.
> 7. **4-mic health check:** a temple that sends no audio for 5 s gets its config re-sent, to that temple only. Mic-control sends now report when they weren't actually queued. The status line says when no audio is arriving, but not after routine restarts like the voice dialog closing.
> 8. **Start failures being permanent:** the session now waits ("Waiting for the glasses microphone...") and the 3 s check retries. Captions start only once the mic is up, so waiting doesn't create empty conversations.
> 9. **Left/right flapping:** the caption feed only switches temples when the other one is clearly louder for 500 ms.
>
> Behaviour changes you'll notice:
> - The watch shows "Listening…" during Transcribe.
> - EvenHub mic apps get no audio while Transcribe runs.
> - In 4-mic mode, an EvenHub mic app still takes over the single native audio listener. Fixing that needs a change on the Kotlin side, so I left it.
>
> On hardware, I'd test:
> - The assistant, Transcribe and an EvenHub mic app taking the mic while Microphones is open, then giving it back.
> - Reconnecting in both stock and 4-mic modes.
> - Killing the app mid-recording, then checking the conversation is recovered at the next start.
>
> I saved these notes to memory as `microphones-reliability.md`.
</details>
2026-10-14 02:17:05 -07:00

216 lines
7.3 KiB
TypeScript

import { Utils } from "@nativescript/core";
import { toUint8Array } from "../../util/array-util";
import { micModelPath } from "./mic-models";
import { SPEAKER_KNOWN_THRESHOLD, encodeEmbedding, speakerRegistry } from "./speakers";
declare const com: any;
declare const android: any;
/**
* Re-diarize a saved conversation: run offline speaker diarization (pyannote
* segmentation + speaker embeddings + clustering) over the session's
* recording, then reconcile the resulting speaker turns against the stored
* transcript segments — matching each cluster back to an enrolled speaker by
* voice-print, or minting a new speaker when no one matches.
*/
export type RediarizeResult = {
turns: number;
clusters: number;
reassignedSegments: number;
newSpeakers: number;
};
export type RediarizeCallbacks = {
onProgress: (fraction: number) => void;
onDone: (result: RediarizeResult) => void;
onError: (message: string) => void;
};
type Turn = { startMs: number; endMs: number; cluster: number };
function store(): any {
const context = Utils.android.getApplicationContext();
return com.faceclaw.app.FaceclawConversationStore.getInstance(context);
}
// Each run's listener is referenced only from Java while the diarizer works;
// hold it here so it isn't GC'd before its callbacks.
const activeListeners = new Set<any>();
export function rediarizeSession(
sessionId: number,
options: { numSpeakers?: number; threshold?: number },
callbacks: RediarizeCallbacks,
): void {
const audioPath = String(store().getSessionAudioPath(sessionId));
if (!audioPath) {
callbacks.onError("This conversation has no saved recording to re-analyze.");
return;
}
const segmentationModel = micModelPath("diarization-segmentation");
const embeddingModel = micModelPath("speaker-embedding");
if (!segmentationModel || !embeddingModel) {
callbacks.onError("Download the re-diarization and speaker voice models first (Microphones > Voice & speakers).");
return;
}
const listener = new com.faceclaw.app.FaceclawDiarizer.Listener({
onProgress: (progress: number) => callbacks.onProgress(Number(progress)),
onDone: (turnsJson: string) => {
activeListeners.delete(listener);
try {
const turns = JSON.parse(String(turnsJson)) as Turn[];
callbacks.onDone(applyTurns(sessionId, turns));
} catch (error) {
callbacks.onError(`Applying diarization failed: ${String(error)}`);
}
},
onError: (message: string) => {
activeListeners.delete(listener);
callbacks.onError(String(message));
},
});
activeListeners.add(listener);
try {
const diarizer = new com.faceclaw.app.FaceclawDiarizer(
segmentationModel,
embeddingModel,
audioPath,
options.numSpeakers ?? -1,
options.threshold ?? 0.5,
listener,
);
diarizer.start();
} catch (error) {
activeListeners.delete(listener);
callbacks.onError(`Re-diarization failed to start: ${String(error)}`);
}
}
type StoredSegment = {
id: number;
speakerId: number | null;
startedAt: number;
endedAt: number;
audioOffsetMs: number | null;
};
function applyTurns(sessionId: number, turns: Turn[]): RediarizeResult {
const segments = (JSON.parse(String(store().querySegments(sessionId))) as Array<Record<string, unknown>>).map(
(row): StoredSegment => ({
id: Number(row.id),
speakerId: row.speakerId === null ? null : Number(row.speakerId),
startedAt: Number(row.startedAt),
endedAt: Number(row.endedAt),
audioOffsetMs: row.audioOffsetMs === null ? null : Number(row.audioOffsetMs),
}),
);
const embeddings = new Map<number, Float32Array>();
for (const row of JSON.parse(String(store().querySegmentEmbeddings(sessionId))) as Array<
Record<string, unknown>
>) {
const decoded = decodeBase64Floats(String(row.embeddingBase64 ?? ""));
if (decoded) embeddings.set(Number(row.id), decoded);
}
// Assign each transcript segment to the diarization cluster it overlaps most.
const segmentClusters = new Map<number, number>();
for (const segment of segments) {
if (segment.audioOffsetMs === null) continue;
const startMs = segment.audioOffsetMs;
const endMs = startMs + (segment.endedAt - segment.startedAt);
let bestCluster = -1;
let bestOverlap = 0;
for (const turn of turns) {
const overlap = Math.min(endMs, turn.endMs) - Math.max(startMs, turn.startMs);
if (overlap > bestOverlap) {
bestOverlap = overlap;
bestCluster = turn.cluster;
}
}
if (bestCluster >= 0) segmentClusters.set(segment.id, bestCluster);
}
// Centroid per cluster from the stored per-segment voice-prints.
const clusterEmbeddings = new Map<number, Float32Array[]>();
for (const [segmentId, cluster] of segmentClusters) {
const embedding = embeddings.get(segmentId);
if (!embedding) continue;
const list = clusterEmbeddings.get(cluster) ?? [];
list.push(embedding);
clusterEmbeddings.set(cluster, list);
}
speakerRegistry.reload();
const clusterSpeaker = new Map<number, number>();
let newSpeakers = 0;
for (const [cluster, prints] of clusterEmbeddings) {
const centroid = meanEmbedding(prints);
if (!centroid) continue;
const best = speakerRegistry.bestMatch(centroid);
if (best && best.similarity >= SPEAKER_KNOWN_THRESHOLD) {
clusterSpeaker.set(cluster, best.profile.id);
} else {
const id = Number(store().createSpeaker(nextAutoName(), "#90A4AE", false, encodeEmbedding(centroid)));
speakerRegistry.reload();
// A failed insert returns -1; leave that cluster's segments as they were.
if (id < 0) continue;
clusterSpeaker.set(cluster, id);
newSpeakers += 1;
}
}
let reassigned = 0;
for (const segment of segments) {
const cluster = segmentClusters.get(segment.id);
if (cluster === undefined) continue;
const speakerId = clusterSpeaker.get(cluster);
if (speakerId === undefined || speakerId === segment.speakerId) continue;
store().reassignSegmentSpeaker(segment.id, speakerId);
reassigned += 1;
}
speakerRegistry.reload();
return {
turns: turns.length,
clusters: clusterEmbeddings.size,
reassignedSegments: reassigned,
newSpeakers,
};
}
function nextAutoName(): string {
const taken = new Set(speakerRegistry.all().map((profile) => profile.name));
for (let index = 1; ; index++) {
const candidate = `Speaker ${index}`;
if (!taken.has(candidate)) return candidate;
}
}
function meanEmbedding(prints: Float32Array[]): Float32Array | null {
if (prints.length === 0) return null;
const mean = new Float32Array(prints[0]!.length);
for (const print of prints) {
if (print.length !== mean.length) continue;
for (let i = 0; i < mean.length; i++) mean[i] += print[i]!;
}
let norm = 0;
for (let i = 0; i < mean.length; i++) norm += mean[i]! * mean[i]!;
norm = Math.sqrt(norm);
if (norm > 0) {
for (let i = 0; i < mean.length; i++) mean[i] /= norm;
}
return mean;
}
function decodeBase64Floats(base64: string): Float32Array | null {
if (!base64) return null;
try {
const bytes = toUint8Array(android.util.Base64.decode(base64, android.util.Base64.NO_WRAP));
if (bytes.length < 4 || bytes.length % 4 !== 0) return null;
return new Float32Array(bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.length));
} catch {
return null;
}
}