Sign two or three glosses, pick a candidate sentence, and the phone speaks it into a call. Hand geometry for recognition, a small local LLM only for unseen sequences — no server, no API key, no account.
Recognising an n-sign sentence succeeds with probability ≈ pⁿ. The architecture attacks both terms: shorten the signed key (autocomplete does the rest) and keep p high with a small, well-separated vocabulary plus a rejection threshold that answers UNKNOWN instead of guessing.
Survival probability at per-sign accuracy p = 0.95 · p = 0.90. A model of the failure mode, not a measurement.
A classifier’s output layer is fixed to its training vocabulary. Adding one sign means data collection, retraining, re-export and shipping a new model. Template geometry lets a user record a new sign in about a second and use it on the next frame.
See the diagram above. Walk through the signing flow:
A guide circle fills over the hold duration; a cooldown forces the hand to visibly re-form before the same gloss can fire again. On the call screen a detection zone restricts matching to hands held at sign height.
Consecutive repeats of the same held sign are de-duplicated. A short key like ME NEED METFORMIN is enough to retrieve or generate a whole sentence.
An exact repeat returns from memory; a sequence whose candidates you chose before returns from sentence memory. Only unseen sequences wake the 3B model — which also keeps the phone cool under camera load.
Prompts ask for multiple candidates because a gloss sequence is genuinely ambiguous. Output is a candidate, never an action — also the guard against a hallucinated sentence.
Android has no public API to inject audio into a cellular call, so output is acoustic coupling or a LAN relay to a second phone that speaks and streams a transcript back.
The only vision model is MediaPipe Hand Landmarker. The sign matcher itself is plain TypeScript — not a model.
21 landmarks per hand. Wrist (red) is the origin.
Translate to the wrist, scale by hand size → translation- and scale-invariant.
Same transform the snapshot trainer used, so bundled and user-recorded templates compare directly.
Rank templates; above 0.42 → UNKNOWN.
| Check | Rejection |
|---|---|
| Landmarks valid | no-hand |
handSpread | too-far |
maxJitter | unstable |
Steadiness is judged across the whole hold window, not just the final frame.
Sentence memory (shipped) records which candidate you picked per exact gloss sequence; a different pick later adds an option. The standalone translation-memory engine handles corpora bigger than a JSON file.
| Route | How | Trade-off |
|---|---|---|
| Acoustic coupling | Speakerphone; the call’s open mic picks up TTS | Works on any carrier call; lower quality, echo cancellation can suppress it |
| LAN relay | zeroconf aslrelay + TCP :12345; sender transmits text, receiver speaks and streams transcript back | Clean audio, no cloud telephony; both phones on one network (hotspot as fallback) |
Committed in the repo. The only vision model.
~2 GB GGUF, sideloaded (debug) or bundled (release). 1.5B / 3B / 7B suggested per RAM tier via the llm-testbed harness.
~75 MB, committed. Inbound transcription via whisper.rn.