Roadmap: adopt NLnet open gesture recognizer as glide backend; build two-thumb on top

[AsafMah]

Summary (roadmap / research — not actionable yet)

HeliBoard's glide typing currently requires the abandoned closed-source Google library (downloaded separately; why our offlinelite flavor can't glide — no INTERNET to fetch it). There is a funded effort to replace it with an open recognizer, and our two-thumb engine should be designed to sit on top of it rather than on the Google blob.

What the project is

• NLnet "GestureTyping" project (NGI Mobifree / EU Horizon Europe, grant 101135795), https://nlnet.nl/project/GestureTyping/ — started June 2025.
• Goal: a completely open-source gesture-typing library, developed separately from HeliBoard with a compatibility layer to drop-in replace the closed Google code; reusable by other keyboards (Android + Linux).

Status (from HeliBoard's own gesture_data_* strings + code)

• Current phase = data gathering. HeliBoard ships the data-collection client only (GestureDataGathering.kt, GestureDataDao.kt, settings/screens/gesturedata/).
• Hard deadline in the strings: data gathering ends "end of 2026 latest."
• The accumulated dataset will be published under CC BY-SA 4.0 after gathering; library development comes after the dataset.
• The recognizer library itself does not exist publicly yet — not in any visible Helium314 repo.
• Modes: active gathering live (shown a word → swipe it); passive (real swipes) planned.
• Per-sample data: target word, layout + dimensions, gesture times + coordinates, current-lib suggestions, dict hash, app version, mode.

The two-thumb implication

• Scope is single-finger only, by design — validated against the single-finger Google library; one gesture per word; zero notion of multi-pointer / two-thumb. Two-thumb is this fork's novel contribution; nobody upstream is building it.
• This is an opportunity, not a blocker:
  1. An open single-finger recognizer removes the proprietary-blob dependency → fixes offlinelite glide.
  2. Our two-thumb engine is a layer on top of single-finger glide (BatchInputArbiter merges multiple pointer strokes). When the open lib lands, build two-thumb on top of it → only keyboard with two-thumb on a fully-open recognizer.
  3. Our TraceRecorder (A3a: Pointer-trace recorder & capture format #20) schema (InputPointers + committed word + geometry) is near-identical to their gathering schema → we could contribute single-finger samples and/or align formats.
  4. Dataset is CC BY-SA 4.0 (~end 2026) → usable to train/tune our own recognizer, including the single-finger base the two-thumb layer sits on.

Action items (track, revisit when the library/dataset lands)

• Watch the NLnet project + HeliBoard for the recognizer repo / dataset publication (target ~end 2026).
• Keep TraceRecorder (A3a: Pointer-trace recorder & capture format #20) output schema-compatible with their gathering format (low-cost alignment now).
• When the open recognizer ships: prototype it as the single-finger glide backend behind a flavor/flag; re-platform the two-thumb layer on top of it.
• Decide whether to contribute single-finger gesture samples upstream (via HeliBoard, not this fork — it phones home and our offline flavors have no INTERNET).

Explicitly out of scope

• Do not merge HeliBoard's GestureData* data-gathering subsystem into this fork: it's temporary scaffolding (to be removed when the project finishes), it phones data home, and it conflicts with our no-INTERNET offline/offlinelite flavors.

[cinnabar777]

I was working on a kotlin gesture engine, started with this java engine. Stopped because LeanType went a little off on the java. The kotlin is a modified DTW. it seemed to function better than the L2 java but LeanType never got the indexing working correctly. I threw it into a repo. I'm not a developer but I understand glide typing and the necessity for a good database tied to the gesture engine for it to really be good. I've figured out how to work with multiple AI better now to do this type of project but it isn't something I can do alone.

FUTO swipe has dual gesture tied into their neural engine. Possibly an option but way over my head other than the database and I can pinpoint the things they've gotten wrong that degrade gesture typing.

The HeliBoard developer working on the gesture engine doesn't seemed to have started, still reading 🤷🏻‍♂️, and plans on L2 with no user intents like circle for double letters. So waiting might be a while.

If your are interested and developing the DTW is something you think might fit I would be interested in helping.