Skip to content

feat(search): Kotlin Multiplatform search for iOS and Android 10+ - #134

Draft
kdroidFilter wants to merge 8 commits into
masterfrom
feat/kmp-search
Draft

feat(search): Kotlin Multiplatform search for iOS and Android 10+#134
kdroidFilter wants to merge 8 commits into
masterfrom
feat/kmp-search

Conversation

@kdroidFilter

@kdroidFilter kdroidFilter commented Oct 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Summary

The search becomes Kotlin Multiplatform (JVM, Android 10+, iOS), so the mobile apps only have their UI to write. Inspired by #83 and kdroidFilter/Zayit#606.

  • Common code: SearchEngine/SearchSession, query building, MagicDictionaryIndex (sqlite3 on iOS), ranking, snippets, highlights, hallucination blacklist.
  • Lucene runs on a real JVM only. ART can't keep up with the Java versions Lucene requires (21 for 10, 25 for 11: Bump Lucene 11.0.0 minimum required Java version to 25 apache/lucene#14229) and has no FFM. The bridge is Nucleus JVM Access.
    • search-facade (Kotlin/JVM) is the façade: LuceneIndex (analyze, search, cursors, facets, collectLongs, count, KNN) and the value types, with IndexQuery as a sealed value. LuceneIndexBackend, which is internal, runs Lucene.
    • search applies NJA. Its common code uses the proxies and value types generated from the façade, in …search.index, the package it already used. The query helpers (bool {}) stay in common code.
    • Where the JVM runs: on the desktop it's the app's own JVM; Android runs HotSpot in the app's process; iOS links Zero.
  • Runtimes: jdk-mobile, OpenJDK 27 GA with openjdk/mobile's port, built by NucleusFramework/openjdk-mobile's CI. search downloads release jdk-mobile-27b35.3, checks its SHA-256 and stages it for NJA.
  • What an app gets:
    • Android needs nothing: libnja.so (arm64-v8a, x86_64) and the runtime ship in the library's native libraries and assets.
    • iOS links the static libraries and carries NJA's bundle.
  • Removed: our hand-written bridge (CBOR protocol, BridgedLuceneIndex, IndexBridgeServer, EmbeddedJvm, the Java process and its transport) and lucene-android.
  • Docs: JVM_RUNTIME.md describes the whole setup.

Depends on

NucleusFramework/NucleusJvmAccess#1 (JVM targets, primitive arrays, sealed value hierarchies, default arguments). Until an NJA release ships it, build with -Pnja.local=<NJA checkout>. The catalog already points to NJA 0.2.0.

Verification

  • Parity on the real index (7M documents, SearchDump):
    • The desktop, now through NJA, gives the same dump as before.
    • iOS gives the same facets, snippets, highlights, titles and find-in-book candidates as the desktop; only Lucene's scores differ (they vary from one process to the next).
  • allTests passes.
  • The iOS device test binary links. It hasn't run on hardware.
  • Android 11 emulator (Zayit's searchkit-check, hybrid search with the semantic model): the same dump as the desktop, except one query's hit total, which Lucene counts as a lower bound. It takes 1.9 s once the runtime is extracted.
  • Not verified: the desktop's GraalVM native image. NJA finds its dispatcher through ServiceLoader, which GraalVM registers by default.

@kdroidFilter
feat(search): Kotlin Multiplatform search for iOS and Android 15+
55ee971 
The search (query building, dictionary, ranking, snippets, highlights) moves to common code; Lucene is
reached only through the LuceneIndex port (IndexQuery: plain data, cursors, facets, KNN).

- core, dao: iOS targets; the repository and the catalog loader on kotlinx-io, cachemap and a portable
  PRAGMA helper (the native and Android drivers can't run them through execute).
- search: JVM and Android run Lucene in process (LuceneIndexBackend, androidJvmMain); iOS runs it in an
  embedded Java runtime (OpenJDK's mobile port, Zero) through a JSON bridge over JNI (EmbeddedJvm,
  BridgedLuceneIndex, IndexBridgeServer). The runtime is built by kdroidFilter/openjdk-mobile's CI and
  downloaded by the build, its SHA-256 checked (see IOS_RUNTIME.md).
- lucene-android: lucene-core with two classes patched for ART (Android 15+), checked against the
  upstream classes' SHA-256.
- Parity tests on a real index (SearchDump): the same facets, snippets, highlights, titles and
  find-in-book candidates on the JVM, in the iOS simulator and on Android 15, but for Lucene's score noise.

Signed-off-by: Elie Gambache <elyahou.hadass@gmail.com>
@kdroidFilter
fix(search): release the iOS runtime's threads and close what the eng…
86f0355 
…ine owns

- iOS: threads attached to the embedded Java runtime are detached when they exit (pthread key destructor in C),
  or after each call when no destructor can be registered; static methods cached by class and method.
- SQLite readers: iOS throws on step errors and refuses queries after close (no use of the freed handle); the
  JVM one keeps its statements prepared per thread and opens no connection after close.
- The engine closes its dictionary, including one that finishes loading after close.
- Bridge: closing an index closes its open cursors; a cursor's reads and close take turns.
- Each quoted phrase is analyzed once; the bridged index caches its analyses.

Signed-off-by: Elie Gambache <elyahou.hadass@gmail.com>
@kdroidFilter
refactor(search): carry the iOS bridge's calls in CBOR rather than JSON
063606b 
The same @serializable requests and responses, the result encoded by its own serializer; kotlinx-serialization-cbor
replaces -json in the search and the runtime's jars.

Signed-off-by: Elie Gambache <elyahou.hadass@gmail.com>
@kdroidFilter
fix(search): send the bridge's bytes as CBOR byte strings and refuse …
8545d40 
…NaN vectors

- BridgeCbor writes byte arrays (results, byte vectors) as one byte string, not one integer per byte.
- KnnRequest refuses a vector with a NaN or infinite component, the check the JSON encoding used to make.
- BridgedLuceneIndexTest drives the bridged index through IndexBridgeServer end to end: pages with Hebrew text and
  scores, cursors, analysis, counts, ids and a null result.

Signed-off-by: Elie Gambache <elyahou.hadass@gmail.com>
@kdroidFilter
build(search): run the iOS search on OpenJDK 27 GA (jdk-mobile-27b35.1)
9c9a921 
The runtime now comes from kdroidFilter/openjdk-mobile's versioned releases: a stable OpenJDK 27 with
openjdk/mobile's port, its JNI headers in the archive.

Signed-off-by: Elie Gambache <elyahou.hadass@gmail.com>
@kdroidFilter kdroidFilter mentioned this pull request Oct 16, 2026
Draft
@kdroidFilter
feat(search): Lucene on a real JVM on Android too, from Android 10
e4c1cad 
ART can't keep up with the Java versions Lucene requires (21 for Lucene 10, 25 for Lucene 11, apache/lucene#14229),
and already needed two of its classes patched. Android now runs Lucene as iOS does, behind the same bridge, in a Java
process of the app's own (the Android runtime of the same jdk-mobile release: OpenJDK 27, server VM).

- JavaProcessTransport (androidJvmMain) and IndexBridgeServer.main: the calls on the process's standard input and
  output; the calls sent to a process that ends fail without failing those of the next one.
- LuceneIndexBackend moves to jvmMain; lucene-android and the ART device test go.
- minSdk 29; the Android runtime is downloaded with the iOS ones; iosJvmClasspath becomes mobileJvmClasspath.
- JavaProcessTransportTest runs the transport on the desktop, a java standing for the launcher.

Signed-off-by: Elie Gambache <elyahou.hadass@gmail.com>
kdroidFilter added a commit to kdroidFilter/Zayit that referenced this pull request Oct 16, 2026
@kdroidFilter
feat(search): Lucene in a Java process on Android, from Android 10
9a96582 
As SeforimLibrary now runs Lucene on a real JVM only (ART can't keep up with Lucene's Java versions), Android
reaches it through the bridge, in a Java process of the app's own.

- searchkit: AndroidJavaRuntime (configure(context), a 128 MB heap: ~140 MB for the process), openLuceneIndex
  always through it, assembleAndroidJavaRuntime (the launcher, the JDK, the search's jars), minSdk 29.
- searchkit-check back: the app that runs the search on a device (a library's test APK can't hold the launcher).
- SeforimLibrary at e4c1cad (kdroidFilter/SeforimLibrary#134).

Verified on Android 15 and Android 11 emulators: SearchKitDump as on the desktop, in 2.3 to 4.5 s.

Signed-off-by: Elie Gambache <elyahou.hadass@gmail.com>
@kdroidFilter kdroidFilter changed the title feat(search): Kotlin Multiplatform search for iOS and Android 15+ feat(search): Kotlin Multiplatform search for iOS and Android 10+ Oct 16, 2026
@kdroidFilter
build(search): jdk-mobile-27b35.2, with the Android runtime for x86_6…
043e435 
…4 too

The Android runtimes are now downloaded per architecture (android-aarch64, android-x64: their java.home differs,
UnixConstants among others), for devices and x86_64 emulators; the release is built on Linux for Android and
smoke-tested on an x86_64 emulator.

Signed-off-by: Elie Gambache <elyahou.hadass@gmail.com>
@kdroidFilter
refactor(search): Lucene through Nucleus JVM Access, on every platform
81d9725 
The hand-written bridge (the CBOR protocol, BridgedLuceneIndex, IndexBridgeServer, EmbeddedJvm, the Java
process and its transport) gives way to Nucleus JVM Access (NucleusFramework/NucleusJvmAccess):

- search-facade, a Kotlin/JVM module, is the façade: LuceneIndex (analyze, search, cursors, facets,
  collectLongs, count, KNN) and the value types, IndexQuery a sealed value; LuceneIndexBackend, internal,
  runs Lucene.
- search applies NJA: its common code uses the proxies and value types NJA generates from the façade, in
  io.github.kdroidfilter.seforimlibrary.search.index (the package it used); the query helpers stay common.
  The desktop calls the façade in its own JVM, Android runs HotSpot in the app's process (libnja.so and the
  runtime in the library's native libraries and assets), iOS links Zero.
- The runtimes are jdk-mobile-27b35.3's (NucleusFramework/openjdk-mobile: the Android runtimes' static
  libraries), downloaded, checked and staged for NJA (build/jdkMobile).
- -Pnja.local=<checkout> builds NJA from its sources until a release has what the search needs.
- IOS_RUNTIME.md becomes JVM_RUNTIME.md.

The same dumps as before on the real index: the desktop (through NJA) identical, iOS and Android 11 the same
but for Lucene's score noise and hit totals.

Signed-off-by: Elie Gambache <elyahou.hadass@gmail.com>
kdroidFilter added a commit to kdroidFilter/Zayit that referenced this pull request Oct 16, 2026
@kdroidFilter
refactor(search): the search's JVM through Nucleus JVM Access
ca7dcf2 
SeforimLibrary's search now runs Lucene through NJA (kdroidFilter/SeforimLibrary#134), with the JVM in the
app's process on every platform:

- searchkit: openLuceneIndex is the façade's constructor, common to all platforms; AndroidJavaRuntime,
  IosJavaRuntime and assembleAndroidJavaRuntime go (Android gets the JVM with search's library, nothing to
  configure); iOS binaries link the staged runtime and carry NJA's bundle, assembleIosJavaRuntime* assembles
  both for an app.
- searchkit-check: no runtime to deliver anymore.
- SeforimLibrary at 81d9725.

The same SearchKitDump as before: the desktop byte for byte, iOS identical to it, Android 11 but for one hit
total (Lucene's lower bound), in 1.9 s once the runtime is extracted.

Signed-off-by: Elie Gambache <elyahou.hadass@gmail.com>
@kdroidFilter kdroidFilter mentioned this pull request Oct 16, 2026
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@kdroidFilter