Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
*.iml
.gradle
/.kotlin/
/local.properties
/.idea/caches
/.idea/libraries
Expand Down
54 changes: 42 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ into an existing plain-text (`.txt`) or Markdown (`.md`) file.
1. Download and verify the speech model on first launch.
2. Select an existing writable `.txt` or `.md` file.
3. Select a folder containing audio recordings.
4. Review discovered recordings in the **New** tab and choose **Transcribe all**.
4. Review discovered recordings in the **New** tab and choose **Transcribe all**, or
use the startup-processing policy when the app opens.
5. The application processes new recordings sequentially in foreground work and
appends one entry per successful file containing its filename, recording
time, and recognized text.
Expand All @@ -18,6 +19,20 @@ application preserves existing text and separates each new entry with one blank
line. Folder scanning runs when the application starts and when **Refresh** is
selected. It scans direct children only and does not traverse nested folders.

## Automatic Processing

Android Settings provides two independent automation options:

- **Startup processing** runs after the app opens, restores its selections, and
finishes scanning the audio folder. It defaults to asking before processing;
it can instead transcribe automatically or leave files queued.
- **Nightly transcription** uses Android background work around the selected
time. Android may delay this work to save battery, so exact timing is not
guaranteed.

Both options reuse the same batch transcription worker and exclude failed files,
which remain available through their individual **Retry** actions.

## Incremental Catalog

The application stores folder entries and processing state in a private SQLite
Expand Down Expand Up @@ -83,15 +98,30 @@ Requirements:
The Gradle build extracts ONNX Runtime, builds the Rust JNI library for
`arm64-v8a`, and packages the required native libraries.

## iOS Shell
## iOS Application

The repository includes a SwiftUI iOS application at
`iosApp/VoiceInbox.xcodeproj`. It uses the generated `Shared` Kotlin
Multiplatform framework for the shared catalog, transcription rules, and speech
model metadata, and links the Rust transcription engine through the native iOS
bridge.

The current iOS application supports:

- selecting and scanning an audio inbox folder
- importing individual audio files and receiving files through the share
extension
- selecting an existing transcript output file
- downloading or manually installing the pinned speech model
- previewing audio and transcribing or retrying individual files
- processing queued files when the application starts, with **Ask**, **Yes**, and
**No** startup policies
- viewing processed transcripts and persisting catalog state
- changing storage and startup-processing preferences from Settings

The repository includes a minimal SwiftUI iOS shell at
`iosApp/VoiceInbox.xcodeproj`. It is a KMP wiring milestone: the app launches to
a simple screen and imports the generated `Shared` Kotlin Multiplatform
framework to display speech-model metadata from shared core. The shell also
includes a Settings screen that persists scheduled transcription preferences in
iOS `UserDefaults` and normalizes them through shared scheduling rules; iOS
scheduled execution itself remains future work.
Both applications provide configurable startup processing. Unlike Android, iOS
does not currently schedule nightly background transcription. Its automatic
processing is evaluated when the application starts or becomes active.

Open the project from `notes_recognition`:

Expand Down Expand Up @@ -122,9 +152,9 @@ Xcode SDK/configuration. The lower-level shared compile checks remain:
./gradlew :shared:compileKotlinIosSimulatorArm64 :shared:compileKotlinIosX64 :shared:compileKotlinIosArm64
```

This shell intentionally does not implement document picking, folder scanning,
audio playback, catalog persistence, transcription execution, settings,
scheduling, output-file writing, or Rust/iOS native bridging.
The iOS and Android applications share workflow rules and catalog behavior, but
their platform integrations remain native: SwiftUI and Apple document pickers on
iOS, and Android views, the Storage Access Framework, and WorkManager on Android.

## GitHub Actions

Expand Down
3 changes: 3 additions & 0 deletions app/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,9 @@ dependencies {
implementation(libs.material)
implementation(libs.androidx.activity)
implementation(libs.androidx.constraintlayout)
implementation(libs.androidx.lifecycle.runtime.ktx)
implementation(libs.androidx.lifecycle.viewmodel.ktx)
implementation(libs.androidx.recyclerview)
implementation("com.microsoft.onnxruntime:onnxruntime-android:1.22.0")
implementation("androidx.work:work-runtime-ktx:2.10.1")
implementation("com.squareup.okhttp3:okhttp:4.12.0")
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
package me.maxistar.voiceinbox

import android.content.ClipData
import android.content.Intent
import android.net.Uri
import android.provider.DocumentsContract
import androidx.test.core.app.ActivityScenario
import androidx.test.ext.junit.runners.AndroidJUnit4
import androidx.test.platform.app.InstrumentationRegistry
import me.maxistar.voiceinbox.core.AndroidSqlDelightAudioCatalogFactory
import androidx.work.WorkInfo
import androidx.work.WorkManager
import org.junit.Assert.assertEquals
import org.junit.Assert.assertTrue
import org.junit.Before
import org.junit.Test
import org.junit.runner.RunWith
import java.io.File
import java.util.concurrent.TimeUnit

@RunWith(AndroidJUnit4::class)
class AudioShareIntentInstrumentedTest {
private val context get() = InstrumentationRegistry.getInstrumentation().targetContext

@Before
fun clearState() {
context.deleteDatabase(AndroidSqlDelightAudioCatalogFactory.DATABASE_NAME)
File(context.filesDir, AndroidAudioImportConstants.DIRECTORY_NAME).deleteRecursively()
WorkManager.getInstance(context)
.cancelUniqueWork(TranscriptionWorker.UNIQUE_WORK_NAME)
.result.get(30, TimeUnit.SECONDS)
}

@Test
fun parserNormalizesSingleMultipleClipAndMalformedIntents() {
val wav = documentUri("wav")
val m4a = documentUri("m4a")
val single = Intent(Intent.ACTION_SEND)
.putExtra(Intent.EXTRA_STREAM, wav)
assertEquals(listOf(wav), AudioShareIntentParser.streamUris(single))

val multiple = Intent(Intent.ACTION_SEND_MULTIPLE)
.putParcelableArrayListExtra(Intent.EXTRA_STREAM, arrayListOf(wav, m4a, wav))
.apply { clipData = ClipData.newRawUri("audio", m4a) }
assertEquals(listOf(m4a, wav), AudioShareIntentParser.streamUris(multiple))
assertTrue(AudioShareIntentParser.streamUris(Intent(Intent.ACTION_SEND)).isEmpty())
assertTrue(AudioShareIntentParser.streamUris(Intent(Intent.ACTION_VIEW)).isEmpty())
}

@Test
fun coldWarmAndRecreatedDeliveryCreatesOneDurableImport() {
ActivityScenario.launch(MainActivity::class.java).use { scenario ->
scenario.onActivity { activity ->
MainActivity::class.java.getDeclaredMethod("handleShareIntent", Intent::class.java)
.apply { isAccessible = true }
.invoke(activity, shareIntent(documentUri("wav")))
}
awaitImportFinished(scenario)
assertEquals(1, importedRows())
assertTrue(importedDocumentIds().single().startsWith("content://${context.packageName}.files/"))
assertTrue(
WorkManager.getInstance(context)
.getWorkInfosForUniqueWork(TranscriptionWorker.UNIQUE_WORK_NAME)
.get(30, TimeUnit.SECONDS)
.none { it.state in ACTIVE_WORK_STATES },
)

scenario.onActivity { activity ->
MainActivity::class.java.getDeclaredMethod("onNewIntent", Intent::class.java)
.apply { isAccessible = true }
.invoke(activity, shareIntent(documentUri("wav")))
}
awaitImportFinished(scenario)
assertEquals(1, importedRows())
}
}

@Test
fun pickerStyleMultipleInputKeepsAcceptedItemWhenAnotherIsRejected() {
ActivityScenario.launch(MainActivity::class.java).use { scenario ->
scenario.onActivity { activity ->
MainActivity::class.java.getDeclaredMethod("ingestAudioUris", List::class.java)
.apply { isAccessible = true }
.invoke(activity, listOf(documentUri("m4a"), documentUri("text")))
}
awaitImportFinished(scenario)
assertEquals(1, importedRows())
}
}

@Test
fun telegramStyleOggAndOpusMultipleShareImportsBothItems() {
val multiple = Intent(Intent.ACTION_SEND_MULTIPLE)
.setType("audio/*")
.putParcelableArrayListExtra(
Intent.EXTRA_STREAM,
arrayListOf(documentUri("ogg"), documentUri("opus")),
)
ActivityScenario.launch(MainActivity::class.java).use { scenario ->
scenario.onActivity { activity ->
MainActivity::class.java.getDeclaredMethod("handleShareIntent", Intent::class.java)
.apply { isAccessible = true }
.invoke(activity, multiple)
}
awaitImportFinished(scenario)
assertEquals(2, importedRows())
}
}

private fun awaitImportFinished(scenario: ActivityScenario<MainActivity>) {
repeat(100) {
var active = true
scenario.onActivity { activity ->
active = MainActivity::class.java.getDeclaredField("ingestionActive")
.apply { isAccessible = true }
.getBoolean(activity)
}
if (!active) return
Thread.sleep(25)
}
error("Timed out waiting for audio ingestion")
}

private fun importedRows(): Int {
val repository = AndroidSqlDelightAudioCatalogFactory(context).create()
return try {
repository.importedFiles(AndroidAudioImportConstants.SOURCE_ID).size
} finally {
repository.close()
}
}

private fun importedDocumentIds(): List<String> {
val repository = AndroidSqlDelightAudioCatalogFactory(context).create()
return try {
repository.importedFiles(AndroidAudioImportConstants.SOURCE_ID).map { it.documentUri }
} finally {
repository.close()
}
}

private fun shareIntent(uri: Uri): Intent =
Intent(context, MainActivity::class.java)
.setAction(Intent.ACTION_SEND)
.setType("audio/wav")
.putExtra(Intent.EXTRA_STREAM, uri)

private fun documentUri(id: String): Uri =
DocumentsContract.buildDocumentUri(TestAudioDocumentsProvider.AUTHORITY, id)

private companion object {
val ACTIVE_WORK_STATES = setOf(
WorkInfo.State.ENQUEUED,
WorkInfo.State.BLOCKED,
WorkInfo.State.RUNNING,
)
}
}
Loading
Loading