You can find the full source code here: https://github.com/shaikmohammed8/LocalMind

The Architecture: Bridging Two Worlds

The core challenge of this project was that the inference engine for Whisper is written in C/C++ (ggml), but modern Android development happens in Kotlin. To make them talk, I had to utilize the Java Native Interface (JNI).

This is where things get dangerous. In Kotlin, memory is managed for you. In C++, if you forget to release a pointer, you leak memory. If you access a pointer from the wrong thread, the app crashes.

Here is a look at the JNI layer I wrote to handle the model initialization. I had to manually map Java input streams to C++ buffers to load the model from Android assets without exploding the RAM.

// inside native-lib.cpp

extern "C" JNIEXPORT jlong JNICALL
Java_com_example_localmind_WhisperLib_00024Companion_initContextFromAsset(
        JNIEnv *env, jobject /* thiz */, jobject assetManager, jstring asset_path_str) {

    // 1. Convert Java String to C string
    const char *asset_path = env->GetStringUTFChars(asset_path_str, nullptr);
    
    // 2. Initialize the Whisper context using the asset manager
    whisper_context *context = whisper_init_from_asset(env, assetManager, asset_path);
    
    // 3. CRITICAL: Release the memory for the string immediately
    env->ReleaseStringUTFChars(asset_path_str, asset_path);
    
    // 4. Return the pointer address as a Long to Kotlin
    return reinterpret_cast<jlong>(context);
}

By returning reinterpret_cast<jlong>(context), I’m essentially passing the raw memory address of the C++ object back to Kotlin. Kotlin holds this address (as a Long) and passes it back whenever it needs to transcribe something.

The Hardware Problem: Not All Cores Are Equal

One of the most interesting problems I ran into was performance tuning. On Linux desktops, you just throw threads at the problem. On mobile, big.LITTLE architecture complicates everything.

if I spun up 8 threads on an 8-core phone, the OS might schedule the heavy AI math on the “efficiency” cores to save battery, causing the transcription to be 5x slower.

I had to write a hardware abstraction layer that digs into /proc/cpuinfo and /sys/devices/system/cpu to identify the high-performance cores and explicitly tell Whisper how many threads to use.

// LocalMindCpuConfig.kt

object LocalMindCpuConfig {
    val preferredThreadCount: Int
        // Always ensure we have at least 2 threads, 
        // but prioritize high-performance cores
        get() = CpuInfo.getHighPerfCpuCount().coerceAtLeast(2)
}

// ... internal logic reading clock speeds ...
private fun getMaxCpuFrequency(cpuIndex: Int): Int {
    val path = "/sys/devices/system/cpu/cpu${cpuIndex}/cpufreq/cpuinfo_max_freq"
    val maxFreq = BufferedReader(FileReader(path)).use { it.readLine() }
    return maxFreq.toInt()
}

Managing Concurrency in the ViewModel

On the Android side, I used Jetpack Compose and Coroutines. The tricky part was managing the state. Recording audio is an I/O operation, but transcribing it is a heavy CPU operation.

I separated these concerns using specific Dispatchers. If I ran Whisper on Dispatchers.IO, it would crowd out disk operations. If I ran it on Main, the UI would freeze.

Here is how I structured the processAudioData function in the ViewModel to handle the pipeline:

1. IO Thread: Save the raw WAV file to disk.
2. Database: Insert a “Loading” placeholder message.
3. Default Thread: Run the C++ Inference (Whisper).
4. Database: Update the message with the transcribed text.

// LocalMindViewModel.kt

private suspend fun processAudioData(fullAudioData: FloatArray) {
    try {
        // Step 1: Disk IO (Save WAV)
        val wavFile = withContext(Dispatchers.IO) {
            val file = File(getApplication<Application>().filesDir, filename)
            WavHelper.saveWavFile(file, fullAudioData)
            file
        }

        // Step 2: Insert into Room Database (Get the ID!)
        val message = Message(
            isAudio = true,
            audioPath = wavFile.absolutePath, 
            // ...
        )
        val rowId = dao.insertMessage(message)

        // Step 3: The Heavy Lifting (C++ Inference)
        val text = withContext(Dispatchers.Default) {
            // This calls the JNI bridge
            whisperContext?.transcribeData(fullAudioData, printTimestamp = false) ?: ""
        }

        // Step 4: Update UI and Database
        if (text.isNotEmpty()) {
            val newMessage = message.copy(
                id = rowId.toInt(),
                content = text
            )
            dao.updateMessage(newMessage)
        }
    } catch (e: Exception) {
        Log.e("LocalMind", "Transcription failed", e)
    }
}

Visualizing the Data

Since I enjoy hacking around with UI as much as backend code, I didn’t want a standard progress bar. I wanted a raw audio visualization.

I built a custom Compose Canvas that draws the audio amplitude. The challenge here was blending. I needed the bars to be gray by default, but fill with white as the audio played—pixel perfect, without drawing "half" bars.

I solved this using clipRect logic in the draw scope:

// AudioWaveform.kt

Canvas(modifier = modifier) {
    // 1. Draw the "Inactive" Gray layer
    drawWaveformBars(color = MindDarkGray.copy(alpha = 0.25f))

    // 2. Draw the "Active" White layer, clipped to progress
    clipRect(
        left = 0f,
        top = 0f,
        right = size.width * progress, // The magic cut-off line
        bottom = size.height
    ) {
        drawWaveformBars(color = MindWhite)
    }
}

Screen Shots

Final Thoughts

Building LocalMind forced me to step out of the comfortable world of high-level APIs and deal with the messy reality of memory pointers, CPU topology, and thread management.

It reminded me why I love development: sometimes the best way to understand how something works is to stop importing libraries and start building the bridge yourself.

Check out the code, fork it, and let me know what you think. https://github.com/shaikmohammed8/LocalMind