MediaPipe LLM API Is No Longer Updated: Migrate to LiteRT-LM and Add ABI Checks

LiteRT-LM Replaces the MediaPipe LLM Inference API

Google's MediaPipe documentation has posted a notice: the LLM Inference API (Android, iOS, and Web) has entered maintenance-only mode. New features and optimizations will be focused on LiteRT-LM, and projects should migrate. This means new code should no longer be built on the MediaPipe LLM Inference API, but should start directly with LiteRT-LM.

LiteRT-LM is Google's language model runtime built on LiteRT. Its Kotlin API is fairly lean: Engine loads the .litertlm model, Conversation holds the conversation state, and sendMessageAsync streams replies as a Kotlin Flow. As of writing (October 2026), the current version on Google Maven is 0.18.0, not yet at 1.0. The dependency declaration is simple:

dependencies {
    implementation("com.google.ai.edge.litertlm:litertlm-android:0.18.0")
}

A version number below 1.0 means the API may still change, so it is advisable to pin the version when integrating and keep an eye on the changelog.

LiteRT is Google's unified on-device AI runtime launched in 2024, replacing previously fragmented components such as TensorFlow Lite and the TensorFlow Lite GPU Delegate. Its goal is to provide a cross-framework (including PyTorch, JAX, and TensorFlow) inference execution engine for mobile and embedded devices. LiteRT-LM is an upper-layer runtime built on LiteRT and designed specifically for large language model scenarios, responsible for model loading, KV cache management, sampling decoding, and streaming output orchestration. Compared with the MediaPipe LLM Inference API, LiteRT-LM is closer to the underlying inference stack and updates faster, which is why its API is not yet stable.

.litertlm is a custom container format for LiteRT-LM that packages weights, tokenizer, model metadata, and chat template into a single file, allowing the runtime to complete initialization without additional configuration files. This is similar in spirit to the .task package used by MediaPipe, but the internal structure has been redesigned for LiteRT's operators and quantization schemes.

Model Acquisition and Selection Strategy

LiteRT-LM reads files in the .litertlm format. Google points to the LiteRT Community organization on Hugging Face for ready-made models, including builds of the Gemma series.

For model selection, a rule from real Android app development is: prefer the smallest CPU-friendly quantization (int4 or q8) rather than f32, so that mobile data download volume stays reasonable and the model actually fits in RAM. Here are two typical entries:

Directory entry	File	Approximate size
Gemma 3 270M IT (q8)	gemma3-270m-it-q8.litertlm	304 MB
Gemma 3 1B IT (int4)	gemma3-1b-it-int4.litertlm	584 MB

The following assumes the model file is already in app-private storage and you have its path.

Engine and Conversation Lifecycle

The documentation explicitly states that engine.initialize() can take a long time (the example given is "up to 10 seconds"), so it must run on a background thread or coroutine. The Engine should be loaded once and retained, with a new Conversation created for each chat session. A wrapper example:

class LocalLlm(private val context: Context, private val modelPath: String) {
    private var engine: Engine? = null
    private val lock = Mutex()

    suspend fun engine(): Engine = lock.withLock {
        engine ?: withContext(Dispatchers.Default) {
            Engine(EngineConfig(
                modelPath = modelPath,
                backend = Backend.CPU(),
                cacheDir = context.cacheDir.path, // speeds up subsequent loads
            )).also { it.initialize() }
        }.also { engine = it }
    }

    fun release() { engine?.close(); engine = null }
}

Conversation accepts optional system instructions, history messages, and sampling settings. System instructions are used to set the assistant persona, initialMessages brings existing conversation into a new session, and SamplerConfig controls topK, topP, and temperature.

Streaming Replies with Flow

sendMessageAsync(text) without a callback returns Flow<Message>, which is the recommended approach for coroutine users according to the documentation. In practice, it is usually hidden behind a TextGenerator interface so the UI does not care whether the backend is a local model or a remote API.

A key detail: inside the flow { } builder, use flowOn, not withContext. The reason is at the runtime level: the flow { } builder cannot emit values from a different dispatcher than the collector, and wrapping the body in withContext will throw "Flow invariant is violated." This pitfall has already been written into code comments in real projects and is a rule you must remember.

Flow's context preservation mechanism requires that emissions inside the flow { } builder occur in the same coroutine context as the collector. withContext switches the block to another dispatcher, so emit occurs in a different context, and the runtime detects this and throws Flow invariant is violated. flowOn only changes the execution context of the upstream (i.e., the flow { } body) while guaranteeing context consistency for the downstream collector, which is why it is the officially recommended approach.

In LLM scenarios, inference is a computationally intensive blocking operation that must be switched to Dispatchers.Default or a dedicated thread pool—exactly the typical reason flowOn is needed. If wrapping a callback-style API with callbackFlow, use trySend and clean up resources in awaitClose.

ABI Checks Are Not Optional

This part comes from real crash reports. The litertlm-android AAR only ships native libraries for arm64-v8a and x86_64, with no 32-bit ARM build. Phones that do not support these ABIs will throw UnsatisfiedLinkError the moment the native library is loaded.

The fix has two steps. First, check Build.SUPPORTED_ABIS before loading:

override suspend fun availability(): Availability =
    if (Build.SUPPORTED_ABIS.any { it == "arm64-v8a" || it == "x86_64" })
        Availability.Ready
    else Availability.Unavailable("This phone's processor is not supported")

Second, catch Throwable rather than Exception when creating the Engine. UnsatisfiedLinkError is an Error, not an Exception, so the old catch block would miss it. This detail determines whether the app degrades gracefully or crashes outright.

ABI (Application Binary Interface) refers to the binary contract between the app and the underlying CPU architecture, determining whether a native .so library can load on a specific device. Common Android ABIs include 32-bit armeabi-v7a and x86, and 64-bit arm64-v8a and x86_64. Because large model inference requires a 64-bit address space to hold several GB of model weights and KV cache, LiteRT-LM simply dropped 32-bit builds.

UnsatisfiedLinkError is a subclass of java.lang.Error, which is a parallel branch to Exception rather than an inheritance relationship, so catch (e: Exception) cannot catch it. Similarly, NoClassDefFoundError and OutOfMemoryError fall into this category. Code involving native libraries or resource allocation should catch Throwable and then handle by type.

GPU First, CPU Fallback

The GPU backend is usually faster for language models, but it is not available on every phone and every model. The LiteRT-LM Android guide requires declaring OpenCL-related native libraries in the manifest and marking them as not required:

<application>
    <uses-native-library android:name="libvndksupport.so" android:required="false"/>
    <uses-native-library android:name="libOpenCL.so" android:required="false"/>
</application>

Then try GPU first, falling back to CPU on failure. A more robust approach is to build a list of backend attempts and, within each backend, try different history lengths: full compressed history, half, one round of conversation, no history. Only when all levels fail should an error be shown to the user. This multi-layer fallback encapsulates the uncertainty of on-device models internally, so the user sees available or unavailable rather than various low-level exceptions.

<uses-native-library> is a manifest tag introduced in Android 12 (API 31) to declare vendor native libraries that the app depends on. When android:required="false", the app can still be installed even if the device lacks the library—loading will only fail at runtime. This is exactly the semantics the GPU backend needs: the same APK must be installable on devices without an OpenCL driver.

libvndksupport.so is Android's vendor dynamic library loading bridge, through which the GPU delegate accesses the vendor-provided OpenCL implementation; libOpenCL.so is the OpenCL runtime itself. Both are marked as not required to avoid being filtered out by app stores on devices that do not support GPU inference.

Fitting Conversations into the Context Window

Language models on phones have small context windows, and conversations grow with every message sent. When history does not fit, generation fails. One strategy for trimming history by token budget before each request is:

1. Roughly estimate tokens by dividing character count by four plus a small overhead per message.
2. Allow history to use 55% of the model's token limit, leaving the rest for the new message and reply.
3. Keep the most recent rounds that fit, add back the earliest round if there is space, then include up to a few older conversations whose keywords overlap with the new message.

The chat log itself still lives in Room; only the copy sent to the model is trimmed. This preserves the full history visible to the user while avoiding context overflow.

Releasing Model Memory

A loaded model occupies a large amount of native memory. Close the Conversation when the chat ends (the use block above handles this), and close the Engine when the user leaves the AI part of the app or the system signals memory pressure:

class AssistantApp : Application() {
    lateinit var localLlm: LocalLlm

    override fun onTrimMemory(level: Int) {
        super.onTrimMemory(level)
        if (level >= ComponentCallbacks2.TRIM_MEMORY_BACKGROUND) {
            localLlm.release()
        }
    }
}

Responding to background memory pressure in onTrimMemory is essential for on-device model apps to avoid being killed by the system.

Migration Checklist

• New code uses LiteRT-LM, not the MediaPipe LLM Inference API.
• engine.initialize() runs off the main thread and only once.
• Use flowOn inside flow { }, never withContext.
• Check before loading that Build.SUPPORTED_ABIS contains arm64-v8a or x86_64.
• Catch Throwable at Engine creation.
• GPU first, CPU fallback, with manifest entries marked as not required.
• Release the Engine under memory pressure.

Migrating from the MediaPipe LLM Inference API to LiteRT-LM is not just swapping a dependency coordinate. ABI checks, Flow scheduling rules, context trimming, and memory release—these details together determine whether an on-device AI feature is stably usable or frequently crashes. LiteRT-LM is still at 0.18.0 and the API may continue to evolve, but the engineering practices above will remain relevant until 1.0.