Integrating llama.cpp into AAOS: on-device LLM inference as a platform service
The shape of the integration, ahead of the full write-up: llama.cpp built by Soong as a platform component, a binder daemon that owns the model, and weights that ship on their own release cadence.
llama.cpp is vendored into the AOSP tree and compiled by Soong as ordinary native libraries (no upstream patches, no NDK prebuilt), so the inference engine is just another platform component built for the target ABI. It's wrapped in a binder/AIDL daemon (system_ext) that owns model load, KV cache and multi-turn session state; the GGUF weights ship as a versioned APEX so the model updates independently of the system image. Client apps — e.g. the Compose chat demo — bind to the AIDL and stream tokens back, keeping the model resident in one process rather than per-app, and confined by its own SELinux domain.
gemma4 E2B Q4_0 on llama.cpp b10499, resident in the daemon — the chat app only holds a binder handle, and the answer arrives a token at a time.The full walkthrough — the Soong blueprints, the AIDL surface, the APEX layout and the SELinux policy — is still being written. Come back soon to see the complete article.