Engineering notes
Run a 27 billion parameter LLM on your phone. PrismML's 1-bit Bonsai reasoning models are live in the Wally apps on iOS, Android, and macOS, including the first true 1-bit model ever to run on any NPU.
A solo developer ported his $2K MRR local-AI app to iOS in six weeks with the Wally SDK.
QHexRT is officially live — the first inference engine built to run LLM, VLM, STT, TTS, and embeddings 100% on Qualcomm Hexagon NPUs. First model: LFM 2.5 230M at 12,540 tok/s prefill and 36ms flat TTFT on v81.
MetalRT adds native speech-to-speech support. 1.68s end-to-end latency, 123 tok/s generation throughput, 1.52x faster than mlx-audio on a single M4 Max.
MetalRT adds VLM support and wins every decode benchmark. 279 tok/s vision decode, 92ms time-to-output, 1.22x faster than mlx-vlm across all resolutions on a single M4 Max.
A deep-dive into how PickleRite — a pickleball performance tracker — runs a specialized LLM entirely on-device using Wally SDK. Zero cloud costs, full offline support, complete privacy.
MetalRT becomes the first inference engine to handle LLMs, Speech-to-Text, and Text-to-Speech on Apple Silicon. 101ms to transcribe 70 seconds of audio. 178ms to synthesize speech. 4.6x faster than Apple MLX.
MetalRT delivers 658 tok/s decode and 6.6ms time-to-first-token, winning decode on 3 of 4 models we tested on a single M4 Max.
We added hybrid retrieval (BM25 + vector search) to our on-device voice pipeline. Retrieval adds less than 4ms. The real cost is LLM prefill — but word-level flushing absorbs it. Sub-200ms first-audio on 5,016 chunks with zero cloud dependencies.
FastVoice achieves 63ms first-audio latency — well under the 200ms perceptual threshold — by composing STT, LLM, and TTS into a single C++ pipeline on Apple Silicon. No cloud. No network. Just speed.
No server. No API key. No internet. Just a phone doing things on its own.
And the rabbit hole that taught me more about Android internals than 3 years of app development.
Automate web tasks with natural language using a Chrome extension powered by on-device AI. No API keys, no data leaving your browser, complete privacy.