Why on-device inference matters
Cloud inference requires a network round-trip, exposes user data to a server, and fails in offline scenarios. For consumer apps (messaging, health, productivity) on-device inference is often a requirement, not a nice-to-have.
Gemini Nano and LiteRT
Google's Gemini Nano is a 1.8B parameter model quantized to run on mobile NPUs (Neural Processing Units). The Flutter integration uses the google_ai_dart_sdk package with GeminiNanoModel, falling back to cloud inference when the device model is unavailable.
LiteRT (formerly TensorFlow Lite) handles vision and custom small models. For classification and embedding tasks, a 50MB quantized model runs in under 20ms on a mid-range Android device.
Streaming UX without a network
The key insight: users tolerate slightly slower responses if they can see text appearing token by token. Even on-device inference can stream. Gemini Nano's Dart SDK exposes a generateContentStream method. Pipe tokens directly to a Flutter StreamBuilder for a responsive feel regardless of total generation time.
Battery and thermal management
On-device inference heats the chip. Implement thermal throttling: check DeviceInfo.thermalState (iOS) or subscribe to the battery API on Android. Reduce maxTokens from 512 to 128 during sustained load. Schedule background inference tasks during charging. Users notice neither the throttling nor the scheduling. They notice when their phone gets too hot.