Abstract
The use of multimodal LLMs (MLLMs) for egocentric video understanding with wearable devices is constrained by the token budget. Memory and compute cost scale with the number of visual tokens, and high-resolution video quickly becomes expensive to transmit and process at scale. Prior work (GazeLLM) addresses this by cropping the video around the camera wearer's gaze. This reduces the number of visual tokens by about tenfold while maintaining or improving the quality of full-resolution descriptions. However, this compression strategy depends on dedicated eye-tracking hardware, which is unavailable on consumer smart glasses. Building a software-only substitute poses a joint constraint: the predictor must be accurate enough to preserve downstream description quality, yet light enough to run on-device, within the power and compute budget of a smartphone. We address this with EgoGazeLite, a lightweight dual-process gaze predictor for egocentric video. Across two MLLMs, three automated metrics, and two LLM judges, predicted-gaze crops show no significant difference from ground-truth-gaze crops. Equivalence is confirmed in all ten cases. EgoGazeLite achieves this at 15.7M parameters, 6.71 GFLOPs, and runs the full gaze-and-crop pipeline end-to-end in real time (21.6,ms/frame) on consumer accelerator hardware. Together, these results remove the need for eye-tracking hardware for token-efficient, gaze-conditioned egocentric video understanding with MLLMs.
| Original language | English |
|---|---|
| Publication date | 2026 |
| Number of pages | 16 |
| DOIs | |
| Publication status | Published - 2026 |
| Event | WearableAI Workshop: Towards Real-time Multimodal Contextual Assistants - Malmö, Sweden Duration: 8 Sept 2026 → 9 Sept 2026 https://wearable-ai-workshop.github.io/ |
Workshop
| Workshop | WearableAI Workshop |
|---|---|
| Country/Territory | Sweden |
| City | Malmö |
| Period | 08/09/2026 → 09/09/2026 |
| Other | Advancing egocentric vision, proactive AI, and long-context multimodal understanding for next-generation wearable devices. |
| Internet address |
Keywords
- Gaze prediction
- Wearable AI
- Multimodal-LLM
Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver