Skip to main navigation Skip to search Skip to main content

EgoGazeLite: On-Device Egocentric Gaze Prediction for Token-Efficient Multimodal LLM Video Input

  • Copenhagen Business School

Research output: Contribution to conference › Paper › Research › peer-review

5 Downloads (Pure)

Abstract

The use of multimodal LLMs (MLLMs) for egocentric video understanding with wearable devices is constrained by the token budget. Memory and compute cost scale with the number of visual tokens, and high-resolution video quickly becomes expensive to transmit and process at scale. Prior work (GazeLLM) addresses this by cropping the video around the camera wearer's gaze. This reduces the number of visual tokens by about tenfold while maintaining or improving the quality of full-resolution descriptions. However, this compression strategy depends on dedicated eye-tracking hardware, which is unavailable on consumer smart glasses. Building a software-only substitute poses a joint constraint: the predictor must be accurate enough to preserve downstream description quality, yet light enough to run on-device, within the power and compute budget of a smartphone. We address this with EgoGazeLite, a lightweight dual-process gaze predictor for egocentric video. Across two MLLMs, three automated metrics, and two LLM judges, predicted-gaze crops show no significant difference from ground-truth-gaze crops. Equivalence is confirmed in all ten cases. EgoGazeLite achieves this at 15.7M parameters, 6.71 GFLOPs, and runs the full gaze-and-crop pipeline end-to-end in real time (21.6,ms/frame) on consumer accelerator hardware. Together, these results remove the need for eye-tracking hardware for token-efficient, gaze-conditioned egocentric video understanding with MLLMs.
Original languageEnglish
Publication date2026
Number of pages16
DOIs
Publication statusPublished - 2026
EventWearableAI Workshop: Towards Real-time Multimodal Contextual Assistants - Malmö, Sweden
Duration: 8 Sept 2026 → 9 Sept 2026
https://wearable-ai-workshop.github.io/

Workshop

WorkshopWearableAI Workshop
Country/TerritorySweden
CityMalmö
Period08/09/2026 → 09/09/2026
OtherAdvancing egocentric vision, proactive AI, and long-context multimodal understanding for next-generation wearable devices.
Internet address

Keywords

  • Gaze prediction
  • Wearable AI
  • Multimodal-LLM

Cite this