Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading

Rucha Kulkarni; Kritika   Jain; Himanshu  Bansal; Srinivas  Bangalore; Michael Carl

doi:10.1145/2535948.2535953

Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading

Rucha Kulkarni , Kritika Jain, Himanshu Bansal, Srinivas Bangalore, Michael Carl

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

Abstract

Researchers are proposing interactive machine translation as a potential method to make language translation process more efficient and usable. Introduction of different modalities like eye gaze and speech are being explored to add to the interactivity of language translation system. Unfortunately, the raw data provided by Automatic Speech Recognition (ASR) and Eye-Tracking is very noisy and erroneous. This paper describes a technique for reducing the errors of the two modalities, speech and eye-gaze with the help of each other in context of sight translation and reading. Lattice representation and composition of the two modalities was used for integration. F-measure for Eye-Gaze and Word Accuracy for ASR were used as metrics to evaluate our results. In reading task, we demonstrated a significant improvement in both Eye-Gaze f-measure and speech Word Accuracy. In sight translation task, significant improvement was found in gaze f-measure but not in ASR.

Original language	English
Title of host publication	GazeIn '13. Proceedings of the 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction : Gaze in Multimodal Interaction
Editors	Roman Bednarik , Hung-Hsuan Huang , Kristiina Jokinen , Yukiko I. Nakano
Place of Publication	New York
Publisher	Association for Computing Machinery
Publication date	2013
Pages	35-40
ISBN (Print)	9781450325639
DOIs	https://doi.org/10.1145/2535948.2535953
Publication status	Published - 2013
Event	The 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction. GazeIn '13: Gaze in Multimodal Interaction - Sydney, Australia Duration: 13 Dec 2013 → 13 Dec 2013 Conference number: 6 http://cs.uef.fi/~rbednari/GazeIn2013/

Workshop

Workshop	The 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction. GazeIn '13
Number	6
Country/Territory	Australia
City	Sydney
Period	13/12/2013 → 13/12/2013
Other	Connected to at ACM ICMI 2013
Internet address	http://cs.uef.fi/~rbednari/GazeIn2013/

Access to Document

10.1145/2535948.2535953

Handle.net

Persistent link

Cite this

Kulkarni , R., Jain, K., Bansal, H., Bangalore, S., & Carl, M. (2013). Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading. In R. Bednarik , H-H. Huang , K. Jokinen , & Y. I. Nakano (Eds.), GazeIn '13. Proceedings of the 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction: Gaze in Multimodal Interaction (pp. 35-40). Association for Computing Machinery. https://doi.org/10.1145/2535948.2535953

Kulkarni , Rucha ; Jain, Kritika ; Bansal, Himanshu et al. / Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading. GazeIn '13. Proceedings of the 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction: Gaze in Multimodal Interaction . editor / Roman Bednarik ; Hung-Hsuan Huang ; Kristiina Jokinen ; Yukiko I. Nakano. New York : Association for Computing Machinery, 2013. pp. 35-40

@inproceedings{35aaee7f36c54d3ca758e75db7245332,

title = "Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading",

abstract = "Researchers are proposing interactive machine translation as a potential method to make language translation process more efficient and usable. Introduction of different modalities like eye gaze and speech are being explored to add to the interactivity of language translation system. Unfortunately, the raw data provided by Automatic Speech Recognition (ASR) and Eye-Tracking is very noisy and erroneous. This paper describes a technique for reducing the errors of the two modalities, speech and eye-gaze with the help of each other in context of sight translation and reading. Lattice representation and composition of the two modalities was used for integration. F-measure for Eye-Gaze and Word Accuracy for ASR were used as metrics to evaluate our results. In reading task, we demonstrated a significant improvement in both Eye-Gaze f-measure and speech Word Accuracy. In sight translation task, significant improvement was found in gaze f-measure but not in ASR.",

author = "Rucha Kulkarni and Kritika Jain and Himanshu Bansal and Srinivas Bangalore and Michael Carl",

year = "2013",

doi = "10.1145/2535948.2535953",

language = "English",

isbn = "9781450325639 ",

pages = "35--40",

editor = "{Bednarik }, {Roman } and {Huang }, {Hung-Hsuan } and {Jokinen }, {Kristiina } and Nakano, {Yukiko I. }",

booktitle = "GazeIn '13. Proceedings of the 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction",

publisher = "Association for Computing Machinery",

address = "United States",

note = "null ; Conference date: 13-12-2013 Through 13-12-2013",

url = "http://cs.uef.fi/~rbednari/GazeIn2013/",

}

Kulkarni , R, Jain, K, Bansal, H, Bangalore, S & Carl, M 2013, Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading. in R Bednarik , H-H Huang , K Jokinen & YI Nakano (eds), GazeIn '13. Proceedings of the 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction: Gaze in Multimodal Interaction . Association for Computing Machinery, New York, pp. 35-40, The 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction. GazeIn '13, Sydney, Australia, 13/12/2013. https://doi.org/10.1145/2535948.2535953

Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading. / Kulkarni , Rucha; Jain, Kritika ; Bansal, Himanshu et al.

GazeIn '13. Proceedings of the 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction: Gaze in Multimodal Interaction . ed. / Roman Bednarik ; Hung-Hsuan Huang ; Kristiina Jokinen ; Yukiko I. Nakano. New York : Association for Computing Machinery, 2013. p. 35-40.

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

TY - GEN

T1 - Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading

AU - Kulkarni , Rucha

AU - Jain, Kritika

AU - Bansal, Himanshu

AU - Bangalore, Srinivas

AU - Carl, Michael

N1 - Conference code: 6

PY - 2013

Y1 - 2013

N2 - Researchers are proposing interactive machine translation as a potential method to make language translation process more efficient and usable. Introduction of different modalities like eye gaze and speech are being explored to add to the interactivity of language translation system. Unfortunately, the raw data provided by Automatic Speech Recognition (ASR) and Eye-Tracking is very noisy and erroneous. This paper describes a technique for reducing the errors of the two modalities, speech and eye-gaze with the help of each other in context of sight translation and reading. Lattice representation and composition of the two modalities was used for integration. F-measure for Eye-Gaze and Word Accuracy for ASR were used as metrics to evaluate our results. In reading task, we demonstrated a significant improvement in both Eye-Gaze f-measure and speech Word Accuracy. In sight translation task, significant improvement was found in gaze f-measure but not in ASR.

AB - Researchers are proposing interactive machine translation as a potential method to make language translation process more efficient and usable. Introduction of different modalities like eye gaze and speech are being explored to add to the interactivity of language translation system. Unfortunately, the raw data provided by Automatic Speech Recognition (ASR) and Eye-Tracking is very noisy and erroneous. This paper describes a technique for reducing the errors of the two modalities, speech and eye-gaze with the help of each other in context of sight translation and reading. Lattice representation and composition of the two modalities was used for integration. F-measure for Eye-Gaze and Word Accuracy for ASR were used as metrics to evaluate our results. In reading task, we demonstrated a significant improvement in both Eye-Gaze f-measure and speech Word Accuracy. In sight translation task, significant improvement was found in gaze f-measure but not in ASR.

U2 - 10.1145/2535948.2535953

DO - 10.1145/2535948.2535953

M3 - Article in proceedings

SN - 9781450325639

SP - 35

EP - 40

BT - GazeIn '13. Proceedings of the 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction

A2 - Bednarik , Roman

A2 - Huang , Hung-Hsuan

A2 - Jokinen , Kristiina

A2 - Nakano, Yukiko I.

PB - Association for Computing Machinery

CY - New York

Y2 - 13 December 2013 through 13 December 2013

ER -

Kulkarni R, Jain K, Bansal H, Bangalore S, Carl M. Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading. In Bednarik R, Huang H-H, Jokinen K, Nakano YI, editors, GazeIn '13. Proceedings of the 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction: Gaze in Multimodal Interaction . New York: Association for Computing Machinery. 2013. p. 35-40 doi: 10.1145/2535948.2535953