Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading

Rucha Kulkarni; Kritika   Jain; Himanshu  Bansal; Srinivas  Bangalore; Michael Carl

doi:10.1145/2535948.2535953

Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading

Rucha Kulkarni
, Kritika Jain
, Himanshu Bansal
, Srinivas Bangalore
, Michael Carl

Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › peer review

Abstract

Researchers are proposing interactive machine translation as a potential method to make language translation process more efficient and usable. Introduction of different modalities like eye gaze and speech are being explored to add to the interactivity of language translation system. Unfortunately, the raw data provided by Automatic Speech Recognition (ASR) and Eye-Tracking is very noisy and erroneous. This paper describes a technique for reducing the errors of the two modalities, speech and eye-gaze with the help of each other in context of sight translation and reading. Lattice representation and composition of the two modalities was used for integration. F-measure for Eye-Gaze and Word Accuracy for ASR were used as metrics to evaluate our results. In reading task, we demonstrated a significant improvement in both Eye-Gaze f-measure and speech Word Accuracy. In sight translation task, significant improvement was found in gaze f-measure but not in ASR.

Originalsprog	Engelsk
Titel	GazeIn '13. Proceedings of the 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction : Gaze in Multimodal Interaction
Redaktører	Roman Bednarik , Hung-Hsuan Huang , Kristiina Jokinen , Yukiko I. Nakano
Udgivelsessted	New York
Forlag	Association for Computing Machinery
Publikationsdato	2013
Sider	35-40
ISBN (Trykt)	9781450325639
DOI	https://doi.org/10.1145/2535948.2535953
Status	Udgivet - 2013
Begivenhed	The 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction. GazeIn '13: Gaze in Multimodal Interaction - Sydney, Australien Varighed: 13 dec. 2013 → 13 dec. 2013 Konferencens nummer: 6 http://cs.uef.fi/~rbednari/GazeIn2013/

Workshop

Workshop	The 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction. GazeIn '13
Nummer	6
Land/Område	Australien
By	Sydney
Periode	13/12/2013 → 13/12/2013
Andet	Connected to at ACM ICMI 2013
Internetadresse	http://cs.uef.fi/~rbednari/GazeIn2013/

Adgang til dokumentet

10.1145/2535948.2535953

Handle.net

Persistent link

Citationsformater

Kulkarni , R., Jain, K., Bansal, H., Bangalore, S., & Carl, M. (2013). Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading. I R. Bednarik , H.-H. Huang , K. Jokinen , & Y. I. Nakano (red.), GazeIn '13. Proceedings of the 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction: Gaze in Multimodal Interaction (s. 35-40). Association for Computing Machinery. https://doi.org/10.1145/2535948.2535953

Kulkarni , Rucha ; Jain, Kritika ; Bansal, Himanshu et al. / Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading. GazeIn '13. Proceedings of the 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction: Gaze in Multimodal Interaction . red. / Roman Bednarik ; Hung-Hsuan Huang ; Kristiina Jokinen ; Yukiko I. Nakano. New York : Association for Computing Machinery, 2013. s. 35-40

@inproceedings{35aaee7f36c54d3ca758e75db7245332,

title = "Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading",

abstract = "Researchers are proposing interactive machine translation as a potential method to make language translation process more efficient and usable. Introduction of different modalities like eye gaze and speech are being explored to add to the interactivity of language translation system. Unfortunately, the raw data provided by Automatic Speech Recognition (ASR) and Eye-Tracking is very noisy and erroneous. This paper describes a technique for reducing the errors of the two modalities, speech and eye-gaze with the help of each other in context of sight translation and reading. Lattice representation and composition of the two modalities was used for integration. F-measure for Eye-Gaze and Word Accuracy for ASR were used as metrics to evaluate our results. In reading task, we demonstrated a significant improvement in both Eye-Gaze f-measure and speech Word Accuracy. In sight translation task, significant improvement was found in gaze f-measure but not in ASR. ",

author = "Rucha Kulkarni and Kritika Jain and Himanshu Bansal and Srinivas Bangalore and Michael Carl",

year = "2013",

doi = "10.1145/2535948.2535953",

language = "English",

isbn = "9781450325639 ",

pages = "35--40",

editor = "\{Bednarik \}, \{Roman \} and \{Huang \}, \{Hung-Hsuan \} and \{Jokinen \}, \{Kristiina \} and Nakano, \{Yukiko I. \}",

booktitle = "GazeIn '13. Proceedings of the 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction",

publisher = "Association for Computing Machinery",

address = "United States",

note = "The 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction. GazeIn '13 ; Conference date: 13-12-2013 Through 13-12-2013",

url = "http://cs.uef.fi/\textasciitilde{}rbednari/GazeIn2013/",

}

Kulkarni , R, Jain, K, Bansal, H, Bangalore, S & Carl, M 2013, Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading. i R Bednarik , H-H Huang , K Jokinen & YI Nakano (red), GazeIn '13. Proceedings of the 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction: Gaze in Multimodal Interaction . Association for Computing Machinery, New York, s. 35-40, The 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction. GazeIn '13, Sydney, Australien, 13/12/2013. https://doi.org/10.1145/2535948.2535953

Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading. / Kulkarni , Rucha; Jain, Kritika ; Bansal, Himanshu et al.
GazeIn '13. Proceedings of the 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction: Gaze in Multimodal Interaction . red. / Roman Bednarik ; Hung-Hsuan Huang ; Kristiina Jokinen ; Yukiko I. Nakano. New York: Association for Computing Machinery, 2013. s. 35-40.

Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › peer review

TY - GEN

T1 - Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading

AU - Kulkarni , Rucha

AU - Jain, Kritika

AU - Bansal, Himanshu

AU - Bangalore, Srinivas

AU - Carl, Michael

N1 - Conference code: 6

PY - 2013

Y1 - 2013

N2 - Researchers are proposing interactive machine translation as a potential method to make language translation process more efficient and usable. Introduction of different modalities like eye gaze and speech are being explored to add to the interactivity of language translation system. Unfortunately, the raw data provided by Automatic Speech Recognition (ASR) and Eye-Tracking is very noisy and erroneous. This paper describes a technique for reducing the errors of the two modalities, speech and eye-gaze with the help of each other in context of sight translation and reading. Lattice representation and composition of the two modalities was used for integration. F-measure for Eye-Gaze and Word Accuracy for ASR were used as metrics to evaluate our results. In reading task, we demonstrated a significant improvement in both Eye-Gaze f-measure and speech Word Accuracy. In sight translation task, significant improvement was found in gaze f-measure but not in ASR.

AB - Researchers are proposing interactive machine translation as a potential method to make language translation process more efficient and usable. Introduction of different modalities like eye gaze and speech are being explored to add to the interactivity of language translation system. Unfortunately, the raw data provided by Automatic Speech Recognition (ASR) and Eye-Tracking is very noisy and erroneous. This paper describes a technique for reducing the errors of the two modalities, speech and eye-gaze with the help of each other in context of sight translation and reading. Lattice representation and composition of the two modalities was used for integration. F-measure for Eye-Gaze and Word Accuracy for ASR were used as metrics to evaluate our results. In reading task, we demonstrated a significant improvement in both Eye-Gaze f-measure and speech Word Accuracy. In sight translation task, significant improvement was found in gaze f-measure but not in ASR.

U2 - 10.1145/2535948.2535953

DO - 10.1145/2535948.2535953

M3 - Article in proceedings

SN - 9781450325639

SP - 35

EP - 40

BT - GazeIn '13. Proceedings of the 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction

A2 - Bednarik , Roman

A2 - Huang , Hung-Hsuan

A2 - Jokinen , Kristiina

A2 - Nakano, Yukiko I.

PB - Association for Computing Machinery

CY - New York

T2 - The 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction. GazeIn '13

Y2 - 13 December 2013 through 13 December 2013

ER -

Kulkarni R, Jain K, Bansal H, Bangalore S, Carl M. Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading. I Bednarik R, Huang HH, Jokinen K, Nakano YI, red., GazeIn '13. Proceedings of the 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction: Gaze in Multimodal Interaction . New York: Association for Computing Machinery. 2013. s. 35-40 doi: 10.1145/2535948.2535953