Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading

Rucha Kulkarni , Kritika Jain, Himanshu Bansal, Srinivas Bangalore, Michael Carl

    Research output: Contribution to conferencePaperResearchpeer-review

    Abstract

    Researchers are proposing interactive machine translation as a potential method to make language translation process more efficient and usable. Introduction of different modalities like eye gaze and speech are being explored to add to the interactivity of language translation system. Unfortunately, the raw data provided by Automatic Speech Recognition (ASR) and Eye-Tracking is very noisy and erroneous. This paper describes a technique for reducing the errors of the two modalities, speech and eye-gaze with the help of each other in context of sight translation and reading. Lattice representation and composition of the two modalities was used for integration. F-measure for Eye-Gaze and Word Accuracy for ASR were used as metrics to evaluate our results. In reading task, we demonstrated a significant improvement in both Eye-Gaze f-measure and speech Word Accuracy. In sight translation task, significant improvement was found in gaze f-measure but not in ASR.
    Original languageEnglish
    Publication date2013
    Number of pages6
    Publication statusPublished - 2013
    Event10th International Conference on Natural Language Processing - Centre for Development of Advanced Computing, Noida, India
    Duration: 18 Dec 201320 Dec 2013
    Conference number: 10
    http://ltrc.iiit.ac.in/icon/2013/index.php

    Conference

    Conference10th International Conference on Natural Language Processing
    Number10
    LocationCentre for Development of Advanced Computing
    CountryIndia
    CityNoida
    Period18/12/201320/12/2013
    Internet address

    Bibliographical note

    CBS Library does not have access to the material

    Cite this

    Kulkarni , R., Jain, K., Bansal, H., Bangalore, S., & Carl, M. (2013). Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading. Paper presented at 10th International Conference on Natural Language Processing, Noida, India.
    Kulkarni , Rucha ; Jain, Kritika ; Bansal, Himanshu ; Bangalore, Srinivas ; Carl, Michael. / Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading. Paper presented at 10th International Conference on Natural Language Processing, Noida, India.6 p.
    @conference{29a0b0e0cbac4a948f9a2d5ac63cc1a8,
    title = "Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading",
    abstract = "Researchers are proposing interactive machine translation as a potential method to make language translation process more efficient and usable. Introduction of different modalities like eye gaze and speech are being explored to add to the interactivity of language translation system. Unfortunately, the raw data provided by Automatic Speech Recognition (ASR) and Eye-Tracking is very noisy and erroneous. This paper describes a technique for reducing the errors of the two modalities, speech and eye-gaze with the help of each other in context of sight translation and reading. Lattice representation and composition of the two modalities was used for integration. F-measure for Eye-Gaze and Word Accuracy for ASR were used as metrics to evaluate our results. In reading task, we demonstrated a significant improvement in both Eye-Gaze f-measure and speech Word Accuracy. In sight translation task, significant improvement was found in gaze f-measure but not in ASR.",
    author = "Rucha Kulkarni and Kritika Jain and Himanshu Bansal and Srinivas Bangalore and Michael Carl",
    note = "CBS Library does not have access to the material; null ; Conference date: 18-12-2013 Through 20-12-2013",
    year = "2013",
    language = "English",
    url = "http://ltrc.iiit.ac.in/icon/2013/index.php",

    }

    Kulkarni , R, Jain, K, Bansal, H, Bangalore, S & Carl, M 2013, 'Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading' Paper presented at, Noida, India, 18/12/2013 - 20/12/2013, .

    Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading. / Kulkarni , Rucha; Jain, Kritika ; Bansal, Himanshu ; Bangalore, Srinivas ; Carl, Michael.

    2013. Paper presented at 10th International Conference on Natural Language Processing, Noida, India.

    Research output: Contribution to conferencePaperResearchpeer-review

    TY - CONF

    T1 - Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading

    AU - Kulkarni , Rucha

    AU - Jain, Kritika

    AU - Bansal, Himanshu

    AU - Bangalore, Srinivas

    AU - Carl, Michael

    N1 - CBS Library does not have access to the material

    PY - 2013

    Y1 - 2013

    N2 - Researchers are proposing interactive machine translation as a potential method to make language translation process more efficient and usable. Introduction of different modalities like eye gaze and speech are being explored to add to the interactivity of language translation system. Unfortunately, the raw data provided by Automatic Speech Recognition (ASR) and Eye-Tracking is very noisy and erroneous. This paper describes a technique for reducing the errors of the two modalities, speech and eye-gaze with the help of each other in context of sight translation and reading. Lattice representation and composition of the two modalities was used for integration. F-measure for Eye-Gaze and Word Accuracy for ASR were used as metrics to evaluate our results. In reading task, we demonstrated a significant improvement in both Eye-Gaze f-measure and speech Word Accuracy. In sight translation task, significant improvement was found in gaze f-measure but not in ASR.

    AB - Researchers are proposing interactive machine translation as a potential method to make language translation process more efficient and usable. Introduction of different modalities like eye gaze and speech are being explored to add to the interactivity of language translation system. Unfortunately, the raw data provided by Automatic Speech Recognition (ASR) and Eye-Tracking is very noisy and erroneous. This paper describes a technique for reducing the errors of the two modalities, speech and eye-gaze with the help of each other in context of sight translation and reading. Lattice representation and composition of the two modalities was used for integration. F-measure for Eye-Gaze and Word Accuracy for ASR were used as metrics to evaluate our results. In reading task, we demonstrated a significant improvement in both Eye-Gaze f-measure and speech Word Accuracy. In sight translation task, significant improvement was found in gaze f-measure but not in ASR.

    M3 - Paper

    ER -

    Kulkarni R, Jain K, Bansal H, Bangalore S, Carl M. Mutual Disambiguation of Eye Gaze and Speech for Sight Translation and Reading. 2013. Paper presented at 10th International Conference on Natural Language Processing, Noida, India.