Evaluating the Effects of Interactivity in a Post-Editing Workbench

Nancy  Underwood; Bartolomé  Mesa-Lao; Mercedes Garcia Martinez; Michael Carl; Vicent Alabau; Jesus Gonzalez-Rubio; Luis A. Leiva; Germán Sanchis-Trilles; Daniel  Ortíz-Martínez; Francisco Casacuberta

Evaluating the Effects of Interactivity in a Post-Editing Workbench

Nancy Underwood, Bartolomé Mesa-Lao, Mercedes Garcia Martinez, Michael Carl, Vicent Alabau, Jesus Gonzalez-Rubio, Luis A. Leiva, Germán Sanchis-Trilles, Daniel Ortíz-Martínez , Francisco Casacuberta

Polytechnic University of Valencia

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

Abstract

This paper describes the field trial and subsequent evaluation of a post-editing workbench which is currently under development in the EU-funded CasMaCat project. Based on user evaluations of the initial prototype of the workbench, this second prototype of the workbench includes a number of interactive features designed to improve productivity and user satisfaction. Using CasMaCat's own facilities for logging keystrokes and eye tracking, data were collected from nine post-editors in a professional setting. These data were then used to investigate the effects of the interactive features on productivity, quality, user satisfaction and cognitive load as reflected in the post-editors’ gaze activity. These quantitative results are combined with the qualitative results derived from user questionnaires and interviews conducted with all the participants.

Original language	English
Title of host publication	Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)
Editors	Nicoletta Calzolari, Khalid Choukri , Thierry Declerck , Hrafn Loftsson , Bente Maegaard, Joseph Mariani , Asuncion Moreno , Jan Odijk , Stelios Piperidis
Number of pages	7
Place of Publication	Paris
Publisher	European Language Resources Association
Publication date	2014
Pages	553-559
ISBN (Print)	9782951740884
Publication status	Published - 2014
Event	The 9th International Conference on Language Resources and Evaluation. LREC 2014 - Reykjavik, Iceland Duration: 26 May 2014 → 31 May 2014 Conference number: 9 http://lrec2014.lrec-conf.org/en/

Conference

Conference	The 9th International Conference on Language Resources and Evaluation. LREC 2014
Number	9
Country/Territory	Iceland
City	Reykjavik
Period	26/05/2014 → 31/05/2014
Internet address	http://lrec2014.lrec-conf.org/en/

Access to Document

http://www.lrec-conf.org/proceedings/lrec2014/summaries/289.html

Handle.net

Persistent link

Cite this

Underwood, N., Mesa-Lao, B., Martinez, M. G., Carl, M., Alabau, V., Gonzalez-Rubio, J., A. Leiva, L., Sanchis-Trilles, G., Ortíz-Martínez , D., & Casacuberta, F. (2014). Evaluating the Effects of Interactivity in a Post-Editing Workbench. In N. Calzolari, K. Choukri , T. Declerck , H. Loftsson , B. Maegaard, J. Mariani , A. Moreno , J. Odijk , & S. Piperidis (Eds.), Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14) (pp. 553-559). European Language Resources Association. http://www.lrec-conf.org/proceedings/lrec2014/summaries/289.html

Underwood, Nancy ; Mesa-Lao, Bartolomé ; Martinez, Mercedes Garcia et al. / Evaluating the Effects of Interactivity in a Post-Editing Workbench. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14). editor / Nicoletta Calzolari ; Khalid Choukri ; Thierry Declerck ; Hrafn Loftsson ; Bente Maegaard ; Joseph Mariani ; Asuncion Moreno ; Jan Odijk ; Stelios Piperidis. Paris : European Language Resources Association, 2014. pp. 553-559

@inproceedings{5363a78f93cb4ec3966752d7f3009f7d,

title = "Evaluating the Effects of Interactivity in a Post-Editing Workbench",

abstract = "This paper describes the field trial and subsequent evaluation of a post-editing workbench which is currently under development in the EU-funded CasMaCat project. Based on user evaluations of the initial prototype of the workbench, this second prototype of the workbench includes a number of interactive features designed to improve productivity and user satisfaction. Using CasMaCat's own facilities for logging keystrokes and eye tracking, data were collected from nine post-editors in a professional setting. These data were then used to investigate the effects of the interactive features on productivity, quality, user satisfaction and cognitive load as reflected in the post-editors{\textquoteright} gaze activity. These quantitative results are combined with the qualitative results derived from user questionnaires and interviews conducted with all the participants.",

keywords = "Post-editing, Machine translation, Evaluation, Interactive translation prediction",

author = "Nancy Underwood and Bartolom{\'e} Mesa-Lao and Martinez, {Mercedes Garcia} and Michael Carl and Vicent Alabau and Jesus Gonzalez-Rubio and {A. Leiva}, Luis and Germ{\'a}n Sanchis-Trilles and Daniel Ort{\'i}z-Mart{\'i}nez and Francisco Casacuberta",

year = "2014",

language = "English",

isbn = "9782951740884",

pages = "553--559",

editor = "Calzolari, {Nicoletta } and {Choukri }, {Khalid } and {Declerck }, {Thierry } and {Loftsson }, Hrafn and Maegaard, {Bente } and {Mariani }, {Joseph } and {Moreno }, {Asuncion } and {Odijk }, {Jan } and Piperidis, {Stelios }",

booktitle = "Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)",

publisher = "European Language Resources Association",

address = "France",

note = "The 9th International Conference on Language Resources and Evaluation. LREC 2014 ; Conference date: 26-05-2014 Through 31-05-2014",

url = "http://lrec2014.lrec-conf.org/en/",

}

Underwood, N, Mesa-Lao, B, Martinez, MG, Carl, M, Alabau, V, Gonzalez-Rubio, J, A. Leiva, L, Sanchis-Trilles, G, Ortíz-Martínez , D & Casacuberta, F 2014, Evaluating the Effects of Interactivity in a Post-Editing Workbench. in N Calzolari, K Choukri , T Declerck , H Loftsson , B Maegaard, J Mariani , A Moreno , J Odijk & S Piperidis (eds), Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14). European Language Resources Association, Paris, pp. 553-559, The 9th International Conference on Language Resources and Evaluation. LREC 2014, Reykjavik, Iceland, 26/05/2014. <http://www.lrec-conf.org/proceedings/lrec2014/summaries/289.html>

Evaluating the Effects of Interactivity in a Post-Editing Workbench. / Underwood, Nancy ; Mesa-Lao, Bartolomé ; Martinez, Mercedes Garcia et al.
Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14). ed. / Nicoletta Calzolari; Khalid Choukri ; Thierry Declerck ; Hrafn Loftsson ; Bente Maegaard; Joseph Mariani ; Asuncion Moreno ; Jan Odijk ; Stelios Piperidis. Paris: European Language Resources Association, 2014. p. 553-559.

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

TY - GEN

T1 - Evaluating the Effects of Interactivity in a Post-Editing Workbench

AU - Underwood, Nancy

AU - Mesa-Lao, Bartolomé

AU - Martinez, Mercedes Garcia

AU - Carl, Michael

AU - Alabau, Vicent

AU - Gonzalez-Rubio, Jesus

AU - A. Leiva, Luis

AU - Sanchis-Trilles, Germán

AU - Ortíz-Martínez , Daniel

AU - Casacuberta, Francisco

N1 - Conference code: 9

PY - 2014

Y1 - 2014

N2 - This paper describes the field trial and subsequent evaluation of a post-editing workbench which is currently under development in the EU-funded CasMaCat project. Based on user evaluations of the initial prototype of the workbench, this second prototype of the workbench includes a number of interactive features designed to improve productivity and user satisfaction. Using CasMaCat's own facilities for logging keystrokes and eye tracking, data were collected from nine post-editors in a professional setting. These data were then used to investigate the effects of the interactive features on productivity, quality, user satisfaction and cognitive load as reflected in the post-editors’ gaze activity. These quantitative results are combined with the qualitative results derived from user questionnaires and interviews conducted with all the participants.

AB - This paper describes the field trial and subsequent evaluation of a post-editing workbench which is currently under development in the EU-funded CasMaCat project. Based on user evaluations of the initial prototype of the workbench, this second prototype of the workbench includes a number of interactive features designed to improve productivity and user satisfaction. Using CasMaCat's own facilities for logging keystrokes and eye tracking, data were collected from nine post-editors in a professional setting. These data were then used to investigate the effects of the interactive features on productivity, quality, user satisfaction and cognitive load as reflected in the post-editors’ gaze activity. These quantitative results are combined with the qualitative results derived from user questionnaires and interviews conducted with all the participants.

KW - Post-editing

KW - Machine translation

KW - Evaluation

KW - Interactive translation prediction

M3 - Article in proceedings

SN - 9782951740884

SP - 553

EP - 559

BT - Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)

A2 - Calzolari, Nicoletta

A2 - Choukri , Khalid

A2 - Declerck , Thierry

A2 - Loftsson , Hrafn

A2 - Maegaard, Bente

A2 - Mariani , Joseph

A2 - Moreno , Asuncion

A2 - Odijk , Jan

A2 - Piperidis, Stelios

PB - European Language Resources Association

CY - Paris

T2 - The 9th International Conference on Language Resources and Evaluation. LREC 2014

Y2 - 26 May 2014 through 31 May 2014

ER -

Underwood N, Mesa-Lao B, Martinez MG, Carl M, Alabau V, Gonzalez-Rubio J et al. Evaluating the Effects of Interactivity in a Post-Editing Workbench. In Calzolari N, Choukri K, Declerck T, Loftsson H, Maegaard B, Mariani J, Moreno A, Odijk J, Piperidis S, editors, Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14). Paris: European Language Resources Association. 2014. p. 553-559