Word Transition Entropy as an Indicator for Expected Machine Translation Quality

Michael Carl; Moritz Schaeffer

Word Transition Entropy as an Indicator for Expected Machine Translation Quality

Michael Carl, Moritz Schaeffer

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

Abstract

While most machine translation evaluation techniques (BLEU, NIST, TER, METEOR) assess translation quality based on a set of reference translations, we suggest to evaluate the literality of a set of (human or machine generated) translations to infer their potential quality. We provide evidence which suggests that more literal translations are produced more easily, by humans and machine, and are also less error prone. Literal translations may not be appropriate or even possible for all languages, types of texts, and translation purposes. However, in this paper we show that an assessment of the literality of translations allows us to (1) evaluate human and machine translations in a similar fashion and (2) may be instrumental to predict machine translation quality scores,

Original language	English
Title of host publication	Proceedings of the Workshop on Automatic and Manual Metrics for Operational Translation Evaluation. MTE 2014
Editors	Keith J. Miller, Lucia Specia, Kim Harris, Stacey Bailey
Place of Publication	Paris
Publisher	European Language Resources Association
Publication date	2014
Pages	45-50
Publication status	Published - 2014
Event	The Workshop on Automatic and Manual Metrics for Operational Translation Evaluation. MTE 2014 - Harpa Conference Centre , Reykjavik, Iceland Duration: 26 May 2014 → 26 May 2014 http://mte2014.github.io/

Workshop

Workshop	The Workshop on Automatic and Manual Metrics for Operational Translation Evaluation. MTE 2014
Location	Harpa Conference Centre
Country/Territory	Iceland
City	Reykjavik
Period	26/05/2014 → 26/05/2014
Other	Held in connection to the LREC 2014: The 9th edition of the Language Resources and Evaluation Conference
Internet address	http://mte2014.github.io/

Bibliographical note

Workshop on The Ninth International Conference on Language Resources and Evaluation. LREC 2014

Access to Document

http://mte2014.github.io/MTE2014-Workshop-Proceedings.pdf

Cite this

Carl, M., & Schaeffer, M. (2014). Word Transition Entropy as an Indicator for Expected Machine Translation Quality. In K. J. Miller, L. Specia, K. Harris, & S. Bailey (Eds.), Proceedings of the Workshop on Automatic and Manual Metrics for Operational Translation Evaluation. MTE 2014 (pp. 45-50). European Language Resources Association. http://mte2014.github.io/MTE2014-Workshop-Proceedings.pdf

@inproceedings{73de7b49a8f84c96b3595aa7bc60ac2a,

title = "Word Transition Entropy as an Indicator for Expected Machine Translation Quality",

abstract = "While most machine translation evaluation techniques (BLEU, NIST, TER, METEOR) assess translation quality based on a set of reference translations, we suggest to evaluate the literality of a set of (human or machine generated) translations to infer their potential quality. We provide evidence which suggests that more literal translations are produced more easily, by humans and machine, and are also less error prone. Literal translations may not be appropriate or even possible for all languages, types of texts, and translation purposes. However, in this paper we show that an assessment of the literality of translations allows us to (1) evaluate human and machine translations in a similar fashion and (2) may be instrumental to predict machine translation quality scores,",

keywords = "Word translation entropy,, Edit distance, Literality threshold",

author = "Michael Carl and Moritz Schaeffer",

note = "Workshop on The Ninth International Conference on Language Resources and Evaluation. LREC 2014; null ; Conference date: 26-05-2014 Through 26-05-2014",

year = "2014",

language = "English",

pages = "45--50",

editor = "Miller, {Keith J. } and Specia, {Lucia } and Harris, {Kim } and Bailey, {Stacey }",

booktitle = "Proceedings of the Workshop on Automatic and Manual Metrics for Operational Translation Evaluation. MTE 2014",

publisher = "European Language Resources Association",

address = "France",

url = "http://mte2014.github.io/",

}

Carl, M & Schaeffer, M 2014, Word Transition Entropy as an Indicator for Expected Machine Translation Quality. in KJ Miller, L Specia, K Harris & S Bailey (eds), Proceedings of the Workshop on Automatic and Manual Metrics for Operational Translation Evaluation. MTE 2014. European Language Resources Association, Paris, pp. 45-50, The Workshop on Automatic and Manual Metrics for Operational Translation Evaluation. MTE 2014, Reykjavik, Iceland, 26/05/2014. <http://mte2014.github.io/MTE2014-Workshop-Proceedings.pdf>

Word Transition Entropy as an Indicator for Expected Machine Translation Quality. / Carl, Michael; Schaeffer, Moritz.

Proceedings of the Workshop on Automatic and Manual Metrics for Operational Translation Evaluation. MTE 2014. ed. / Keith J. Miller; Lucia Specia; Kim Harris; Stacey Bailey. Paris : European Language Resources Association, 2014. p. 45-50.

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

TY - GEN

T1 - Word Transition Entropy as an Indicator for Expected Machine Translation Quality

AU - Carl, Michael

AU - Schaeffer, Moritz

N1 - Workshop on The Ninth International Conference on Language Resources and Evaluation. LREC 2014

PY - 2014

Y1 - 2014

N2 - While most machine translation evaluation techniques (BLEU, NIST, TER, METEOR) assess translation quality based on a set of reference translations, we suggest to evaluate the literality of a set of (human or machine generated) translations to infer their potential quality. We provide evidence which suggests that more literal translations are produced more easily, by humans and machine, and are also less error prone. Literal translations may not be appropriate or even possible for all languages, types of texts, and translation purposes. However, in this paper we show that an assessment of the literality of translations allows us to (1) evaluate human and machine translations in a similar fashion and (2) may be instrumental to predict machine translation quality scores,

AB - While most machine translation evaluation techniques (BLEU, NIST, TER, METEOR) assess translation quality based on a set of reference translations, we suggest to evaluate the literality of a set of (human or machine generated) translations to infer their potential quality. We provide evidence which suggests that more literal translations are produced more easily, by humans and machine, and are also less error prone. Literal translations may not be appropriate or even possible for all languages, types of texts, and translation purposes. However, in this paper we show that an assessment of the literality of translations allows us to (1) evaluate human and machine translations in a similar fashion and (2) may be instrumental to predict machine translation quality scores,

KW - Word translation entropy,

KW - Edit distance

KW - Literality threshold

M3 - Article in proceedings

SP - 45

EP - 50

BT - Proceedings of the Workshop on Automatic and Manual Metrics for Operational Translation Evaluation. MTE 2014

A2 - Miller, Keith J.

A2 - Specia, Lucia

A2 - Harris, Kim

A2 - Bailey, Stacey

PB - European Language Resources Association

CY - Paris

Y2 - 26 May 2014 through 26 May 2014

ER -