This paper reports on our participation with the MUCOW test suite at the WMT 2020 news translation task. We introduced MUCOW at WMT 2019 to measure the ability of MT systems to perform word sense disambiguation (WSD), i.e., to translate an ambiguous word with its correct sense. MUCOW is created automatically using existing resources, and the evaluation process is also entirely automated. We evaluate all participating systems of the language pairs English -{ extgreater} Czech, English -{ extgreater} German, and English -{ extgreater} Russian and compare the results with those obtained at WMT 2019. While current NMT systems are fairly good at handling ambiguous source words, we could not identify any substantial progress - at least to the extent that it is measurable by the MUCOW method - in that area over the last year.

Scherrer, Y., Raganato, A., Tiedemann, J. (2020). The MUCOW word sense disambiguation test suite at WMT 2020. In Proceedings of the Fifth Conference on Machine Translation (pp.365-370). Association for Computational Linguistics.

The MUCOW word sense disambiguation test suite at WMT 2020

Raganato, A
;
2020

Abstract

This paper reports on our participation with the MUCOW test suite at the WMT 2020 news translation task. We introduced MUCOW at WMT 2019 to measure the ability of MT systems to perform word sense disambiguation (WSD), i.e., to translate an ambiguous word with its correct sense. MUCOW is created automatically using existing resources, and the evaluation process is also entirely automated. We evaluate all participating systems of the language pairs English -{ extgreater} Czech, English -{ extgreater} German, and English -{ extgreater} Russian and compare the results with those obtained at WMT 2019. While current NMT systems are fairly good at handling ambiguous source words, we could not identify any substantial progress - at least to the extent that it is measurable by the MUCOW method - in that area over the last year.
paper
machine translation; word sense disambiguation; benchmark
English
Fifth Conference on Machine Translation
2020
Proceedings of the Fifth Conference on Machine Translation
978-1-948087-81-0
2020
365
370
reserved
Scherrer, Y., Raganato, A., Tiedemann, J. (2020). The MUCOW word sense disambiguation test suite at WMT 2020. In Proceedings of the Fifth Conference on Machine Translation (pp.365-370). Association for Computational Linguistics.
File in questo prodotto:
File Dimensione Formato  
2020.wmt-1.40.pdf

Solo gestori archivio

Dimensione 249.79 kB
Formato Adobe PDF
249.79 kB Adobe PDF   Visualizza/Apri   Richiedi una copia

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/10281/361589
Citazioni
  • Scopus 8
  • ???jsp.display-item.citation.isi??? ND
Social impact