When a company decides to integrate machine translation (MT) into its processes—whether to localise content, serve customers in multiple languages or speed up documentation production—sooner or later the same question arises: how do we know the system translates well?
The answer seems simple, but it isn't. Measuring the quality of a machine translation requires methodological decisions that, if not addressed properly, can lead to erroneous conclusions. This is the first in a series of articles in which we will review the key elements to bear in mind in the machine translation evaluation process.
1. Evaluation datasets
1.1. What do I want to measure?
In a previous post we already discussed the importance of datasets as the invisible foundation on which artificial intelligence rests. When it comes to evaluating machine translation, choosing evaluation datasets is not a minor decision. The first step is to be clear about one question: what are we interested in measuring?
Evaluating the following is not the same thing:
- the linguistic accuracy of a translation (grammar, terminology, fluency);
- its discursive coherence throughout a complete text;
- its performance in a specific domain (legal, medical, technical, etc.);
- its behaviour when faced with specific phenomena (translation of URLs, codes, place names, etc.).
Each of these objectives requires a different type of dataset. A company that translates legal contracts does not have the same needs as one that translates product descriptions for a catalogue. Clearly defining the objective of the evaluation is the first step and, often, also one of the most overlooked.
1.2. Types of datasets
1.2.1. At sentence level
Historically, evaluation datasets in MT have been built at sentence level: sets of original sentences with their reference translations, evaluated independently of one another (Costa-jussà, 2022; Federico et al., 2012; Tiedemann, 2020).
This approach has been used for decades for practical and historical reasons. Since Statistical Machine Translation (SMT) in the 1990s, the training corpora of MT systems were divided into sentences. Even with the arrival of the first Transformers (Vaswani et al., 2017), the translation capacity of models was, to a large extent, limited to this level. However, this approach has a fundamental problem: when evaluating isolated sentences, all contextual information is lost (Castilho, 2021).
This means a system can obtain very good sentence-by-sentence scores and still produce a final text that is incoherent, with poorly resolved pronouns, inconsistent terminology between paragraphs, or a register that changes without reason throughout the document. In short, sentence-level evaluation can make a system look "good" when, in real use, it generates text that is unnatural or difficult to read when considered as a whole.
1.2.2. At paragraph level
In recent years, the evaluation paradigm has been changing. As an intermediate solution between the isolated sentence and the complete document, there are evaluation datasets at paragraph level. This format allows part of the discursive context to be captured—such as coherence between nearby sentences or the resolution of short-range coreferences—without requiring the complexity of evaluating entire documents (Andrews et al., 2025).
This is a particularly useful option when working with content structured into relatively self-contained units, such as paragraphs of a product sheet or sections of an FAQ, and it allows for a more realistic evaluation than at sentence level.
1.2.3. At document level
Evaluating at document level means observing how the system behaves when faced with a complete text, with all its structure, internal coherence and dependencies between sentences (coreferences, verb tense, consistent terminology, etc.).
As mentioned above, there are translation errors that are only detectable when the complete document is evaluated instead of isolated sentences. Phenomena such as anaphora resolution, terminological consistency throughout the text, or stylistic coherence disappear—or simply cannot be evaluated—when working with loose sentences.
Evaluations based solely on individual sentences tend to overestimate the real quality of systems, since they conceal errors that only emerge in the broader context of the document (Barrault et al., 2019; Castilho, 2020; Toral et al., 2018).
For a company, this is especially relevant: if the content being translated consists of manuals, articles, emails, web pages, documents or any text with continuity, a document-level evaluation will give a much truer picture of the quality that end users will actually receive.
1.2.4. Test-suites
Test-suites are a type of evaluation resource different from the above. Rather than attempting to measure the overall quality of a system, they are specifically designed to detect how it behaves when faced with specific linguistic phenomena: gender agreement, verb tenses, idiomatic expressions, lexical ambiguity, negation, units of measurement, etc.
Each example in a test-suite is constructed in a controlled way to isolate a very specific phenomenon, so that it can be determined precisely whether the system resolves it correctly or not. This makes them particularly useful for:
- diagnosing specific weaknesses in a system (for example, whether it mistranslates URLs, codes, proper nouns, etc.);
- comparing systems on specific aspects relevant to a given domain or language;
- tracking a system's evolution following updates or new training runs.
Test-suites can be designed at either sentence or paragraph level (Balkan, 1994; Savoldi et al., 2023; Kocmi et al., 2025). They do not replace general datasets but rather complement them: while a general dataset offers an overall view of quality, a test-suite makes it possible to understand how the system behaves in specific cases and to diagnose possible weaknesses.
2. Problems arising from poorly constructed datasets
Having evaluation datasets is not enough. If they are poorly built, the results can be misleading and lead to erroneous decisions. Some claims about parity between the quality of machine translation and human translation have been linked to inadequate designs of evaluation datasets (Hassan et al., 2018).
2.1. The source text and the problem of translationese
A basic but frequently ignored rule is that the source text of an evaluation dataset must be written directly in the source language, and must not already be a translation from another language.
When this is not the case, the phenomenon known as translationese appears (Baker, 1993): a translated text—even one produced by professional translators—tends to display linguistic patterns different from those of a source text originally written by a native speaker, such as simpler syntactic structures, calques from the source language, or reduced lexical variety.
If the "source" text used in an evaluation dataset is, in fact, a translation from another language, the MT system ends up translating a text that already displays features typical of translation. This can make the dataset easier for machine translation systems to handle (Graham et al., 2020; Zhang et al., 2019). For this reason, in order to carry out a reliable evaluation, it is essential to verify the real origin of the source texts.
2.2. References should not be MT post-edits
Another common and potentially problematic error is to use, as a "human reference", a text that is, in fact, a machine translation post-edited by a person.
Although post-editing does involve human intervention, the resulting text can retain decisions and structures characteristic of the automatic system that generated it. If this reference is subsequently used to evaluate MT systems—potentially including the very system that originally generated it—a systematic bias can be introduced: systems whose output more closely resembles the style of the system that produced the post-edit may obtain artificially higher scores, not necessarily because they translate better, but because the "reference" is already closer to their own style.
For this reason, the references used in an evaluation should always be translations produced by expert translators from scratch, without starting from a machine translation output. If this is not possible, the bias introduced must be taken into account before drawing conclusions or making decisions based on the results.
3. Conclusions
Measuring the quality of machine translation is not simply a matter of running a metric and getting a number. It requires:
- Being clear about what you want to measure before choosing a dataset.
- Choosing the appropriate level of granularity (sentence, paragraph or document) depending on the actual use the translation will be put to.
- Supplementing the evaluation with test-suites when it is necessary to diagnose specific linguistic phenomena.
- Evaluating the system in the specific domain in which it will be used, since the same system can produce very different results depending on the field. For example, it may perform very well in the legal domain and show more problems in the journalistic one.
- Ensuring that the datasets are well constructed: source texts that are genuinely original (not translationese) and genuine human references, not MT post-edits.
For a company that depends on machine translation to communicate with its customers, investing time in a rigorous evaluation is not an academic exercise, it is what makes it possible to check that the technology being used truly meets the needs of its intended use, and not merely delivers good results on paper.
BIBLIOGRAPHY
Costa-Jussà, Marta R., et al. "No language left behind: Scaling human-centered machine translation." arXiv preprint arXiv:2207.04672 (2022).
Tiedemann, Jörg. "The tatoeba translation challenge–realistic data sets for low resource and multilingual MT." Proceedings of the fifth conference on machine translation. 2020.
Federico, Marcello, et al. "Overview of the IWSLT 2012 evaluation campaign." Proceedings of the 9th International Workshop on Spoken Language Translation: Evaluation Campaign. 2012.
Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems 30 (2017).
Castilho, Sheila. "Towards document-level human MT evaluation: On the issues of annotator agreement, effort and misevaluation." Proceedings of the workshop on human evaluation of NLP systems (HumEval). 2021.
Andrews, Pierre, et al. "BOUQuET: dataset, benchmark and open initiative for universal quality evaluation in translation." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
Castilho, Sheila, Maja Popović, and Andy Way. "On context span needed for machine translation evaluation." Proceedings of the Twelfth Language Resources and Evaluation Conference. 2020.
Toral, Antonio, et al. "Attaining the unattainable? reassessing claims of human parity in neural machine translation." Proceedings of the third conference on machine translation: Research papers. 2018.
Barrault, Loïc, et al. "Findings of the 2019 conference on machine translation (WMT19)." Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1). 2019.
Balkan, Lorna. "Test Suites: some issues in their use and design." Proceedings of the Second International Conference on Machine Translation: Ten years on. 1994.
Kocmi, Tom, et al. "Findings of the wmt25 general machine translation shared task: Time to stop evaluating on easy test sets." Proceedings of the tenth conference on machine translation. 2025.
Hassan, Hany, et al. "Achieving human parity on automatic chinese to english news translation." arXiv preprint arXiv:1803.05567 (2018).
Graham, Yvette, Barry Haddow, and Philipp Koehn. "Statistical power and translationese in machine translation evaluation." Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP). 2020.
Zhang, M., & Toral, A. (2019). The effect of translationese in machine translation test sets. Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers).
Baker, M. (1993). Corpus: Linguistics and Translation Studies: Implications and Applications. In M. Baker, G. Francis, & E. Tognini-Bonelli (Eds.), Text and Technology: In honor of John Sinclair. Amsterdam (pp. 233-250). John Benjamins.
Savoldi, Beatrice, et al. "Test suites task: Evaluation of gender fairness in MT with MuST-SHE and INES." Proceedings of the Eighth Conference on Machine Translation. 2023.
