Evaluating Machine Translation: The Limits and Potential of Large Language Models in Quality Assessment

Large Language Models have increasingly emerged across the technology sector as a faster and more cost-effective alternative to human judgment for machine translation evaluation. However, a new study led by researchers Hala Almaghout, Christian Federmann, and Qin Gao reveals that despite their efficiency, LLMs remain unreliable for most translation evaluation settings, underscoring the enduring complexity of accurately judging machine-translated text.

Machine translation evaluation spans a vast array of language pairs, specialized domains, and levels of annotation granularity. Because of this complexity, the research team emphasizes that Large Language Models must be thoroughly evaluated across all of these dimensions before they can be reliably adopted as substitutes for traditional human evaluation.

In their research paper, the authors evaluate the performance of LLMs across two prominent machine translation quality evaluation schemes: Multidimensional Quality Metrics and Error Span Annotation. To measure how effectively these models perform, the researchers compared their agreement directly against human annotators, shedding light on the current capabilities and distinct limitations of artificial intelligence in linguistic quality control.

The investigation presents comprehensive results drawn from a demanding long-context test set encompassing 70 distinct language pairs, alongside the publicly available data from the widely recognized WMT23 and WMT25 evaluations. By examining both numerical scoring and precise error span annotation agreement across a diverse variety of language pairs and operational domains, the researchers were able to paint a detailed picture of how automated systems stack up against human linguistic experts.

The findings reveal a nuanced reality regarding the current state of AI-assisted translation evaluation. While the study shows that LLM agreement with human annotators can occasionally exceed the baseline agreement observed between different human annotators themselves on certain specific evaluation tasks, both human and machine agreement vary substantially. This variability spans across different annotation schemes, language pairs, and operational domains, ultimately leading the researchers to conclude that LLMs remain unreliable for most practical settings.

Furthermore, the study identifies distinct operational challenges facing both human and artificial intelligence annotators during the evaluation process. Human evaluators were found to be particularly challenged when dealing with fine-grained Multidimensional Quality Metrics annotations and low-resource language pairs, where linguistic data is scarce and context can be difficult to establish. Conversely, Large Language Models struggle with a different set of linguistic hurdles, demonstrating notable difficulties when handling minor translation errors, wrong language variants, and precise error span annotations.

These complementary strengths and weaknesses highlight a significant potential for performance improvement across both human and LLM annotation pipelines. The researchers suggest that future advancements may lie in collaborative annotation frameworks that combine human expertise with artificial intelligence capabilities, specifically designed to address and mitigate the reliability issues identified in the study.

Related readings and updates.

Safety policies define what constitutes safe and unsafe AI outputs, guiding data annotation and model development. However, annotation disagreement is pervasive and can stem from multiple sources such as operational failures where annotators misunderstand or misexecute the task, policy ambiguity where policy wording leaves room for interpretation, or value pluralism where different annotators hold different perspectives on safety. Distinguishing these factors remains a critical ongoing area of research within the broader field of artificial intelligence development and deployment.

At the same time, machine learning and artificial intelligence systems rely heavily on human-annotated data for both initial training and rigorous evaluation. A major challenge in this context is the occurrence of annotation errors, as their downstream effects can severely degrade overall model performance. Addressing this, parallel research presents predictive error models trained to detect potential errors in search relevance annotation tasks across industry-scale machine learning applications, including music streaming, video streaming, and related digital domains.