Why AI Translation Benchmarks Miss the Hardest Languages

A New Benchmark That Still Leaves Millions Behind

Researchers have begun crowdsourcing the hardest cases in machine translation to build a more honest benchmark for AI systems — one that goes beyond clean, high-resource language pairs and forces models to confront the sentences they actually fail on. The project, covered by Slator, represents a genuine step forward for MT evaluation.

But here is the problem: the languages most likely to appear in that benchmark are Spanish, German, Chinese, Arabic, and a handful of other well-resourced pairs. Chuukese and Pohnpeian will not make the list. Neither will most of the Pacific Island languages spoken daily in US healthcare waiting rooms, public school classrooms, and courtrooms.

That gap is not a minor methodological footnote. It is a patient safety issue.

What "Hard Cases" Actually Look Like

The crowdsourcing approach works by inviting translators and linguists to submit sentences that consistently break AI systems — idioms that shift meaning under clinical context, culturally specific constructions, low-frequency terminology. For high-resource languages, this produces useful stress tests.

For Chuukese, almost every sentence is a hard case.

Chuukese is a Micronesian language with verb-heavy morphology, where a single word can encode what English requires an entire clause to express. A Chuukese verb form can simultaneously indicate the subject, object, direction of action, and whether the action is ongoing or completed. Current large language models trained primarily on English and European languages have no reliable framework for this structure.

Pohnpeian adds another layer of complexity through its honorific register system. The same concept — asking someone to sit down, for example — requires entirely different vocabulary depending on whether you are addressing a commoner, a titled person, or a chief. In a medical intake form, getting the register wrong does not just sound rude. It can cause a patient to disengage from care entirely.

No current public MT benchmark captures these failure modes, because no benchmark builders have enough Chuukese or Pohnpeian data to construct meaningful test sets.

Why Healthcare and Legal Workflows Cannot Wait for Better Benchmarks

Federated States of Micronesia citizens hold Compact of Free Association status, meaning they can live and work in the United States without a visa. Roughly 50,000 to 60,000 Micronesians currently reside in the US, with the largest concentrations in Hawaii, Guam, Arkansas, and Oregon. Many arrived with limited English proficiency.

These communities interact with US healthcare systems, school districts, and government agencies every day. Federal language access obligations under Title VI of the Civil Rights Act require meaningful access regardless of English proficiency. "Meaningful" does not mean running a Chuukese intake form through Google Translate and hoping for the best.

The table below shows where MT currently stands for these language pairs versus what clinical and legal workflows actually require.

The conclusion is not subtle. For Chuukese and Pohnpeian, machine translation is not a starting point that needs refinement. It is not a draft that a post-editor can fix efficiently. It is a liability.

What Genuine Rare-Language MT Failure Looks Like in Practice

Here are concrete scenarios drawn from the kinds of work that crosses our desk regularly.

A hospital discharge summary asks a Chuukese-speaking patient to follow up with a specialist "if symptoms worsen." The MT output inverts the conditional phrasing in a way that reads, in Chuukese, as an instruction to wait until symptoms worsen before doing anything. A patient who trusts the document and follows it accordingly could delay care for a serious post-surgical complication.

A school district sends home a Pohnpeian-language consent form for a special education evaluation, generated by an AI tool. The form uses base register vocabulary throughout. The family receiving it is from a high-status lineage for whom the unmarked register reads as disrespectful. They do not return the form. The child's evaluation is delayed by months.

These are not hypothetical edge cases. They are predictable outcomes of applying MT to languages it was never trained to handle, in contexts where precision is not optional.

What Better Benchmarking Would Actually Require

The crowdsourcing model for MT benchmarks is a good idea. Expanding it to cover Chuukese, Pohnpeian, Marshallese, Chamorro, and other Pacific languages spoken by US-resident populations would require investment in three things: trained bilingual linguists willing to contribute test cases, domain-specific corpora in healthcare and legal contexts, and evaluation frameworks that account for morphological complexity and register.

That is a significant ask, and it will not happen quickly. Which means that for the foreseeable future — realistically, for the next several years at minimum — anyone serving Micronesian communities through healthcare, education, or government channels needs qualified human translators and interpreters.

Benchmarks are useful for measuring progress. They do not substitute for the progress itself.

The Practical Takeaway

If your organization works with Chuukese or Pohnpeian speakers and you are currently relying on any AI translation tool for patient-facing, legal, or educational documents, stop and audit those materials now. The benchmark research confirms what specialists have known for years: MT fails hardest where testing is thinnest, and the testing is thinnest precisely where your highest-risk populations are.

If you need qualified Chuukese or Pohnpeian translation for a healthcare or government project, reach out to TXLOC to discuss your specific workflow.