Machine Translation Performance Leaderboard

Translation quality of 63 models on classical Asian languages → English.

Language pairs: Tibetan → English (bo-en) · Sanskrit → English (sa-en) · Chinese → English (zh-en).

Ranked by Elo from a head-to-head arena: every pair of models was judged against every other on the same 100 test sentences, and the ratings are the strengths that best explain all 122,500 verdicts. The overall column is Elo across all three language pairs; the Tibetan, Sanskrit and Chinese columns are Elo fitted within that pair alone. BLEU, chrF and BLEURT are still measured — click any model row to see them, and the per-language heatmap below shows the raw values. The main table lists closed-book models; retrieval-augmented runs, which get similar sentences pasted into the prompt, are ranked in a separate table further down. Click a column header to sort.

Leaderboard

Closed-book models — each translates from the source text alone, with no retrieved examples. Ranked by overall Elo across all three language pairs; the Tibetan, Sanskrit and Chinese columns are Elo fitted within that language pair on its own, so they are comparable down a column but not across columns. Higher and greener is better. Click any row for the confidence interval, win rate, and the full per-language BLEU / chrF / BLEURT breakdown. Retrieval-augmented runs are in their own table below.

Tip: click a row to see Tibetan / Sanskrit / Chinese results for that model.

Provenance note: the row marked “live API” — MITRA (gemini-3-flash-preview, live API) — is the only run measured end-to-end against the deployed dharmamitra translation API (server-side retrieval, prompting and generation, i.e. exactly what dharmamitra.org users get). All other MITRA rows replicate the production pipeline in the standalone evaluation harness (mitra-evaluation) and call the model APIs directly; the remaining rows are the raw models under the harness's own prompting.

High thinking budget (tb = 1024)

These models were run with a large 1024-token reasoning budget, which makes them slower and more expensive, so they get their own table. They did take part in the same arena, and their Elo is on the same scale as the main board — they are simply listed apart rather than ranked head-to-head with the minimal-reasoning models above.

Retrieval-augmented (RAG / ICL)

These runs see retrieved reference translations at inference time — similar sentences pulled from a parallel corpus and pasted into the prompt, plus dictionary entries for the MITRA pipeline — instead of translating from the source alone. They are listed separately because a rating here measures the retrieval pipeline as much as the model, so putting them next to closed-book models would not compare like with like. Elo is on the same scale as the tables above: all of these models fought the same arena, against closed-book opponents included, so a number here and a number above are directly comparable.

Retrieval is decontaminated against the evaluation sets — near-duplicate source/target pairs were removed from the index, since 81% of the Tibetan test rows were originally present in it near-verbatim.

Scored, but not in the arena

These models were evaluated on the same test sets, but their translations were not retained — only their scores — so they could not be entered into the head-to-head arena and have no Elo. They are ranked here by BLEURT, shown as a relative 0–100 score (min–max normalized across every model on this page, so 100 is the best-scoring model anywhere here). That is not the same scale as the Elo tables above and the two should not be compared row-for-row. Note that this group mixes both kinds: the three MITRA rows are retrieval-augmented (including the production “live API” run), while the gpt / claude / grok rows are closed-book.

Sanskrit → Japanese (contextual, passage-level)

A separate contextual benchmark: 100 passages of 20 consecutive sentences each (held-out from the Sanskrit↔Japanese corpus, grouped by source document in reading order). Each model translates the whole 20-sentence passage at once into Japanese, testing long-sequence / contextual translation rather than isolated sentences. Ranked by GEMBA (an LLM judge — gemini-3-flash-preview — rating each translation 1–100 against the human Japanese reference); BLEU (ja-mecab) and chrF are surface-overlap metrics. BM25 rows augment the prompt with per-sentence retrieval (k=2 examples per source sentence, decontaminated against the eval set): JA = 2 sa→ja examples from the SanskritJapaneseTranslation train split; EN = 2 sa→en examples from the 1.55M gemini-cleaned sa-en set; EN+JA = both (2 each). The EN / EN+JA variants (run on gemini-3-flash-preview) measure how much English references help a Japanese-target translation. BLEURT is omitted (it is English-only). Click a column header to sort.

Vanilla = zero-shot; BM25 = retrieval-augmented (per-sentence, k=2). JA / EN / EN+JA = language(s) of the retrieved reference examples.

Per-language detail (raw scores)

This heatmap shows the raw, un-rescaled scores behind the leaderboard — one cell for every model, metric and language pair. Each block of three columns is a language pair (bo-en = Tibetan→English, sa-en = Sanskrit→English, zh-en = Chinese→English), and within it the three metrics are BLEU and chrF (word/character overlap with the reference, 0–100) and BLEURT-20 (a learned semantic-similarity score, roughly 0.4–0.6 here). Color is normalized per metric so each is comparable down its own column.