Machine Translation Performance Leaderboard

Translation quality of 61 models on classical Asian languages → English.

Language pairs: Tibetan → English (bo-en) · Sanskrit → English (sa-en) · Chinese → English (zh-en).

Ranked by BLEURT (learned semantic-quality metric), averaged over the three language pairs; columns show per-language BLEURT. Because raw BLEURT-20 scores cluster tightly (~0.47–0.60), they are displayed as a relative 0–100 score — min–max normalized across the models listed here (100 = best, 0 = worst). Raw BLEURT-20 values appear in the per-language heatmap below and when you click any model row. BLEU and chrF are surface-overlap metrics — also in the row breakdown. Click a column header to sort.

Leaderboard

Each row is one model, ranked by its overall BLEURT (0–100) score — translation quality averaged across all three language pairs. The Tibetan, Sanskrit and Chinese columns are that same 0–100 score for each individual language pair. Higher and greener is better. Click any row for the full per-language breakdown including BLEU, chrF, and the raw BLEURT-20 value.

Tip: click a row to see Tibetan / Sanskrit / Chinese results for that model.

Provenance note: the row marked “live API” — MITRA (gemini-3-flash-preview, live API) — is the only run measured end-to-end against the deployed dharmamitra translation API (server-side retrieval, prompting and generation, i.e. exactly what dharmamitra.org users get). All other MITRA rows replicate the production pipeline in the standalone evaluation harness (mitra-evaluation) and call the model APIs directly; the remaining rows are the raw models under the harness's own prompting.

High thinking budget (tb = 1024)

Same scoring as the main board (BLEURT 0–100), but these models were run with a large 1024-token reasoning budget. They are listed separately because the extra reasoning makes them slower and more expensive, so they are not ranked head-to-head with the minimal-reasoning models above.

Sanskrit → Japanese (contextual, passage-level)

A separate contextual benchmark: 100 passages of 20 consecutive sentences each (held-out from the Sanskrit↔Japanese corpus, grouped by source document in reading order). Each model translates the whole 20-sentence passage at once into Japanese, testing long-sequence / contextual translation rather than isolated sentences. Ranked by GEMBA (an LLM judge — gemini-3-flash-preview — rating each translation 1–100 against the human Japanese reference); BLEU (ja-mecab) and chrF are surface-overlap metrics. BM25 rows augment the prompt with per-sentence retrieval (k=2 examples per source sentence, decontaminated against the eval set): JA = 2 sa→ja examples from the SanskritJapaneseTranslation train split; EN = 2 sa→en examples from the 1.55M gemini-cleaned sa-en set; EN+JA = both (2 each). The EN / EN+JA variants (run on gemini-3-flash-preview) measure how much English references help a Japanese-target translation. BLEURT is omitted (it is English-only). Click a column header to sort.

Vanilla = zero-shot; BM25 = retrieval-augmented (per-sentence, k=2). JA / EN / EN+JA = language(s) of the retrieved reference examples.

Per-language detail (raw scores)

This heatmap shows the raw, un-rescaled scores behind the leaderboard — one cell for every model, metric and language pair. Each block of three columns is a language pair (bo-en = Tibetan→English, sa-en = Sanskrit→English, zh-en = Chinese→English), and within it the three metrics are BLEU and chrF (word/character overlap with the reference, 0–100) and BLEURT-20 (a learned semantic-similarity score, roughly 0.4–0.6 here). Color is normalized per metric so each is comparable down its own column.