Cross-lingual LLM evaluation

LangMatchCan language itself compress a prompt?

A controlled study of whether English, compact Modern Chinese, and Classical Chinese move an LLM to a different quality–cost frontier — with tasks and model backends held fixed.

View repository See results Read report
LangMatch project illustration comparing English, Simplified Chinese, and WenYan prompts
Wényán is not universally cheaper or better — but on strong models it can become a genuine Pareto contender.
4modelsclosed and open-weight model families
3languagesEnglish, Simplified Chinese, WenYan
3benchmarksIFEval, MATH-500, and MMLU-Pro
2048token budgetcorrected output budget with truncation auditing

What the study finds

The result is not “one language wins.” Prompt language interacts with model capacity, task type, and reasoning visibility.

Robust default

Compact Modern Chinese travels best.

zh_compact is the most stable choice across model families, often preserving quality while controlling completion length.

Conditional gain

Wényán helps selectively.

Classical Chinese is strongest overall for GPT-5.4 in the corrected matrix, but its advantage does not transfer reliably to smaller open models.

Reasoning format

Visibility changes the frontier.

Hidden reasoning mainly cuts visible-token cost; compact-visible reasoning preserves the strongest GPT-5.4/Wényán point at much lower cost than explicit process.

One task, three prompt languages

Only the system-prompt language changes. The task subset, model backend, evaluator, and output budget stay fixed.

base

Compact English

The conventional control: clear, terse, and familiar to every model family in the study.

“Follow the user’s instructions carefully… eliminate redundancy and aim for compact, clear expression.”

zh_compact

Simplified Chinese

A compact Chinese control that separates language effects from the special literary style of Wényán.

“请严格遵循用户要求……删繁就简，要求表达精炼、意思清楚。”

wy

Classical Chinese

An information-dense natural language whose ellipsis and terse grammar make it a plausible prompt-compression regime.

“凡码与引文外，释理叙事宜从简。禁绝白话冗词，务求辞约义明。”

Corrected main matrix

The 2048-token rerun removes the largest truncation artifact from the pilot and reports score together with total inference tokens.

MATH-500 prompt-language comparison
MATH-500.GPT-4o favors compact Chinese; GPT-5.4 favors Wényán.
IFEval prompt-language comparison
IFEval.High-capacity models satisfy the benchmark under multiple languages.
MMLU-Pro prompt-language comparison
MMLU-Pro.The most conservative slice; language effects remain model-dependent.

Reasoning visibility matters too

A second three-run study varies whether reasoning is explicit, hidden, or compact-visible while keeping the language conditions fixed.

Explicit reasoning overall results
Explicit process.GPT-5.4/Wényán reaches 0.833 SR.
Hidden reasoning overall results
Hidden.Much cheaper for closed models, but language gaps flatten.
Compact-visible reasoning overall results
Compact visible.The same 0.833 GPT-5.4/Wényán score at 381.92 tokens instead of 578.42.

Manifest-driven and auditable

Every run records the exact task subset, prompt condition, configured token budget, finish reason, token usage, and evaluator result. The pipeline keeps raw execution, aggregation, and plotting separate.

fixed manifest
  → task × language × model runs
  → results.jsonl + usage metadata
  → truncation audit
  → cross-run aggregation
  → score / token Pareto plots
Interpretation boundary. LangMatch is an exploratory study, not evidence that Classical Chinese is universally superior. Token counts are source-native and should be compared within a model/backend; GPT-5.4’s Wényán gain is real in this release, but strongly conditional.

Reports and reproducibility

The public repository includes manifests, runners, aggregated reports, plotting scripts, and publish-safe result artifacts.

RepositoryCode, manifests, tests, and project overview. Main matrixCorrected 2048-token report and complete tables. Three-run studyReasoning-visibility comparison and caveats. ManifestsFrozen task subsets and experiment definitions.