German Language LLM Index
a PeerBench project

German LLM Benchmark · Model Profile

Qwen3.8 27B (FP8)

Alibabaopen weights · Apache-2.0fp8run 2026-08-15
1582

Elo

#9 among open-weights models

Benchmark breakdown

GermEval

81.6%

n=1024

Native German named-entity recognition — identify persons, locations, organisations and misc entities in German text, emitted as JSON. Scored with seqeval micro-F1 excluding the noisy MISC class. Run reasoning-off.

Named-entity recognitionnative · Native German
via GermEval (via EuroEval) ↗ View run details →

INCLUDE

86.9%

n=107

Native German exam and licensing questions covering region-specific knowledge — history, law, civics and culture. Written by humans in German, not translated.

4-option multiple choicenative · Native German
via CohereLabs/include-base-44 ↗ View run details →

MMLU-ProX

80.0%

n=11737

Hard academic questions across 14 subjects — STEM, law, health, economics, philosophy and more. Professionally translated to German, with up to ten answer options per question.

10-option multiple choicetranslated · Professional translation
via li-lab/MMLU-ProX ↗ View run details →

SB10K

62.7%

n=1024

Native German social-media sentiment classification — positive, neutral or negative. Human-annotated German text, not translated. Run reasoning-off; headline score is MCC on the predicted label.

3-class sentimentnative · Native German
via SB10K (via EuroEval) ↗ View run details →

ScaLA

66.3%

n=2048

Native German linguistic acceptability — does the sentence read as grammatical German (ja / nein)? Built from clean vs. minimally-corrupted German sentences. Run reasoning-off.

Binary acceptabilitynative · Native German
via ScaLA-de (via EuroEval) ↗ View run details →

Comparable models