German Language LLM Index
a PeerBench project

German LLM Benchmark · Model Profile

Nemotron 3.5 Bolt 74B-A9B (IQ2)

NVIDIAopen weights · OpenMDWiq2run 2026-10-16
1172

Elo

#22 among open-weights models

Benchmark breakdown

GermEval

73.4%

n=1024

Native German named-entity recognition — identify persons, locations, organisations and misc entities in German text, emitted as JSON. Scored with seqeval micro-F1 excluding the noisy MISC class. Run reasoning-off.

Named-entity recognitionnative · Native German
via GermEval (via EuroEval) ↗ View run details →

INCLUDE

73.8%

n=107

Native German exam and licensing questions covering region-specific knowledge — history, law, civics and culture. Written by humans in German, not translated.

4-option multiple choicenative · Native German
via CohereLabs/include-base-44 ↗ View run details →

MMLU-ProX

80.5%

Hard academic questions across 14 subjects — STEM, law, health, economics, philosophy and more. Professionally translated to German, with up to ten answer options per question.

10-option multiple choicetranslated · Professional translationpreliminary
via li-lab/MMLU-ProX ↗

MuSR

76.9%

n=553

Multi-step soft reasoning over long narrative contexts — murder mysteries, object placement and team allocation. Requires chaining clues across several paragraphs to reach the correct answer. Translated to German from the original English MuSR benchmark. Public results use the corrected v1.1 evaluation with 553 effective test cases.

2–5 option multiple choicetranslated · German translation
via TAUR-Lab/MuSR ↗ View run details →

SB10K

52.8%

n=1024

Native German social-media sentiment classification — positive, neutral or negative. Human-annotated German text, not translated. Run reasoning-off; headline score is MCC on the predicted label.

3-class sentimentnative · Native German
via SB10K (via EuroEval) ↗ View run details →

ScaLA

58.9%

n=2048

Native German linguistic acceptability — does the sentence read as grammatical German (ja / nein)? Built from clean vs. minimally-corrupted German sentences. Run reasoning-off.

Binary acceptabilitynative · Native German
via ScaLA-de (via EuroEval) ↗ View run details →

Comparable models