What LLM Benchmarks Actually Measure
A vendor tells you their model scores 92% on some benchmark. It sounds precise, so it gets treated as proof. Most buyers nod along without asking the one question that actually matters: 92% at what, exactly?
Benchmark names get used as shorthand for "this model is good", and they're not that. Each one is a narrow exam built to test a specific skill, scored a specific way, on a specific kind of question. A model can be excellent at one and mediocre at the task you actually need done, and the benchmark score alone won't tell you which situation you're in.
This is the first of three pieces on reading AI benchmark claims like a buyer instead of a spectator. Here we'll open up four benchmarks you'll see quoted constantly and explain, in plain terms, what each one is actually testing.
MMLU: a multiple-choice exam across 57 subjects
MMLU stands for Massive Multitask Language Understanding. It's a set of multiple-choice questions spanning subjects from elementary maths to US law to professional medicine, roughly 14,000 questions across 57 subject areas. A model reads a question and four answer options and picks one.
What it tells you
How broad the model's factual knowledge is, and how well it recognises the right answer when it's sitting right there among the options. It's closer to a school exam than a work task: no research, no drafting, no back-and-forth, just recognition.
What it doesn't tell you: whether the model can produce something useful unprompted, hold a long conversation, follow your specific instructions, or avoid making things up when there's no multiple-choice option to fall back on.
HumanEval: can it write a short function that passes the tests
HumanEval gives the model a short, plain-English description of a programming problem and asks it to write a Python function that solves it. The function is then run against a set of hidden test cases. Pass all of them, it counts as solved.
What it tells you
Whether the model can turn a clear, self-contained coding prompt into working code, on problems small enough to fit in a single function. It's a reasonable signal for "can this thing code at all".
What it doesn't tell you: whether it can work inside an existing codebase, navigate a real project's structure, understand a vague bug report, or avoid introducing a regression somewhere else in the code. Real engineering work rarely looks like a self-contained function with a clean spec.
SWE-bench: a real GitHub issue, a real codebase, a real fix
SWE-bench is closer to actual software work. The model is given a real, previously filed issue from an open-source GitHub repository, along with the codebase as it existed at the time, and asked to produce a patch. The patch is scored by running the project's own test suite against it.
What it tells you
Whether the model can find its way around an unfamiliar, sizeable codebase and produce a change that actually resolves a real problem without breaking existing tests. This is a meaningfully harder and more realistic task than HumanEval.
What it doesn't tell you: how it performs outside the mostly Python, mostly popular-open-source-project slice of the software world the benchmark draws from, or how it handles a codebase with poor test coverage, which describes a lot of real business software.
GPQA: graduate-level questions built to resist lookup
GPQA stands for Graduate-Level Google-Proof Q&A. The questions are written by subject-matter experts in biology, physics and chemistry, and deliberately constructed so that a skilled non-expert with unrestricted internet access still struggles to answer them correctly by searching.
What it tells you
Raw depth of reasoning and domain knowledge on hard scientific questions, tested in a way that resists simple lookup or pattern-matching.
What it doesn't tell you: almost anything about how the model will perform on the kind of work most businesses actually need. Very few SMB use cases involve graduate-level physics reasoning. This benchmark matters far more if you're evaluating a model for scientific research than if you're evaluating one to draft customer emails or summarise reports.
The pattern underneath all four
Every benchmark is a narrow exam, and a high score tells you the model is good at that exam. It doesn't tell you the model is good, full stop, and it especially doesn't tell you it's good at your job. A model that tops the leaderboard on graduate-level science questions can still write clumsy customer emails. A model that scores well on a coding benchmark built from tidy open-source projects can still struggle in your fifteen-year-old internal codebase with no tests.
Once you know what a benchmark is actually testing, a vendor's headline number stops being a verdict and starts being one data point, useful only once you know whether it's testing anything close to the job you have in mind.
The next piece in this series looks at why a strong benchmark score still doesn't reliably predict how a model performs on your actual work, and what tends to cause the gap. After that, a practical way to read a model leaderboard without needing a machine learning background.
Evaluating AI tools for your business? Alvento helps businesses cut through vendor claims and deploy AI systems that hold up on their actual work, not just a leaderboard. Discuss your project at alvento.uk or email hello@alvento.uk, first conversation is free.