How to Evaluate LLMs: Benchmarks, Leaderboards, and Testing Models on Your Own Work

November 11, 2026 · 15 min read

Every few weeks a new AI model arrives with a launch post claiming it "beats" its rivals on a list of benchmarks. A chart shows the new model's bar slightly taller than everyone else's. A leaderboard reshuffles. Then a team actually tries the model on its own work and finds that it writes worse support replies than the model they already use, costs twice as much, or takes four seconds longer to answer.

That gap between headline scores and real-world usefulness is the subject of this guide. LLM evaluation is the practice of measuring how well a language model, or a system built around one, performs a specific job. Public benchmarks are one form of evaluation. Leaderboards built on human votes are another. The most important form for any business, though, is the one almost nobody publishes: a test built from your own tasks, your own data, and your own definition of a good answer.

This guide explains, in plain English for U.S. B2B SaaS teams, what benchmarks measure, why scores can mislead, how human-preference leaderboards and "AI as judge" grading work, and how to build a small, practical evaluation that tells you which model to use. It ends with what evaluation thinking reveals about measuring your brand's visibility in AI answers. If you are new to the technology itself, our practical guide to large language models is a good starting point.

What It Means to Evaluate an LLM

Evaluating traditional software is mostly about correctness: given this input, does the program return the expected output every time? Language models are harder to evaluate for three reasons.

1. Outputs vary. The same prompt can produce different answers on different runs, so a single result proves little.
2. Many tasks have no single right answer. There are thousands of acceptable ways to summarize a contract or draft a sales email. "Correct" becomes a judgment about accuracy, completeness, tone, and format.
3. Performance depends on the whole system. The model is only one part. Instructions, retrieved documents, tool access, and the length of the input all change the result, as explained in our guides to prompt engineering and tokens and context windows.

Because of this, it helps to separate two questions that often get blurred together. Model evaluation asks how capable a model is in general. System evaluation asks whether your specific application, with its prompts, data, and guardrails, does its job well enough to ship. Public benchmarks speak to the first question. Only your own tests answer the second.

The Three Families of LLM Evaluation

Almost every evaluation method falls into one of three families.

• Automated benchmarks: fixed sets of questions with known answers, scored by software. They are cheap, fast, and repeatable, which is why model makers report them.
• Human judgment: people compare or grade outputs. This captures quality that is hard to score automatically, such as helpfulness and clarity, but it is slow, costly, and varies between graders.
• Model-graded evaluation: one language model grades another's output against a rubric. It scales like automation while approximating human judgment, but it carries its own biases.

Strong evaluation programs use all three, each where it fits. The rest of this guide covers how each works and where each breaks down.

Public Benchmarks: What the Famous Scores Measure

A benchmark is a standardized exam. Researchers assemble questions, publish them, and report how often each model answers correctly. A few appear in nearly every model launch.

Knowledge and reasoning exams

MMLU (Massive Multitask Language Understanding), introduced in a 2020 research paper, contains multiple-choice questions across 57 subjects, from elementary mathematics to U.S. history, law, and medicine. For several years it was the default measure of broad knowledge. As top models began scoring very high, it lost much of its power to separate them, a pattern researchers call saturation.

Harder exams followed. GPQA, described in a 2023 paper, contains 448 graduate-level questions in biology, physics, and chemistry, written by domain experts and designed to be "Google-proof," meaning that skilled non-experts with web access still struggle to answer them.

Truthfulness

TruthfulQA, from a 2021 paper, contains 817 questions across 38 categories designed to tempt models into repeating common misconceptions. It measures something different from knowledge: whether a model resists plausible-sounding falsehoods. That makes it relevant to the problems covered in our guide to LLM hallucinations.

Real-world tasks

Newer benchmarks try to measure useful work rather than exam answers. SWE-bench, introduced in a 2023 paper, gives models 2,294 real issues drawn from popular open-source Python projects on GitHub and checks whether the model's code change makes the project's tests pass. Task-style benchmarks like this are closer to how AI agents and tool use work in practice, and its maintainers publish public leaderboards.

Broad evaluation suites

Stanford's HELM (Holistic Evaluation of Language Models), described in a 2022 paper and maintained as a public project, argues that a single accuracy number is not enough. It evaluates models across many scenarios and reports several metrics side by side, including accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency. The idea is useful for any business: decide in advance which qualities matter, then measure each one separately.

Why Benchmark Scores Can Mislead

Benchmarks are valuable, but a launch chart is a marketing document, not a buying recommendation. Six problems are worth understanding before you trust any score.

Contamination

Models are trained on vast amounts of public text. If benchmark questions, or discussions of their answers, appear in that training data, a model can score well partly by recognition rather than reasoning. Researchers tested this for grade-school math by writing a fresh set of problems that matched the style and difficulty of a popular benchmark but had never been published. In that 2024 study, several model families scored noticeably lower on the new problems than on the original set, evidence of overfitting to the familiar test.

Saturation

When most leading models score above 90 percent, differences of a point or two are often within normal run-to-run variation. A saturated benchmark tells you a model clears a bar; it no longer tells you which model is better.

Teaching to the test

Economists know this as Goodhart's law: when a measure becomes a target, it stops being a good measure. Once a benchmark becomes famous, there is pressure to optimize for it specifically. Scores rise faster than real-world ability.

Setup differences

Scores depend on details: how the question is worded, how many examples are included in the prompt, whether the model may reason step by step, how answers are extracted and graded, and how many attempts are allowed. Two organizations can report different scores for the same model on the same benchmark. Compare numbers only when the setup is documented and matched.

Format mismatch

Most benchmarks are multiple-choice or have short exact answers. Most business work is open-ended writing, summarizing, extracting, and deciding. A model that excels at picking the right letter can still produce a rambling, poorly formatted customer email.

Missing dimensions

A leaderboard rarely reports what your finance team and your customers will notice: cost per thousand requests, response time, consistency across runs, refusal behavior, and how the model handles your industry's vocabulary. Those often decide which model is right.

Human-Preference Leaderboards

To capture qualities that exams miss, some leaderboards rely on people. The best-known approach, Chatbot Arena, now operated as LMArena and described in a 2024 research paper, shows a visitor two anonymous model answers to the visitor's own prompt and asks which is better. Hundreds of thousands of these pairwise votes are combined into ratings using statistical methods borrowed from ranking chess players.

This design has real strengths. Prompts are fresh and varied, so contamination is less of a problem, and the votes reflect what people actually prefer. It also has limits worth knowing:

• Voters are not your customers. The prompts come from a self-selected public audience, which may look nothing like your support queue or your buyers' questions.
• Style can beat substance. People often prefer longer, more confident, better-formatted answers, even when a shorter answer is more accurate. Leaderboard operators have introduced adjustments for style, which is itself an admission of the effect.
• Preference is not correctness. A casual voter cannot verify a tax rule or a drug interaction. They judge what sounds right.

Treat human-preference rankings as a signal about general helpfulness, not as evidence that a model will be accurate in your domain.

When a Model Grades Another Model

Grading thousands of open-ended answers by hand is expensive, so many teams use a strong model as an automated grader, an approach known as LLM-as-a-judge. The grader receives the question, the answer, and a rubric, then returns a score or a preference.

A 2023 study found that strong model judges agreed with human preferences more than 80 percent of the time, roughly the same rate at which humans agree with each other. The same study documented consistent biases:

• Position bias: favoring whichever answer appears first, or second, regardless of quality.
• Verbosity bias: favoring longer answers.
• Self-preference: favoring answers written in the grader's own style or by the same model family.

These biases are manageable. Swap the order of answers and grade twice. Give the grader a specific rubric instead of asking which answer is "better." Ask for a short justification before the score. Most importantly, check a sample of the grader's decisions against your own experts, and keep doing so. An automated grader is a tool to scale human judgment, not a replacement for it.

Build Your Own Evaluation: A Practical Method

The most useful evaluation for a business is small, specific, and repeatable. Major providers recommend the same core approach in their documentation, including OpenAI's guide to evals and Anthropic's guide to defining success criteria and building evaluations. Here is a version any team can run.

Step 1: Define the job and what success looks like

Write one sentence describing the task, such as "Draft a first reply to inbound support tickets about billing." Then list the criteria a good output must meet: correct facts drawn only from the help center, the right next step, under 150 words, no promises about refunds, and an explicit escalation when the answer is unknown. Criteria written before testing prevent you from rationalizing whatever the model produces.

Step 2: Collect real examples

Gather 30 to 100 realistic inputs from your own work. Include ordinary cases, difficult ones, ambiguous ones, and a few that should trigger a refusal or escalation. Remove personal and confidential data first, following the practices in our guide to LLM security and privacy. For each input, note what a correct answer must include.

Step 3: Choose how to grade

Use the simplest grader that works for each criterion:

• Code checks for anything objective: required format, length limits, presence of a required link, valid structured output.
• Model grading with a rubric for qualities such as accuracy against a source document or appropriate tone.
• Expert grading for a sample of outputs, to calibrate the automated graders and catch what they miss.

Step 4: Run each candidate several times

Test each model or prompt version on every input at least three times. Record the pass rate per criterion, not just an overall impression. Variation between runs is itself a finding: a model that passes 95 percent of the time and fails unpredictably may be worse for customer-facing work than one that passes 90 percent of the time and fails in predictable ways.

Step 5: Measure cost and speed alongside quality

Record tokens used, cost per request, and response time for each candidate. Hidden reasoning steps and long instructions can multiply cost without improving results. A slightly less capable model that is five times cheaper and twice as fast is often the right choice for high-volume tasks.

Step 6: Keep the evaluation and rerun it

Save every evaluation run with the date, model version, prompt version, and results. Providers update and retire models. When that happens, or when you change prompts or data sources, rerun the same evaluation. Without a saved baseline, you cannot tell whether a change made things better or worse.

Evaluating Systems, Not Just Models

When a model is connected to documents or tools, failures can come from any stage. In a system using retrieval-augmented generation, a wrong answer may mean the search step found the wrong passage, the right passage was cut off, or the model ignored good evidence. Evaluate each stage separately: did retrieval return the right sources, and did the model use them faithfully? For agents, check whether the right tool was chosen, whether its inputs were correct, and whether the overall task finished within a sensible number of steps.

Safety belongs in the same test set. Include inputs that try to make the system reveal its instructions, follow instructions hidden in a document, or take an action without approval. The NIST AI Risk Management Framework, published in 2023 and widely used by U.S. organizations, organizes this work into four functions: Govern, Map, Measure, and Manage. Evaluation is the Measure function made concrete.

Common Evaluation Mistakes

• Choosing a model from a launch chart alone. Use public scores to build a shortlist, then decide with your own tests.
• Testing with five hand-picked examples. Small, friendly samples make every model look good.
• Judging from a single run. One excellent answer demonstrates possibility, not reliability.
• Changing several things at once. Switch the model or the prompt, not both, so you know what caused the difference.
• Trusting an automated grader without checking it. Compare its decisions with expert judgment on a sample.
• Evaluating once and never again. Model updates can quietly change behavior. Rerun on a schedule and whenever something changes.
• Ignoring cost and latency. The best answer is not useful if it is too slow or too expensive at your volume.

What Evaluation Teaches You About AI Visibility

The same principles apply when the question is not "Which model should we use?" but "What do AI engines say about our brand?"

Many teams check their AI visibility the way people read launch charts: someone types one prompt into one AI engine, sees the brand mentioned or missing, and draws a conclusion. That is a single run on a single example with no saved baseline. It proves almost nothing.

A sound visibility measurement looks exactly like a sound model evaluation:

• A defined prompt set. Write down the questions real buyers ask, including "best for," "alternatives to," and comparison prompts, as described in our guide to tracking LLM recommendations.
• Repeated, dated runs. Record the exact prompt, the engine, the date, and the verbatim answer, then repeat on a schedule, because answers vary between runs and change as engines update.
• Criteria set in advance. Decide what counts as a mention, a recommendation, an accurate description, and a cited source before you look at the answers.
• Separated stages. When an engine searches the web before answering, check whether it found your pages at all, then whether it described them accurately. A crawling problem and a content problem need different fixes, as our guide to structuring your website for LLM crawlers explains.
• Honest unknowns. If a surface was not tested, report it as not tested, not as zero.

This is the method behind our observation articles: documented prompts, named engines, dates, and verbatim answers, repeated over time. If you suspect AI engines are leaving your product out of buyer answers, our article on why your brand is invisible in ChatGPT covers the common causes.

A 30-Day Plan to Start Evaluating

Week 1: Pick one task. Choose the AI use case that matters most to your team. Write the task description and five to eight success criteria.

Week 2: Build the set. Collect 30 to 50 real, cleaned examples, including difficult and refusal cases, with notes on what each correct answer must contain.

Week 3: Run and compare. Test two or three candidate models or prompt versions, each at least three times per input. Record pass rates, cost, and speed. Have an expert check a sample of the grading.

Week 4: Decide and save. Choose based on results, document why, and save the full evaluation with dates and versions. Then apply the same discipline outward: run ten buyer questions about your category across the AI engines your buyers use, and save the answers as your first visibility baseline.

The Bottom Line

Benchmarks tell you which models are broadly capable. Human-preference leaderboards tell you which answers people tend to like. Neither tells you which model will do your job well, at your volume, within your budget. That answer comes only from a small, specific evaluation built from your own tasks, graded against criteria you wrote in advance, run more than once, and saved so you can repeat it.

The same habits, defined prompts, repeated dated runs, clear criteria, and honest unknowns, are what separate real AI visibility measurement from anecdotes. If you want to see how AI engines answer the questions your buyers actually ask, and what evidence they are missing about your brand, book a strategy call with LLM Recommend.

Frequently Asked Questions

What is an LLM benchmark?

A benchmark is a standardized set of questions or tasks with known answers, scored automatically, used to compare language models. Examples include MMLU for broad knowledge, GPQA for expert-level science questions, TruthfulQA for resisting misconceptions, and SWE-bench for fixing real software issues.

Why do benchmark scores not match my experience with a model?

Benchmarks measure narrow, often multiple-choice skills under specific setups, and some questions may have appeared in training data. Your work is usually open-ended and depends on your prompts, data, cost limits, and quality standards, which public scores do not capture.

Are leaderboards like Chatbot Arena reliable?

They are a useful signal of general helpfulness because they use fresh prompts and real human votes. But voters are not your customers, people often favor longer and more confident answers, and preference is not the same as correctness in your domain.

Can I use an AI model to grade another AI model?

Yes, with care. Research found strong model graders agree with human preferences more than 80 percent of the time, but they show position, length, and self-preference biases. Use a specific rubric, swap answer order, and regularly check a sample against expert judgment.

How many examples do I need to evaluate a model for my business?

Thirty to one hundred realistic examples is a practical start for one task, including difficult and refusal cases. Run each candidate at least three times per example and record pass rates per criterion, plus cost and response time.

How does LLM evaluation relate to my brand's visibility in AI answers?

Measuring AI visibility needs the same discipline: a defined set of buyer prompts, repeated dated runs across named engines, criteria set in advance, saved verbatim answers, and untested surfaces reported as not tested rather than zero. A single prompt on a single day proves very little.