Evaluation of Large Language Models

• NOTE
• LLM Leaderboards
• Classification of LLM Evaluation
  • Commonsense Reasoning Datasets
  • Multi-Hop Reasoning Datasets
• Benchmarks
• How to Calculate a Benchmark
  • A Concrete Example: TruthfulQA
  • The Evaluation Harness
  • Zero-Shot vs. Few-Shot Evaluation
  • Dataset Splits and Leakage
• Takeaways

The landscape of large language model evaluation is rich and sometimes overwhelming. Dozens of leaderboards, hundreds of benchmarks, and no single agreed-upon standard for what “good” means. This post maps out the major categories of evaluation, the key benchmarks within each, and how to actually run them yourself.

NOTE

Stanford CS224U is an excellent resource for building a deeper theoretical foundation in LLM evaluation beyond what is covered here.

LLM Leaderboards

A useful starting point is to see which benchmarks are prominent enough to appear on major leaderboards. The following are the most widely referenced:

1. Hugging Face Open LLM Leaderboard
2. LMSYS Chatbot Arena — human preference-based ranking
3. Can AI Code — coding-specific evaluation

For a comprehensive academic treatment, the survey paper Evaluating Large Language Models: A Comprehensive Survey covers the full taxonomy in depth.

Classification of LLM Evaluation

LLM evaluation broadly splits into two axes: what is being evaluated (knowledge, reasoning, alignment, safety, etc.) and how it is evaluated (automatic scoring, human judgment, LLM-as-judge).

The knowledge and capability progression across model generations:

Commonsense Reasoning Datasets

Multi-Hop Reasoning Datasets

Benchmarks

Once you have a dataset, you need a benchmark — a standardized procedure that produces a comparable score. The following are the most commonly used:

For holistic evaluation across many tasks simultaneously:

How to Calculate a Benchmark

Understanding the mechanics of benchmark calculation matters — especially when you need to trust the numbers or reproduce them.

A Concrete Example: TruthfulQA

The TruthfulQA dataset is designed to evaluate whether a model generates factually accurate answers rather than plausible-sounding ones. It is one of the six benchmarks that appear on the Hugging Face Open LLM Leaderboard.

The Evaluation Harness

Rather than implementing evaluation from scratch for each benchmark, the standard tool is lm-evaluation-harness by EleutherAI. It provides a unified interface for running a model against hundreds of benchmarks with consistent prompting and scoring.

Zero-Shot vs. Few-Shot Evaluation

The evaluation protocol — zero-shot or few-shot — significantly affects scores and must be reported alongside results:

• Few-shot: The model is primed with k examples before the test question. A prompt suffix like "thus, the choice is:" can steer the model toward a structured answer format (e.g., A, B, C).
• Zero-shot: No examples are provided. The model must interpret the task from the question alone. Multiple prompts are sometimes needed: one to elicit a free-form response, another to coerce it into a parseable answer.

Dataset Splits and Leakage

Pay attention to train/val/test splits. For benchmarks like HellaSwag, models should be fine-tuned only on train and val, with the test set held out strictly for evaluation. Contamination — where test data appears in the pretraining corpus — is a known problem with many public benchmarks and is an active area of research.

Takeaways

Benchmarks are necessary but imperfect. A few principles worth keeping in mind:

• No single benchmark is sufficient. Use a diverse set covering different capabilities and domains.
• Report the evaluation protocol. Zero-shot vs. few-shot, prompt format, and framework version all affect results.
• Be skeptical of leaderboard rankings. Many benchmarks have been saturated or are potentially contaminated in large pretraining corpora.
• Human evaluation remains the gold standard for open-ended generation quality — Chatbot Arena’s Elo-based approach is the closest thing to a reliable real-world signal.