LLM Evals: Everything You Need to Know – Hamel’s Blog - Hamel Husain
A comprehensive guide to LLM evals, drawn from questions asked in our popular course on AI Evals. Covers everything from basic to advanced topics.
Using LLM-as-a-Judge For Evaluation: A Complete Guide – Hamel's Blog - Hamel Husain
hamel.devSaved February 18, 2026
Summary
This guide outlines a 7-step process for building LLM-as-a-Judge systems to evaluate AI outputs.
Steps include finding a domain expert, creating diverse datasets of real or synthetic user interactions, directing the expert for pass/fail judgments with critiques, fixing errors, iteratively building the judge prompt, performing error analysis, and creating specialized judges. It covers dataset structuring, prompt optimization, error rates on unseen data, and an FAQ on models, fine-tuning, and scaling.
