Types of evals for AI products, sorted by judge

The main LLM evaluation methods, or types of evals, differ in who does the judging: code, a decision model, an LLM judge, or people. Which one you need depends on what's being judged, how often, and whose idea of good the score should reflect.

Most guides list three types of evals (code-based checks, LLM-as-a-judge and human evaluation) and split them once more into offline and online. That list is missing a grader that arrived in September 2026, the decision model. It treats "human evaluation" as one thing, when asking your team, a small panel or your users at scale gets you three different measurements. It also rarely asks what's being judged, a single output or the experience a person had.

Code-based evals

A code-based eval is a function that checks something exactly. Did the response parse as valid JSON? Did the agent call the refund tool before issuing the refund? Is the answer under the length limit? Does it contain a phrase you've banned? Code checks are exact, free to run and immune to the biases that affect model judges, so they should cover everything they can.

Their limit is that the right answer has to be writable in advance. A function can confirm that an agent called the weather tool. It can't easily tell whether the agent used the result to answer the question well, because many answers would be acceptable.

Decision-model evals

Decision models are the newest type. Jev, launched by TypeSafe AI in early access on September 15, 2026, is the one eval tools have started to support. It answers questions whose possible answers you define in advance (yes or no, pick one of several, or place on a scale), returns a probability with each answer, and never generates text.

In the first benchmarks, Jev matched an LLM judge on closed checks where the evidence is in hand, such as whether an answer sticks to its source document, and its confidence was better calibrated. It cost about two cents per thousand judgments and answered in 171 milliseconds at the median. It fell behind on criteria that need step-by-step reasoning, like checking a derivation, or a close read of a long text. That profile makes decision models the right grader for closed criteria at high volume, including live traffic, where an LLM judge would be too slow or too expensive. DeepEval, Langfuse and Mastra support Jev today, and LangChain has tested it through an alpha integration.

LLM-as-a-judge evals

An LLM judge is a language model prompted with your rubric and asked to grade. It handles what code can't express and a decision model can't answer, such as open-ended quality, step-by-step reasoning over a derivation or a long document, and anything where someone needs to read why a response failed. Strong LLM judges agreed with expert and crowd preferences more than 80% of the time, about as often as the people agreed with each other.

The costs are latency, price and a set of known weaknesses. LLM judges lean toward longer answers and toward whichever answer comes first, and they can be confidently wrong. Those weaknesses get managed the same way, by measuring the judge against human labels before trusting it, and again whenever the rubric or the model changes.

Human evals come in three kinds

Most taxonomies file human evaluation as one type and describe it as domain experts reviewing outputs. Which humans you ask decides whose answer key you're grading against. In practice, teams climb through three kinds, and each one replaces more of the team's guesses about the rubric and the test scenarios with evidence.

Your team. Engineers write the first checks and PMs draft the first rubric and scenarios, then both read traces, dogfood the product and do error analysis. Nobody is faster at spotting broken behavior, and every eval practice should start here, because error analysis is how you figure out what to measure in the first place. But your team knows what the system was built to do, so its rubric and scenarios are educated guesses about users, graded against the builder's intent.

A small panel of users and experts. The next step is to show the draft rubric and scenarios to a handful of outsiders: design partners, a few friendly customers, and a domain expert when correctness needs credentials, like a legal or medical answer. They grade a sample of outputs, flag criteria that don't matter to them, and name tasks the team never thought to test. It's cheap and it catches the biggest misreads. It's still a handful of people, so it can't tell you how often each scenario comes up across your users, or whether this panel's taste is typical.

Your users, at scale. The last step moves user input upstream and makes it big enough to trust. Hundreds of people recruited to match your customers use the product on real tasks before launch, on camera, and explain why they graded each experience the way they did. Their sessions tell you which scenarios to test and how often each one happens. Their reasons become the rubric, and their grades become the golden set that an automated judge is calibrated to match, once before launch and again on a cadence. They're the only group that knows what users come to do, which makes them the reference for experienced quality.

Few teams reach this step, because it has meant running a research project: sourcing a panel, moderating and recording hundreds of sessions, then coding the transcripts into criteria. Voicepanel runs all three in one place. It recruits from its own consumer panels, screened to your criteria, runs AI-moderated sessions that ask follow-up questions about every grade, and turns what people say into findings your team can build a rubric from.

Offline and online evals

Offline evals run on a fixed dataset before release, in CI or as part of a launch review. Online evals grade samples of live production traffic to catch regressions and drift after release. Every grader type above can run either way, and cost usually decides which ones do. Decision models cut that cost. At a fraction of a cent per judgment, you can score far more of your traffic instead of sampling a sliver of it.

An output or the experience

The last axis is what gets judged. Most evals grade an output, meaning one response, one step or one tool call, against what the builder intended. That's system quality, and it lives in traces and test suites.

Experienced quality asks whether a real person got what they came for, and how it felt along the way. It can only be graded against what that person intended. The unit being graded is a treatment, a snapshot of what the user saw and got, such as a rendered results page, a feed, or a side-by-side of two versions.

The two drift apart for two reasons. Quality is relational, since whether results are good depends on a person's taste and goal, and neither appears in any log. Experiences also fail by composition, because every step can pass its eval while the session as a whole frustrates the person in it. Our guide to AI product quality goes deeper on both.

How the types fit together

A mature eval program uses all six. Code and decision models cover the checkable and the high-volume. LLM judges take the open criteria. Your team drafts the rubric and triages what breaks, and a small panel checks the draft. Your users set the reference that every automated grader is calibrated against. We call that humans once, judges always. People establish what good means, judges measure it on every release, and people recheck the judges when the product or its users change.

Grade builds that last layer for you. We recruit your target users and record them using your product before launch, on hundreds of user scenarios, help you build a rubric from what they say and do, and deliver an autograder calibrated to them. It recalibrates once you're live. Join the Grade waitlist.