Deterministic vs Probabilistic Evals

Deterministic evaluations use fixed, rule-based criteria with binary pass/fail outcomes (does the output match the expected regex? does the code compile? do all tests pass?), while probabilistic evaluations use statistical methods or LLM-as-judge to assess quality on a spectrum (is this response helpful? is this code well-structured?). The key architectural decision is which type to use for each dimension of quality: deterministic evals are reliable, reproducible, and cheap to run, but can only measure what can be reduced to a rule; probabilistic evals can capture nuanced quality dimensions like helpfulness, correctness of reasoning, and code quality, but they introduce variance that requires larger sample sizes for confident conclusions. The best eval suites layer both: deterministic checks as fast guardrails (format validation, type checking, regression tests) and probabilistic assessments for the subtler quality dimensions that determine user satisfaction. Understanding this distinction prevents the common mistake of either over-relying on vibes-based assessment (pure probabilistic) or missing important quality signals that can't be reduced to rules (pure deterministic). This concept connects to eval frameworks for the tooling that runs both types, quality metrics for defining what each eval type measures, regression testing for the primarily deterministic eval category, and the LLM-as-judge subtopic for the most common probabilistic eval approach.