LLM Evaluation
A methodology for assessing the performance of large language models on specific tasks or business scenarios. It encompasses metric definition, dataset construction, offline and online evaluation, and continuous monitoring.
Core Facts
Timeline (last 90 days)
The usable resolution of LLM assessment scores is about 0.5 points, not 0.1 points.
In LLM assessment, comparing differences at 0.1-point precision such as 8.3 versus 8.7 is noise and meaningless.
The difference between the "9-point band" and the "7-point band" is meaningful in LLM assessment; the digits after the decimal point are not.
The core problem that LLM evaluation addresses is how to systematically measure output quality when model outputs are open-ended natural language, with common dimensions including accuracy, faithfulness, relevance, and safety.
LLM evaluation engineering typically relies on frameworks such as RAGAS and DeepEval, combined with human annotation or using another LLM as a judge for scoring.
AI evaluation targets are divided into three layers—large language model evaluation, knowledge base (RAG) evaluation, and agent (Agent) evaluation—forming a progressive relationship of increasing complexity and knowledge depth.
All Facts (6)
AI evaluation targets are divided into three layers—large language model evaluation, knowledge base (RAG) evaluation, and agent (Agent) evaluation—forming a progressive relationship of increasing complexity and knowledge depth.
80%UnverifiedThe usable resolution of LLM assessment scores is about 0.5 points, not 0.1 points.
50%UnverifiedIn LLM assessment, comparing differences at 0.1-point precision such as 8.3 versus 8.7 is noise and meaningless.
50%UnverifiedThe difference between the "9-point band" and the "7-point band" is meaningful in LLM assessment; the digits after the decimal point are not.
50%UnverifiedThe core problem that LLM evaluation addresses is how to systematically measure output quality when model outputs are open-ended natural language, with common dimensions including accuracy, faithfulness, relevance, and safety.
50%UnverifiedLLM evaluation engineering typically relies on frameworks such as RAGAS and DeepEval, combined with human annotation or using another LLM as a judge for scoring.
50%