Test Set

Test sets in RAG scenarios usually need to cover three types of cases: normal questions (to verify basic capability), edge-case questions (such as multi-hop reasoning and questions requiring synthesis across documents), and adversarial questions (such as cases where the retrieval source itself lacks the answer, or distracting documents are present); the latter two types are most likely to expose defects in the coordination between retrieval and generation.

Training a vision model requires dividing the data into a training set, a validation set, and a test set: the training set is used to update model parameters, the validation set is used to tune hyperparameters and select the model version, and the test set is used to evaluate the model's true performance on unseen data.

When evaluating a model, data is usually split into a training set and a test set, with the test set used to measure the true level of generalization.

The correct practice is to complete all hyperparameter selection on an independent validation set, run the final evaluation on the test set only once, and report the results regardless of outcome.

On a small test set of only one hundred scenarios, scores can swing by several percentage points between different runs, and judgment should be made on the full dataset.