Chinese Online Review Pragmatic Inference Benchmark
The first large-scale benchmark for pragmatic inference in Chinese online reviews, comprising 4,735 human-verified diagnostic test items. It is designed to evaluate large language models' understanding of pragmatic phenomena in Chinese social media comments, including indirectness, sarcasm, and banter.
Core Facts
Timeline (last 90 days)
The study constructed the first large-scale benchmark for pragmatic reasoning on Chinese online reviews.
The research team screened more than 200,000 publicly available Chinese social media interaction records and ultimately constructed a diagnostic test set of 4,735 human-verified items.
Each test item consists of three parts: a target review, a reconstructed context, and plausible misreading options.
The study conducted a dual evaluation of 8 large language models, having them serve both as question setters generating misleading options and as answer solvers resolving the problems.
Under the leave-one-author-out cross-validation setting, the strongest model achieved an accuracy of 81.42%.
The human baseline accuracy is 90.8%.
The study adopts a design that uses plausible misreadings as distractors, requiring models to select the correct answer from multiple seemingly reasonable interpretations rather than making a simple binary classification judgment.
All Facts (7)
Each test item consists of three parts: a target review, a reconstructed context, and plausible misreading options.
80%UnverifiedThe research team screened more than 200,000 publicly available Chinese social media interaction records and ultimately constructed a diagnostic test set of 4,735 human-verified items.
50%UnverifiedThe study constructed the first large-scale benchmark for pragmatic reasoning on Chinese online reviews.
50%UnverifiedUnder the leave-one-author-out cross-validation setting, the strongest model achieved an accuracy of 81.42%.
50%UnverifiedThe human baseline accuracy is 90.8%.
50%UnverifiedThe study adopts a design that uses plausible misreadings as distractors, requiring models to select the correct answer from multiple seemingly reasonable interpretations rather than making a simple binary classification judgment.
50%UnverifiedThe study conducted a dual evaluation of 8 large language models, having them serve both as question setters generating misleading options and as answer solvers resolving the problems.
50%