SWORD
A Systematic Wikidata-based Object-Relation Distortion benchmark that evaluates the cross-lingual fact-checking consistency of large language models through controlled perturbations, using an inverse evaluation approach to test a model's ability to reject false statements.
Timeline (last 90 days)
SWORD's perturbations include two types: random entity replacement and semantically plausible attribute-based replacement.
SWORD generates grammatically fluent but factually incorrect statements by fixing the subject and relation while replacing only the object, encompassing two types of perturbation: random entity replacement and semantically plausible attribute-based replacement
Models exhibit significant performance degradation on East Asian languages (such as Chinese, Japanese, and Korean) when faced with distorted statements, with cross-lingual performance gaps of up to 28 percentage points and relative drops as high as 49%
Adversarial evaluation in the NLP field has evolved from word-level substitution attacks such as TextFooler, to behavioral testing frameworks such as CheckList, and then to systematic factual perturbation methods such as SWORD
SWORD (Systematic Wikidata-based Object-Relation Distortion) is a systematic Wikidata-based object-relation distortion benchmark that evaluates the fact-checking ability of large models by testing whether models can consistently reject false statements.
SWORD research found that model accuracy on semantically plausible distorted statements was actually higher than on meaningless random substitutions.
SWORD researchers believe that models rely on distributional familiarity rather than a genuine fact-verification mechanism.
The evaluation of large models should go beyond the multiple-choice paradigm; testing only whether correct answers are selected is insufficient to measure genuine factual understanding, and adversarial evaluation that tests rejection of incorrect statements must be introduced.
SWORD (Systematic Wikidata-based Object-Relation Distortion) is a fact-checking evaluation benchmark that generates grammatically fluent but factually incorrect statements through controlled perturbations based on Wikidata triples.
The article recommends that when deploying multilingual AI systems, one should build dedicated distortion test sets for target languages, introduce multi-level fact verification pipelines, and set differentiated confidence thresholds for different languages.
7 more timeline events
All Facts (17)
SWORD's perturbations include two types: random entity replacement and semantically plausible attribute-based replacement.
50%UnverifiedSWORD generates grammatically fluent but factually incorrect statements by fixing the subject and relation while replacing only the object, encompassing two types of perturbation: random entity replacement and semantically plausible attribute-based replacement
50%UnverifiedModels exhibit significant performance degradation on East Asian languages (such as Chinese, Japanese, and Korean) when faced with distorted statements, with cross-lingual performance gaps of up to 28 percentage points and relative drops as high as 49%
50%UnverifiedAdversarial evaluation in the NLP field has evolved from word-level substitution attacks such as TextFooler, to behavioral testing frameworks such as CheckList, and then to systematic factual perturbation methods such as SWORD
50%UnverifiedSWORD (Systematic Wikidata-based Object-Relation Distortion) is a systematic Wikidata-based object-relation distortion benchmark that evaluates the fact-checking ability of large models by testing whether models can consistently reject false statements.
50%UnverifiedSWORD research found that model accuracy on semantically plausible distorted statements was actually higher than on meaningless random substitutions.
50%UnverifiedSWORD researchers believe that models rely on distributional familiarity rather than a genuine fact-verification mechanism.
50%UnverifiedThe evaluation of large models should go beyond the multiple-choice paradigm; testing only whether correct answers are selected is insufficient to measure genuine factual understanding, and adversarial evaluation that tests rejection of incorrect statements must be introduced.
50%UnverifiedSWORD (Systematic Wikidata-based Object-Relation Distortion) is a fact-checking evaluation benchmark that generates grammatically fluent but factually incorrect statements through controlled perturbations based on Wikidata triples.
50%UnverifiedThe article recommends that when deploying multilingual AI systems, one should build dedicated distortion test sets for target languages, introduce multi-level fact verification pipelines, and set differentiated confidence thresholds for different languages.
50%UnverifiedSWORD adopts an inverse evaluation approach, testing whether models can consistently reject incorrect statements, rather than the traditional benchmark approach of selecting the correct answer.
50%UnverifiedThe SWORD benchmark covers eight widely used languages to systematically examine models' cross-lingual fact-checking consistency.
50%UnverifiedSWORD's perturbations include two levels: random entity replacement and semantically plausible attribute-based replacement.
50%UnverifiedThe study found that models' accuracy on semantically plausible distorted statements is actually higher than their performance on meaningless random replacements.
50%UnverifiedThe study suggests that models rely on distributional familiarity rather than genuine fact-verification mechanisms to judge the truth or falsity of statements.
50%UnverifiedThe study shows that certain models exhibit significant performance degradation on distorted statements in East Asian languages (Chinese, Japanese, and Korean), with cross-lingual performance gaps of up to 28 percentage points and relative declines as high as 49%.
50%UnverifiedThe study points out that aggregated accuracy metrics systematically obscure asymmetric capabilities in multilingual factual reasoning, which is a manifestation of Simpson's paradox.
50%