SCF
An evidence aggregation method for user inference tasks on language models, based on Fisher's combined probability test. By truncating document-level evidence and averaging within sources, it determines whether a given data source participated in model training.
Timeline (last 90 days)
SCF's accompanying calibrated decision rule uses another set of known non-participating sources to calibrate the p-values, thereby testing the source participation hypothesis.
Under the source exchangeability assumption, SCF's calibrated decision rule provides finite-sample Type-I error control.
The researchers conducted experiments on four real-world datasets covering four different domains; the results show that among the evaluated aggregation methods, SCF achieved the highest AUROC in all four domains.
Combined with the calibrated decision rule, SCF consistently provides competitive detection capability across the tested settings, achieving the highest power in many cases, while the empirical false positive rate is typically near or below the nominal level.
Discussions of the related work by Bilibili UP hosts and YouTube channels both point out that SCF's capping mechanism effectively mitigates the influence of extreme document scores, while the calibration rule ensures statistical rigor.
Some discussions raise questions about SCF's performance at extremely small source sizes (e.g., only 2-3 documents), suggesting that capping may excessively weaken the evidence; but the original experiments did not specifically test this edge case.
All Facts (6)
SCF's accompanying calibrated decision rule uses another set of known non-participating sources to calibrate the p-values, thereby testing the source participation hypothesis.
50%UnverifiedUnder the source exchangeability assumption, SCF's calibrated decision rule provides finite-sample Type-I error control.
50%UnverifiedThe researchers conducted experiments on four real-world datasets covering four different domains; the results show that among the evaluated aggregation methods, SCF achieved the highest AUROC in all four domains.
50%UnverifiedCombined with the calibrated decision rule, SCF consistently provides competitive detection capability across the tested settings, achieving the highest power in many cases, while the empirical false positive rate is typically near or below the nominal level.
50%UnverifiedDiscussions of the related work by Bilibili UP hosts and YouTube channels both point out that SCF's capping mechanism effectively mitigates the influence of extreme document scores, while the calibration rule ensures statistical rigor.
50%UnverifiedSome discussions raise questions about SCF's performance at extremely small source sizes (e.g., only 2-3 documents), suggesting that capping may excessively weaken the evidence; but the original experiments did not specifically test this edge case.
50%