Inter-Run Variance
A phenomenon in semantic query systems where the same query produces different results across multiple executions. This can occur even under deterministic parameter settings, originating from non-determinism in hardware floating-point operations and implementation details of serving frameworks.
Timeline (last 90 days)
In many evaluations conducted on small datasets, scores fluctuate by several percentage points across different runs.
A study based on 1.2 million model calls systematically quantified the inter-run variance of semantic query engines
The rows that change are precisely the rows on which models disagree with one another
All Facts (3)
In many evaluations conducted on small datasets, scores fluctuate by several percentage points across different runs.
50%UnverifiedA study based on 1.2 million model calls systematically quantified the inter-run variance of semantic query engines
50%UnverifiedThe rows that change are precisely the rows on which models disagree with one another
50%