Scientific Agents

An arXiv study evaluated an open-source corpus called Scientific Agents, which contains 503 occupation-specific AGENTS.md profile files.

The study concludes that for the models and tasks tested, loading the full occupational profile by default neither improves accuracy nor reduces cost—in fact, it significantly raises costs.

The study compared matched occupational profiles against four control conditions: a minimal baseline, a role-opening sentence, a general scientific rigor guideline, and a cross-domain mismatched profile.

The evaluation covered nine text-based scientific benchmarks, sampling 4,531 questions and 100 matched profiles, with 4,488 items completing all five conditions in full.

The mean accuracy difference of occupational profiles relative to the minimal baseline was -0.6 percentage points, with a 95% bootstrap interval of [-1.5, +0.2].

Matched profiles generated 1.5 to 2.3 times the number of output tokens compared to the baseline, and the cost per successful call was 2.2 to 4.5 times higher than the baseline.

On the BioMysteryBench bioinformatics task, the average problem-solving rate with occupational profiles was 46.7%, while the minimal baseline reached 56.7%, a gap of -10.0 percentage points (95% interval [-16.7, -3.3]).

On the SuperGPQA benchmark, the minimal baseline answered only 54.0% of questions correctly on the first attempt, while occupational profiles reached 71.6%, but this advantage stems from prompt length or format rather than domain expertise.

The study's conclusions are based on the specific model Gemini 3.8 Flash and closed-ended evaluation tasks; whether selectively retrieving partial sections of profiles or facing open-ended scientific tasks would change the conclusions remains to be verified.

Before adding complexity to a prompt, its usefulness should first be demonstrated through controlled experiments; defaulting to stacking occupational profiles amplifies token consumption and cost.