TCSExam
A large language model evaluation benchmark for theoretical computer science (TCS), comprising four tracks: theorem proving, solver engineering, construction, and open conjectures. It uses the Lean kernel or executable checkers for hard verification and provides two versions: Full (100 tasks) and Lite (20 tasks).
Timeline (last 90 days)
TCSExam is a unified benchmark covering research capabilities across the full spectrum of theoretical computer science (TCS), comprising four parallel tracks: theorem proving, solver engineering, construction, and direct attempts to resolve open conjectures.
The research team evaluated eight agents on TCSExam-Full and nine agents on TCSExam-Lite.
The TCSExam evaluation process produced improved solver outputs, mathematical constructions, and independently verified special cases and bounds obtained from attempts on open conjectures.
TCSExam is positioned as an extensible framework, and the paper provides task sources, Docker build recipes, and evaluation tools for reproducible use by the community.
In TCSExam, the Lean kernel serves as the hard judge for the theorem-proving track and some construction tasks, while solver engineering and executable construction tasks are verified by actually running task checkers, and together the two form the objectivity basis of dual verification.
All Facts (5)
TCSExam is a unified benchmark covering research capabilities across the full spectrum of theoretical computer science (TCS), comprising four parallel tracks: theorem proving, solver engineering, construction, and direct attempts to resolve open conjectures.
50%UnverifiedThe research team evaluated eight agents on TCSExam-Full and nine agents on TCSExam-Lite.
50%UnverifiedThe TCSExam evaluation process produced improved solver outputs, mathematical constructions, and independently verified special cases and bounds obtained from attempts on open conjectures.
50%UnverifiedTCSExam is positioned as an extensible framework, and the paper provides task sources, Docker build recipes, and evaluation tools for reproducible use by the community.
50%UnverifiedIn TCSExam, the Lean kernel serves as the hard judge for the theorem-proving track and some construction tasks, while solver engineering and executable construction tasks are verified by actually running task checkers, and together the two form the objectivity basis of dual verification.
50%