3DCodeEdit
A benchmark for evaluating the ability of language models to edit Blender 3D program code through natural language instructions. It contains 2,592 tasks covering parameter modification, structural changes, stylistic redesign and scene composition, and judges rendered results against human-checked lists.
Timeline (last 90 days)
The researchers introduced 3DCodeEdit, a benchmark containing 2,592 instruction-guided editing tasks, covering 108 Blender programs in which objects are built entirely with code.
The task types of 3DCodeEdit include parameter modification, structural change, style redesign, and scene composition.
3DCodeEdit is judged starting from the rendered results, against human-reviewed checklists.
The researchers tested 17 models on 3DCodeEdit, and the average success rate was only 37%.
Even the best-performing models achieved a completion rate of less than 60% on structural editing and style editing.
In the multi-turn feedback experiments of 3DCodeEdit, feedback from an independent reviewer model (describing the differences between the rendered result and the request) worked best, outperforming increasing reasoning effort and using agent frameworks.
The 3DCodeEdit benchmark exposes the general shortcomings of current models in structural editing and style editing.
All Facts (7)
The researchers tested 17 models on 3DCodeEdit, and the average success rate was only 37%.
50%UnverifiedThe task types of 3DCodeEdit include parameter modification, structural change, style redesign, and scene composition.
50%Unverified3DCodeEdit is judged starting from the rendered results, against human-reviewed checklists.
50%UnverifiedThe 3DCodeEdit benchmark exposes the general shortcomings of current models in structural editing and style editing.
50%UnverifiedEven the best-performing models achieved a completion rate of less than 60% on structural editing and style editing.
50%UnverifiedIn the multi-turn feedback experiments of 3DCodeEdit, feedback from an independent reviewer model (describing the differences between the rendered result and the request) worked best, outperforming increasing reasoning effort and using agent frameworks.
50%UnverifiedThe researchers introduced 3DCodeEdit, a benchmark containing 2,592 instruction-guided editing tasks, covering 108 Blender programs in which objects are built entirely with code.
50%