Pelican Test
A large language model evaluation method that prompts a model to generate code for rendering an image of a pelican riding a bicycle. The benchmark simultaneously assesses spatial reasoning, biological knowledge, understanding of mechanical structures, and code generation capability.
Timeline (last 90 days)
The pelican test is a composite evaluation combining code generation and visual reasoning, typically using Python with drawing libraries such as matplotlib, turtle, or SVG.
The result code from the first test in minimal mode placed certain elements in unreasonable positions; the overall execution worked but the detail handling was not refined.
When conducting pure capability evaluations, it is recommended to disable search to avoid results being influenced by external information.
Regardless of the mode, complex spatial relationship details such as limb contact between the pelican and the bicycle and wheel structure are prone to errors.
The "draw a pelican riding a bicycle" test was originally proposed by developer Simon Willison to evaluate the capability boundaries of large language models and image generation models.
All Facts (4)
The "draw a pelican riding a bicycle" test was originally proposed by developer Simon Willison to evaluate the capability boundaries of large language models and image generation models.
60%UnverifiedThe pelican test is a composite evaluation combining code generation and visual reasoning, typically using Python with drawing libraries such as matplotlib, turtle, or SVG.
50%UnverifiedWhen conducting pure capability evaluations, it is recommended to disable search to avoid results being influenced by external information.
50%UnverifiedRegardless of the mode, complex spatial relationship details such as limb contact between the pelican and the bicycle and wheel structure are prone to errors.
50%