RefCraft
A benchmark for evaluating AI agents on visually reference-conditioned web page generation, comprising 300 multimodal tasks that combine natural-language instructions with reference screenshots and are assessed through a combination of key points, programmatic judging and VLM judging.
Timeline (last 90 days)
The research paper "RefCraft: Benchmarking Visual-Reference-Conditioned Agentic Generation through Web Development" proposes the RefCraft benchmark.
The RefCraft benchmark contains 300 real multimodal web development tasks.
Each RefCraft task combines natural language instructions with 1 to 5 reference webpage screenshots.
RefCraft's evaluation employs independently verifiable keypoints, along with complementary programmatic judging and vision-language model (VLM) judging.
The researchers tested nine frontier (multimodal) large language models, and the strongest system scored only 52.12 on a 100-point scale.
Existing models often struggle to extract effective signals from visual feedback after the first-round generation, resulting in limited iterative gains, which is one of the important reasons for RefCraft's low scores.
RefCraft's low scores reveal that models struggle to extract and fuse relevant information from multiple reference screenshots, often relying excessively on text instructions or simply copying the references.
All Facts (7)
The research paper "RefCraft: Benchmarking Visual-Reference-Conditioned Agentic Generation through Web Development" proposes the RefCraft benchmark.
50%UnverifiedThe RefCraft benchmark contains 300 real multimodal web development tasks.
50%UnverifiedEach RefCraft task combines natural language instructions with 1 to 5 reference webpage screenshots.
50%UnverifiedRefCraft's evaluation employs independently verifiable keypoints, along with complementary programmatic judging and vision-language model (VLM) judging.
50%UnverifiedThe researchers tested nine frontier (multimodal) large language models, and the strongest system scored only 52.12 on a 100-point scale.
50%UnverifiedExisting models often struggle to extract effective signals from visual feedback after the first-round generation, resulting in limited iterative gains, which is one of the important reasons for RefCraft's low scores.
50%UnverifiedRefCraft's low scores reveal that models struggle to extract and fuse relevant information from multiple reference screenshots, often relying excessively on text instructions or simply copying the references.
50%