RefCraft Benchmark: Visual Reference-Driven AI Web Generation Scores Only 52

How can AI agents leverage visual references in web development? A recent study shows that even the strongest models score only 52.

Research Background: From Idealized to Real-World Workflows

Existing web development benchmarks largely oversimplify real-world scenarios: they either only require generating pages from text instructions or reconstructing a given target screenshot. However, a more common real-world workflow is that a developer receives several reference webpage screenshots, combines them with natural language requirements, and creatively develops a new page. This "visual-reference-conditioned generation" capability is crucial for AI agents to be deployed in productivity scenarios.

According to the research paper RefCraft: Benchmarking Visual-Reference-Conditioned Agentic Generation through Web Development, current benchmarks overlook this common workflow. To address this, the researchers proposed RefCraft, a benchmark containing 300 real multimodal web development tasks, each combining natural language instructions with 1 to 5 reference webpage screenshots. Agents need to selectively interpret and utilize these references while rendering, visually inspecting, and iteratively refining.

Visual-Reference-Conditioned Generation refers to models generating content not only based on text instructions but also extracting visual information (such as layout, color schemes, component styles, interaction patterns) from one or more reference images and selectively incorporating it into the output. Unlike "screenshot reconstruction" tasks, the reference images are not targets to be reproduced pixel by pixel but sources of design inspiration; the model must determine which elements can be borrowed and which need innovation based on text requirements. This capability requires comprehensive abilities such as cross-modal alignment, disentanglement and recombination of visual elements, and intent understanding—a key step for agents to evolve from "code generators" to "collaborative developers."

RefCraft Benchmark Design: Multimodal Tasks and Comprehensive Evaluation

The core of RefCraft lies in simulating the complexity of real development scenarios. Each task provides text instructions and visual references, requiring the agent to not only understand the instructions but also extract design elements, layout logic, or interaction patterns from the reference screenshots and apply them to the generation of a new page. The agent needs to repeatedly render outputs, visually compare, and adjust code, just like a human developer.

For evaluation, the research team adopted independently verifiable keypoints along with complementary programmatic judging and vision-language model (VLM) judging. This combination ensures both objectivity in evaluation and consideration of the subjective quality of visual outputs.

Keypoints evaluation is a method that decomposes open-ended generation tasks into several independently verifiable check items. Each keypoint corresponds to a specific functional or visual requirement (such as "navigation bar contains a search box" or "button changes color on hover"), and each is judged individually, transforming subjective overall quality into a quantifiable pass rate. Programmatic judging verifies keypoints through automated scripts (such as DOM structure checks, screenshot pixel comparison, interaction simulation); Vision-language model (VLM) judging uses multimodal large models to score rendered results at the semantic and aesthetic levels. The two complement each other: programmatic judging is objective and reproducible, while VLM judging can capture dimensions that are difficult to formalize, such as layout harmony and visual consistency, together forming a comprehensive measure of agent generation quality.

Evaluation of Nine Frontier Models: Top Score Only 52.12

The researchers tested nine frontier (multimodal) large language models. The results showed that the strongest system scored only 52.12 out of 100. This result indicates that visual-reference-conditioned agentic generation remains a significant challenge for current models. Models perform poorly in understanding reference intent, selectively applying visual elements, and iterative refinement, leaving a clear gap from practical productivity applications.

Agentic iterative refinement refers to the loop process in which the model, after generating an initial page, gradually approaches the target by rendering screenshots, visually comparing, locating discrepancies, and modifying code. This process relies on two core capabilities: first, utilization of visual feedback, i.e., whether the model can accurately identify deviations from reference images or requirements (such as improper spacing, color mismatch, misaligned elements) from rendered results; second, self-correction strategies, i.e., generating effective code modifications based on deviations rather than blindly retrying. RefCraft's task design mandates multiple rounds of iteration, yet existing models often struggle to extract effective signals from visual feedback after the first round, resulting in limited iteration gains—one of the important reasons for the low scores.

Challenges and Insights: Visual Reference Understanding Is the Key Bottleneck

The low scores of RefCraft reveal several key issues. First, models struggle to extract and fuse relevant information from multiple reference screenshots, often over-relying on text instructions or simply copying references. Second, in the iterative refinement phase, models lack the ability to effectively utilize visual feedback, leading to deviations between output and expectations. Additionally, evaluating open-ended generation tasks is inherently challenging, but RefCraft provides a reproducible measurement standard through keypoints and dual judging mechanisms.

This research reminds us that for AI agents to truly become productivity tools, improving code generation alone is insufficient; multimodal understanding and integration of visual references must also be strengthened. In the future, with the promotion of the benchmark and iteration of models, breakthroughs in this field are expected.

For more details, please refer to the original paper.

One-line Summary

New benchmark RefCraft reveals that AI agents score only 52 in visual-reference web development, with multimodal understanding as the bottleneck.

Paragraph Summary

The research paper proposes the RefCraft benchmark, containing 300 real web development tasks that combine natural language instructions with 1-5 reference screenshots, simulating the workflow of developers creatively developing with design references. Evaluation combines keypoints, programmatic judging, and VLM judging. Testing nine frontier multimodal large models shows a top score of only 52.12 (out of 100), indicating that visual-reference-conditioned agentic generation remains a significant challenge. Models underperform in extracting and fusing information from multiple references and utilizing visual feedback for iterative refinement. The study points out that enhancing multimodal understanding and visual reference integration is key for AI agents to become productivity tools.