AesCode: A New Paradigm for Aesthetic Code Generation with Decoupled Cross-Modal Rewards

The Aesthetic Dilemma of Code Generation

Getting AI to generate information-dense visual artifacts like slides, posters, or dashboards has always involved a core contradiction: code models can produce structured, editable, verifiable code, but they "can't see" how layouts and colors ultimately come together on the canvas. This has long limited them in spatial planning and aesthetic quality.

The other path is to use image generators to directly produce visual output. They excel at design but often fail at content rendering—garbled text, misplaced icons, missing elements. Treating such unreliable outputs as targets for direct copying is highly risky.

The paper "AesCode: Aesthetic Code Generation with Decoupled Cross-Modal Rewards" proposes a third path: use images generated from the same prompt as aesthetic guidance, but still deliver code as the final output.

Design Graph: Making Every Attribute Attributable

AesCode's core approach is to represent the target as a design graph. This graph describes the entire canvas and makes every attribute attributable—who decided the position, who decided the color scheme, who decided the hierarchy—all can be traced back to specific graph elements and relationships.

The advantage of this representation is that it breaks down a vague "does it look good" question into an actionable set of attributes. Attributes that code can directly parse and render are scored by a deterministic verifier; attributes that are hard to parse at the code level are judged by a vision-language model (VLM).

Decoupled Cross-Modal Reward Mechanism

AesCode's most critical innovation is the design of complementary cross-modal rewards:

• Deterministic verifier: responsible for evaluating attributes that can be parsed from code and rendering, such as whether elements are aligned, whether sizes match, whether content is correctly presented. These rewards are verifiable and reproducible.
• VLM judge: responsible for evaluating attributes that cannot be parsed. It uses a sample-level Visual Graph Rubric (VGR) bound to design graph elements and relationships to judge aesthetic quality.

After being decoupled, the two types of rewards are jointly optimized through reinforcement learning with per-channel normalization, thereby simultaneously improving spatial planning, aesthetic quality, and verifiable correctness.

The value of this decoupled design is that it avoids using a single reward signal to simultaneously constrain "is the content correct" and "does it look good"—these two things are fundamentally different, and mixing them together easily leads to sacrificing one for the other.

Per-channel normalization is a technique in reinforcement learning for handling multi-objective rewards. When a model simultaneously optimizes multiple reward signals (such as a deterministic verifier and a VLM judge), the numerical scales of different rewards may differ greatly—for example, alignment scores range from 0 to 1, while aesthetic scores may range from 0 to 10. If directly weighted and summed, rewards with larger scales will dominate gradient updates, causing the model to ignore objectives with smaller scales. Per-channel normalization subtracts the mean and divides by the standard deviation for each reward channel separately, mapping them to comparable scales before joint optimization. This makes rewards from different sources contribute more evenly during training, avoiding one dimension being suppressed.

VLM (Vision-Language Model) is a type of model that can understand both images and text, with typical representatives including GPT-4V, Qwen-VL, etc. In AesCode, the VLM judge receives the rendered canvas image and design graph information, and outputs an aesthetic quality score. It excels at perceiving attributes that humans can perceive but code struggles to quantify, such as visual balance, color harmony, and whitespace rhythm. However, VLM judging also has limitations: scores may be affected by prompt wording, and multiple scores for the same image may not be completely consistent, so it needs to be used complementarily with the deterministic verifier.

Benchmark: Scoring Correctness and Visual Quality Separately

To evaluate this setting, the researchers built a new benchmark that scores verifiable correctness and visual quality separately. The visual scores are highly consistent with human judgments, indicating that this automatic scoring standard can reflect people's true feelings well.

There are two notable findings in the experiments:

1. Reference images significantly improve the visual quality of code generated by existing models. Even for existing models, as long as they are given a reference image, visual quality improves markedly.
2. Ablation experiments show that AesCode benefits from reference images in two ways: both as model input and as a source of visual reward. This means the reference image is not just a "take a look" hint, but can also participate in constructing the training signal.

Scoring verifiable correctness and visual quality separately corresponds to two fundamentally different objectives in code-generated visual artifacts. Verifiable correctness focuses on objective facts: whether text content is consistent with the input, whether the number of elements is correct, and whether the layout satisfies hard constraints (such as no overlap and no out-of-bounds). Such metrics can be automatically determined by deterministic rules or parsers without subjective judgment. Visual quality involves aesthetic preferences: whether the color scheme is harmonious, whether the hierarchy is clear, and whether the overall look is beautiful, which must rely on human ratings or VLM approximations of human judgment.

The significance of separating the two is to avoid samples that are "correct but ugly" or "beautiful but wrong or missing content" being concealed by a single score. For example, if a poster has beautiful colors but a typo in the title, and only a composite score is given, the model may mistakenly think it is on the right track. After separating the scores, both training and evaluation can identify specific weaknesses, which is also an echo of AesCode's decoupled reward mechanism at the evaluation level.

Performance Comparison: Approaching Closed-Source Frontier Models

Under the same reference image conditions, compared with a Qwen3-VL baseline of the same scale:

• AesCode-8B improved visual quality by 31.1 percentage points
• AesCode-32B improved by 22.4 percentage points

More importantly, the scores of these two models are already comparable to closed-source frontier models such as GPT-5.5 and Claude Opus 4.8. For a model at the open-source scale, this result is quite convincing.

Qwen3-VL baseline refers to a Qwen3-VL model of the same scale and architectural foundation as AesCode, but without AesCode's reward optimization training. Choosing a same-scale baseline for comparison is to exclude the advantage brought by differences in model parameter count and purely measure the gain from AesCode's training method. The improvements of 31.1 and 22.4 percentage points mean that under the same reference image input conditions, the model optimized with decoupled cross-modal rewards greatly surpasses the original baseline in visual quality.

GPT-5.5, Claude Opus 4.8 represent current closed-source frontier multimodal models, usually with larger parameter counts and training resources far exceeding the open-source scale. The fact that AesCode-8B and AesCode-32B can be comparable to them in visual quality scores shows that improvements in training paradigms can approach top-tier results without blindly expanding parameter counts, which is especially important for the open-source community and resource-constrained researchers.

Why This Matters

The route of code-generated visual artifacts essentially turns "design" into "editable, verifiable structured content." AesCode's contribution is not just raising scores, but providing a reusable idea:

When a task contains both objective correctness and subjective aesthetics, do not try to unify them with a single reward. Splitting them apart, modeling them separately, and then jointly optimizing them makes it easier for the model to do well in both dimensions.

For application scenarios that need to produce presentation decks, report covers, and data dashboards in bulk, this paradigm of "using images as aesthetic guidance and code as the deliverable" may be more practical than pure image generation or pure code generation.

Limitations and Outlook

From the paper information, AesCode relies on generating reference images from the same prompt, and the quality of this step itself affects the final result. In addition, the introduction of the VLM judge also means that training and evaluation costs are not low. How the design graph representation is constructed and the generalization ability of VGR are directions worth observing in the future.

However, based on the results disclosed so far, decoupling verifiable correctness from aesthetic quality has already provided a solid technical starting point for aesthetic code generation.

The construction of the design graph currently relies on parsing elements and relationships from prompts or reference images. If the prompt is vague or the reference image quality is low, the design graph may lack key attributes, resulting in incomplete reward signals. In addition, the schema design of the design graph—including which element types, relationships, and attributes are supported—directly affects generalization ability. For new layout types unseen during training, the design graph representation and VGR scoring standard may not accurately cover them, which is a gap that must be crossed from benchmarks to real-world applications.

VGR (Visual Graph Rubric) binds scoring standards to the elements and relationships of the design graph, making VLM judging more targeted. But VGR itself needs to be manually designed or induced from data, and its quality determines the upper limit of VLM judging. If VGR omits some aesthetic dimension, the model will not optimize in that direction. How to automatically generate or adaptively extend VGR in the future is one of the key directions for improving generalization.

One-line Summary

AesCode uses images from the same prompt as aesthetic guidance and delivers code, improving visual quality through decoupled cross-modal rewards.

Paragraph Summary

To address the contradiction that code generation for visual artifacts "cannot see" layout aesthetics while image generation is unreliable in content, AesCode proposes a third path: using images generated from the same prompt as aesthetic guidance while ultimately delivering editable, verifiable code. The core method represents the canvas as an attributable design graph and decouples two complementary rewards—a deterministic verifier evaluates parseable attributes, and a VLM judge evaluates aesthetics based on a visual graph rubric. The two rewards are jointly optimized after per-channel normalization, simultaneously improving spatial planning, aesthetics, and correctness. A new benchmark scores verifiable correctness and visual quality separately. Under same-scale comparison, AesCode-8B improves visual quality by 31.1 percentage points and AesCode-32B by 22.4 percentage points, approaching closed-source frontier models. Ablation experiments show that reference images benefit both as input and as a reward source. Limitations include dependence on reference image quality, high VLM judging cost, and the generalization ability of the design graph and VGR.