Character-Level Transformer Generates URLs: Most Realistic ≠ Most Useful?

Character-level Transformer generates synthetic URLs to improve phishing detection, but realism and usefulness are not equivalent.
The author trained a 25.5M-parameter character-level Transformer to generate synthetic URLs from scratch to alleviate phishing detection data scarcity. Since traditional AUC evaluation is unreliable, he designed a realism evaluation method that scores by URL field category with real data as a baseline; 29 of 32 checks passed. Preregistered experiments showed that synthetic data improved detector recall at a 1% false positive rate across 6 settings and outperformed simple generators. However, on the PhreshPhish test set, URLs generated by a prompt-driven 7B LLM were more helpful despite having the lowest realism. This unexpected finding suggests that the realism and practical value of generated data may not be consistent, and evaluating synthetic data requires considering both. The project code and evaluation are open-sourced.
Why Do We Need Synthetic URL Data?
Phishing detectors need large amounts of training data, but real phishing data is scarce and quickly goes stale—it can become useless within days. Reddit user youssefismail-dev trained a character-level Transformer specifically to generate synthetic URLs. The model has 25.5M parameters and was trained from scratch, without pretrained models or templates. The project code, weights, and full evaluation have been open-sourced under the Apache-2.0 license for defensive research use.

How to Measure "Realism": A Harder Problem Than Modeling
The author originally wanted to use a classifier to judge the authenticity of generated URLs, targeting an AUC of 0.5. But the reality is: even two real URL samples evaluated against each other can't reach 0.5. So he designed a more detailed evaluation method: score each URL field (domain, suffix, subdomain, path, query string) by category, and calculate its excess value relative to a "real vs. real" baseline. Of 32 checks, 29 passed. However, the model still has clear weaknesses: a detector that checks whether each word appears in real data can identify words fabricated by the model; in free mode, phishing query strings can also be detected.
AUC (Area Under the Curve) in binary classification evaluation typically refers to the area under the ROC curve, ranging from 0 to 1. 0.5 corresponds to random guessing, and 1.0 indicates perfect discrimination. In the task of "judging whether a synthetic URL is realistic," if a classifier cannot distinguish between real and synthetic samples, its AUC should be close to 0.5. However, the author found that when two real URL sample sets were evaluated against each other, the AUC was also below 0.5, indicating that the real data itself has distribution differences, and directly using AUC as a realism metric is unreliable. To address this, the author turned to finer-grained field-level scoring and introduced a "real vs. real" baseline to correct this natural bias, thereby more fairly measuring how close generated samples are to real samples.
Core Experiment: Preregistered Question—Does the Data Help Detectors?
The author preregistered the question he cared about most: can synthetic data improve detector performance? Across 6 settings, adding synthetic data improved recall at a 1% false positive rate. Compared with the best-performing simple generator, on the main test set, this model led in 5 of 6 items, with an advantage of 0.7 to 2.5 percentage points, and the gap in 1 item was proven statistically significant by testing.
Surprising Finding: Realism and Usefulness Are Not Equivalent
However, on another test set, PhreshPhish, the results reversed: URLs generated by a prompt-driven 7B large language model were more helpful in 4 of 6 items, even though this LLM was the least realistic among the learned generators. According to the author, he has not yet fully explained this phenomenon. This suggests that the realism of generated data and its practical value may not be consistent, and when evaluating synthetic data, we cannot look only at realism.
PhreshPhish is a public benchmark dataset for phishing URL detection, characterized by containing relatively recent phishing samples and commonly used to evaluate a model's generalization ability to unseen attack patterns. Compared with the main test set, PhreshPhish's distribution may be closer to real-world concept drift scenarios—phishing tactics evolve over time, and detectors trained on old data easily become outdated. Therefore, performance on this test set better reflects whether synthetic data truly enhances the detector's robustness to novel threats. The author observed that although URLs generated by the prompt-driven 7B LLM had the lowest realism, they brought greater improvements on PhreshPhish. This may be because the LLM's generated diversity is higher, covering a broader range of potential attack patterns, rather than simply imitating the surface features of real samples.
Openness and Collaboration
In the README, the author thanked Claude Code for its help and made all code and evaluation details public, welcoming the community to point out errors in the evaluation. This openness helps advance defensive research.
Project address: https://github.com/youssefismail-dev/limnex-uie
Note: This article is based on a share by Reddit user youssefismail-dev, and all data and conclusions come from their original description.
Related articles

GPT-6.2 Astra and Sol Released: Dual-Model Architecture and Community Buzz
OpenAI releases GPT-6.2 with Astra and Sol models. This article summarizes the official announcement and Hacker News discussion, analyzing the potential impact of the dual-model architecture on developers and offering selection advice.

Gmod TTT Egyptian-Themed Map: Space Capitalism vs. the Pharaoh
Explore the fun of a Gmod TTT Egyptian-themed map: players pick roles like pharaoh, teleport to space via a cube, and engage in a humorous capitalism vs. communism showdown with chaotic prop fighting.

GraphRAG: Beyond Search, Toward Global Knowledge Discovery
GraphRAG is a knowledge-graph-enhanced retrieval technology from Microsoft Research that goes beyond local search in traditional RAG to enable global knowledge discovery. This article introduces its core ideas, generational evolution, and healthcare use cases.