After AI-Generated Test Data Failed Me, I Switched to Faker Scripts

Title

After AI-Generated Test Data Failed Me, I Switched to Faker Scripts

Meta Description

AI-generated test data always falls short? This article shares a developer's lessons from a 15-hour prompt-tuning battle: use AI to brainstorm edge cases, then use Faker scripts to precisely generate JSON test data—balancing diversity and controllability.

Article Content

While developing a new user profile API, preparing test data became a major obstacle. We needed realistic, diverse data full of edge cases—not the same old john.doe@example.com and password123. As a developer in 2026, my first instinct was naturally to let AI handle it—but that led to a 15-hour tug-of-war.

The Three Pitfalls of AI-Generated Test Data

The initial prompt was straightforward: "Generate 20 social media user JSON profiles with diverse names, emails, bios, and follower counts." The returned JSON was perfectly formatted, but the content was disappointingly bland: bios were all generic sentences, follower counts clustered in the hundreds, and emails were uniformly firstname.lastname@example.com. This kind of "too perfect" data is useless for finding real bugs.

When I refined the requirements to include edge cases, the problem got thornier. The AI would either repeat itself (every missing email was "email": null, every long bio was the same copy-pasted paragraph), or be type-inconsistent (is_verified should be a boolean but returned the string "true", causing API validation to fail outright), or miss the point ("invalid email" only produced "notanemail" instead of samples like user@domain@domain.com that bypass simple regex but fail stricter validation).

According to the original author, he spent a full 15 hours across three days, mainly反复 adjusting ChatGPT 4.0 prompts, trying to make the AI understand the nuances of "edge cases." The result was like wrestling with a very polite, very literal robot—the output format was impeccable, but it just didn't meet the specific needs.

The Breakthrough: AI as Consultant, Faker Does the Work

The turning point came from a shift in perspective: don't expect AI to generate data directly; instead, let it help you define the problem. AI excels at understanding schemas and divergent thinking, but it's not a perfect direct generator for truly random, deliberately corrupted, or highly specific data.

Step 1: Have AI Brainstorm Edge Cases

The author gave ChatGPT the basic API schema and asked: "List 15 distinct edge cases for testing. For each, describe the scenario, point out which fields will be abnormal or erroneous, and explain why. Be specific about data types or expected values."

This step worked remarkably well. The AI produced high-quality descriptions, such as: "Name contains international characters: the name field includes non-ASCII characters like 'Björn' or '你好'." Or "Friends list is an empty array: the friends field is [] rather than null or a populated array." These descriptions formed a clear testing roadmap.

Edge case testing is one of the core methods in software quality assurance, referring to test cases designed specifically for extreme values, boundary values, or abnormal combinations within the input domain. For structured data like JSON user profiles, typical boundaries include: missing or null fields, type errors (e.g., passing a string to a boolean field), length violations (overly long strings or empty arrays), special characters (Unicode, emoji, control characters), malformed formats (illegal variants of emails, dates, UUIDs), and logical contradictions (e.g., is_verified is true but email is invalid). The goal of boundary testing is not to verify the normal path, but to probe whether the system can still correctly error, degrade, or reject under "unexpected but possible" inputs—thereby avoiding silent failures or security vulnerabilities in production.

Step 2: Implement Precisely with Faker

Armed with the AI's suggestions, the author wrote a script using Python's Faker library (version 18.10.1). The approach: first generate realistic base data via Faker, then programmatically override specific fields based on the edge cases described by the AI.

For example, when generating invalid emails, the script first has Faker generate a valid email, then mutates it into nastier samples like foo@bar or user@@domain.com. Long bios are generated with faker.paragraph(nb_sentences=15). The core code structure is as follows:

from faker import Faker
import random

faker = Faker()

def generate_base_profile():
    return {
        "id": faker.uuid4(),
        "name": faker.name(),
        "email": faker.email(),
        "bio": faker.paragraph(nb_sentences=3),
        "followers_count": random.randint(0, 100000),
        "is_verified": faker.boolean(),
        "privacy_level": random.choice(["public", "private", "friends_only"]),
        "last_login_at": faker.iso8601(),
        "tags": faker.words(nb=random.randint(0, 5))
    }

def generate_invalid_email_profile():
    profile = generate_base_profile()
    profile["email"] = faker.word() + "@" + faker.word()
    if random.random() < 0.5:
        profile["email"] = faker.word() + "@@" + faker.domain_name()
    return profile

def generate_long_bio_profile():
    profile = generate_base_profile()
    profile["bio"] = faker.paragraph(nb_sentences=15, variable_nb_sentences=True)
    return profile

In the end, a roughly 150-line script became the primary tool. The author could mix and match different functions to ensure "good" and "bad" data were truly randomly distributed, with the underlying structure always correct—something AI struggles to guarantee when generating from scratch.

Faker is a third-party Python library specifically designed to generate realistic-looking fake data, covering dozens of data types such as names, addresses, emails, phone numbers, dates, and text paragraphs. Its core value lies in "controllable randomness": each call returns a different value, but the format always conforms to real-world conventions (e.g., emails have @ and a domain, dates are in ISO 8601 format). Unlike AI generation, Faker's output never suffers from field type drift, developers can precisely specify generation rules (e.g., nb_sentences=15 to control paragraph length), and it supports fixing the random seed via seed to reproduce results. In test data scenarios, Faker can reliably produce large volumes of structurally correct base samples, which can then be combined with a small amount of programmatic overrides to construct diverse boundary data—this is the fundamental reason it's more reliable than "repeatedly prompting AI."

Key Insight: Treat AI as a Consultant, Not a Coder

The value of this case isn't that "AI is bad," but rather a misallocation of roles. AI excels at brainstorming, outlining, and even generating code snippets, but in test data generation scenarios that require precise control and programmatic validation, it's better suited to play the role of consultant: define the problem, propose solutions, and then let traditional coding tools (like Faker) implement them in a precise, controllable way.

This combined strategy not only saved the hassle of manually constructing hundreds of JSON objects but also achieved more robust test coverage than simply "repeatedly prompting." For any team that needs to build complex test datasets for APIs, this is a reusable path: let AI tell you what to test, and let scripts decide how to create it.

One-line Summary

AI-generated test data kept failing; switching to AI as a consultant to define boundaries and Faker scripts to implement them greatly improved efficiency.

Paragraph Summary

A developer needed realistic, diverse test data for a new user profile API. He initially tried to have ChatGPT generate JSON directly but encountered issues like repetitive content, inconsistent field types, and missed edge cases, spending 15 hours with unsatisfactory results. The breakthrough came from changing the division of labor: first have AI brainstorm 15 specific edge case descriptions based on the schema, then use Python's Faker library to write a roughly 150-line script that generates structurally correct base data and programmatically overrides specific fields. This strategy ensures both randomness and diversity of data while keeping the structure always compliant, ultimately achieving more robust test coverage than repeatedly tweaking prompts. The core insight: AI is good for consulting on "what to test," while traditional scripts handle "how to create it."