AI-Generated Docstrings Gone Wrong: A Hybrid Workflow for Legacy Python Code

Title

AI-Generated Docstrings Gone Wrong: A Hybrid Workflow for Legacy Python Code

Meta Description

An indie developer shares a real-world experience generating docstrings for legacy Python code with AI: fully automated approaches frequently produce hallucinations and vague descriptions, but a hybrid workflow of "human writes the core + AI expands the structure + manual review" boosted efficiency by 60%–70%.

Article Content

Taking over a legacy Python project cluttered with uncommented functions and cryptic one-liners, most people's first instinct is: let AI generate docstrings in bulk. But a real-world experience from an indie developer tells a different story—AI-generated output was full of platitudes and outright fabrications, and the truly effective solution was to demote AI from "creator" to "polisher."

Step One: The Fantasy of Throwing the Entire Repository at AI

The developer's goal was clear: generate Google-style docstrings for every function, class, and method—not just with uniform formatting, but with accurate descriptions of what the code does, the meaning of parameters, and return values. He first tried the most direct approach—opening GitHub Copilot in VS Code, or feeding ChatGPT 4o a prompt like "generate a Google-style docstring for this function," then pasting in the function code.

Taking an invoice calculation function as an example, the AI's docstring was fine format-wise, with parameters and return values correctly written, but it completely failed to mention the discount codes actually supported in the code (SAVE10, FREESHIP), or to explain that the FREESHIP branch only sets a flag and that the real shipping logic lives elsewhere. This kind of "templated" output looks usable but actually loses the business context that legacy code most needs to convey.

Google-style docstrings are a widely adopted Python documentation string convention that originated from Google's internal Python style guide. They use structured sections to describe function behavior, with a typical format consisting of three main parts: Args: (parameter descriptions), Returns: (return value descriptions), and Raises: (exception descriptions). Each parameter is listed as param_name (type): description, and return values specify both type and meaning. Compared to reStructuredText (used by Sphinx) and NumPy style, Google style maintains structure while being more readable, making it especially suitable for quickly locating information while reading source code. Many documentation tools such as Sphinx (via the napoleon extension) and pydocstyle can parse this format. For this reason, the developer wanted legacy code to adopt this style uniformly for later automatic API documentation generation. However, format conventions are just the shell—the real difficulty lies in filling in accurate business semantics.

Step Two: Hallucinations and Inconsistencies in the Face of Complex Logic

The real problems emerged when dealing with complex functions. The AI started fabricating non-existent parameters, claiming a function returned a boolean when it actually returned a dictionary, or completely misunderstanding the purpose of a conditional branch.

A typical scenario is a function like process_user_data(user_id, config)—where config is a deeply nested dictionary passed around everywhere in the system, prone to errors when keys are missing. The AI-generated description would only say "config: User configuration settings"—technically correct, but useless for understanding which keys are required or what the structure looks like.

The developer spent about 3 hours repeatedly tweaking prompts, asking the AI to "be more specific" and "analyze the code carefully," but the output quality was extremely unstable: one docstring would be long-winded, the next too terse. The AI was generating "some kind" of docstring, but not "his" docstring, nor accurate documentation that could be trusted without line-by-line review.

LLM hallucination refers to a model generating content that appears plausible but doesn't match the facts. In code documentation scenarios, hallucinations manifest as fabricating non-existent parameters, incorrectly declaring return types, or making unfounded assumptions about what a function does. The root cause is that LLMs are fundamentally predicting the next token based on probability, rather than truly "understanding" code execution logic. When function bodies are short and patterns are common, the model can infer fairly accurately; but when facing deeply nested data structures, cross-module implicit dependencies, or highly customized business branches, the model lacks sufficient context to reason accurately and tends to "fill in the blanks" using common patterns from training data. This is why AI claims a function returning a dictionary returns a boolean—because in its training distribution, similarly named functions often return booleans. Understanding this helps determine when to trust AI output and when human verification is essential.

Step Three: A Hybrid Process Where Humans Write the Core and AI Fills in the Structure

The turning point was repositioning AI's role: its strength in complex, unfamiliar code isn't "creation," but "refinement" and "unification" given a solid starting point. The new workflow has three steps:

Humans write the core idea first. For each function, spend 30 seconds to 1 minute writing one or two sentences—based on understanding of the code and business logic—explaining what it actually does, and manually identifying parameters and their meanings.

AI handles structural expansion. Give the function and that rough description to ChatGPT 4o, with a prompt explicitly requesting "accurately describe parameters, return values, and possible exceptions, keeping Google docstring format consistent." For tricky parameters like user_permissions or config, an additional paragraph of context is provided. At this point, the AI can produce quite polished docstrings, including permission-checking logic and exception descriptions for PermissionError, ValueError, etc.

Manual review for corrections. This step cannot be skipped. Verify each parameter type, return value, and description against the actual code logic—often catching cases where int was written as str, or a missing None return. Because a complete AI-generated structure already exists, this step is much faster than writing from scratch or repeatedly debugging a fully automated approach.

Force multiplier is a concept from military strategy referring to tools or methods that can significantly amplify the effectiveness of existing resources. In the context of AI-assisted programming, it means AI doesn't replace human judgment but multiplies human expertise. The opposite is autopilot mode—completely letting AI handle tasks independently. The core insight of this article is: for tasks like docstring generation that require deep business understanding, AI works remarkably well as a force multiplier, because humans provide the business context and accuracy judgment that AI lacks, while AI contributes formatting, structuring, and language organization. But if used as autopilot—letting AI generate documentation from scratch—hallucinations and platitudes are inevitable. This distinction has general guiding significance for understanding the capability boundaries of current AI programming tools: the most effective way to use AI is often to augment rather than replace human professional judgment.

Efficiency Gains and Boundary Awareness

According to the developer's statistics, the hybrid approach reduced docstring generation time by about 60% to 70%, and the output was more consistent and accurate than letting AI run fully automatically.

This case reveals a boundary with general significance for AI programming: when tasks require deep contextual understanding and absolute accuracy, especially in unfamiliar codebases, humans must still remain in the loop. AI isn't a silver bullet, but it can significantly shorten that loop. Treating AI as a force multiplier rather than autopilot may be the most pragmatic stance for collaborating with code assistants at the current stage.

One-line Summary

Fully automated AI docstring generation frequently hallucinates on legacy code, while a hybrid process of human-written core + AI expansion + manual review can boost speed by 60%–70%.

Paragraph Summary

An indie developer shares real-world experience generating docstrings for a legacy Python codebase. He initially tried having ChatGPT 4o or GitHub Copilot generate Google-style docstrings directly from code, but the AI frequently fabricated parameters, misdescribed return values, and lost key business context. After about 3 hours of prompt debugging with unstable results, he repositioned AI's role as "polisher" rather than "creator," adopting a three-step hybrid process: humans first write one or two sentences capturing the function's core intent and manually identify parameters; AI then expands it into a structurally complete docstring with exception descriptions and consistent formatting; finally, item-by-item review for corrections. This method reduced docstring generation time by about 60%–70%, with more consistent and accurate output. The article's conclusion: when tasks require deep contextual understanding and absolute accuracy, humans must remain in the loop—AI is a force multiplier, not an autopilot.