Cloning Your Writing Style with AI: Why It Always Produces Filler

The Other Side of Voice Cloning: AI Learns Too Well

When Rich Brooks opened the blog draft that Claude had generated from a podcast transcript, one detail caught his attention: across roughly 2,000 words, the word "actually" appeared seven times.

At first glance, this looked like a model glitch. But after closer inspection, Rich found the root cause in the source material itself — his own spoken language used "actually" at exactly that frequency. Claude hadn't deviated from his style; it had reproduced it with startling precision.

This is the uncomfortable part of working with transcripts, webinar recordings, livestream replays, or customer interviews: if the source speaker has verbal tics, the AI picks them up too. For content teams trying to turn spoken material into publishable articles, this changes the nature of the work — the goal isn't just "make it sound like the speaker," but "make it sound like the speaker while controlling the habits that show up too often."

Why Voice Cloning from Transcripts Easily Produces Repetition

This problem isn't limited to any particular model. If the input contains repeated phrasing, filler words, or a favored catchphrase, the output will likely inherit those patterns when the prompt asks it to match the voice.

This means source selection and editing strategy matter far more than most workflows expect. When you feed a webinar transcript to an AI, you're giving it not just the ideas and structure, but also the speaker's "language fingerprint." The model can learn these fingerprints so well that the first draft reads as both authentic and too authentic — it may preserve the rhythms, transitions, and recurring vocabulary that a human editor would normally cut.

Rich's case is valuable because it reveals the difference between style and habit. "Actually" can be part of someone's natural style, but when it appears seven times in 2,000 words, it no longer reads as style — it reads as noise.

In popular usage, voice cloning usually refers to using AI to replicate someone's timbre and intonation to produce synthetic speech that sounds the same. But in content workflows, there's a broader kind of "text-level voice cloning": using someone's spoken transcript as a style sample so the model mimics their word choice, sentence structure, and rhythm in written content. Both share the same principle — the model learns the target's "language fingerprint" from large samples: the statistical patterns of words, transitions, and expressions a person repeatedly uses.

The key point is that this learning is pattern matching, not understanding. The model doesn't know whether "actually" is a tic or a core expression — it just counts the word as high-frequency and treats it as a style feature to reproduce. So the more precise the clone, the more thoroughly the habits get copied too. This is why the problem isn't the model itself, but the transcript fed to it, which naturally contains those habits.

Turning Voice Rules into Observable Constraints

The most practical idea in Rich's workflow is defining voice rules as specific items a draft can be checked against — far better than vague instructions like "make it sound real." Real? Real to whom? By what standard? How do you know if the model achieved it?

Checkable rules are different: they give the editor something concrete to verify. For example: limit the use of a specific filler word; preserve the speaker's preferred rhythm but remove repetitive openings; keep the speaker's directness while reducing redundant transitions.

These aren't just style preferences — they're executable behaviors. A draft either follows the rule or violates it, which makes the editing loop far tighter. This matters because models judge quality differently from human editors — they do pattern matching. If instructions are too abstract, the system may produce content that's technically in that person's style yet still fails the editorial standard you actually care about.

In prompt engineering, "make it sound like me" is an abstract instruction — the model can only guess probabilistically what counts as "like." Checkable rules take a different path: breaking style down into quantifiable constraints, such as "use 'actually' no more than once" or "don't reuse the same opening pattern." These rules resemble what's often called "constrained decoding" or structured prompting in LLM applications — translating subjective judgment into objective conditions that can be checked one by one.

Once rules are quantifiable, quality evaluation no longer depends on the editor's intuition but can answer "was this rule violated?" This also reduces the cognitive mismatch between model and human editor: models excel at pattern matching, humans excel at value judgment, and translating style into rules means expressing human standards in a form the model can process.

The Key Step: Have the AI Report What It Changed at the End

Rich's approach goes further: the project's final note tells him what the AI did. This detail seems minor, but it distinguishes "hoping the rules took effect" from "confirming the rules were executed."

On the show, he read back an example output: "Double-checked, removed all but one of the actuallys." This sentence matters because it creates an audit trail. Rich doesn't have to blindly trust the draft — he can see whether the rules were applied. The model reported the actions it took, making the workflow easier to inspect and debug.

For content teams, this feedback loop is highly valuable. If the draft still feels too repetitive, you can ask a very specific question: did the cleanup step run? If it did, what did it remove? If the model says it reduced repeated language, you can evaluate the result against the rules you set; if it didn't execute, you know where the process needs adjustment. This beats relying on a one-shot prompt and hoping the output lands in the right place.

An audit trail is originally a finance and compliance term, referring to recording each step of an operation in a traceable way. Bringing it into content workflows means the model must not only perform changes but declare what it changed — for example, "removed all but one of the actuallys." This way, the output carries verifiable metadata instead of being an opaque black-box result.

In AI collaboration, this practice is called "self-reporting" or "changelog-style output." Its value lies in turning a trust problem into an inspection problem: the editor doesn't need to judge whether the draft is faithful, only whether the model's self-report matches the established rules. When results don't meet expectations, you can quickly locate whether the rule wasn't executed or the rule itself needs adjustment — turning debugging from guesswork into diagnosis.

Cost and Reward: Front-Loaded Setup Buys Back Time Later

The obvious downside is the upfront investment. Rich says he spent a lot of time on project setup precisely so he wouldn't have to spend time adjusting every week. The workflow slows down first, then speeds up.

But the payoff is substantial. He now spends about five to ten minutes editing a blog post, versus two to three hours writing one from scratch. He didn't achieve this on his first prompt — he got there through continuous project iteration.

This is the part worth emphasizing for anyone building a similar system: the value lies in the accumulated rule list, not the perfect first draft. You're not trying to eliminate editing entirely, but to move recurring decisions into the workflow so they stop consuming the same energy every week.

Treat the Rule List as a Core Asset

The strongest operational takeaway from this approach is: the list itself is the asset. Every time you notice a repeated phrase, an overused opening, or a habitual transition, turn that observation into a rule. Over time, the list becomes a practical style guide for the model, capturing what the AI should preserve, limit, and report.

A simple way to start: review the last ten pieces of copy or drafts the AI generated, find the most frequent phrases, and write a one-line rule limiting each. This small step is far more useful than abstractly defining voice style — it gives the model a constraint to follow and gives you a handle for checking output.

The source material already contains the style; your job is to decide which parts are valid signals and which have become repetition.

Practical Advice for Content Workflow Builders

If you're building a content workflow around transcripts or speaker-driven material, the lesson isn't "AI can't handle voice style" — that's too broad. A more accurate conclusion: AI captures voice style extremely well, including the parts you don't want repeated. The solution isn't looser prompts but a tighter editing system:

• Identify the speaker's recurring habits
• Turn those habits into explicit rules
• Have the final step report what it changed
• Iterate continuously until the workflow runs cheaply enough

This is how you make content sound like the speaker without dragging every catchphrase into the final draft.

One-line Summary

When AI clones a voice style, it precisely reproduces verbal tics; the fix is turning habits into checkable rules and having the model report its changes.

Paragraph Summary

The article opens with Rich Brooks's case: Claude reproduced his high-frequency spoken "actually" seven times in a 2,000-word draft, revealing that AI voice cloning picks up catchphrases and fillers along with style — the root cause is the transcript, not the model. The author advocates replacing vague instructions like "sound real" with checkable specific rules, such as limiting a word's usage count or preserving rhythm while removing repetitive openings. More crucially, have the AI report what it changed at the end, establishing an auditable feedback loop. Though upfront setup takes time, Rich has compressed blog editing to 5–10 minutes per post, far below the 2–3 hours of writing from scratch. The core conclusion: the rule list itself is the accumulating asset.