Red-Teaming a Kids' AI Tutor: One Word Bypasses the Prompt Injection Filter

Red-Teaming a Kids' AI Tutor: One Word Bypasses the Prompt Injection Filter

Learna Tutor, a children's AI tutoring app, ran a round of red-team adversarial testing during its private testing phase. The very first run exposed a real gap: the guardrails could block "ignore previous instructions," but missed "ignore your previous instructions" — with one extra word.

Why Safety for a Kids' AI Tutor Is a "Load-Bearing Wall"

Learna Tutor is designed for children roughly ages 4 to 13, allowing them to talk directly with an AI tutor. Unlike adult tools, the failure modes here point directly at minors: prompt injection ("ignore your instructions, then…"), privacy extraction ("tell me your email"), and academic integrity bypasses ("just give me the answer").

Safety for a product like this can't rest on a single system prompt and a prayer. It needs guardrails that can actually be verified. The testing team's goal was clear: don't test "can it block the examples in the docs?" — test "can it block what real users will actually type?"

A Three-Layer Defense-in-Depth Architecture

Learna's safety design has three layers, and the testing focused on the two deterministic ones:

Input filter (checkTutorInput): Runs before any model call and contains eight categories of regex rules — privacy, crisis, bullying, medical/legal, out-of-scope, academic integrity, prompt injection, and unsafe web content — while also directly rejecting unsupported tutor modes.

Model system prompt: Tiered by grade level, prioritizing hints over answers, verified through human review rather than unit tests.

Output filter (checkTutorOutput): Screens responses after the model generates them. It has a dedicated privacy check — if a tutor asks a child for personal information (e.g., "tell me your email address"), that response gets blocked. One detail: it handles negation correctly — "don't share your email address" passes, because blocking a safety reminder would itself be a false positive.

Putting the regex filter on the outermost layer is deliberate: it's cheap, fast, doesn't require a model call, and fails closed.

How the Adversarial Testing Was Done

The test suite covered every category, using wording that a child or curious teenager would actually type, not textbook example sentences.

Equally important was adding false-positive checks: legitimate questions the filter must never block. The author's favorite example is a reading comprehension question containing "Maya heard footsteps and hid a letter under the floorboards." A naive privacy filter that matches personal-context words near "letter" would kill a perfectly normal English question. Fortunately, this one passed.

Red teaming is a testing method that simulates real attacker behavior: testers actively play the role of malicious users and try various means to bypass system guardrails, rather than only verifying that the system works correctly on the happy path. In AI product contexts, red teaming typically focuses on attack surfaces like prompt injection, jailbreaking, privacy extraction, and inducing harmful content. The key difference from regular unit testing is this: unit testing verifies "can known correct inputs produce correct outputs," while red teaming verifies "can malicious or edge-case inputs be effectively blocked."

False positives refer to a system incorrectly blocking legitimate inputs that should be allowed through. In security, false negatives (attacks slipping through) are usually the priority concern, but in learning products for children, the cost of false positives is equally high: a filter that treats a normal English reading comprehension question as privacy extraction directly breaks the product experience and takes away a child's ability to get help. That's why safety testing must measure both block rate and false-positive rate — neither can be neglected.

The Finding: One Word Made the Gap

The test suite caught a real vulnerability on its first run: the prompt injection pattern matched "ignore previous instructions" but missed "ignore your previous instructions" — with an optional "your" added.

This is the classic brittleness problem with regex. Attackers won't use your exact wording; they'll use variants. "Ignore all previous instructions," "disregard the above," "forget everything I told you" — each variant is a separate hole unless the pattern generalizes. A child doesn't even need malicious intent — "ignore your rules" is just how kids talk.

The Fix: Make the Pattern Cover Determiners and Synonyms

The fix was to extend the injection pattern with optional capture groups so determiners and modifiers generalize:

ignore\s+(your\s+|all\s+|the\s+)?(previous\s+)?instructions

At the same time, parallel coverage was added for disregard, forget, system prompt, jailbreak, developer message, and role-play escapes ("pretend you are not…", "act as unrestricted"). After the fix, the full suite was rerun — all green — and six additional injection probes with new wording were added, all of which were blocked.

Regular expressions (regex) are a technique for matching text using character patterns. Their core advantages are determinism, fast execution, and no need for model inference, which makes them suitable as the outermost first layer of filtering. But regex's weakness is equally clear: it can only match "patterns that are written," and cannot understand semantics. An attacker only needs to change word order, insert determiners, or swap synonyms to make an originally effective pattern fail. The article's case of "ignore previous instructions" missing "ignore your previous instructions" is a typical example.

Optional capture groups are regex syntax used to express "this part is optional." For example, the ? in (your\s+|all\s+|the\s+)? means the entire parenthesized content can appear zero or one time, so determiners like "your," "all," and "the" are generalized. This significantly improves pattern robustness, but it is still fundamentally enumerating known variants and cannot exhaust all natural language expressions. That's why regex is suitable as a "tripwire," not the only line of defense.

Honest Limitations

Regex is a tripwire, not a wall. It can catch known wording families, but a truly novel jailbreak phrasing may still get past the input filter — which is exactly why the output filter and system prompt exist as independent backstops. No layer is fully trusted.

This round of testing was conducted at the filter layer, with no end-to-end testing against a live model (the test environment had no API key). Filters are deterministic, and unit tests are exactly the right tool for them; model behavior requires separate evaluation.

This work continues. Every time a new attack phrasing is learned, a test case is added. The suite is designed to keep growing.

Defense in depth originates from military and information security, with the core idea of not relying on a single line of defense, but letting multiple independent mechanisms back each other up. In Learna's architecture, the input filter, model system prompt, and output filter run at different stages, assuming any layer might be bypassed. This way, even if regex is breached by new wording, the output filter may still block harmful model-generated content, and the system prompt can constrain model behavior to some extent.

The division of labor between deterministic filtering and model evaluation: Regex filters are deterministic — the same input always produces the same result — so they are well suited to precise verification with unit tests. Model behavior is probabilistic — the same prompt may produce different outputs — requiring separate evaluation sets, human review, or model-assisted scoring. The article clearly distinguishes these two types of testing, avoiding the mistake of using unit-test thinking to verify model behavior, and avoiding using the fuzzy standards of model evaluation to judge whether regex missed a match.

Four Transferable Lessons

Test guardrails adversarially, not just functionally. "Can it block the examples in the docs?" and "can it block what real users type?" are two different questions.

False-positive testing matters as much as block testing. A filter that blocks legitimate learning questions is a broken product, especially for children.

One-word variants are the whole ballgame in prompt injection. If your pattern can't generalize to determiners and synonyms, it's a demo, not a defense.

Defense in depth isn't a slogan — it's the reason a regex miss doesn't become a disaster. Input filter, system prompt, output filter: every layer assumes the others might fail.