AI Agents That Use Customer Data Without Leaking It? Testing Aura's Local Anonymization

Nearly every company that tries to deploy an AI Agent runs into the same wall: the most valuable questions have answers buried in their own data.

Which customers are over a month overdue? Who's still waiting on a reply? What are people complaining about most? To answer these, the Agent needs access to your customer lists, invoices, and emails. And those are exactly the things you'd never want to upload to a cloud AI model — names, email addresses, phone numbers, account IDs, all personal information that was never consented to be shared externally.

That leaves two options: either abandon the question, or paste it in anyway and hope for the best. Aura's developers wanted a third path, and they put the entire approach through rigorous testing.

The Core Idea: The Agent Sees Tokens, You See Real Names

Before an Agent touches any table, Aura generates a local copy and replaces every name, email address, phone number, and account ID with a stable token.

Your original data	What the Agent actually sees
Olena Petrenko, olena@firm.ua, balance due 12,000	name_1, email_1, balance due 12,000
"Call Mr. Kowalski, he wants the invoice sent to jan@firma.pl"	"Call name_2, he wants the invoice sent to email_2"

The key is that the same person always maps to the same token. This lets the Agent still count, group, join tables, and track a single customer across rows. When you ask it to "send me the overdue list," the result gets written to a local file on your machine with real names restored — the Agent only ever knows the filename.

The obvious follow-up question: does hiding real names make the AI worse?

This approach has a formal name in data privacy: pseudonymization, explicitly defined under Article 4 of the GDPR. It replaces direct identifiers with reversible substitutes, making data impossible to attribute to a specific individual without an additional mapping table. The key distinction from "anonymization" is reversibility: a locally stored mapping table links tokens to real names, which is exactly what enables consistent cross-table tracking — but it also means that if the mapping table is ever compromised, the protection collapses. GDPR permits pseudonymized data to carry a reduced compliance burden under certain conditions, but this doesn't mean the data falls entirely outside data protection obligations. This is the legal context behind the original text's emphasis that "this is pseudonymization, not anonymization."

Does the AI Get Worse? Barely.

The developers used the public benchmark BIRD to find out. It's a test suite where an AI writes SQL to answer questions against real databases — 179 questions across 4 databases — with Google's gemini-3.8-flash generating the queries.

What the Agent sees	Accuracy	Names/emails/numbers sent to the cloud
Real data	68.2%	849
Tokens replacing names (Aura)	67.6%	0
Schema only, no data rows	65.9%	0
Local small model, nothing sent externally	35.2%	0

The results speak for themselves: the tokenized approach trails unrestricted access by just 0.6 percentage points — well within noise — while reducing leaked personal information from 849 to zero. A second model, Qwen, produced consistent results (tokens 67.0% vs. real data 64.8%).

The approach many people intuitively reach for — "just run everything locally" — performed far worse. Local small models answered only about half the questions correctly (35.2%). This comparison neatly illustrates the value of a privacy gateway: it preserves the capability of cloud-scale models while enforcing a data boundary.

BIRD (BIg Bench for Text-to-SQL) is a Text-to-SQL benchmark released by the University of Hong Kong in collaboration with Google and others, designed to measure AI performance on natural language queries against real-world business databases. Unlike earlier benchmarks such as Spider, BIRD includes dirty data, numerical reasoning, and external knowledge dependencies — challenges widely considered to better reflect real enterprise database complexity than synthetic datasets. The core metric is execution accuracy: can the AI's generated SQL actually run against the database and return results matching the reference answer, not just be syntactically correct. This benchmark matters for validating the anonymization approach: if tokenization degraded analytical capability, businesses wouldn't adopt it. The 0.6% accuracy gap shows that replacing real names with tokens barely affects SQL reasoning, because the core SQL operations (counting, grouping, joining) rely on consistency of field values, not their semantic meaning.

The Hard Part: Free-Form Text

Structured tables are the easy case: a column named "Name" contains names. But the bulk of real business data is text — notes, comments, email bodies. A person's name can appear anywhere, in any language, in any grammatical form.

The industry standard approach is to use a small AI model to find personal information in text. To run on ordinary office computers, the model can't be a multi-gigabyte behemoth. The developers tested the four lightest multilingual models they could find, evaluating them on a public test set covering 21 languages (court rulings, EU legal texts, synthetic financial documents).

The conclusion: models alone aren't good enough. Even the best performers only caught 69–78% of names, meaning roughly one in four would slip through to the cloud.

What actually works is using information the gateway already has — who this text belongs to. An email has a sender and recipients; a note is attached to a specific customer record. Those people's names, addresses, and numbers are known, so they can be reliably found in text in any form (for example, the Polish genitive "Kowalskiego" mapping back to Kowalski, or "Mr Benham" matching Stephen Andrew Benham) — no language-specific word lists needed. The model only needs to catch incidental mentions of "strangers."

Text type	Share of people hidden
Financial documents (7 languages)	95%
Court rulings	~95% of those named (82% of all mentions; the remainder are court-anonymized isolated initials like "A.")
EU legal texts (21 languages, most people not in any records)	55%

The developers also ran a test on 85 real emails from their own inbox: Aura replaced all senders, recipients, and over 750 names, addresses, phone numbers, and links in the message bodies. It then ran a self-check — searching the Agent copy for each of the 254 values it had hidden. The result: no complete names, addresses, or numbers remained, with only three isolated words left behind (a city name and a standalone first name).

This self-check is now built into the app and runs automatically after every upload — you don't have to take the developer's word for it.

In NLP, the task of identifying personal information such as names, addresses, and phone numbers in unstructured text is called Named Entity Recognition (NER). Traditional NER models rely on a language model's contextual understanding, but they face two inherent bottlenecks in privacy protection scenarios: recall ceiling — models can't recognize name variants or rare inflections they've never seen — and size vs. performance tradeoff — high-accuracy models capable of handling 21 languages are typically too large to run in real time on ordinary office hardware. The "use text attribution" approach described here essentially converts the NER problem into the simpler task of string matching and fuzzy lookup: known variants of a person's name (such as Polish declension forms) can be pre-enumerated using morphological rules or edit-distance algorithms, without relying on a language model's generalization ability. This gives far higher recall for known individuals than a pure model approach, at minimal computational cost.

What It Can't Do

Privacy promises are easy to oversell, and the original write-up is admirably candid about the limits.

• This is pseudonymization, not anonymization. The Agent can still see the content itself: subjects, amounts, dates, company names. If the content alone can identify someone (e.g., "the CEO of our largest client"), tokens won't help.
• Free-text coverage depends on your data. It's highest when the people mentioned are already in your records; it's lower for strangers. Isolated words — a standalone first name or city name — may slip through.
• Names across different scripts can't be linked. A Cyrillic-script record won't match a Latin-script signature unless you enable the optional model.

How to Try It

Aura is available as a free Windows app at aurasdk.dev/desktop. The Data tab has two sections:

• Upload: The Agent sees the file as-is, for documents you're not concerned about.
• Safe upload: Names, contact details, and numbers (including those inside emails and notes) are replaced with tokens, and the copy is run through the self-check automatically.

It supports Excel, CSV, SQLite, and email formats (.eml, .mbox from Gmail/Thunderbird/Apple Mail), with one-click integration for Claude Code, Claude Desktop, Cursor, and other Agents. The optional model is around 380 MB, downloaded only when you enable it, and is used to catch people not already in your data. All processing happens on-device.

This is still an early version. For teams that want to use AI on customer data but face privacy compliance constraints, the concept of a "local anonymization gateway" is worth watching — it doesn't ask you to choose between capability and privacy. It tries to preserve both.