Gemini Outputs a Stranger's Name: The Privacy Risks Behind AI Memory Features

A Disturbing AI Anomaly

Recently, a Reddit user described an unsettling experience: during a normal conversation with Google Gemini, the AI suddenly began repeatedly "spamming" his mother's name in the middle of a response — and according to the user, he had never mentioned his mother in any conversation, let alone her name.

This phenomenon quickly sparked discussion in the community. For ordinary users, seeing an AI produce a real name from their private life out of nowhere inevitably brings intense confusion and alarm: Where did this information come from? Is the AI "peeking" at my other data? Or is it simply a random model glitch?

Three Technical Explanations for Gemini's Abnormal Output

To understand this type of phenomenon, we need to analyze it from two perspectives: how large language models work and the product architecture.

Model Hallucination

The most common and most likely explanation is "hallucination." Large language models are fundamentally systems that predict the next token based on probability. Here, a token is the basic unit the model uses to process text — it could be a complete English word, a Chinese character, or part of a word. A model's vocabulary typically contains tens of thousands to over a hundred thousand tokens. During each generation step, the model uses the self-attention mechanism in the Transformer architecture to compute a probability score for every token in the vocabulary, then decides which token to actually output based on the decoding strategy (such as greedy decoding, top-k sampling, nucleus sampling, etc.).

Under certain abnormal conditions, the model may fall into a repetition loop, repeatedly outputting the same word or phrase. The technical cause of this phenomenon lies in the following: when the model's attention weights become abnormally concentrated on certain already-generated token sequences, subsequent generation continually reinforces this pattern, forming a positive feedback loop. Although modern models have introduced mechanisms like repetition penalty to mitigate this issue, they can still fail under specific context conditions or decoding parameter configurations.

If during generation the model randomly samples a common personal name and falls into repetition due to the decoding strategy or contextual anomalies, it could produce the "spamming of a name" scenario. The fact that this name happens to match the user's mother's name may simply be a coincidence — common names appear with extremely high frequency in training corpora, and their corresponding tokens or token combinations occupy significant positions in the model's probability distribution.

Cross-Session Memory Feature Leaking Personal Information

More noteworthy is Gemini's recently launched personalization memory feature. Google has introduced cross-conversation memory capabilities to Gemini, allowing the AI to remember information users provided in previous conversations to deliver a more coherent, personalized experience.

This feature belongs to the "stateful AI assistant" design paradigm that has emerged in the industry in recent years. Traditional large language model conversations are stateless — after each conversation ends, the model retains no contextual information, and the next conversation starts from scratch. Memory features achieve cross-session continuity by maintaining a persistent user information storage layer outside the model. Technically, the system typically auto-extracts key information during conversations (such as user preferences, important facts, interpersonal relationships, etc.), stores them as structured or semi-structured memory entries, and retrieves and injects relevant memories into the system prompt or context window at the start of subsequent conversations. OpenAI's ChatGPT Memory, Anthropic's memory exploration, and Google Gemini's personalization features all employ similar architectures, but differ in information extraction granularity, memory retrieval strategies, and user control permissions.

This means that if a user mentioned relevant information in a past conversation (even a long time ago, or in another connected application), or if Gemini has access to the user's Google account data (such as Gmail, Contacts, Google Photos, etc.), the model could theoretically "retrieve" this information. As one of the world's largest internet service providers, Google's ecosystem covers virtually every dimension of a user's digital life: Gmail handles communications (over 1.8 billion users worldwide), Google Calendar manages schedules, Google Photos stores personal images, Google Maps records location trails, and Google Search reflects user interests and needs. When Gemini is authorized to access these services, it can theoretically construct an extremely detailed user profile — knowing the user's social relationships, family members' names, work schedules, and even health conditions.

A user's subjective memory of "never having mentioned" something doesn't necessarily mean the relevant data doesn't exist in the system. An email from their mother, a photo tagged with a person's name, or a contact entry in their address book could all be sources from which the AI obtained this information.

Account or Session Data Crosstalk

Although the probability is low, in extreme cases we cannot rule out backend data crosstalk — where different users' or sessions' contexts are incorrectly mixed on the server side.

Modern AI services typically run on large-scale GPU clusters, using load balancing, request routing, and distributed caching to handle massive concurrent requests. Data crosstalk can occur at multiple points: GPU memory not being properly cleared, leaving residual KV Cache (key-value cache, intermediate computation results used to accelerate Transformer inference) from a previous request; key collisions in distributed caches causing different users' session data to be incorrectly associated; or session ID collisions in database shards. In March 2023, ChatGPT experienced a race condition in its Redis cache that caused some users to see other users' chat titles and payment information fragments — one of the publicly confirmed data crosstalk incidents in the industry.

Such bugs occasionally occur in large-scale online services, and when they do, they constitute serious privacy incidents affecting not only individual user privacy but also the entire platform's credibility.

The Core Controversy Between AI Personalization and Privacy Boundaries

The reason this individual case deserves in-depth discussion isn't because of how serious it is in itself, but because it reflects the increasingly blurred boundary between "personalization" and "privacy" in AI assistants.

As products like Gemini and ChatGPT race to launch memory features and account data integration, the scope of user information accessible to AI is expanding dramatically. Gemini's deep integration with the Google ecosystem means it can theoretically access emails, calendars, photos, search history, and other massive amounts of personal data. While AI becomes "better at understanding you," users' sense of control over "how much does the AI actually know about me" is declining. This "personalization paradox" is becoming a core tension in AI product design: users expect AI to be smarter and more attentive, yet feel uneasy about AI "knowing" too much about them.

For the user in this case, the core source of unease comes from inexplicability: the AI output information that the user believed they had never provided, yet the system cannot offer a transparent trace of its source. This is precisely what current AI products universally lack — data source traceability.

AI system explainability and data provenance are important research directions in both academia and industry. The EU's AI Act was officially passed in 2024 and explicitly requires high-risk AI systems to have transparency and explainability. On the technical side, the industry is exploring multiple approaches: Influence Functions can trace associations between model outputs and training data; attention visualization can show which parts of the input the model "focused on" during generation; and Retrieval-Augmented Generation (RAG) architectures inherently possess certain source attribution capabilities since they can clearly label which retrieved document information came from. However, for personalized AI assistants that deeply integrate multiple data sources, achieving end-to-end information traceability remains an unsolved engineering challenge.

How Users Can Protect Personal Privacy in AI Conversations

Facing similar anomalies, users can take the following measures to protect themselves:

• Check Gemini memory settings: In Gemini's settings, check whether "personalization" or "memory" features are enabled, review saved memory entries, and delete unnecessary content. The typical path is Gemini app settings > Extensions/Personalization > Manage memories, where users can review item by item what information the AI has remembered about them.
• Review Google account authorizations: Confirm which Google services Gemini has data access permissions for, and disable unnecessary connections as needed. In the "Data and Privacy" settings of your Google account, you can view and manage access permissions for various applications, including Gemini's access to Gmail, Drive, Calendar, and other services.
• Preserve evidence and provide feedback: If you encounter an obvious anomaly, take screenshots and report through official channels — this type of feedback is crucial for vendors to identify potential bugs. Complete records including timestamps, conversation context, and abnormal output content help engineering teams reproduce the issue and determine the root cause.
• Stay vigilant about sensitive information: Avoid entering highly sensitive personal identity information in AI conversations, such as ID numbers and bank accounts. Even if AI service providers promise data security, minimizing sensitive information exposure is always the most fundamental security principle.
• Regularly clear conversation history: Periodically delete conversation records that are no longer needed to reduce the total amount of personal information that AI systems can retain long-term and potentially access.

Transparency Is the Foundation of AI Trust

The most likely truth behind this seemingly bizarre "spamming mom's name" incident is perhaps just an ordinary model hallucination coincidence. But the alarm it raises is real: as AI assistants increasingly integrate personal data, vendors must find a balance between powerful functionality and privacy transparency.

Users have the right to know what the AI "remembers," "where it got the information from," and "how to delete that data." This is not only a best practice in product design but is increasingly becoming a hard legal requirement — the EU AI Act, various US state privacy laws, and China's Personal Information Protection Law all emphasize, to varying degrees, users' right to know about and control how AI systems process their personal information.

Only by establishing explainable, controllable, and traceable data mechanisms can the value of AI personalization truly earn user trust and acceptance. Otherwise, every "accidental" leak-like output will continuously erode users' fundamental trust in AI products — and once trust is lost, the cost of rebuilding it will far exceed the initial investment in establishing transparent mechanisms.