Can AI Chat Logs Be Used as Evidence? Understanding the Privacy Boundaries of ChatGPT Conversations

Case Recap: AI Chat Logs Enter the Judicial Spotlight for the First Time

A recent overseas court case has drawn widespread attention: an analyst was sentenced to probation after confessing violent criminal intentions against an ex-partner to ChatGPT—including plans of rape and murder. While the case itself involves serious violent threats, from a technology and privacy perspective, it reveals a much deeper question: Just how "private" are our conversations with AI chatbots?

As more and more users treat ChatGPT as a confessional, a therapist, or even a personal diary, this case serves as a wake-up call: words that seem to exist only between a human and a machine may, under certain circumstances, become admissible evidence in court. Notably, this isn't the first time AI-related conversational data has intersected with the judicial system—in recent years, search engine histories, smart speaker recordings, and other digital traces have been cited as supporting evidence in criminal cases. However, the direct use of large language model (LLM) chat logs as corroborating evidence of criminal intent makes this case a landmark.

Why AI Conversations Are Not "Absolutely Private"

Many users operate under a misconception: that talking to an AI is like talking to yourself, with no one else ever seeing it. That's simply not the case.

How ChatGPT Stores and Retrieves Conversation Data

Taking OpenAI as an example, conversations with ChatGPT are recorded on the company's servers by default for purposes including model improvement, safety reviews, and regulatory compliance. This means:

• Conversation content is stored on the service provider's servers, not solely on your local device;
• When served with a valid judicial investigation order or subpoena, companies are generally obligated to cooperate with law enforcement and provide relevant data;
• When conversations involve high-risk content such as violent threats, self-harm, or child safety concerns, the platform's safety mechanisms may proactively trigger manual review.

From a technical architecture standpoint, every conversation a user has with ChatGPT is sent as an API request to OpenAI's cloud servers for inference computation. The conversation context—including system prompts, user inputs, and model outputs—is logged in server-side databases. According to OpenAI's publicly available data retention policies, even if a user deletes their chat history through the interface, the relevant data may still be retained on the server for a period of time (typically 30 days) for safety and abuse detection purposes. Furthermore, OpenAI's privacy policy explicitly states that the company may disclose user information to law enforcement "to the extent required or permitted by applicable law."

Under the U.S. legal framework, this type of data retrieval is primarily governed by the Stored Communications Act (SCA), which is part of the 1986 Electronic Communications Privacy Act (ECPA). Law enforcement typically needs a court-issued search warrant to obtain stored communication content, but in certain emergency situations—such as imminent threats to personal safety—service providers may voluntarily disclose information to law enforcement without a formal legal instrument. This mechanism makes AI chat data far more accessible in judicial proceedings than most users would expect.

In other words, once a user reveals explicit criminal plans in a conversation, those chat logs can absolutely be used as evidence in legal proceedings. This case is a real-world manifestation of exactly that logic.

The Paradox Between Privacy Protection and Public Safety

This incident also exposes the inherent tension within AI services between privacy protection and public safety.

The Dual Responsibility of AI Platforms

On one hand, AI companies need to protect user privacy and avoid excessive surveillance that could trigger a crisis of trust. On the other hand, when conversations involve real-world threats to others, platforms face both moral and legal obligations to intervene. This contradiction isn't unique to AI—it also exists on social media platforms, messaging apps, and even in traditional psychotherapy.

In the traditional psychotherapy industry, there's a classic legal precedent that serves as a reference point: the "Tarasoff Duty," established by the California Supreme Court in 1976. The ruling determined that when a psychotherapist has reasonable grounds to believe a patient may pose a serious threat to a specific third party, the therapist has a duty to take reasonable steps to protect the potential victim—even if that means breaking therapist-patient confidentiality. Since then, most U.S. states have adopted some form of "duty to warn" or "duty to protect." The dilemma AI platforms face today is strikingly similar: when a user expresses explicit violent intent in a conversation, should the platform bear a comparable "duty to warn"? The difference is that psychotherapists are trained human professionals, whereas AI systems rely on algorithmic models to assess intent—making the attribution of responsibility significantly more complex.

You may not have noticed, but looking at the outcome, this type of content review mechanism may have served as an early warning in this case, preventing potential escalation of violence. This reminds us that AI safety guardrails aren't just about filtering "inappropriate outputs"—they also include identifying and responding to high-risk "inputs."

These so-called "safety guardrails" typically consist of multiple layers of defense at the technical level: the first layer is a rule-based filter for keywords and phrases, designed to intercept the most obvious violations; the second layer consists of specially trained Content Classifiers—classification models that perform multi-dimensional risk assessments on input text, determining whether it involves violence, self-harm, illegal activities, and other categories; the third layer is embedded in the LLM's training process itself—through RLHF (Reinforcement Learning from Human Feedback), the model is guided during training to refuse generating harmful content or to respond to dangerous requests with dissuasive replies. When the system detects high-risk input, in addition to displaying safety prompts in the chat interface, it may also flag the session for entry into an internal safety queue for manual review by a dedicated Trust & Safety team.

The Privacy Awareness Users Should Develop

For everyday users, the key takeaway is: never treat any internet-connected AI tool as an absolutely secure private space. Any content you input could theoretically be stored, analyzed, and even disclosed under extreme circumstances. This should be consistent with how we approach email and cloud documents.

In fact, there's a widely cited rule of thumb in information security: any information transmitted over the internet should be treated as a "postcard" rather than a "sealed letter." In the traditional postal system, sealed letters are legally protected, and unauthorized opening is illegal. But the technical nature of electronic communication means that data can be accessed by intermediaries at multiple points during transmission and storage. For AI conversations, this characteristic is particularly pronounced—every sentence you type must travel over the network to a remote server, undergo inference computation on a GPU cluster, and then have the results sent back. The data pipeline involved is far more complex than a "whispered conversation between two people."

AI's Role as a "Knowing Party": A Tech Ethics Perspective

This case also raises a more controversial topic: when an AI "learns" of a user's criminal intent, what role should it play?

Should AI Be a Passive Recorder or an Active Intervener?

Currently, mainstream AI products typically provide dissuasive responses when they detect explicit violent or self-harm intent, and may trigger internal safety protocols. But the boundaries of such intervention remain blurry:

• To what extent should AI proactively report to law enforcement?
• How do you distinguish between genuine threats and emotional venting or creative writing?
• How can privacy violations from false positives be avoided?

On the question of "proactive reporting," regulations vary significantly across global jurisdictions. The United States has not yet enacted legislation specifically mandating reporting by AI platforms, but the National Center for Missing & Exploited Children (NCMEC) related regulations require internet service providers to report suspected child sexual abuse material (CSAM) to authorities—an obligation that has been interpreted as applying to AI platforms as well. In the EU, the AI Act, which took effect in 2024, imposes transparency and risk management requirements on high-risk AI systems, but specific obligations regarding proactive reporting of criminal intent are still under legislative discussion. China's Interim Measures for the Management of Generative AI Services explicitly require providers to "promptly take remedial measures" and "report to relevant authorities" when illegal content is discovered.

Regarding "distinguishing genuine threats from emotional venting," this is one of the core challenges facing current natural language processing (NLP) technology. Human language is rife with irony, hyperbole, metaphor, and context dependency—a statement like "I want to kill my boss" is, in the vast majority of cases, merely an emotional expression of workplace stress rather than a genuine criminal premeditation. Existing content classifiers still have a relatively high false positive rate when processing such ambiguous expressions. Research shows that even the most advanced intent classification models fall far short of satisfactory accuracy when distinguishing between "genuine threats" and "rhetorical expressions," meaning any automated reporting mechanism must make difficult trade-offs between "erring on the side of over-reporting" and "protecting privacy."

These questions currently have no definitive answers. It's foreseeable that as AI penetrates deeper into daily life, legal and ethical debates surrounding "responsibility attribution when AI is in the know" will only intensify.

Technical Pathways for Privacy Protection

It's worth noting that the tech community is also actively exploring technical solutions that could fundamentally ease this tension.

End-to-End Encryption is a privacy protection technology already widely adopted in instant messaging. Its core principle ensures that only the communicating parties can decrypt message content, while the server merely relays ciphertext without being able to read the plaintext. However, applying end-to-end encryption to AI conversation scenarios faces a fundamental contradiction: large language models must "read" user input on the server side to generate responses, meaning traditional end-to-end encryption cannot be directly applied under current cloud-based inference architectures.

Homomorphic Encryption offers a theoretical solution—it allows computation to be performed directly on encrypted data, enabling servers to complete inference without decryption. However, the computational overhead of homomorphic encryption is currently enormous (typically tens of thousands to millions of times slower than plaintext computation), and it remains a long way from practical application in real-time LLM inference.

On-Device Inference is currently one of the most practically feasible approaches. With advances in model compression techniques (such as quantization, distillation, and pruning), an increasing number of lightweight large language models (such as smaller parameter versions of Meta's Llama series, Google's Gemma, and Microsoft's Phi series) can run locally on smartphones or personal computers, with conversation data never needing to leave the user's device. Apple Intelligence's strategy of "processing on-device whenever possible" is a commercial implementation of this approach. Of course, local models typically underperform compared to cloud-based large models, requiring users to make trade-offs between privacy protection and model performance.

Differential Privacy offers yet another approach: by injecting carefully designed mathematical noise into data, it ensures that no individual user's specific information can be reverse-engineered from aggregated data. Apple and Google have already widely adopted differential privacy techniques in their data collection practices. In AI training scenarios, differential privacy can ensure that models learn overall patterns from user data but cannot "remember" or leak any specific user's conversation content.

Practical Advice for Users and the Industry

Overall, while this incident is extreme, the AI privacy issues it reflects have universal significance.

For Everyday Users

1. Maintain privacy awareness: AI conversations are not absolutely confidential. Statements involving sensitive or illegal content may leave a permanent record;
2. Use emotional support features responsibly: AI can serve as an emotional outlet, but it cannot replace professional mental health intervention. While mainstream AI chatbots demonstrate a degree of empathy in conversations, they lack the professional judgment, ethical constraints, and legal accountability of licensed therapists. Professional organizations such as the American Psychological Association (APA) have repeatedly reminded the public that AI tools should not be considered substitutes for mental health services—especially when dealing with suicide risk, post-traumatic stress, and other serious mental health issues, professional human intervention is indispensable;
3. Understand the terms of service: Know exactly how the product you're using handles and stores conversation data. Users are advised to carefully read the privacy policies of the AI services they use, paying particular attention to the following key provisions: data retention periods, whether data is used for model training, whether there's an "opt-out" option, cross-border data transfer policies, and under what circumstances data may be disclosed to third parties. Taking OpenAI as an example, users can disable the "Chat History & Training" option in settings, which prevents new conversations from being used for model training—but this does not mean conversation content is completely unrecorded by the server. OpenAI still retains conversation data for a period of time for safety monitoring.

For the AI Industry

1. Make data policies transparent: Give users a clear understanding of where their conversation data goes and how it's used;
2. Improve safety mechanisms: Establish reasonable response protocols for high-risk content while maintaining privacy protections;
3. Advocate for legislative clarity: The legality, boundaries, and procedures for using AI conversations as evidence urgently need a clearer legal framework. Looking at global legislative trends, the EU AI Act has taken the lead in establishing a risk-tiered regulatory framework, classifying AI systems into four risk levels—unacceptable risk, high risk, limited risk, and minimal risk—with different compliance requirements for each category. The United States tends toward a combination of industry self-regulation and sector-specific oversight, and has not yet enacted comprehensive federal AI legislation, though multiple states have begun advancing local legislation. China has already issued several specialized regulations targeting algorithmic recommendations, deepfakes, and generative AI. These different legislative approaches will profoundly influence the legal status and evidentiary weight of AI conversation data within their respective jurisdictions.

Conclusion: Redefining the Privacy Boundaries of Human-AI Conversations

The judicial outcome of this particular case isn't the main point. What truly deserves reflection is that the nature of "human-AI conversation" in the AI era is being redefined. As we grow increasingly accustomed to confiding in AI, we also need to soberly recognize that the records on the other side of the screen may be far more "real" and "permanent" than we imagine.

From a broader perspective, this case is yet another footnote in the ongoing evolution of "privacy rights" in the digital age. From the late 18th-century Fourth Amendment to the U.S. Constitution establishing protections against "unreasonable searches and seizures" of "persons, houses, papers, and effects," to Justice Brandeis's early 20th-century articulation of the "right to be let alone," to the 21st-century EU General Data Protection Regulation (GDPR) establishing data subject rights, the boundaries of privacy have continuously expanded and been reshaped alongside technological progress. The legal characterization of AI conversation data—whether it more closely resembles a "personal diary" (entitled to stronger privacy protections) or a "communication record" (subject to lawful retrieval under specific conditions)—will be one of the key questions the legal community must answer in the years ahead.

While enjoying the conveniences AI brings, understanding its privacy boundaries is a required course for every citizen of the digital age.