Together.ai Reportedly Exposes Prompt Leakage Vulnerability: Request Handling Mechanism Raises Concerns

Together.ai is accused of prompt leakage in requests, exposing isolation risks in multi-tenant inference platforms.
A Hacker News discussion about Together.ai's "Prompt Leakage for Requests" has brought the request isolation issue of multi-tenant inference platforms to the forefront. The article points out that prompts have evolved into "new code" carrying business logic and sensitive data, and their leakage is more covert than model weight leakage and difficult to detect through conventional monitoring. Multi-tenant platforms need to achieve isolation across request queues, KV Cache cleanup, log access control, and batching logic; a lapse in any link may lead to cross-user data interference. Developers should avoid writing keys or customer identity information into system prompts, desensitize and secondarily process sensitive business operations, and monitor service providers' security announcements. Because public details are limited, the incident scope and fix status still await official explanation, but the industry's attention to request lifecycle security will increase accordingly.
Event Overview
Recently, a discussion about AI cloud service provider Together.ai appeared on Hacker News, titled "Together.ai: Prompt Leakage for Requests." The post received 8 points and 4 comments. Although discussion activity is not particularly high, the issue involved—prompt leakage—is highly sensitive for developers who rely on large model APIs.
Prompts typically contain an enterprise's business logic, system instructions, and even sensitive data. Once a leak occurs during request processing, the consequences may go far beyond an ordinary technical failure. At present, the post does not provide complete technical details, but the title clearly indicates that the issue points to "prompt leakage in requests," rather than leakage of model output content.

Why Prompt Leakage Warrants Vigilance
In large model application architecture, prompts have evolved into a kind of "new code." System prompts often specify role settings, output format constraints, and rules for invoking internal knowledge bases; some enterprises also splice customer data fragments into the context. Once such information is seen by other tenants or unauthorized parties, it is equivalent to making the application's "design blueprint" public.
Unlike model weight leakage, prompt leakage is often more covert and harder to detect through conventional monitoring. Developers usually only focus on whether the API returns normally and rarely verify whether requests are isolated during server-side processing. The issue for which Together.ai was called out this time falls precisely on this easily overlooked link.
System prompt is an instruction layer injected before a conversation begins to constrain model behavior, and it is usually invisible to end users. It carries role settings, security policies, tool invocation rules, output format templates, and sometimes embedded retrieved knowledge fragments. Unlike user input, the system prompt is the application provider's intellectual property and is also a primary theft target for attackers conducting "prompt injection." Once it is read by other tenants or retained in plaintext in logs, attackers can use it to construct inputs that bypass protections or directly copy a competitor's application logic, so its confidentiality level should be treated like source code.
Common Challenges of Multi-Tenant Inference Platforms
Together.ai positions itself as providing inference services for a variety of open-source models, and its core value lies in enabling developers to call models at lower cost. Such platforms are inherently multi-tenant architectures, where requests from multiple users are scheduled and processed on the same batch of GPU clusters.
Isolation in a multi-tenant environment must be achieved at multiple levels: isolation of request queues, cleanup of KV Cache in video memory, access control for logging systems, and correctness of batching logic. A lapse in any one link may cause User A's prompt to appear somewhere visible to User B. This is not a challenge unique to Together.ai, but an engineering problem that all inference service platforms must face.
KV Cache is a core optimization mechanism in the large model inference process. When a Transformer model autoregressively generates each token, it needs to recompute the attention key (Key) and value (Value) vectors of all previous tokens; to avoid repeated computation, the inference engine caches the already-computed K/V tensors in video memory, and subsequent generation directly reuses them. This can significantly reduce latency and compute overhead, but it also means that intermediate data strongly bound to a specific user request remains resident in video memory for a long time. On a multi-tenant platform, if the cache is not cleared promptly after a request ends, or cache blocks from different requests are incorrectly reused by the scheduler (for example, due to an error in block allocation in paged attention), subsequent users may read residual features of the previous user's prompt. Therefore, KV Cache lifecycle management and video memory reclamation strategies are the most technically difficult part of tenant isolation.
Batching is a key means by which inference platforms improve GPU utilization. To lower per-request cost, platforms combine requests from multiple users within the same time window into a batch for joint forward computation, and continuous batching also dynamically inserts new requests during generation. If batching is implemented incorrectly—for example, improper construction of the attention mask, errors in sequence length alignment, or incorrect concatenation of contexts from different requests—cross-request context interference may occur. Such defects often reproduce only under specific concurrency and timing conditions and are difficult to discover through conventional functional testing, making them one of the most typical technical sources of prompt leakage.
Practical Impact on Developers
For teams currently using or planning to use third-party inference APIs, this discussion provides an opportunity to re-examine supply chain security. A few practical recommendations:
- Avoid writing keys, internal URLs, real customer identity information, or other content that must not be disclosed into system prompts;
- Adopt desensitized prompt structures for sensitive business operations and place critical logic in your own server for secondary processing;
- Pay attention to service providers' security announcements and incident response records, not just price and throughput metrics.
It should be noted that public information currently consists only of the post title and limited discussion, and it is not yet possible to confirm the specific scope of the leak, the trigger conditions, or whether it has been fixed. Before an official statement appears, it is inappropriate to draw conclusions about the severity of the incident.
Industry-Level Implications
This discussion reflects a broader trend: as more and more enterprises hand inference workloads to third-party platforms, the security boundary of AI infrastructure is expanding from "whether the model is secure" to "whether the request lifecycle is secure." Alignment research at the model level is certainly important, but engineering-level tenant isolation, log desensitization, and cache cleanup also determine the actual risk in production environments.
For inference service providers, prompt leakage issues are likely to become a standard question in customer due diligence. Whoever can establish verifiable isolation mechanisms and transparent incident disclosure processes earlier will be better positioned to win the trust of enterprise customers.
One-line Summary
Together.ai is accused of prompt leakage in requests, exposing the isolation risks of multi-tenant inference platforms.
Paragraph Summary
A discussion on Hacker News about Together.ai's "Prompt Leakage for Requests" has brought the request isolation issue of multi-tenant inference platforms to the forefront. The article points out that prompts have evolved into "new code" that carries business logic and sensitive data, and their leakage is more covert than model weight leakage and difficult to detect through conventional monitoring. Multi-tenant platforms need to achieve isolation at multiple levels, including request queues, KV Cache cleanup, log access control, and batching logic; a lapse in any link may lead to cross-user data interference. For developers, keys or customer identity information should not be written into system prompts, sensitive business operations should be desensitized and processed secondarily, and service providers' security announcements should be monitored. Because public details are limited, the scope of the incident and its fix status still await official explanation, but the industry's attention to request lifecycle security will increase accordingly.
Related articles

Codex for Beginners: 13 Hands-On Projects from Installation to Building Websites and Blender Modeling
Complete Bilibili Codex beginner tutorial: from registration and installation, permissions and model selection, to 13 hands-on scenarios including environment setup, GPT image generation for personal websites, Skill packaging, MCP connection to Blender modeling, CapCut control, scheduled tasks, and mobile control.

Python + AI from Scratch: Environment Setup and Core Concepts Guide
An introduction to a beginner Python and AI course: why Python is the top choice for AI, how to set up a versatile development environment, five core use cases, and AI talent demand analysis.

OpenAI Codex Complete Guide: From Installation to Hands-On Practice, a Full Breakdown of the AI Coding Agent
OpenAI Codex complete guide: the essential difference between Codex as an AI coding agent and ChatGPT, its four usage forms—CLI, web, plugin, and APP—and their scenarios, plus why developers should learn Codex in the AI era.