LiteLLM Supply Chain Poisoning Incident: A Deep Dive into 40 Minutes of Full-Stack Credential Theft

A Supply Chain Attack That Lasted 40 Minutes

On March 24, 2026, two malicious versions of LiteLLM—1.82.7 and 1.82.8—were pushed to PyPI, remaining online for approximately 40 minutes before being pulled. The window was brief, but the damage potential was staggering. This attack was part of a larger "TeamPCP" campaign, traced back to a leaked Trivy automation token.

Software supply chain attacks are among the most threatening attack vectors in recent years—attackers don't directly target systems but instead compromise upstream components that the target depends on. PyPI, as the largest third-party package repository in the Python ecosystem, sees over 1 billion downloads per day, meaning any poisoned package can spread to vast numbers of production environments in an extremely short time. Trivy is an open-source vulnerability scanner developed by Aqua Security, widely used in CI/CD pipelines for security auditing of container images and dependencies. After obtaining Trivy's automation token, the attackers were able to inject malicious code through the official maintenance process—an attack path that exploits security tools themselves, which is both ironic and devastatingly effective.

Notably, the FBI has issued a FLASH advisory (TLP: CLEAR, publicly shareable) regarding this incident, indicating it has entered official security response channels. FBI FLASH advisories are the Bureau's mechanism for rapidly sharing cyber threat intelligence with the private sector, and the TLP:CLEAR marking means the information can be shared without restriction—signaling that the severity of this incident triggered a national-level response. For any team running an LLM gateway in production, this isn't an episode to brush off—it's a hard lesson in software supply chain security.

Attack Mechanism: Why Python .pth Files Are So Dangerous

What should truly alarm every MLOps engineer about this attack is its technical implementation.

Silent Execution That Bypasses Import

The malicious package contained a .pth file. Unlike regular Python modules, .pth files are executed at interpreter startup, not when code is imported.

To understand the danger of this mechanism, you need to understand Python's startup process: CPython traverses all .pth files in the site-packages directory when initializing the site module, parsing them line by line—ordinary path lines are added to sys.path, while lines beginning with import are directly executed via exec(). This mechanism has existed since Python 2.x, originally designed to facilitate package initialization configuration (for example, setuptools' .pth files enable pkg_resources), but it effectively provides an entry point for executing arbitrary code before any user code runs.

This means:

• Whether your code actually calls litellm is completely irrelevant;
• As long as the package is installed and any Python process starts, the malicious payload runs;
• Most security scanners treat "import time" as the detection point, but .pth payloads have already finished executing long before that.

This design renders traditional runtime detection nearly useless. Static analysis tools like Bandit and import hook-based runtime monitoring solutions struggle to catch this type of attack because the malicious code's execution path completely bypasses their detection points. The attackers precisely exploited a long-overlooked execution entry point in the Python ecosystem.

Theft Targets: API Keys, Cloud Credentials, and SSH Keys

Once the payload runs, it collects environment variables, SSH keys, cloud credentials, Kubernetes service account tokens, and various provider API keys. In containerized MLOps environments, this information is typically provided to application processes via environment variable injection or mounted Secrets—a design that aligns with twelve-factor app best practices, but also means any code running within the process memory space can access all sensitive information through os.environ or filesystem reads.

Why LiteLLM Is the Highest-Value Attack Target in MLOps

The poster (from CI/CD security company InvisiRisk) highlighted a critical insight: LiteLLM's design positioning makes it the highest-value credential theft landing point in the entire ML tech stack.

LiteLLM is an open-source unified proxy layer for LLM APIs, unifying the APIs of dozens of model providers—OpenAI, Anthropic, Google Vertex AI, AWS Bedrock, Azure OpenAI, and more—into an OpenAI-compatible format. In typical MLOps architectures, LiteLLM acts as a reverse proxy: all internal applications only need to communicate with LiteLLM, which handles routing, load balancing, rate limiting, and cost tracking. This architectural design borrows from the API Gateway concept in microservices (like Kong, Envoy), but in AI scenarios there's a critical difference—it must hold authentication credentials for all downstream services.

As a unified gateway, LiteLLM inherently "sits in front of all model services." This means a single process often simultaneously holds:

• Your OpenAI Key
• Your Anthropic Key
• Your Bedrock credentials
• And all other Provider keys routed through it

Traditional API gateways typically rely on downstream services' own authentication mechanisms, but LLM gateways must store all Provider keys in their own process memory because they need to proxy API calls. In other words, compromising LiteLLM at a single point equals obtaining an organization's entire model access permissions in one shot. And during those fatal 40 minutes, it was not only the highest-value target but also the easiest to exploit. This is precisely the security paradox of AI infrastructure centralization: convenience is directly proportional to attack surface.

Post-Incident Investigation: Why Standard Detection Methods Almost All Failed

The truly thorny issue is: months later, can you still confirm whether you installed the malicious version? The poster acknowledged that most intuitive checking methods don't work here.

The Trap of Loose Version Pinning

If your dependency declaration was loose, such as litellm>=1.82, and a build happened to occur during that time window—you were very likely compromised.

Dependency management in the Python ecosystem operates at multiple levels: loose declarations in requirements.txt only express compatibility constraints, while lock files (like precise requirements generated by pip-tools, Poetry's poetry.lock, PDM's pdm.lock) record exact versions and hashes determined during a specific resolution. The problem is that many teams only keep loose declarations without committing lock files to version control, or use lock files but allow automatic updates in CI/CD. More critically, even if lock files are preserved, the actual installation behavior during Docker image builds may vary due to cache layers and build timing differences.

Resolved Dependency Manifests Get Discarded

Worse still, resolved dependency manifests are typically discarded after the build. A resolved manifest is the complete record of packages and their exact versions that pip/poetry actually downloaded and installed at build time. So the question "what version did we actually install on March 24" often cannot be answered months later. This exposes a widespread gap in many teams' build artifact retention and auditability—precisely the core problem that Software Bill of Materials (SBOM) aims to solve. SBOMs are designed to generate a complete ingredient list for every software build artifact, making it possible to trace back "what exactly is in this artifact" at any point in time.

Two Investigation Approaches: Version History vs. Credential Exposure Detection

For this incident, the post mentioned a more efficient method and clarified the distinction between two different types of questions.

Question One: Did I pull the malicious package?

This requires version history to answer—checking lock file history, checking registry pull logs. But as discussed, these records are often no longer available.

Question Two: Were my keys actually stolen?

CloudSEK published a query tool for this incident (exposure.cloudsek.com/ai-supply-chain-incident). It answers a question that's closer to the outcome: whether your keys appear in the data actually collected by the attackers. According to the poster, this is a check that "can be completed in 30 seconds."

It's important to emphasize: even if there's a hit, it does not confirm a breach. The FBI advisory similarly notes—discovering a malicious dependency does not prove the malicious code actually executed. A hit should be treated as "grounds for deeper investigation," not "a conclusion about the incident itself."

Practical Defense Recommendations for MLOps Teams

This incident provides a clear action checklist for all teams running AI gateways:

1. Immediately verify the time window: If you use loose version pinning, trace back build records around March 24.
2. Leverage exposure query tools: When lock history cannot be reconstructed, use credential exposure queries to quickly assess risk level.
3. Assume and rotate: Many teams take the pragmatic approach—rather than spending time on "lock file archaeology," directly rotate all potentially exposed credentials (API keys, SSH keys, cloud credentials, etc.). Credential rotation is the core action in security response; when the scope of impact cannot be determined, assuming the worst case and rotating comprehensively is an industry-recognized best practice.
4. Tighten dependency locking strategies: Use exact version pins and complete lock files, and retain resolved manifests long-term to make future audits possible. Consider integrating SBOM generation tools (such as Syft, CycloneDX) into your CI/CD pipeline to automatically generate traceable dependency manifests for every build.
5. Re-evaluate the gateway's credential isolation architecture: Since a single gateway holds full-stack keys, consider defense-in-depth measures like least privilege, key isolation, and short-lived credentials. Specifically, use dynamic secrets from key management systems like HashiCorp Vault to generate temporary credentials for each API call; leverage AWS STS temporary credentials (default validity of 1 hour) or Google Cloud Workload Identity Federation instead of long-lived static keys; use separate, permission-restricted service accounts for each downstream Provider to ensure a single point of leakage doesn't lead to total compromise.

Conclusion: The Security Cost of AI Infrastructure Centralization

The LiteLLM incident is a microcosm of AI infrastructure security. When we centralize all model access into a unified gateway for convenience, we inadvertently create a "single point of fatal failure" attack target. The 40-minute window, .pth's silent execution, and the difficulty of reconstructing dependency history after the fact together reveal the fragile underbelly of the modern ML supply chain.

This incident also reflects the structural challenge facing the entire cloud-native ecosystem: in our pursuit of development efficiency, we've built ever-deeper dependency chains and increasingly centralized trust points. From SolarWinds to Log4Shell to this LiteLLM incident, every supply chain attack reminds us—the security boundary of modern software is no longer the few lines of code you wrote yourself, but rather the weakest link in the entire dependency graph.

The real question is perhaps not "who got hit this time," but "can we find out faster next time."