DEV Community

Anoymask
Anoymask

Posted on Edited on

LiteLLM 1.107.0 Supply Chain Compromise: Python Startup Hooks Steal Cloud and AI Credentials

1. Basic Information

2. Executive Summary

The LiteLLM release workflow was compromised, introducing an info-stealer and a remote Python loader into officially signed PyPI wheels and Docker images (v1.107.0). In Python environments where the malicious files are loaded during startup, simply running Python or pip commands triggers the malicious code even without explicitly importing LiteLLM. Official root cause analysis confirmed that an Agent Platform master key exposed in a public repository in May, which was never revoked, served as the initial vector. Note that the presence of collection-suppression environment variables and success markers means that data exfiltration did not necessarily complete in every execution.

Key Updates as of October 25

  • On October 23, LiteLLM published a canonical root cause analysis that supersedes previous hypotheses. Attackers used the master key exposed on October 10 to access the Agent Platform API and obtain an unrestricted GitHub CLI OAuth token belonging to an organization owner.
  • Using the owner token, they executed a temporary workflow to exfiltrate two maintainer tokens and additional secrets. They then used the owner token to trigger a staging workflow, approved the production gate with one maintainer token, and deleted the release run logs using the other maintainer token. Although main was reverted and force-pushed to restore history, the audit logs and commit SHAs remain as evidence.

3. Attack Chain

Attack and Exploitation Chain Reconstructed from Public Information

  1. Attackers used an Agent Platform master key that was accidentally committed to a public repository in May 2026 and never revoked. On October 10, they obtained an unrestricted GitHub CLI OAuth token for an organization owner via the Agent Platform API, and used that token to run a temporary workflow that additionally stole two maintainer tokens and other secrets.
  2. The attackers abused repository owner privileges to disable branch protection rules and redirect the release workflow source reference to a malicious repository (Berri-AI/litellm).
  3. They published official wheel packages and container images containing malicious code through PyPI's Trusted Publisher and legitimate signing and build channels.
  4. Python startup hooks (.pth / sitecustomize) in victim environments automatically executed the malicious script, harvested and exfiltrated credentials, while a background loader fetched and executed additional code from external sources.

The table below organizes the public information above, with identifiers serving as candidate checks for internal investigation (Inference). Specific implementations and subsequent actions that remain private are not extrapolated.

Attack Stage Attacker Action / Technique Tools / Communication Defender Identifiers (Prerequisites / Artifacts)
1. Credential Theft Obtained an owner token using an Agent Platform master key exposed in a public repository, then exfiltrated two maintainer tokens and other secrets via a temporary workflow Agent Platform API, GitHub CLI OAuth token, GitHub Actions, webhook.site API requests on 2026-10-10, ci-tmp-* branches, secret-exfiltration workflows, deleted runs, audit log token hashes
2. Pipeline Tampering Lifted branch protection and changed the release workflow source reference to an external repository GitHub Actions, repository settings changes Branch protection removal logs, workflow changes referencing external repositories
3. Malicious Package Distribution Signed and published via modified legitimate workflows, leveraging PyPI's Trusted Publishing mechanism PyPI, Docker Hub, GHCR Unusually timed releases (v1.107.0), legitimate-signed malicious wheel and image hashes
4. Execution & Info-Stealing Harvested and exfiltrated environment variables and secrets via cook.pth / sitecustomize.py during Python startup HTTP POST (www.litellm.dev/healthcheck), external loader communication File access during process startup, /tmp/remote_code_loader.lock, marker creation

4. Attacker Position and Execution Location

  • A remote attacker holding stolen GitHub credentials with permissions to manipulate the LiteLLM release repository and workflows.

5. Victim and Administrator Perspective

Victims

  • Packages introduced as legitimate distributions via standard pip install or docker pull, with malware executing in the background upon Python execution.

Administrators and SOC

  • Observable via affected package versions, container image digests, Python startup files (.pth), suspicious external communications, and anomalous access in cloud audit logs.

6. Success and Failure Conditions

Success Conditions

  • Introduction of a malicious wheel or affected container image, executed in an environment where its Python startup hooks are loaded (distinguishing clearly between mere download/pull and actual execution).
  • Residual artifacts may remain in local caches, mirrors, and saved images even after public pull-down. Persistence is not strictly required for all attacks; verify the exact running digest and installed files rather than relying solely on tag names.
  • Successful credential theft requires target secrets to be readable by the executing process and network reachability to the exfiltration destination (distinguishing between hook execution, data collection, and external transmission success).

Failure Conditions and Mitigations

  • If affected artifacts are not executed, malicious code execution via startup hooks does not occur (the absence of collection-suppression markers does not prove safety).
  • Immediately isolate affected hosts, preserve forensic evidence without launching contaminated Python/pip environments, and rebuild from clean environments or virtual machines.
  • Immediately revoke and reissue all credentials and secrets that were reachable from the compromised environment.

7. What Happens Upon Success

  • Theft of credentials and SSH private keys for cloud environments (AWS, GCP, Azure, etc.), AI providers, CI/CD environments, and developers.
  • Remote execution of additional Python payloads via the background loader.
  • Lateral movement into CI/CD environments, cloud infrastructure, and downstream supply chain services using the stolen credentials.

8. Observable Logs

Inference: The following are candidate checks for internal investigations. Availability depends on products, permissions, configurations, and retention periods, and should be distinguished from confirmed observations in source reports. A single event alone does not confirm attribution to this incident or successful compromise.

Email

  • No specific email vectors have been publicly disclosed.

Proxy / SWG / DNS

  • Check for HTTP POST traffic from compromised environments to www.litellm.dev/healthcheck, as well as communication history with GitHub-hosted loader configuration endpoints or instructed external servers (distinguish clearly from normal LiteLLM API requests).

Endpoint / EDR

  • Check for the presence of cook.pth, cook.py, and sitecustomize.py, affected container image digests, Python process execution history, access to target collection files (SSH keys, cloud configs, browser data), independent loader processes, and /tmp/remote_code_loader.lock (do not launch Python in contaminated environments for investigation purposes).

Identity / IdP

  • Investigate unusual usage histories for cloud IAM roles, AI provider API keys, GitHub/npm/PyPI tokens, and SSH keys that were accessible from the compromised environment.

SaaS / Cloud

  • Review package registry retrieval logs, CI/CD runners and build caches, and host-related cloud audit logs (such as CloudTrail) to investigate API calls originating from unusual IP addresses or regions.

Network

  • Track connections to data exfiltration destinations and subsequent traffic utilizing stolen credentials in chronological order (traffic attempts alone do not confirm completed data transmission).

9. Attack Success Determination

Confirmed in Public Information

The introduction of malicious artifacts into official distribution channels, along with the implementation of startup hooks and remote loaders, has been confirmed by LiteLLM official statements and multiple security researchers. However, this does not imply that execution or data theft succeeded in every organization that deployed the software.

Determination Criteria for Your Organization

  • Initial Execution Confirmed: Identified process execution history or related files where startup hooks were loaded, in addition to the introduction of affected artifacts (mere download or pull does not constitute execution success).
  • Credential Theft or Session Compromise Confirmed: Supported by the presence of the success marker file /tmp/.pc_once_72129118c59f5636c0bb or confirmed HTTP 200 completion logs to exfiltration destinations. The absence of markers or connection attempts alone does not confirm success.

10. Investigation Playbook

  • Trigger: Initiate investigations based on the deployment of LiteLLM 1.107.0, affected container digests, detection of cook.pth / sitecustomize.py, or traffic destined for litellm.dev.
  • Initial Checks: Verify package deployment history, container digests, execution times around 2026-10-21, outbound traffic, and IAM roles assigned to hosts.
  • Endpoint and Server Investigation: Do not launch Python within affected environments; preserve and investigate site-packages, lock files, processes, and persistence settings offline (e.g., via mounted storage).
  • Credential and Cloud Investigation: Check usage histories for AWS CloudTrail (such as GetCallerIdentity), AI provider keys, GitHub/npm/PyPI tokens, and SSH private keys.
  • Follow-up Actions: Track external cloud API executions, repository tampering, deployment of additional payloads, internal reconnaissance, and data exfiltration.
  • Containment: Immediately isolate affected hosts from the network, rebuild them, pin to verified artifacts, and revoke and reissue all secrets that were accessible.
  • Assessment Categories: Clearly distinguish between contact, execution, authentication success, data acquisition, and subsequent compromise stages, without presuming unconfirmed success.

11. Defense and Detection Ideas

  • Single Events: Detect the generation of cook.pth / sitecustomize.py, creation of /tmp/remote_code_loader.lock, and traffic destined for litellm.dev/healthcheck.
  • Chronological Correlation: Correlate the flow of package retrieval -> Python process execution -> credential file access -> archive generation -> litellm.dev communication -> cloud API usage.
  • Threat Hunting: Comprehensively search for affected artifacts across PyPI installation logs, container pull logs, SBOMs, image digests, and CI runner caches.
  • Log Limitations: The absence of marker files does not prove the absence of compromise, as container terminations or temporary file deletions may have erased traces.
  • Priority Mitigations: Prioritize pinning to artifacts confirmed safe, verifying digests, checking build provenance, enforcing least privilege on secrets, strictly limiting outbound traffic, and comprehensively reissuing all accessible secrets (avoid relying solely on official signatures for safety determinations).

12. Facts / Inference / Hypothesis

Facts

  • LiteLLM announced on October 21, 2026, that its release pipeline was compromised, resulting in malware injected into all 7 wheel variants of PyPI litellm 1.107.0 and multiple Docker/GHCR tags.
  • On May 27, 2026, an Agent Platform master key was committed to a public repository; although history was overwritten via force-push, the key was not revoked. Attackers used this key on October 10 to obtain an organization owner GitHub CLI OAuth token from the Agent Platform API.
  • Using the owner token, they ran a temporary workflow to additionally exfiltrate two maintainer tokens and other secrets. The attackers used the owner token to bypass branch protection, triggered a staging workflow, approved the production gate with one maintainer token, and deleted run records with the other maintainer token. Finally, they used the owner token to force-push main and revert surface history.
  • cook.pth and sitecustomize.py execute automatically upon Python startup, collecting and encrypting credentials, private keys, and browser data for cloud, AI provider, and CI/CD environments before transmitting them to litellm.dev.
  • An independently operating loader periodically checks GitHub pointers and fetches/executes additional Python code from external servers.
  • The seven PyPI wheels were affected between 11:39:51 and 13:23 UTC on October 21, and Docker Hub tags (v1.107.0, latest, main-latest, main-stable) between 11:46:56 and 14:20 UTC. Affected GHCR tags became unavailable by 13:51 UTC. The official GitHub v1.107.0 source tag contained no malicious files and is distinct from distribution artifacts originating from attacker-controlled repositories; Enterprise Private releases were unaffected.

Inference

  • If executed on AI gateways or CI runners, compromises may cascade beyond AI provider keys to host IAM roles and downstream systems.

Hypothesis

No additional hypotheses. Unconfirmed items are listed under "Unknowns and Additional Investigations".

13. MITRE ATT&CK Mapping

ID Technique Confidence Basis
T1195.002 Compromise Software Supply Chain high Confirmed that malicious wheels and container images were distributed through legitimate release and distribution channels.
T1546 Event Triggered Execution medium Automatic execution via .pth and sitecustomize upon Python startup (distinguished from Unix shell configuration modification T1546.004; parent technique used as candidate).
T1555 Credentials from Password Stores high Analyzed collection scripts confirmed functionality to extract browser saved passwords, cookies, and other credentials. Success of collection in each deployment environment requires separate verification.
T1105 Ingress Tool Transfer high Implementation confirmed where background loaders download and execute additional Python code from external sources.

14. Unknowns and Additional Investigations

  • While the master key exposure in the public repository and GitHub token acquisition path have been established, the specific method by which attackers discovered the exposed key, the full scope of use for exfiltrated secrets, and potential downstream unauthorized usage remain unknown.
  • The number of organizations that actually installed and executed the software during the distribution window, and the success scope of data exfiltration across environments.

15. Impact on SOCs and Organizations

Organizations deploying LiteLLM as an AI gateway, development environment, CI/CD pipeline, or Kubernetes environment should immediately check execution history for version 1.107.0 or subsequent tracking tags, promptly isolate and rebuild affected hosts, and reissue all accessible secrets from a clean environment.

16. Summary by Target Audience

  • For SOCs: Correlate and monitor artifact digests, cook.pth / sitecustomize.py, /tmp/remote_code_loader.lock, traffic destined for litellm.dev, and external cloud credential usage.
  • For Administrators: Isolate hosts without launching Python or pip in affected environments, pin to officially verified safe versions (temporary recommendation at announcement was 1.106.2), and rebuild. Revoke and reissue all accessible secrets from a clean environment. Verify latest versions and digests through official advisories.
  • For Users: General developers and users who do not directly manage LiteLLM should follow instructions from infrastructure administrators to inspect devices and reset passwords.

Top comments (6)

Collapse
 
gautamtalksdev profile image
Gautam K • • Edited

The step that worries me most is number 2: one owner token was enough to switch off branch protection and repoint the release workflow, and nothing asked a second question at that moment. Changing branch protection is a single API call. In an approval broker I'm building, a PUT or DELETE on a branch protection rule waits for a person to approve it with a passkey, so a valid token alone can't do it.

That only helps if the raw token never sits where logs can catch it, which is where LiteLLM suspects these came from. Did LiteLLM say whether the owner token was a long lived CLI token or something scoped and short lived?

Collapse
 
anoymask profile image
Anoymask •

Thanks, Gautam. LiteLLM's October 23 RCA clarifies a couple of important details.

The credential was an unscoped GitHub CLI OAuth token belonging to an organization owner. The report doesn't specify its lifetime, but it explicitly states that GitHub CLI tokens are treated as post-2FA and are not subject to GitHub's "super-user" 2FA reauthentication for sensitive organization or repository actions. That directly relates to your concern about the lack of a second authentication challenge.

The RCA also corrects the initial theory about token leakage from internal logs. A LiteLLM Agent Platform master key was accidentally committed to a public repository in May and never revoked. On October 10, the attacker used that key to access the Agent Platform API and retrieve the owner's GitHub token.

Your passkey-based approval mechanism is particularly relevant here. Independently authorizing changes to security-critical controls could provide a valuable additional boundary, even when a privileged token has been compromised.

Source: docs.litellm.ai/blog/security-inci...

Collapse
 
gautamtalksdev profile image
Gautam K •

Thanks for digging that out. A token treated as already past 2FA means a protection change needs no extra prompt, which is the gap a passkey per sensitive action would cover. The master key sitting in a public repo since May is the part that stands out: did the RCA say whether anything was scanning for committed keys?

Thread Thread
 
anoymask profile image
Anoymask •

Good question. The RCA doesn't specify whether automated secret scanning was in place or how the exposed key was initially detected.

It does say the mistake was identified quickly, and the file was removed about three hours later. However, the master key was never revoked, and the original commit remained in the public Git history until October 22.

So the confirmed failure was in following through on key revocation, regardless of how the exposure was initially detected. That's an important distinction.

Collapse
 
cryptostrapon profile image
cryptostrapon •

Worth stressing for readers: a .pth file runs on every Python start, not only at install, so installing with scripts disabled does not help here. Since the release was signed through the real pipeline, hash pinning would not have caught it either. What would have: a release cooldown in CI (minimumReleaseAge in Renovate, exclude-newer in uv), so a version only gets pulled after it has been public for a few days.

Collapse
 
anoymask profile image
Anoymask •

Good points—especially the repeated execution risk from the .pth startup hook on subsequent Python launches in an affected environment.

I agree that a dependency cooldown (minimumReleaseAge or exclude-newer) could have prevented this particular release from being pulled, especially since the poisoned PyPI wheels were removed within roughly two hours.

One small nuance on hash pinning: while verifying the hash of a newly pulled release wouldn't catch an artifact poisoned inside the official pipeline, keeping lockfiles pinned to previously verified safe versions and artifact hashes still helps prevent unintended upgrades.

Ultimately, the broader lesson is to layer lockfile pinning, release-age cooldowns, and independent artifact and behavioral validation rather than treating a valid signature or hash as proof of safety. Thanks for highlighting those operational controls!