PEP 668 Blocks Runtime Package Installation: A Postmortem of SQLAlchemy Implicit-Flush Deadlocks

A PoC script meant to reproduce a SQLAlchemy autoflush deadlock crashed before it ever touched the database. The cause wasn't a flaw in the ORM logic—it was that the script tried to dynamically install missing packages via pip install at its entry point, and Debian's PEP 668 protection mechanism refused outright. This failure turned out to be more valuable than the concurrency experiment it was meant to run—it exposed both a boundary problem in dependency management and a lock-ordering trap in implicit flushing.

The Failure Scene: Exit Before Startup

The script's execution path was clear: first check whether orjson exists, install it via sys.executable -m pip install if missing, then import pydantic and sqlalchemy, and finally launch two threads to simulate a transfer deadlock. At runtime, the very first round of dependency checks triggered an externally-managed-environment error, and subprocess.CalledProcessError caused the process to exit with a non-zero status.

One detail worth noting: the error occurred inside an already-activated virtual environment. By conventional wisdom, pip within a venv shouldn't be bound by system-level protections, but pip determines environment ownership based on sys.prefix and the location of the EXT-MARKER file. If the venv was created incompletely, if PYTHONPATH was inherited from outside, or if the site-packages symlink is abnormal, pip may identify the root installation as "externally managed" and refuse to write. These edge conditions are uncommon on developer machines but easy to reproduce in containers and sandboxes.

What PEP 668 Is Actually Protecting

PEP 668 introduces the externally-managed-environment marker, allowing the operating system to declare via an EXT-MARKER file that this Python environment is managed by the system package manager (such as apt) and that pip must not arbitrarily modify it. The goal is to prevent apt and pip from writing to the same site-packages simultaneously, avoiding upgrade conflicts and irreversible dependency pollution.

For application developers, the implication is direct: the filesystem location of the Python interpreter determines who is allowed to install things into it. The Python maintained by a distribution belongs to the operating system—to apt; a project's own dependencies should live in the project's own venv or container image. Stuffing installation actions into runtime means letting business code touch the permission boundary of infrastructure.

The full filename of EXT-MARKER is EXTERNALLY-MANAGED, a plain text file placed in the stdlib directory of a Python environment (for CPython, that's lib/python3.x/EXTERNALLY-MANAGED), following INI format. The file typically contains an [externally-managed] section and an Error= field; when pip hits this file, it prints the content of Error as a prompt. Debian/Ubuntu have placed this file in the system interpreter by default since 23.04, and distributions like Fedora and Arch have gradually adopted it. To be clear, PEP 668 only makes pip perform a check before writing and refuse by default—it doesn't change filesystem permissions itself, so --break-system-packages or directly invoking setup.py can still bypass it. This protection relies on convention and default behavior, not enforced isolation.

Anti-Pattern: In-App Dynamic Dependency Resolution

The ensure_dependency function in the original script is a typical "self-healing" design—install whatever is missing, which seems to make the script more self-contained. But it doesn't hold up across multiple dimensions.

• Guaranteed failure in container environments: Read-only root filesystems, non-privileged users, and VPCs without external network access are all the production norm—runtime installation has no room to succeed.
• Non-deterministic state: Multiple threads or processes triggering bootstrap simultaneously can produce half-installed states, making import behavior random.
• Permission and audit violations: Runtime code should not have the ability to access the network and write to system directories—this is usually a hard red line in compliance audits.

Dependency provisioning belongs to the orchestration layer—requirements.txt, pyproject.toml, uv, Poetry, Dockerfile. Runtime is only responsible for consuming an environment that's already in place. When dependencies are missing, the correct approach is to fail immediately with clear installation instructions, not to attempt self-repair.

The Interrupted Experiment: How Implicit Flushing Creates Deadlocks

The problem the script was originally meant to verify is itself valuable. SQLAlchemy's sessionmaker defaults to autoflush=True—before any Query or select() executes, the session flushes pending dirty data to the database first, to ensure query results are consistent with in-memory state.

The problem lies in the timing of lock acquisition. Two threads modify accounts in opposite orders: Thread-1 modifies account 1 then queries account 2, while Thread-2 modifies account 2 then queries account 1. After a barrier sync, both sides issue their second query almost simultaneously—at which point autoflush triggers, flushing each thread's earlier UPDATE to the database early and holding row locks, then each waits for resources held by the other, forming a classic lock-order-inversion deadlock.

SessionLocal = sessionmaker(autocommit=False, autoflush=True, bind=engine)

`autoflush` and `autocommit` are two independent switches that are easy to confuse. `autocommit` (replaced by `autobegin` semantics in SQLAlchemy 1.4/2.0) determines whether statements automatically commit transaction boundaries; `autoflush` only determines whether the Session's `dirty` and `new` objects are flushed to the database as INSERT/UPDATE before a Query/`select()` is issued. Flush merely sends the SQL out—the transaction is still open, and row locks are not released until the transaction commits or rolls back, which is exactly why the lock can be observed and awaited by another thread. Additionally, the specific granularity of row locks depends on the database and isolation level: MySQL InnoDB under the default REPEATABLE READ may escalate to gap locks for non-indexed conditions, while PostgreSQL's default READ COMMITTED locks only matched rows—so the same reverse lock-ordering produces different deadlock behavior and reproduction probabilities on different backends.
# After acc1.balance -= 100
acc2 = session.query(Account).filter_by(id=secondary_id).first()
# The implicit flush before this SELECT locks the row containing acc1 early

In other words, implicit flushing turns an otherwise harmless read query into a write transaction that holds locks early. In single-threaded code this is convenient; under concurrency it's a catalyst for deadlocks.

Three Boundaries for Engineering Practice

First, respect environment boundaries. PEP 668 is not an obstacle but a protection of determinism. Any --break-system-packages that bypasses it should only appear in one-off debugging scenarios and must never enter the codebase.

Second, fail fast. Missing dependencies are a deployment problem, not a runtime problem. Letting the process exit with a clear error is far safer than letting it continue running in an uncertain state.

Third, isolate the validation environment. To test ORM lock contention and transaction isolation, provision dependencies in advance so that measurements reflect database behavior rather than being polluted by environment anomalies. As for autoflush, explicitly setting it to False on concurrent write paths, or enforcing a uniform lock ordering, are both cheaper choices than troubleshooting deadlocks after the fact.

One-line Summary

A script meant to reproduce a SQLAlchemy deadlock got blocked by PEP 668 first, exposing dual traps in dependency management and implicit flushing.

Paragraph Summary

The article revisits an unexpected failure of a PoC script: intended to reproduce a two-thread reverse-lock-ordering deadlock in SQLAlchemy, the script was instead rejected at its entry point by Debian's PEP 668 mechanism with externally-managed-environment when it tried to install dependencies via runtime pip install, exiting before touching the database. The author follows two threads: first, PEP 668 declares via an EXT-MARKER file that system Python is governed by the package manager, so runtime installation is out of bounds—and can even be misjudged under venv edge conditions; in-app dynamic dependency resolution inevitably fails in read-only containers, non-privileged users, and no-external-network environments, and brings risks of half-installed states and audit issues. Second, the interrupted experiment itself: autoflush=True flushes dirty data before SELECT, causing read queries to hold locks early, forming a deadlock under reverse lock ordering. The conclusion is that dependency provisioning should be left to the orchestration layer, missing runtime dependencies should fail fast, and concurrent write paths should explicitly disable autoflush or enforce a uniform lock ordering.