How OpenAI Scales Git Repositories

OpenAI's official blog published an article titled "Scaling Git at OpenAI," which quickly sparked widespread discussion on Hacker News, earning 235 upvotes and 149 comments. For a company centered on AI research, scaling Git repository management may seem like an infrastructure detail, yet it reflects a long-underestimated challenge in ultra-large-scale code collaboration.

Why Git Scaling Becomes a Problem at OpenAI

Git was designed for distributed collaboration from its inception, and Linux-kernel-scale projects have long validated its feasibility in large open-source communities. But OpenAI faces a somewhat different situation: repositories must not only carry large amounts of code but also intertwine with non-traditional software assets such as model training, experiment configurations, and evaluation scripts.

When repository size, commit history, and branch count all grow simultaneously, Git's default behaviors—such as cloning entire histories and computing full diffs—quickly become bottlenecks in developers' daily operations. This is not a matter of some single operation slowing down, but rather latency accumulating across every pull, branch switch, and code review in the entire development loop.

The article's core premise is precisely this degradation in experience: when a foundational tool starts slowing down engineers' rhythm, it shifts from "transparent infrastructure" to "a problem that needs to be engineered."

Key Challenges of Scaling

Repository Size and History Depth

The typical pain points of large monorepos are clone time and disk usage. The more complete the history and the more participants, the higher this cost. For teams that need to switch branches frequently and conduct code reviews, waiting for full history synchronization on every operation is unacceptable.

OpenAI's practice shows that scaling Git is not simply a matter of swapping in a faster server, but rather finding a balance between retaining critical history information and controlling transfer costs. Partial history isolation and on-demand fetching are common engineering directions.

Partial clone and shallow clone are the two main mechanisms for coping with repository bloat. Shallow clone uses the --depth parameter to fetch only the most recent commits, at the cost of losing full history, which affects git blame, bisection, and merge-base computation. Partial clone (--filter=blob:none or --filter=tree:0) retains commit and tree objects and only lazily downloads file contents (blobs) on demand, so history queries remain available—though first access to certain files requires an extra network round trip. Partial clone has matured gradually since Git 2.19, and together with promisor remotes and git backfill, common objects can be prefetched in the background.

Another approach is the commit-graph file, which precomputes and stores the parent-child relationships and generation numbers of commits, turning git log --graph, reachability checks, and merge-base lookups from object-graph traversal into table lookups—yielding several-fold speedups on repositories with millions of commits. In practice, partial clone, commit-graph, filesystem monitoring (FS Monitor), and sparse-checkout are typically used in combination, addressing the four dimensions of transfer cost, history computation, working-tree scanning, and disk usage respectively.

Branches and Collaboration Patterns

When multiple teams work in parallel on the same repository, the number of branches grows rapidly. Git's storage cost for branches themselves is low, but reference updates, merge computations, and state synchronization around branches amplify with scale. The more complex the collaboration pattern, the more directly the underlying protocol is tested.

Reference advertisement is a key protocol detail for understanding the cost of branch scale. At the start of a fetch/push, the server first sends the names and SHAs of all its references to the client, which uses them to determine which objects are missing. When branches and tags number in the hundreds of thousands, this advertisement alone consumes considerable bandwidth and parsing time, even when there are actually no new objects to transfer.

Ref-filtering (such as ref-prefix in ls-refs), introduced in Git protocol v2, allows clients to request only the namespaces they care about, significantly reducing advertisement overhead. On the server side, packed-refs and the reftable format are commonly used to replace per-file loose refs, avoiding the inode and stat pressure of massive numbers of small files in a directory. In addition, choosing fetch.negotiationAlgorithm, enabling core.commitGraph, and configuring mirrors or cache layers for high-frequency read-only scenarios are all common measures to mitigate the amplification of multi-branch collaboration. These optimizations yield limited benefits individually, but only when combined can branch-intensive workflows maintain interactive response times.

Toolchain and Development Loop

Git rarely exists in isolation. CI systems, code review tools, and IDE integrations all read and write repository state. If a repository scaling solution only optimizes Git itself while ignoring the surrounding toolchain, developers' actual experience may not improve. This is why large-scale transformation often requires systematically examining the entire development loop.

Lessons for Engineering Teams

OpenAI's case highlights a frequently overlooked fact: the infrastructure challenges of AI companies are not limited to GPUs and training clusters, but also lie in the "software supply chain" that supports engineers' daily collaboration. Git repository scaling problems are common in organizations with multiple teams and large codebases—only the scale threshold differs.

For teams currently experiencing repository bloat, the value of this article lies in providing a thinking framework—treating Git performance issues as observable, engineerable problems rather than as inevitable friction. Reasonable repository splitting strategies, history management approaches, and toolchain adaptation can all significantly improve the development experience.

The active discussion in the Hacker News community also shows that Git scaling is not an isolated case for any one company, but an infrastructure topic faced collectively by many engineering organizations. Specific implementation paths vary by team, but the direction of "proactively designing and governing repository scale" has universal reference value.

This article is compiled based on the OpenAI official blog post and Hacker News discussion information. For specific technical solutions, please refer to the original article.

The monorepo vs. polyrepo debate is not an either-or proposition. Companies like Google and Meta rely on self-developed virtual filesystems and on-demand materialization layers (such as Google's Piper and Meta's Sapling) for ultra-large monorepos, an investment that ordinary teams find difficult to replicate. More realistic middle-ground approaches include: splitting repositories by lifecycle or permission boundaries, managing shared components with Git submodule or subtree, reducing full builds through affected-target analysis in build systems like Bazel/Nx, and using sparse-checkout within a single repository to delineate each team's workspace.

To determine whether the threshold for splitting has been reached, several quantifiable signals can be observed: P50/P95 durations of clone and fetch, the proportion of the checkout phase in CI, git status response time in large working trees, and indexing latency of code review tools. Only by incorporating these metrics into daily monitoring can the repository itself be treated like any other service—and this is precisely the concrete embodiment of making Git performance issues "observable."

One-line Summary

OpenAI shares Git repository scaling practices, revealing an underestimated infrastructure challenge in ultra-large-scale code collaboration.

Paragraph Summary

OpenAI's official blog "Scaling Git at OpenAI" sparked heated discussion on Hacker News, reflecting another category of infrastructure challenges for AI companies beyond GPUs. The article points out that when repository size, commit history, and branch count grow simultaneously, Git's default behaviors—cloning full history and computing full diffs—cause latency to accumulate across pulls, branch switching, and code reviews, degrading a foundational tool from transparent infrastructure into a problem requiring engineering governance. The core challenges center on three aspects: repository size and history depth, branches and collaboration patterns, and the read/write amplification of repository state by surrounding toolchains such as CI, code review, and IDEs. The article emphasizes that scaling Git is not about swapping in faster servers, but about balancing retention of critical history against controlling transfer costs, and advocates proactively designing repository scale as an observable, governable engineering problem.