AI Can Write Ruby But Can't Navigate Codebases? A 5-Model, 13-Project Benchmark Reveals the Truth

The Hidden Blind Spot in AI Programming: Can Write, Can't Read

When we talk about AI coding assistants, we almost always fixate on the same question: can it write correct code? Can it implement an algorithm or complete a logic block? Yet a benchmark test covering 5 major large language models and 13 real-world codebases has produced a surprising answer — AI agents are good at generating Ruby code, but they frequently stumble when it comes to navigating existing codebases.

This finding is striking because it cuts to the core reality of software engineering: the vast majority of development work isn't writing code from scratch — it's understanding, locating, modifying, and maintaining code within large, existing systems. A tool that can only "write" but can't "read and move through" a codebase is considerably less valuable in production environments.

Why "Generation" and "Navigation" Are Two Fundamentally Different Skills

Local Tasks vs. Global Tasks

Code generation is fundamentally a local task: given a clear requirement, the model outputs a self-contained snippet. This aligns closely with how large models are trained — vast amounts of open-source code provide abundant pattern material, and Ruby's syntax and idioms have been deeply absorbed by these models.

It's worth understanding why this "natural fit" exists. LLM pretraining is essentially probabilistic modeling of large text sequences — given prior context, predict the next token. OpenAI's Codex model (behind GitHub Copilot) was trained on over 54GB of public code; subsequent models like GPT-4 and Claude were trained on several times that amount. This training paradigm almost perfectly matches the "local code generation" task: models have seen enough "requirement description → code implementation" patterns to naturally complete logic given clear context. But training data contains almost no complete examples of "tracing a call chain through a 100,000-line unfamiliar codebase" — which is precisely why training signals for navigation ability are so scarce.

Code navigation, by contrast, is a global task: the model must trace method call chains, understand module dependencies, and locate where a piece of business logic actually lives across potentially hundreds of files and tens of thousands of lines of code. This requires building a "mental map" of the entire project structure — not just processing the code snippets currently in view.

Ruby Metaprogramming: A Natural Navigation Fog

Ruby's highly dynamic nature becomes a unique obstacle here. Metaprogramming capabilities like method_missing, runtime dynamic method definition, and send calls make code extremely flexible — but also make static analysis extremely difficult. A method may not explicitly appear anywhere in the source code, instead being dynamically generated at runtime.

Specifically, method_missing is a hook method that fires automatically when an object receives an undefined message. Rails' ActiveRecord uses it to implement hundreds of seemingly "conjured" query methods like find_by_name and find_by_email. define_method allows creating methods dynamically at runtime, while send can bypass access control to call any method by name — including names assembled from string concatenation. This means in a Ruby codebase, there may be no statically traceable link between "a method being called" and "that method's definition appearing in source code." For navigation tools that rely on AST (Abstract Syntax Tree) parsing and symbol indexing, this is a structural obstacle, not an implementation detail.

By comparison, statically typed languages like Java and TypeScript have explicit type signatures and declarations that are much easier for both tools and models to navigate. Ruby's "convention over configuration" philosophy — especially in the Rails ecosystem — compounds this challenge: large amounts of core logic are hidden in framework conventions rather than explicit lines of code.

The Benchmark: Design Philosophy and Key Findings

Why "Real Codebases" Matter

The core value of this test lies in its authenticity. Many mainstream programming evaluations rely on carefully crafted synthetic problems, while 13 real-world codebases means models must confront the full complexity of production environments: unconventional naming habits, legacy technical debt, implicit cross-file dependencies — the real challenges engineers face every day.

Through a cross-model comparison of 5 models, the test focused on several key questions:

• How large is the capability gap between "generation" and "navigation" tasks for each model?
• Is this gap a universal trait across all models, or are there notable exceptions?
• How do codebase size and complexity affect navigation performance?

A Warning for Industry Evaluation Standards

The test findings point to a deeper issue: the standards we use to measure AI coding ability may themselves have a systematic bias. Mainstream benchmarks like HumanEval and MBPP mostly focus on function-level code generation — precisely where AI excels — which objectively overstates AI's practical value in real engineering scenarios.

Understanding the scale of this bias requires knowing what these benchmarks were designed for: HumanEval was released by OpenAI in 2021 with 164 function-level Python problems — each providing a function signature and docstring, asking for a completed implementation. MBPP (Mostly Basic Python Problems), released by Google, contains 374 beginner-oriented programming problems. Both share the same characteristic: tasks are entirely self-contained, inputs and outputs are explicit, and no understanding of external dependencies is required. This design enables automated evaluation but systematically avoids the core complexity of real engineering scenarios. SWE-bench — currently the closest benchmark to real-world conditions, requiring models to resolve actual GitHub issues — has a pass rate below 50% even for the strongest models, confirming the enormous capability gap between "generation problems" and "real engineering tasks."

An AI agent that passes every generation-focused benchmark but cannot locate where changes need to be made in a real codebase still has a long way to go before it can "autonomously complete engineering tasks."

The Evolution of AI Coding Tools

Code Indexing and Symbol Resolution: A More Practical Fix Than Scaling Parameters

Closing the navigation gap may not be best addressed by simply scaling model parameters or expanding context windows. A more promising direction is deeply integrating large models with traditional engineering tools like code indexing, symbol resolution, and call graph analysis — letting deterministic tools handle "locating," while large models handle "understanding and reasoning."

This direction already has mature engineering foundations. The "jump to definition" and "find all references" features in traditional IDEs rely on the Language Server Protocol (LSP) — a communication protocol between editors and language servers, pioneered by Microsoft and now an industry standard. LSP servers continuously maintain symbol tables, call graphs, and type information for a codebase in the background, enabling millisecond-precision jumps. Combining LSP toolchains with large models — having the model obtain precise symbol definition locations via tool calls rather than relying on the model to "guess" where code lives — is the core architectural approach of current agentic coding tools like Cursor and Cline. For dynamic languages like Ruby, tools such as Solargraph use a hybrid strategy combining heuristic static analysis with type annotations; runtime information (actual execution paths, dynamically generated methods) is often more valuable than static source code. Truly practical AI agents of the future may need the capability to "run code in order to understand code."

The Division of Labor Developers Should Embrace

Based on the current test findings, making clear-headed judgments about task division is more important than blindly relying on AI coding assistants:

• Better suited for AI: writing standalone functions, generating boilerplate code, implementing well-defined algorithmic logic
• Be cautious about over-reliance: locating the root cause of a problem in a large, unfamiliar codebase, tracing complex call chains, understanding cross-module business logic

Honestly, AI currently resembles an efficient "code typist" more than an experienced "codebase navigator." Recognizing this boundary is what allows you to genuinely play to its strengths.

Summary

The "can write but can't read" phenomenon is a vivid illustration of the uneven development of current AI coding capabilities. It reminds us that the core difficulty of software engineering has never been just about writing code — it's about maintaining holistic command over large, complex systems.

As more benchmarks focused on "code navigation" and "codebase comprehension" continue to emerge, evaluation of AI coding ability will become more multidimensional, driving tools toward genuine engineering utility.

One detail worth noting: the benchmark's sample size remains relatively limited (5 models, 13 codebases), and its conclusions await validation at larger scale. But the direction of the problems it reveals is undoubtedly worth the entire industry taking seriously.

Key Takeaways