Why Does AI Get 'Lost' in Multi-Turn Conversations? A Microsoft Researcher Reveals the Truth Behind the Failures

In a Microsoft Research podcast, Jennifer Neville — Professor of Computer Science and Statistics at Purdue University and Partner Research Manager at Microsoft Research — joined applied scientist Chad Atalla for an in-depth conversation about AI evaluation, failure mechanisms, and user expectations. The "AI Interaction and Learning" team she leads focuses on a critically underexplored question: where exactly do current AI systems fail, and what can those failures teach us?

From Astrophysics to AI: An Unconventional Research Path

Neville's research career began with a detour. As a child, she wanted to be an astrophysicist. In college, she deliberately avoided computer science — not wanting to follow in her father's footsteps. She studied mathematics, physics, and cognitive science, even dropped out of university for a time, and only stumbled into the world of AI through a growing interest in "data" after returning to school to study computer science.

What truly hooked her was the process of doing research itself. She candidly recalls the repeated frustrations of a mandatory honors research project — "there were times I felt like I was spinning my wheels, nothing was working, and I'd get really discouraged." But her advisor's encouragement kept her going, and when she finally understood something no one had understood before, that feeling of standing at the frontier of knowledge became the "addiction" that sustained more than two decades of research.

She joined Microsoft from academia in 2021 but remains a professor at Purdue. In her view, the fundamental questions in industry and academic research are similar — the difference lies in the "target": industry can apply work at scale on real systems, real data, and real users, while academia tends toward more abstract, cross-domain generalization. Her advice is unambiguous: "If you want to work on AI systems right now, industry is where you should be."

Why Start with Evaluation?

Neville's team began with a key observation: standard benchmarks in machine learning are far too simple and bear little resemblance to how users actually interact with these systems.

"The first question you have to ask is: what do we actually want from these systems? Then you design evaluations that reflect real-world scenarios." This means testing more complex settings — multi-turn interactions, collaborative environments, and long-horizon tasks. Only by identifying performance gaps and failure points in these evaluations can the team determine where theoretical or algorithmic improvements are needed.

In other words, evaluation is an entry point — a path toward understanding that drives improvements at the frontier. The team's ultimate goal isn't to find fault, but to develop better algorithms, models, and estimation methods to improve performance.

The limitations of benchmarks are one of the central debates in AI evaluation today. Traditional benchmarks (such as MMLU and HellaSwag) typically consist of fixed single-turn question-and-answer items that test a model's knowledge and reasoning in static, isolated settings. While they're standardized, reproducible, and easy to compare across models, they fail to capture how real users actually interact — users rarely ask a single closed-ended question. More commonly, they clarify needs, shift direction, and add constraints across a continuous conversation. Long-horizon tasks present another evaluation challenge: they require the model to maintain goal consistency across multiple steps — for example, helping revise a report through several rounds of edits, or tracking context state continuously during code debugging. Mainstream benchmarks barely cover such scenarios, which is why the gap between high benchmark scores and the actual user experience continues to grow. This is precisely why research focused on real-world evaluation is seen as a critical path toward meaningful model improvement.

The Most Counterintuitive Finding: Models Get 'Lost' in Multi-Turn Conversations

The most striking research discussed in this episode involves performance degradation in large language models during multi-turn dialogues.

The team observed a real-world phenomenon: users in practice tend to "underspecify" — they often don't know how to fully describe their needs upfront, and instead gradually add conditions and clarifications across multiple turns. To test this, the team took fully specified single-turn complex instructions from public benchmark datasets and broke them apart, having simulated users provide the same information incrementally across multiple turns.

The results were surprising: models performed extremely well in the single-turn setting (since model builders optimize heavily for this), but performance "degraded significantly" in the multi-turn setting. "No one had ever evaluated it this way before," Neville said. When the research was published, users responded with strong recognition — "This is exactly my experience. I already knew this would happen. What can we do about it?"

The team is currently using reinforcement learning to train models to improve this behavior, but they also offer an immediately actionable tip: if a model gets confused in a multi-turn conversation, stop the conversation, clear the context, and restate everything you now know as a single, comprehensive instruction — the model will perform noticeably better.

From a technical standpoint, performance degradation in multi-turn conversations is closely tied to how Transformer models process information within their context windows. The attention mechanism assigns weights across the input sequence; as conversation turns accumulate, the model must locate key information within an increasingly long context. Research has shown a phenomenon called "lost in the middle" — information at the very beginning or end of a conversation tends to receive higher attention weights than content in the middle, making critical conditions added in intermediate turns prone to being diluted or overlooked. Underspecification is also tied to cognitive load: users often haven't fully worked out what they need at the start of a task, and the model's early responses in turn shape how the user understands their own requirements — creating a dynamic, bidirectional process of information construction. This is fundamentally different from the static assumption in single-turn benchmarks, where a complete question maps to a complete answer, and it explains why distributing the same information across multiple turns leads to a visible drop in model performance.

'Surprising Failures': Our Human Intuition Is Misleading Us

Neville placed particular emphasis on a category she calls "surprising failures." These are distinct from the well-studied problem of hallucination — they tend to be subtle, hard to isolate, and difficult to reproduce. One example is the "subtle loss of semantic content" that occurs in agentic tasks involving repeated document editing.

She proposed a profound hypothesis: we tend to assume that tasks difficult for humans are also difficult for models, but the opposite is often true. The real question is: "What is computationally difficult for a Transformer? What is hard for its retrieval system?" The surprise comes precisely from this mismatch — tasks that humans intuitively find simple, or that a model got right last time, may not be reliably accomplished by an LLM. And if errors aren't caught promptly, they can accumulate across long-horizon workflows, further disrupting the model's downstream judgment.

On privacy, Neville clarified that her team does not "directly view" user data. Instead, they extract patterns of successes and failures at scale within privacy-protected, restricted environments, and those high-level insights then drive algorithmic improvements. She emphasized that Microsoft does not use user responses to fine-tune models, but that detailed user feedback — especially explicit descriptions of "what went wrong, what you expected, and what you actually got" — is used in subsequent model updates.

The "surprising failures" Neville describes correspond technically to a broader issue: non-monotonicity of model capabilities. Unlike human cognition, the capability boundaries of LLMs don't follow a linear rule where performance decreases as difficulty increases. Research has found, for instance, that some models can solve complex mathematical reasoning problems yet fail at simple counting tasks; they can accurately summarize long documents yet quietly lose semantic detail when tracking changes across multiple rounds of document editing. This inconsistency stems from the fundamental nature of Transformers: they learn statistical associations rather than true logical rules, so they perform robustly on patterns that appear frequently in training data, while producing unexpected errors on tasks requiring precise symbolic manipulation or state tracking. Error cascading is especially dangerous in agentic tasks — when an AI introduces a small deviation at one step of a long-horizon workflow, that deviation propagates as input to subsequent steps, ultimately causing the final result to diverge substantially from the intended outcome in a way that is nearly invisible to the user.

Advice for Users: Don't Expect 100%, but Learn to 'Ask Again'

Cut through the marketing hype, and Neville's advice to everyday users is practical and clear-eyed:

Current AI systems genuinely have capabilities that can improve productivity, but don't expect them to be 100% correct — you should verify the answers they return. If you get a result you think is wrong, don't immediately conclude the model "can't do it." Try rephrasing and asking again — you'll often get a much better answer. She compared this to the early days of search engines, when users had to learn how to query effectively. The shift from search to conversational systems similarly requires users to relearn how to interact.

She also issued an interesting call to action: in conversational systems, users can now provide far richer feedback than a simple thumbs-up or thumbs-down. Clearly describing what went wrong is, in effect, "training the model to do what you want it to do but it can't yet do."

What's Next: Is Transformer Enough?

Looking ahead, Neville believes the research community is simultaneously exploring two paths: first, whether the Transformer architecture is the right underlying model at all — it has known computational limitations; second, how much of those limitations can be compensated for through "agentic harnesses" and backend reasoning layered on top of the model.

She expects to see rapid, parallel development of alternative architectures and complex outer wrappers, with the next generation of work focusing on longer-horizon interactions and enabling models to "understand the world inside the model" in more concrete ways.

She recalls her first internship at AT&T Labs in 2000, when colleagues debated over lunch how to get AI to play Go and how much harder it was than chess. Looking back now, the conservative estimates of "10 years, 25 years" from that era are being shattered at a pace far beyond anyone's imagination, driven by the simultaneous explosion in the number of researchers and in model scale. She is "very optimistic" about the field.

When asked what research legacy she most hopes to leave, she described a vision that borders on science fiction: a work environment where humans and AI form teams to collaborate on innovation and accomplish tasks together. "If we can actually make that work, I'll feel like my research was a success."

The conversation ended with a lesson she says she has had to relearn throughout her career: "look at the data." When results don't match expectations, the problem often isn't that the model is too hard — it's that the data itself is wrong, inconsistent with expectations, or out of distribution from the training data. A simple principle, she admits, that she has had to relearn "at least half a dozen times."

Since its introduction by Google in 2017, the Transformer architecture has become the foundational backbone of virtually all modern large language models. Its core strengths lie in parallel computational efficiency, the ability to capture long-range dependencies via self-attention, and the emergence of surprising capabilities at massive scale. However, it has several well-recognized computational limitations: first, the complexity of the attention mechanism grows quadratically with sequence length, making computation extremely expensive for very long contexts; second, it is inherently stateless — each inference requires reprocessing the entire context, with no true persistent memory; third, pure Transformer models have systematic weaknesses on tasks requiring precise arithmetic, symbolic reasoning, or strict logical deduction. Agentic harnesses have emerged as an engineering paradigm to address these limitations — by equipping models with tool use (e.g., search, code execution), external memory modules, and multi-step planning frameworks (such as ReAct and Chain-of-Thought) — extending the model's effective capability boundary without changing the underlying architecture. Alternative architectures being explored in parallel by the research community include state space models (SSMs, such as Mamba), which attempt to fundamentally resolve Transformer's computational bottlenecks.