LLM 0.32a0 Major Refactor Analysis: How a Backward-Compatible Architecture Upgrade Tackles the Multimodal and Tool-Calling Era

Simon Willison has released the 0.32a0 alpha version of the LLM Python library — a sweeping internal surgery on a tool that was designed two years ago for "text in, text out" and is now squarely confronting the realities of multimodal inputs, tool calling, and reasoning chains. The core challenge of this LLM 0.32a0 refactor: completely redesign the internal architecture while ensuring existing users' code continues to work without changing a single line.

Why This Refactor Was Needed: Settling Abstraction Debt

The LLM library began in April 2023, when modeling large language models as "text prompt in, text response out" was perfectly reasonable — ChatGPT had only been hot for half a year, and multimodal was still just a concept in research papers.

But the brutal reality of the AI field is that your core abstraction can go from "perfectly appropriate" to "seriously outdated" in 18 months. As model capabilities exploded — attachment inputs, structured JSON output, tool calling, reasoning tokens, image generation — the original text-in/text-out pipeline became wholly inadequate.

The LLM library supports thousands of different models through its plugin system, which itself exposes a deep contradiction: the more universal your abstraction layer, the harder it is to keep up with the capability frontier of cutting-edge models. Simon Willison's insight was in refusing to keep patching the old abstraction (which is how most open-source projects die), instead confronting the problem head-on with a structural refactor.

What makes this even more interesting is the phrase "backward compatible" — meaning he not only had to redesign for the future but also carry the baggage of the past. Too aggressive and you lose users; too conservative and you lose the future. This is the hardest balancing act for any library author.

Core Change #1: Prompts Upgraded from Single Text to Message Sequences

This change might look like a mere API shape adjustment, but it's actually a fundamental shift in how LLM interactions are understood.

Pain Points of the Old Design

Previously, the Conversation object could only build dialogues from scratch — you couldn't directly inject existing conversation history. This meant that if you wanted to build an OpenAI-compatible API proxy service or simulate multi-turn interaction scenarios, things got very awkward. The original architecture had a fatal assumption: conversations are linear and start from zero.

How the New Design Works

The new version introduces llm.user() and llm.assistant() builder functions, supporting complete conversation history via a messages=[] array. This transforms conversation from a "process" into "data" — a victory for functional thinking in API design.

The old prompt= parameter still works; LLM automatically upgrades it behind the scenes into a single-message array, maintaining full backward compatibility.

The new response.reply() method is an elegant compromise — it preserves the convenience of chained conversations without forcing users down the old Conversation path. You can reply directly to a response, naturally building multi-turn dialogues.

The sharpest point is the decoupling of the SQLite storage layer. Previously, the CLI tool's mechanism for persisting conversations via SQLite was never a stable API. Binding persistence and conversation logic together was essentially conflating "logging" with "business logic." The new design completely untangles this coupling.

Core Change #2: Streaming Typed Parts — The Most Visionary Design Decision

This is what I consider the most important part of the entire refactor. Model responses are now modeled as a stream of typed message parts, rather than a blob of plain text.

Why Typed Event Streams Are Needed

The output of modern frontier models is far from just text. Within a single response, you might see:

• Reasoning processes (thinking tokens)
• Formal response text
• Structured tool call requests (in JSON format)
• Tool execution results
• Generated images or audio

These things appear interleaved. If your abstraction layer can only express "text chunks," you'll forever be hacking around it.

The New Event Stream API

The new response.stream_events() and response.astream_events() methods return typed events, each with type and chunk attributes. Supported event types include text, tool_call_name, tool_call_args, and more.

This design reminds me of Server-Sent Events and React's event system — using a unified event model to handle heterogeneous data is a pattern validated through countless engineering practices.

Supporting both synchronous and asynchronous API styles also covers different usage scenarios. However, this also means plugin authors' workload doubles — a test for the health of the plugin ecosystem.

There's also a lurking concern: the event type enumeration itself is a new abstraction constraint. When the next frontier model invents some entirely new output type (real-time video streams? 3D models?), whether this event type system is extensible enough remains to be seen.

Practical Demo: Elegant Handling of Tool Calls and Reasoning Display

Explicit Control Over Tool Calls

The demo uses a describe_dog function as a tool, where the model first outputs motivational text then initiates three tool calls. response.execute_tool_calls() executes the functions requested by the model, and response.reply() sends tool return values back to the model to continue reasoning.

This reveals a deep design philosophy: the LLM library chooses to let developers explicitly control the tool execution loop, rather than implementing an automated agent loop like some frameworks do. This is a very restrained and correct choice — automated loops look cool, but in production what you need is controllability, auditability, and interruptibility.

Reasoning Text as Unix Philosophy in Practice

The CLI tool now displays thinking (reasoning) text and final response text in different colors, with reasoning text output to stderr so it doesn't interfere with piped results.

This detail is a textbook example of Unix philosophy in practice. In an era where AI tools generally ignore command-line pipeline compatibility, this respect for Unix tradition allows the LLM tool to truly integrate into developers' workflows rather than becoming an isolated island.

The new -R/--no-reasoning flag can suppress reasoning token output — this is the only CLI-level change in this version. This is both a backward compatibility win and a hint that CLI-level capability releases may wait for subsequent versions.

The demo uses Claude Sonnet 4.6 and the updated llm-anthropic plugin.

Response Serialization: Seemingly Minor but with the Deepest Long-Term Impact

The new response.to_dict() and Response.from_dict() methods support serializing responses into JSON-style dictionaries. The returned dictionary is a TypedDict, defined in the new llm/serialization.py module.

This looks like the least flashy change, but it may have the deepest long-term impact on the library's ecosystem. Serialization capability is essentially saying: "A response is no longer a temporary object that only lives in memory — it's a first-class citizen that can be stored, transmitted, replayed, and compared."

This opens up an enormous possibility space — you can store responses in Redis, send them to message queues, write them to Parquet files for offline analysis, or do snapshot comparisons in tests.

The choice of TypedDict over Pydantic models is also worth noting: lighter weight, fewer dependencies, closer to Python's native type system, but it also means giving up Pydantic's validation capabilities. This is a "library author" choice rather than a "framework author" choice — keeping dependencies minimal and leaving complexity for users to decide.

Roadmap Ahead: Graph-Structured Storage and Ecosystem Upgrades

After the alpha release, Simon Willison plans to upgrade various plugins and test in real environments for several days. The stable 0.32 release is expected to be very close to the alpha.

Graph-Structured Conversation Storage

The most exciting — and most dangerous — part of the roadmap is redesigning the SQLite logging system, with the ideal approach being to model conversations as a graph structure.

This directly addresses a fundamental inefficiency in the chat completions API: every request must send the complete history. If stored linearly, the same conversation prefix gets stored countless times. A graph structure (similar to Git's DAG) can elegantly solve this problem while naturally expressing conversation branching, backtracking, and multi-path exploration.

But this would dramatically increase storage layer complexity. Simon Willison himself is hesitating between putting it in 0.32 or 0.33 — this hesitation itself shows he's aware of the risk.

The Real Test Is the Ecosystem

The real test of the refactor isn't the core library — it's the ecosystem. If plugin authors can't keep up, or if the new abstractions perform poorly with certain edge-case models, then no matter how elegant the design, it's just theory on paper. Releasing the alpha first and letting real users and real plugins expose problems is open-source best practice.

Final Thoughts

This refactor of the LLM library is a mirror for the entire AI toolchain ecosystem: when underlying models upend their own capability boundaries on a quarterly basis, higher-level abstractions must either learn to swap out their skeleton while running, or wait to be replaced by the next competitor starting from scratch.

Simon Willison chose the hardest path — completing a paradigm migration while maintaining backward compatibility. Whether he can outrun the pace of model evolution will determine whether a "one-person open-source library" can sustain its position on the battlefield of AI infrastructure.

For developers currently using the LLM library, now is the time to pay attention to this alpha release. Your old code doesn't need to change, but the new message sequence API and streaming event system will give you an entirely different level of flexibility when building AI applications.