LLM 0.32 Refactoring Deep Dive: Message Sequences and Multimodal Streaming Event Design

Simon Willison just released version 0.32a0 alpha of his open-source project LLM—a major backward-compatible refactoring. Since its inception in April 2023, this Python library and CLI tool has served as an essential bridge for developers accessing various large language models. This update marks the evolution of its core abstraction model from simple "text in, text out" to an entirely new architecture capable of handling the diverse inputs and outputs of modern frontier models.

Simon Willison is the co-creator of the Django web framework and the author of Datasette, a data journalism tool, and has been a long-time active member of the open-source community. His LLM project is positioned as a unified access layer for large language models—with a single CLI command or a few lines of Python code, developers can call OpenAI, Anthropic Claude, Google Gemini, local Llama, and dozens of other models without writing different integration code for each provider. The project uses a plugin architecture, and the community has contributed a plugin ecosystem covering thousands of models.

Why the LLM Library Needed This Refactoring

Two years ago, the LLM library's worldview was simple: send a text prompt, get a text response back. This was perfectly adequate in the GPT-3.5 era. But LLMs evolved far faster than expected—the library successively added attachments support for image/audio/video inputs, schemas for structured JSON output, and tools for function calling.

Meanwhile, frontier models themselves were rapidly evolving: reasoning capabilities, image generation, audio clip output, server-side tool execution... The original "text in, text out" abstraction could no longer support these diverse interaction patterns. With LLM supporting thousands of different models through its plugin system, its core abstraction had to keep pace with the times.

Core Change One: Message Sequences Replace Single Text Prompts

Limitations of the Old Model

Before 0.32, LLM managed multi-turn conversations through conversation objects, but this approach could only build conversations from scratch—it couldn't directly inject an existing conversation history. This made tasks like building an OpenAI-compatible API extremely difficult. While the CLI tool implemented conversation persistence via SQLite, this was never part of a stable Python API.

The New Message Builder API

0.32a0 introduces llm.user() and llm.assistant() builder functions, allowing developers to construct prompts directly as message sequences:

from llm import user, assistant

response = model.prompt(messages=[
    user("Capital of France?"),
    assistant("Paris"),
    user("Germany?"),
])

This design directly aligns with the OpenAI Chat Completions API message format. OpenAI's Chat Completions API uses a message array as input, where each message contains a role field (such as system, user, assistant) and a content field. This format has become the de facto industry standard—Anthropic, Google, Mistral, and other major model providers all adopt similar message sequence structures. By aligning with this format, the LLM library can serve more naturally as an abstraction adapter layer for these APIs, enabling developers to seamlessly switch between different models. The original prompt= parameter remains functional—LLM internally upgrades it to a single-message array.

Additionally, the new version supports response.reply() to directly reply to a response as an alternative way to build conversations, making the API design more flexible and natural.

Core Change Two: Typed Streaming Response Event Flow

The Challenge of Mixed Content Output

Modern model outputs are far from plain text. A single Claude call might return, in sequence: reasoning process (thinking), text content, a JSON-formatted tool call request, followed by more text.

It's worth elaborating on how reasoning models work. Reasoning models (such as OpenAI's o1/o3 series, Anthropic's Claude with extended thinking, DeepSeek-R1) output a "thinking process" (thinking tokens) before generating their final answer. These tokens reveal the model's step-by-step reasoning chain, similar to an internalized version of Chain-of-Thought prompting. Distinguishing reasoning tokens from final response tokens is critically important in practice: the reasoning process typically shouldn't be shown to end users, and reasoning tokens are often billed differently from output tokens.

Server-side tools further increase response complexity. Server-side tools are built-in tools executed by model providers on their own infrastructure—such as OpenAI's Code Interpreter and file search, Anthropic's web search, etc. Unlike developer-defined function calling, these tools don't require client-side participation, but their invocation processes and results are interspersed within the response stream. These server-side tools cause responses to contain a mix of text, tool calls, tool outputs, and other formats. Multimodal output models can even intersperse image or audio clips within streaming responses.

The stream_events Streaming Interface in Detail

The new version introduces response.stream_events() and its async counterpart response.astream_events(), modeling responses as a series of typed events:

for event in response.stream_events():
    if event.type == "text":
        print(event.chunk, end="")
    elif event.type == "tool_call_name":
        print(f"\nTool call: {event.chunk}(", end="")
    elif event.type == "tool_call_args":
        print(event.chunk, end="")

The elegance of this design lies in enabling differentiated handling of different content types. For example, the CLI tool can now display "thinking" text and final responses in different colors, with thinking text output to stderr so it doesn't affect pipe operations. The new -R/--no-reasoning flag can completely suppress reasoning token output.

For tool calls, developers can call response.execute_tool_calls() after the response completes to execute the requested functions, or use response.reply() to automatically execute tools and pass results back to the model, implementing a complete agent loop. An agent loop refers to the iterative process of: model issues tool call request → client executes tool → results are passed back to model → model continues generating. This is the core pattern for building AI Agents. The LLM library encapsulates this multi-step process into a concise API call via the reply() method, dramatically reducing the complexity of building tool-augmented agents.

Serialization and Deserialization: Breaking Free from SQLite Binding

Addressing the rigidity of LLM's current SQLite persistence approach, 0.32a0 provides a universal serialization interface:

serializable = response.to_dict()
# Store anywhere
response = Response.from_dict(serializable)

The returned dictionary is a TypedDict defined in the llm/serialization.py module. Developers can freely choose their storage backend, no longer tied to SQLite.

SQLite is an embedded relational database that stores data as a single file without requiring a separate database server process. Simon Willison is an active advocate for SQLite in the developer tooling space—his Datasette project is built entirely around SQLite. The LLM library uses SQLite to record all conversation histories and model call logs, allowing users to retrospectively query and analyze their LLM usage via SQL. However, this tight coupling also restricts developers who want to use other storage solutions (such as PostgreSQL, cloud storage, or in-memory caching). The new serialization interface addresses this pain point precisely, making persistence strategy a developer's autonomous choice rather than a framework-imposed constraint.

Future Plans and Roadmap

Simon released this as an alpha specifically to validate the new design in real-world environments. He plans to redesign the SQLite logging system in the stable release to capture the rich information returned by the new abstractions at finer granularity. The ideal solution is to model conversations as graph structures, avoiding redundant storage of repeated conversation history in chat completions scenarios.

Modeling conversations as graph structures is a design concept worth understanding deeply. Traditional conversation persistence treats conversations as linear message lists, requiring the full message history to be sent with each API call. When conversations branch (e.g., a user re-asks from some intermediate point) or tool calls produce multiple parallel paths, the linear model leads to massive storage duplication. Modeling conversations as a Directed Acyclic Graph (DAG) allows multiple branches to share common history prefixes, saving storage space while more accurately representing the true topological structure of conversations. This shares conceptual similarities with Git's commit history model—each node points to its parent node, and branching and merging can be naturally expressed.

Backward-Compatible Incremental Refactoring: Design Philosophy Insights

This refactoring embodies the evolutionary wisdom of an excellent open-source project: backward-compatible incremental refactoring. The old prompt= parameter, the conversation() pattern, the simple for chunk in response iteration—all of these continue to work. The new message sequences and event streams are incrementally added capabilities, not destructive replacements.

In an era of explosive growth in LLM capabilities, abstraction design at the tooling layer is critically important. The LLM library's evolution from "text pipeline" to "multimodal event stream" is essentially chasing the frontier of model capabilities. This timely evolution of the abstraction layer determines whether a tool library can maintain its vitality in the rapidly changing AI ecosystem.

Key Takeaways

• LLM 0.32a0 upgrades the core abstraction from "text in, text out" to message sequence input and typed streaming responses, adapting to the complex interaction needs of modern multimodal models
• New llm.user() and llm.assistant() builder functions support direct injection of conversation history, aligning with the OpenAI Chat Completions API message format
• Introduces the stream_events() mechanism to differentially handle text, reasoning processes, tool calls, and other response content types
• Provides universal to_dict()/from_dict() serialization interfaces, removing the hard dependency on SQLite storage
• The overall refactoring maintains backward compatibility—existing APIs continue to work—embodying an incremental evolution design philosophy