LLM 0.32 Refactoring Explained: Message Sequence API and Typed Streaming Responses

Simon Willison just released LLM 0.32a0 alpha — a major but backward-compatible refactoring of his LLM Python library and CLI tool. This update fundamentally redesigns the abstraction layer for model interactions to accommodate the increasingly complex input/output capabilities of today's frontier models.

LLM Library: Positioning and Ecosystem Context

LLM is an open-source project maintained by Simon Willison (Django co-creator, Datasette author), positioned as a unified command-line interface and Python library for large language models. Its core design philosophy uses a plugin system to support arbitrary model backends — from commercial APIs like OpenAI and Anthropic to locally-running open-source models (via plugins like llm-gpt4all, llm-ollama, etc.). This architecture lets developers switch between different models using a unified interface without writing adapter code for each vendor's SDK. As of 2025, LLM's plugin ecosystem covers dozens of model providers, making it an important infrastructure component in the AI toolchain.

Why Did the LLM Library Need This Refactoring?

The LLM library was born in April 2023, when the world was simple: send a string of text to a model, get a string of text back. This "text in, text out" abstraction was perfectly adequate at the time.

But over two years later, large language model capabilities have far exceeded the original design assumptions. The library progressively added attachments for handling image, audio, and video inputs, schemas for structured JSON output, and tools support for function calling. Meanwhile, the models themselves were rapidly evolving — reasoning capabilities, image generation, audio output, and other multimodal abilities kept emerging.

The original abstraction could no longer carry this complexity. LLM supports thousands of different models through its plugin system, and it needed a more flexible underlying architecture to uniformly express these diverse input/output types.

Core Change #1: Message Sequences as Input

Limitations of the Old Approach

Before 0.32, LLM managed multi-turn conversations through conversation objects:

conversation = model.conversation()
r1 = conversation.prompt("Capital of France?")
r2 = conversation.prompt("Germany?")

This approach could only build conversations from scratch — you couldn't directly inject an existing conversation history. If you wanted to build a service compatible with the OpenAI Chat Completions API on top of the LLM library, this became a thorny problem. While the CLI tool implemented conversation persistence via SQLite, this was never part of the stable Python API.

The OpenAI Chat Completions API Paradigm

OpenAI's Chat Completions API, launched in March 2023, established an industry-standard format for conversational interaction: the request body contains a messages array, with each message annotated with a role (system, user, assistant) and content. This design hands complete control of conversation history management to the caller, while the server itself remains stateless. This paradigm was adopted by Anthropic, Google, Mistral, and virtually every other major vendor, becoming the de facto industry standard. LLM 0.32's message sequence design aligns precisely with this paradigm, making it natural to build compatible services on top of the LLM library.

The New Message Sequence API Design

0.32a0 introduces llm.user() and llm.assistant() builder functions, allowing developers to pass in complete message sequences directly:

import llm
from llm import user, assistant

model = llm.get_model("gpt-5.5")
response = model.prompt(messages=[
    user("Capital of France?"),
    assistant("Paris"),
    user("Germany?"),
])
print(response.text())

This design directly aligns with the API paradigm of major vendors like OpenAI — a message list containing role-annotated conversation turns. The existing prompt= parameter still works; LLM automatically upgrades it to a single-message array under the hood.

Additionally, the new version supports replying directly to a response via the response.reply() method, providing another flexible way to build conversations:

response2 = response.reply("How about Hungary?")

Core Change #2: Typed Streaming Responses

The Challenge of Mixed Content Output

Modern LLM output is no longer a single stream of text. A single Claude call might sequentially return: reasoning process, text content, a JSON-formatted tool call request, then more text. Server-side tools like OpenAI's code interpreter and Anthropic's web search make response content even richer. Multimodal output models can even interleave image and audio segments within a streaming response.

Technical Background on Streaming Responses

Streaming is a critical UX optimization in LLM interactions. Since LLMs generate text token by token, waiting for a complete response can take tens of seconds. Streaming output allows clients to receive content incrementally during generation, dramatically reducing Time to First Token (TTFT). Technically, major vendors typically implement streaming via the Server-Sent Events (SSE) protocol — a unidirectional push protocol over HTTP where the server continuously sends event lines prefixed with data: over a maintained connection. LLM 0.32's stream_events interface adds type annotations on top of this, enabling developers not only to receive content chunk by chunk but also to distinguish the semantic type of each chunk — which is critical in mixed output scenarios.

stream_events Interface in Detail

The new LLM models responses as a stream of typed message parts. Through response.stream_events() and its async counterpart response.astream_events(), developers can precisely distinguish between different types of output content:

for event in response.stream_events():
    if event.type == "text":
        print(event.chunk, end="", flush=True)
    elif event.type == "tool_call_name":
        print(f"\nTool call: {event.chunk}(", end="")
    elif event.type == "tool_call_args":
        print(event.chunk, end="")

One tangible improvement this mechanism brings: the CLI tool can now display "thinking" text and final response text in different colors, with thinking content output to stderr so it doesn't interfere with piped results. With the updated llm-anthropic plugin, Claude's reasoning process displays in gray while the final answer appears in normal color.

Function Calling Mechanism

Tool calling (function calling) has been one of the most important capability extensions for LLMs since mid-2023. Here's how it works: developers declare available function signatures (name, parameter JSON Schema) in the request; when the model needs external information or to perform an action, instead of answering directly, it returns a structured function call request. The client executes that function, sends the result back as a new message to the model, and the model then generates its final answer based on the result. This "model decides → client executes → result returned" loop is called the Agentic Loop and is the core pattern for building AI Agents.

In LLM 0.32, after a response completes you can call response.execute_tool_calls() to directly execute the requested functions, or use response.reply() to automatically pass tool execution results back to the model, forming a complete tool-calling loop. This design simplifies what would otherwise require manually orchestrating multi-step interactions into just a few lines of code.

Serialization and Deserialization: Breaking Free from SQLite

SQLite's Historical Role in LLM

Simon Willison is a staunch advocate for SQLite, with several of his projects (Datasette, sqlite-utils) built around it. In the LLM CLI, SQLite serves as the default storage for conversation logs — every interaction's prompt, response, model parameters, etc. are automatically recorded to a local database file (defaulting to ~/.local/share/io.datasette.llm/), convenient for later retrieval and analysis via the llm logs command. However, while SQLite's single-file, embedded nature suits CLI scenarios well, it becomes too rigid for applications requiring distributed deployment, multi-process concurrent writes, or custom storage strategies.

The New Universal Serialization Mechanism

To address this, 0.32a0 provides a universal serialization mechanism:

serializable = response.to_dict()
# Store wherever you like
response = Response.from_dict(serializable)

The returned dictionary is a TypedDict defined in the llm/serialization.py module. Developers are free to choose their storage backend — whether Redis, PostgreSQL, or the filesystem — no longer bound to SQLite. This design follows the "separation of concerns" principle: the core library is only responsible for the structured representation of data, while persistence strategy is entirely up to the application layer.

Future Plans and Version Roadmap

Simon released this version as alpha to validate the new design in real-world environments. He expects the stable 0.32 release to be very close to the alpha, unless testing reveals design flaws.

One remaining major task is redesigning the SQLite logging system to better capture the fine-grained information returned by the new abstractions. The ideal approach would model conversations as a graph structure (Directed Acyclic Graph, DAG), avoiding duplicate data storage in OpenAI-style conversation branching scenarios.

Why a Graph Structure?

Traditional conversation storage models conversations as linear sequences (linked list structure), but in practice, a conversation history can fork — for example, trying different follow-up questions based on the same prefix (similar to Git's branching model). OpenAI's API design naturally supports this pattern: you can use the same prefix message array with different final messages to create branches. With linear storage, each branch would need to redundantly store the complete prefix history, creating massive duplication. A graph structure can share common prefix nodes, only creating new edges at fork points — saving storage space while accurately representing the conversation's topology. This feature may be implemented in 0.32 or 0.33.

Summary: From Text Interaction to a Multimodal Streaming Architecture

This refactoring exemplifies how a well-maintained open-source project evolves: when the underlying technology paradigm undergoes fundamental change, the abstraction layer must evolve with it. LLM 0.32 upgrades from "text in, text out" to "message sequences in, typed streaming parts out," not only adapting to current multimodal AI and tool-calling scenarios but also reserving extension space for future, more complex model capabilities. What's even more impressive is that all of this was accomplished while maintaining backward compatibility — for developers depending on this library, upgrading carries virtually zero migration cost.

From a broader perspective, the evolution of the LLM library also reflects the maturation process of the entire AI infrastructure layer: from the early days of each vendor's SDK going its own way, to the gradual formation of a unified interaction paradigm centered on message sequences, tool calling, and streaming output. Abstraction layer projects like LLM are both beneficiaries and drivers of this standardization trend.

Key Takeaways

• LLM 0.32a0 introduces message sequences (messages) as an input method, allowing direct injection of complete conversation history, replacing the previous limitation of only building conversations from scratch
• New typed streaming responses (stream_events) can precisely distinguish between text, tool calls, reasoning processes, and other output content types
• Provides a universal serialization/deserialization mechanism (to_dict/from_dict), freeing developers from being bound to SQLite storage
• The CLI tool can now display reasoning text and final responses in different colors, with thinking content output to stderr without affecting pipe operations
• The entire refactoring maintains backward compatibility — the existing prompt= parameter and iterative streaming output still work