LLM 0.32 Refactoring Explained: How Message Sequences and Streaming Chunks Are Reshaping This Python LLM Library

Simon Willison has released version 0.32a0 alpha of his Python library LLM — a major, backward-compatible refactoring. This widely used tool for accessing large language models has finally evolved from a simple "text in, text out" model into a new architecture capable of expressing the full capabilities of today's frontier models.

Simon Willison is the co-creator of the Django web framework and the author of the data tool Datasette, enjoying an outstanding reputation in the open-source community. His LLM library is designed as a unified command-line tool and Python API, allowing developers to access OpenAI, Anthropic Claude, Google Gemini, local open-source models, and dozens of other large language models through a consistent interface. The library uses a plugin architecture, with the community having contributed over a hundred plugins to connect to different model providers. This "one interface, multiple backends" design philosophy allows developers to freely switch underlying models without modifying business code, greatly reducing vendor lock-in risk.

Why the LLM Library Needed This Refactoring

The LLM library was born in April 2023, when the world was simple: send text to a model, get text back. But more than two years later, the large language model ecosystem has changed dramatically.

The library itself gradually added attachments for handling image, audio, and video inputs, schemas for structured JSON output, and tools for executing tool calls. Meanwhile, models themselves have been evolving rapidly — reasoning capabilities, image generation, audio output, and other multimodal abilities have emerged one after another. After OpenAI released the o1 model in September 2024, "reasoning models" became an industry hotspot. These models perform an internal Chain of Thought reasoning process before generating their final response. Anthropic's Claude subsequently introduced extended thinking, and open-source reasoning models like DeepSeek-R1 further drove this trend. The reasoning text is semantically distinct from the final response — it's the model's "scratch paper," and users may want to view or hide it differently. The old "text in, text out" abstraction could no longer accommodate this complexity.

The core idea behind this refactoring can be summarized in two points: On the input side, replace single prompts with message sequences. On the output side, replace plain text streams with typed streaming chunks.

Message Sequences: From Single Prompts to Flexible Multi-Turn Conversations

Limitations of the Old Model

Before 0.32, LLM managed multi-turn conversations through a conversation object. This approach worked when building conversations from scratch, but had a fatal flaw: it was impossible to inject an existing conversation history from outside. This made tasks like building an OpenAI-compatible API extremely difficult.

While the CLI tool worked around this by persisting conversations in SQLite, this was never part of a stable API, and in many scenarios developers didn't want to be tied to SQLite as a storage layer.

The New Message Builder Usage

The new version introduces two builder functions, llm.user() and llm.assistant(), allowing developers to construct message sequences directly:

import llm
from llm import user, assistant

model = llm.get_model("gpt-5.5")
response = model.prompt(messages=[
    user("Capital of France?"),
    assistant("Paris"),
    user("Germany?"),
])
print(response.text())

This aligns conceptually with OpenAI's Chat Completions API messages array. When OpenAI released GPT-3.5-turbo in March 2023, it introduced this design of representing conversations as message arrays, with each message carrying a role label (system, user, assistant). This design has become an industry de facto standard — Anthropic, Google, Mistral, and other major model providers have all adopted similar message array formats. The advantage of message arrays is that they are stateless — the server doesn't need to maintain session state, and each request carries the complete conversation context, making load balancing, retries, and conversation forking straightforward. The LLM library's previous lack of native support for this pattern meant developers couldn't easily pass external systems' chat histories (such as existing chat records from a web application) directly into model calls. Now, building compatible APIs on top of the LLM library becomes natural.

The existing prompt= parameter still works — LLM internally upgrades it to a single-element message array, ensuring backward compatibility.

Additionally, the new version supports replying directly to a response via response.reply() as an alternative way to build conversations, making code more concise and intuitive.

Streaming Chunks: A Unified Event Stream for Multiple Output Types

Mixed Output Is Now the Norm

The output of modern large language models is far from plain text. A single call to Claude might sequentially return: reasoning text, formal response text, a JSON tool call request, tool execution results, and more text. Server-side tools like OpenAI's code interpreter and Anthropic's web search further increase output type diversity. Multimodal output models can even intersperse image and audio fragments within streaming responses.

Large language model inference generates tokens one by one, and streaming responses allow the model to send each token to the client immediately after generation, rather than waiting for the entire response to complete. This mechanism is typically implemented via HTTP Server-Sent Events (SSE) and can significantly reduce the user-perceived Time to First Token (TTFT), from several seconds down to a few hundred milliseconds. In the LLM library's old architecture, streaming output was simply treated as a sequence of text fragments, unable to distinguish between content of different semantic types. With the popularization of features like Claude's extended thinking and OpenAI's function calling, a single streaming response may alternate between reasoning traces, formal responses, tool call instructions, and other content types, requiring a more granular event model for expression.

The stream_events Interface Explained

The new version models responses as a typed event stream. Through response.stream_events() (synchronous) or response.astream_events() (asynchronous), developers can precisely distinguish between different types of output fragments:

for event in response.stream_events():
    if event.type == "text":
        print(event.chunk, end="", flush=True)
    elif event.type == "tool_call_name":
        print(f"\nTool call: {event.chunk}(", end="")
    elif event.type == "tool_call_args":
        print(event.chunk, end="")

This design delivers immediate UX improvements. The CLI tool can now display "thinking" text and final response text in different colors, with thinking text output to stderr so it doesn't interfere with pipeline operations. With the updated llm-anthropic plugin, Claude's reasoning process streams in gray while the formal response appears in regular color — something that was impossible under the previous architecture.

Tool Calling (also known as Function Calling) is a key mechanism for allowing large language models to interact with external systems. The model doesn't execute operations directly; instead, it generates a structured JSON request describing the function name and parameters it wants to call. The application is responsible for actually executing that function and returning the result to the model, which then continues generating its response based on the result. This "model request → application execution → result feedback" loop is the foundational pattern for building AI Agents. A complete Agent may need multiple rounds of tool calls to accomplish a task — for example, first searching the web, then reading a file, and finally generating a report.

For tool calling, developers can call response.execute_tool_calls() after the response ends to execute requested functions, or use response.reply() to automatically invoke tools and send return values back to the model, forming a complete agent loop. The design in LLM 0.32a0 where response.reply() automatically executes tool calls and returns results is specifically aimed at simplifying the implementation of these multi-turn Agent loops.

Serialization and Deserialization: Breaking Free from SQLite

Addressing the rigidity of SQLite storage, 0.32a0 provides a universal serialization mechanism:

serializable = response.to_dict()
# Store to any backend: Redis, PostgreSQL, JSON files, etc.
response = Response.from_dict(serializable)

The returned dictionary is a TypedDict defined in the llm/serialization.py module. TypedDict is a type annotation feature introduced in Python 3.8 (defined in the typing module) that allows developers to specify precise value types for each dictionary key. Unlike regular dictionaries, TypedDict provides complete type inference and error detection in static type checking tools (such as mypy and Pyright). The LLM library's choice to define the serialization format with TypedDict means developers get IDE auto-completion and type checking support when using to_dict() return values, reducing runtime errors. This also reflects the growing emphasis on type safety in modern Python library design. Developers can freely choose their storage backend without being tied to SQLite.

Future Plans: Graph-Structured Conversations and Log Redesign

This alpha release is intended to validate the design in real-world environments. Simon Willison states that unless testing reveals design flaws, the stable 0.32 release will be very close to the alpha.

The remaining major task is redesigning the SQLite logging system to capture the finer-grained information returned by the new abstractions. Ideally, he wants to model conversations as graph structures to support OpenAI-style scenarios — where the same conversation keeps extending without needing to redundantly store the complete history in the database.

Traditional conversation storage is typically linear — each message arranged chronologically forming a chain. But in practice, conversations are often tree-shaped or graph-shaped: users might fork multiple different response directions from the same node (similar to ChatGPT's "regenerate" feature), or multiple conversations might share the same prefix (such as the same system prompt with different user questions). Graph-structured storage can avoid redundantly storing shared message history through node references, saving space while precisely expressing conversation branching relationships. OpenAI's Responses API adopts a similar design philosophy, linking responses via previous_response_id rather than repeatedly transmitting complete history. This feature may land in version 0.32 or 0.33.

Conclusion: A Backward-Compatible Abstraction Upgrade

This refactoring demonstrates the evolutionary wisdom of an excellent open-source project: elevating the abstraction layer to match industry reality while maintaining backward compatibility. The two core changes — message sequences and streaming chunks — not only solve current pain points around multi-turn conversation injection and mixed output parsing, but also reserve ample extension space for future scenarios like multimodal output and complex agent workflows. For the LLM library's plugin ecosystem, this is a necessary and timely evolution.