LLM 0.32 Major Refactor: Message Sequences and Streaming Multi-Type Responses Explained

Simon Willison just released LLM 0.32a0 alpha, a major but backward-compatible refactor of his LLM Python library and CLI tool. This update fundamentally redesigns the abstraction for model interactions to accommodate the increasingly complex input/output requirements of today's frontier models.

Simon Willison is the co-creator of the Django web framework and the author of Datasette (an open-source tool for exploring and publishing data). He has long focused on the intersection of data tooling and AI infrastructure. The LLM library is a project he started in early 2023 during the large language model explosion, positioned as a unified command-line and Python interface that lets developers interact with models through a consistent API regardless of whether the underlying provider is OpenAI, Anthropic, a local model, or something else. The library's plugin ecosystem is its core competitive advantage—the community has contributed hundreds of plugins covering nearly every major model service, from Claude and Gemini to Ollama local models.

Why Did the LLM Library Need This Refactor?

The LLM library started in April 2023 with a very clean initial design: send a text prompt, get a text response. This was perfectly adequate at the time, but over two-plus years, the capabilities of large language models have changed dramatically.

LLM provides abstract access to thousands of different models through its plugin system. Over time, it incrementally added attachments for handling image, audio, and video inputs, schemas for outputting structured JSON, and tools for executing tool calls. Meanwhile, LLM models themselves have been constantly evolving—reasoning capabilities, image generation, multimodal output, and other new features keep emerging.

The original "text in → text out" abstraction could no longer support these complex scenarios. The two core changes in 0.32a0 address exactly this: input supports message sequences, and output supports multi-type streaming parts.

Core Change One: Messages API

Limitations of the Old Approach

Before 0.32, LLM managed multi-turn conversations through a conversation object:

conversation = model.conversation()
r1 = conversation.prompt("Capital of France?")
r2 = conversation.prompt("Germany?")

This approach could only build conversations from scratch and couldn't directly inject an existing conversation history. If you wanted to build a service compatible with the OpenAI Chat Completions API on top of the LLM library, you'd face significant difficulties—because the API requires a complete message sequence with every request.

To understand this, you need to grasp the industry paradigm established by the Chat Completions API: OpenAI's API, introduced in March 2023, uses a stateless design where every request carries a complete array of messages, each annotated with a role (system, user, assistant). The server doesn't save conversation context; the client must resend the full history with every request. This pattern was later adopted by Anthropic, Google, Mistral, and virtually every other model provider, becoming the de facto industry standard. The LLM library's previous stateful design based on conversation objects had a fundamental mismatch with this paradigm—developers couldn't directly "feed" existing conversation history from external systems into the LLM library; they had to reconstruct it by prompting one message at a time.

While the CLI tool implemented a workaround for conversation persistence via SQLite, this was never part of a stable Python API, and in many scenarios developers don't want to be tied to SQLite as their storage layer.

New Messages API Usage

0.32a0 introduces llm.user() and llm.assistant() builder functions, allowing developers to pass in complete message sequences directly:

import llm
from llm import user, assistant

model = llm.get_model("gpt-5.5")
response = model.prompt(messages=[
    user("Capital of France?"),
    assistant("Paris"),
    user("Germany?"),
])
print(response.text())

The existing prompt= parameter still works—LLM internally upgrades it to a single-message array. Additionally, the new version supports response.reply() to directly reply to a response as an alternative way to build conversations:

response2 = response.reply("How about Hungary?")

This design maintains backward compatibility while significantly improving flexibility.

Core Change Two: Streaming Parts

The Challenge of Mixed-Type Output

Modern LLM responses are far from pure text. A single Claude call might sequentially return: reasoning process, text content, a JSON tool call request, and then more text. Server-side tools like OpenAI's code interpreter and Anthropic's web search mean responses can mix text, tool calls, tool outputs, and other formats. Multimodal output models can even intersperse image or audio segments within a streaming response.

To understand the complexity here, you need to distinguish between two different tool-calling patterns. Client-side tool calling is when the model requests the client to execute a function (like querying a database or calling an external API) within its response; the client executes it and sends the result back for the model to continue generating. Server-side tool calling is when the model provider directly executes tools on their own infrastructure (like OpenAI's code interpreter running Python in a sandbox, or Anthropic's web search fetching pages server-side), then returns the tool execution results as part of the response stream. Server-side tool calling means a single response can alternate between text segments and tool execution results—traditional pure-text streaming output simply cannot express this structure.

Streaming itself refers to the model returning results token by token during generation rather than waiting for complete generation before returning everything at once, typically implemented via the Server-Sent Events (SSE) protocol. This is crucial for user experience—users can start reading output while the model is "thinking" rather than waiting seconds or even tens of seconds for complete generation.

Using the stream_events API

0.32a0 models responses as a stream of typed message parts. Developers can process different types of events one by one through stream_events() or the async astream_events():

for event in response.stream_events():
    if event.type == "text":
        print(event.chunk, end="", flush=True)
    elif event.type == "tool_call_name":
        print(f"\nTool call: {event.chunk}(", end="")
    elif event.type == "tool_call_args":
        print(event.chunk, end="")

This design delivers immediate practical benefits. For example, the CLI tool can now display "thinking" text and final response text in different colors, with thinking content output to stderr to avoid interfering with pipe operations. With the updated llm-anthropic plugin, Claude's reasoning process can be streamed in real-time in gray text while the final answer appears in normal color.

The "reasoning tokens" involved here are an important concept introduced by newer-generation models like OpenAI's o1/o3 series and Claude 3.5/4. These models first conduct an internal "thinking" process before generating the final answer, producing tokens called reasoning tokens or chain-of-thought. Different models handle reasoning tokens differently: some hide them entirely (like early o1), some return them with special markers (like Claude's thinking blocks), and some let users choose whether to view them. Reasoning tokens typically count toward token usage and billing but don't necessarily appear directly in the final output. LLM 0.32a0's streaming parts type system explicitly distinguishes reasoning text from final response text, giving developers and end users flexible control over whether to display these intermediate processes.

Users can also completely hide reasoning token output via the -R/--no-reasoning flag—this is the only CLI-level change in this release.

Serialization and Deserialization

Addressing the rigidity of the current SQLite persistence approach, 0.32a0 adds a generic serialization mechanism:

serializable = response.to_dict()
# Store anywhere
response = Response.from_dict(serializable)

The returned dictionary is a TypedDict, defined in the new llm/serialization.py module. TypedDict is a feature of Python's type system (from the typing module) that allows specifying value types for each dictionary key, providing static type checking benefits while maintaining dictionary flexibility. This lets developers freely choose their storage backend—whether PostgreSQL, Redis, the filesystem, or any other storage solution—without being tied to SQLite.

Roadmap and Future Plans

Simon released this version as an alpha to validate the new design in real-world environments. He expects the stable 0.32 release to be very close to the alpha unless testing reveals design flaws.

One remaining important task is redesigning the SQLite logging system to better capture the fine-grained information returned by the new abstractions. The ideal approach would model conversations as a graph structure, so that in scenarios like the Chat Completions API, continuously extending conversations can avoid duplicate storage in the database.

Traditional conversation storage is typically linear—each turn is appended sequentially to a record. But in practice, conversations often exhibit tree or graph structures: users might branch from a certain turn into multiple different follow-up directions (such as trying different questioning strategies for the same question), or multiple independent requests might share the same prefix history. With linear storage, every branch requires copying the complete history, creating massive data redundancy. Graph-structured storage treats each message as an independent node, using directed edges to represent sequential relationships, with shared history prefixes stored only once. This is especially critical in typical Chat Completions API usage—since every request carries the full history, there's substantial duplicate data. Graph structure can reduce storage overhead from O(n²) to O(n) through shared prefix nodes. This feature may be implemented in 0.32 or 0.33.

Design Philosophy Insights from This Refactor

This refactor embodies the evolutionary wisdom of an excellent open-source project: redefining core abstractions while maintaining backward compatibility. From "text in, text out" to "message sequences in, multi-type streaming parts out," the LLM library's abstraction layer has finally caught up with the pace of large model capability evolution. For developers building applications on the LLM library, this means fewer hacks, cleaner code structure, and better adaptability to future model capabilities.

This pattern of "abstraction layers catching up with capability layers" recurs throughout software engineering. Early web frameworks only needed to handle synchronous request-response models; later, the proliferation of WebSocket and Server-Sent Events forced frameworks to introduce async abstractions. Database ORMs initially only needed to map simple CRUD operations; later, complex queries and distributed transactions drove the redesign of query builders and transaction managers. The LLM library's refactor follows the same pattern—when underlying capabilities undergo qualitative change, the intermediate abstraction layer must evolve accordingly, or it becomes a bottleneck for innovation.

Key Takeaways

• LLM 0.32a0 introduces a messages mechanism allowing direct injection of complete conversation history, solving the previous pain point of inflexible conversation injection
• The new streaming parts API supports distinguishing and handling different output types—text, tool calls, reasoning processes—within a single response
• The CLI tool can now display reasoning text and final responses in different colors, with reasoning content going to stderr without affecting pipe operations
• New response.to_dict() / Response.from_dict() serialization mechanism frees developers from SQLite storage lock-in
• The refactor maintains full backward compatibility; the existing prompt= parameter and iterative streaming output methods still work