LLM 0.32 Refactor: How Message Sequences and Streaming Chunks Are Reshaping Python LLM Interactions

Simon Willison just released LLM 0.32a0 alpha — a major but backward-compatible refactor of his LLM Python library and CLI tool. This update redefines the core abstraction layer for interacting with large language models, evolving from a simple "prompt-response" model to a modern architecture supporting message sequence inputs and typed streaming outputs.

Why This Refactor Was Needed

When Simon launched the LLM project in April 2023, the abstraction of "send a text prompt, get a text response" was perfectly adequate. But over two years later, the LLM ecosystem has changed dramatically.

The LLM library supports unified access to thousands of different models through its plugin system. This plugin architecture is one of the library's core competitive advantages — through Python's entry_points mechanism, third-party developers can write plugins to connect any model provider, from commercial APIs like OpenAI, Anthropic, and Google Gemini, to local GGUF-format models running via the llm-gguf plugin, to local inference frameworks like Ollama and vLLM. Users simply pip install the corresponding plugin and access models through the unified llm.get_model() interface without worrying about underlying API differences. This design borrows the plugin philosophy from Datasette (another well-known Simon project): keep the core lean and extend capabilities through the ecosystem.

Over time, the LLM library gradually added attachments for handling image/audio/video inputs, schemas for structured JSON output, and tools for executing tool calls. Meanwhile, frontier models kept evolving — reasoning capabilities, image generation, and multimodal outputs emerged one after another. The original "text in, text out" abstraction could no longer accommodate these complex scenarios.

This 0.32a0 alpha brings two core changes: inputs can be expressed as message sequences, and outputs can consist of different types of streaming chunks.

Message Sequences: More Flexible Conversation Modeling

Limitations of the Old Model

Previously, LLM managed multi-turn conversations through conversation objects, but this approach could only build conversations from scratch — it couldn't directly inject an existing conversation history. This made scenarios like building OpenAI-compatible chat completions APIs extremely difficult. While the CLI tool worked around this through SQLite persistence, this was never part of a stable Python API.

The New Message Builder

0.32a0 introduces llm.user() and llm.assistant() builder functions, allowing developers to directly construct complete message sequences:

import llm
from llm import user, assistant

model = llm.get_model("gpt-5.5")
response = model.prompt(messages=[
    user("Capital of France?"),
    assistant("Paris"),
    user("Germany?"),
])
print(response.text())

This design directly maps to OpenAI's Chat Completions API message format. The Chat Completions API is the de facto standard interface format in the LLM industry, requiring input as a message array where each message contains a role (such as system, user, assistant, tool) and a content field. The key advantage of this format is that developers can "prefill" conversation context by injecting assistant-role historical messages, enabling advanced scenarios like few-shot prompting, role-playing, and conversation resumption. Major providers like Anthropic, Google, and Mistral have also adopted compatible or similar message formats, making this the foundation for cross-platform interoperability. LLM 0.32a0's messages parameter directly aligns with this industry standard, making prefilling conversation history effortless.

The existing prompt= parameter still works — LLM automatically upgrades it to a single-message array behind the scenes.

Additionally, the new version supports replying directly to a response via response.reply(), as an alternative to building conversation objects:

response2 = response.reply("How about Hungary?")
print(response2)  # default __str__() calls .text()

Streaming Chunks: Handling Mixed-Type Outputs

The Complexity of Modern Model Outputs

Today's frontier LLMs return far more than plain text. A single Claude call might sequentially return reasoning process, text content, tool call requests, and then more text.

The reasoning tokens involved here represent an important innovation in recent LLMs, exemplified by OpenAI's o1/o3 series and Anthropic Claude's extended thinking feature. These models generate a "Chain-of-Thought" reasoning process before producing the final answer, similar to a human's scratch paper when solving problems. Reasoning tokens typically consume additional compute resources and API costs but can significantly improve accuracy on mathematical reasoning, code generation, and complex logic tasks. In streaming output, reasoning tokens and final response tokens arrive alternately or in segments — this is one of the direct motivations for LLM 0.32a0's introduction of typed event streams, enabling clients to distinguish between "the model is thinking" and "the model is answering."

Meanwhile, tool calling (Tool Use / Function Calling) has become a key capability in LLMs' evolution from pure text generators to intelligent agents. During generation, models can decide to call external tools (such as search engines, code interpreters, or database queries) and integrate the tool results into subsequent generation. Two patterns exist here: client-side tool calling (where the model outputs tool call requests and the client executes them and sends results back) and server-side tool calling (like OpenAI's Code Interpreter or Anthropic's Web Search, executed automatically server-side). Server-side tool response streams mix text fragments, tool call declarations, tool execution results, and other types, making traditional plain-text streaming completely inadequate for expressing the full semantic structure of responses. Multimodal output models can even intersperse image or audio fragments within streaming responses.

Typed Event Streams

The new LLM version models responses as a stream of typed message chunks. Through response.stream_events() (synchronous) or response.astream_events() (asynchronous), developers can precisely distinguish between different output types:

for event in response.stream_events():
    if event.type == "text":
        print(event.chunk, end="", flush=True)
    elif event.type == "tool_call_name":
        print(f"\nTool call: {event.chunk}(", end="", flush=True)
    elif event.type == "tool_call_args":
        print(event.chunk, end="", flush=True)

This mechanism delivers direct UX improvements — the CLI tool can now display "thinking" text and final response text in different colors, with thinking text output to stderr so it doesn't interfere with pipe operations. Combined with the updated llm-anthropic plugin, Claude models' reasoning processes can be displayed in real-time streaming with gray text.

The new -R/--no-reasoning flag can suppress reasoning token output — this is the only user-facing CLI change in this release.

Response Serialization and Deserialization

Addressing LLM's current rigid design of persisting conversations to SQLite, 0.32a0 provides a general-purpose serialization mechanism:

serializable = response.to_dict()
# Store wherever you like
response = Response.from_dict(serializable)

Simon Willison is an active advocate for using SQLite at the application layer. In the LLM CLI tool, all conversation history is stored by default in a local SQLite database (typically in ~/.config/io.datasette.llm/), and users can query past conversations with the llm logs command. As an embedded database requiring no additional service processes, SQLite is well-suited for CLI tools in single-user scenarios. However, when LLM is used as a Python library integrated into web services or distributed systems, SQLite's single-writer limitation and file-level locking become bottlenecks.

The returned dictionary is actually a TypedDict defined in the llm/serialization.py module. TypedDict is a type annotation tool provided by Python's typing module that allows developers to specify precise value types for each dictionary key. Unlike regular dict, TypedDict can be validated by static type checkers like mypy and pyright, ensuring field completeness and type correctness during serialization and deserialization. This means IDEs can provide autocomplete, type checkers can catch field typos or type mismatches at compile time, and third-party storage layer implementers have a clear contract to follow — offering stronger development-time guarantees than plain dictionaries or JSON schemas.

This design means you're no longer tied to SQLite and can freely choose Redis, PostgreSQL, or any other storage solution to persist conversation data.

Next Steps

Simon released this version as an alpha to validate the new API design in real-world environments. He expects the stable 0.32 release to be very close to the alpha, unless testing reveals design flaws.

One remaining major task is redesigning the SQLite logging system to better capture the fine-grained information returned by the new abstractions. The ideal approach would model it as a graph structure to support scenarios like the chat completions API where the same conversation keeps expanding, avoiding duplicate storage in the database. This feature may land in 0.32 or be deferred to 0.33.

Design Philosophy Takeaways

This refactor embodies the evolutionary wisdom of an excellent open-source project: backward-compatible, incremental refactoring. The old prompt= parameter, conversation() pattern, and simple string-iteration streaming — all continue to work. But the new message sequences and typed event streams provide the necessary expressiveness for increasingly complex LLM interaction scenarios.

From a broader perspective, the evolution of the LLM Python library's abstraction layer mirrors the industry's development: from simple text completion, to multi-turn conversations, to tool calling, to multimodal mixed outputs — each leap in model capabilities forces a redesign of the infrastructure layer. LLM 0.32's refactor provides developers with a set of interaction primitives better aligned with real-world needs, and the subsequent stable release is worth watching.

Key Takeaways

• LLM 0.32a0 introduces message sequences (messages) as an input abstraction, supporting direct construction of complete conversation histories and resolving the previous inability to prefill conversations
• New typed streaming events (stream_events) can distinguish between text, tool calls, reasoning processes, and other output chunk types, adapting to the complex responses of modern multimodal models
• Provides response.to_dict()/from_dict() serialization mechanisms, letting developers implement conversation persistence independently of SQLite
• The CLI tool can now display reasoning text and final responses in different colors, with reasoning output to stderr without affecting pipe operations
• All changes are backward-compatible — the old prompt= parameter and conversation() pattern continue to work