LLM 0.32a0 Major Refactor: A Deep Dive into Message Sequences and Typed Streaming Responses

Simon Willison just released LLM 0.32a0 alpha, a major yet backward-compatible refactor of his LLM Python library and CLI tool. This update fundamentally redesigns the abstraction for model interactions to accommodate the increasingly complex input/output requirements of today's frontier models.

Why the LLM Library Needed This Refactor

The LLM library started in April 2023 with a beautifully simple design: send a text prompt, get a text response. That was perfectly adequate at the time, but over the past two-plus years, the capabilities of large language models have changed dramatically.

Simon Willison is the co-creator of the Django framework and the author of Datasette (an open-source tool for exploring and publishing data). The LLM library continues his consistent design philosophy: lowering technical barriers through a clean CLI and Python API. The library's plugin system allows the community to write adapters for different model providers (OpenAI, Anthropic, Google, locally-run Ollama/llama.cpp, etc.), enabling users to switch underlying models without modifying code. This architecture is analogous to an ORM in the database world—providing a unified interface while abstracting away underlying differences.

Through this plugin system, LLM supports abstract access to thousands of different models. Over time, it has progressively added attachments for handling image, audio, and video inputs, schemas for outputting structured JSON, and tools functionality for executing tool calls. Meanwhile, LLM models themselves keep evolving—reasoning capabilities, image generation, multimodal output, and other new features continue to emerge.

The original "text in → text out" abstraction could no longer support these complex scenarios. Version 0.32a0 brings two core changes: inputs can be expressed as message sequences, and outputs can consist of different types of streamed parts.

Core Change #1: Message Sequence Prompts

Limitations of the Old Approach

Before 0.32, LLM managed multi-turn conversations through a conversation object:

conversation = model.conversation()
r1 = conversation.prompt("Capital of France?")
r2 = conversation.prompt("Germany?")

This approach worked when building conversations from scratch, but couldn't inject an existing conversation history from an external source. This made tasks like building an OpenAI-compatible chat completions API extremely difficult. While the CLI tool had a workaround through its SQLite persistence mechanism, this was never part of a stable Python API.

The New Message Construction Approach

The new version introduces user() and assistant() builder functions that allow developers to construct message sequences directly:

from llm import user, assistant

response = model.prompt(messages=[
    user("Capital of France?"),
    assistant("Paris"),
    user("Germany?"),
])

This design directly aligns with the OpenAI Chat Completions API message format. This API, introduced by OpenAI in March 2023, defined a message role model where each message carries a role field (system, user, assistant, tool), forming an ordered array of messages. This format has become the de facto industry standard, with APIs from Anthropic, Google, Mistral, and others adopting similar message sequence structures. LLM 0.32a0's user() and assistant() builder functions are a direct mapping of this paradigm, enabling developers to inject conversation history obtained from any source (databases, files, other APIs) directly into model calls, without having to replay messages one by one through the library's internal conversation object. The existing prompt= parameter still works—LLM automatically upgrades it to a single-message array behind the scenes.

Additionally, the new version supports replying directly to a response via response.reply(), providing a convenient way to build multi-turn conversations:

response2 = response.reply("How about Hungary?")

Core Change #2: Typed Streaming Parts

The Challenge of Mixed-Type Output

Modern LLM responses are no longer pure text streams. A single Claude call might sequentially return reasoning output, text content, a JSON-formatted tool call request, and then more text. Server-side tools like OpenAI's code interpreter and Anthropic's web search mean results can mix text, tool calls, tool outputs, and other formats. Multimodal output models can even intersperse image or audio segments within a streaming response.

To understand the technical context of this challenge, two key concepts are important. First is streaming: models return results token by token during generation rather than waiting for the complete response, typically implemented via the HTTP Server-Sent Events (SSE) protocol. Second is server-side tools—an important feature introduced by major model providers over the past year. Anthropic's Claude supports web search and code execution; OpenAI's Assistants API provides a code interpreter and file retrieval. Unlike client-side tool calls, server-side tool execution happens on the model provider's infrastructure, with responses mixing model-generated text and tool execution results. This is precisely why LLM needs a typed event stream.

Meanwhile, reasoning models are another major trend in the LLM space in 2024-2025. OpenAI's o1/o3 series, Anthropic's Claude extended thinking, DeepSeek-R1, and similar models perform explicit "thinking" before generating their final answer. Some APIs return these thinking steps as separate content blocks, distinguished from the final response, further increasing the complexity of the response stream.

The stream_events Interface Explained

The new version provides typed event streams via stream_events() and astream_events() (the async version):

for event in response.stream_events():
    if event.type == "text":
        print(event.chunk, end="")
    elif event.type == "tool_call_name":
        print(f"\nTool call: {event.chunk}(", end="")
    elif event.type == "tool_call_args":
        print(event.chunk, end="")

Each event carries an explicit type identifier (text, tool_call_name, tool_call_args, etc.), allowing developers to implement different handling logic based on type. This enables the CLI tool to now display "thinking" text and final response text in different colors. Thinking text is output to stderr to avoid interfering with pipe operations—an elegant Unix philosophy design ensuring that only the final useful output passes through pipes, while debug and intermediate information is presented to users via stderr.

For tool calls, developers can call response.execute_tool_calls() after the response completes to execute the requested functions, or use response.reply() to automatically invoke tools and send the return values back to the model.

Serialization and Deserialization

Addressing the rigidity of the current SQLite persistence approach, 0.32a0 adds a general-purpose serialization interface:

serializable = response.to_dict()
# Store anywhere
response = Response.from_dict(serializable)

The returned dictionary is actually a TypedDict, defined in the new llm/serialization.py module. TypedDict is a type hinting feature introduced in Python 3.8 (PEP 589) that allows specifying the expected value type for each dictionary key, providing static type checking support while maintaining dictionary flexibility. LLM's choice of TypedDict over Pydantic models or dataclasses as the serialization format reflects a pursuit of lightweight design and interoperability—what's returned is a plain Python dictionary that can be serialized directly with json.dumps(), or stored in MongoDB, Redis, PostgreSQL JSONB columns, or any JSON-capable storage system. This gives developers the flexibility to break free from SQLite and choose their own storage solution based on project requirements.

Future Plans and Roadmap

Simon released this version as an alpha to validate the new design's reliability in real-world environments. He expects the stable 0.32 release to be very close to the alpha, unless testing reveals design flaws.

One important remaining task is redesigning the SQLite logging system to better capture the fine-grained information returned by the new abstractions. Ideally, he wants to model conversations as graph structures to support the expanding and repeating conversation scenarios in chat completions APIs, avoiding duplicate storage in the database.

Traditional conversation storage treats conversations as linear message lists, but in practice conversations are often tree-shaped or graph-shaped: users might fork new conversation branches from an intermediate point (similar to ChatGPT's "edit and regenerate" feature), or send the same conversation history to different models for comparison. With linear list storage, each branch requires a complete copy of all preceding messages, creating significant data redundancy. A graph structure (specifically a directed acyclic graph, or DAG) allows multiple conversation branches to share common prefixes, with each node stored only once and edge relationships expressing the conversation's evolution path. This design has mature precedent in Git's commit history management. This feature may be implemented in 0.32 or 0.33.

Summary

This refactor demonstrates how a mature open-source project can fundamentally upgrade its core abstractions while maintaining backward compatibility. From "text in, text out" to "message sequences in, typed streaming parts out," the LLM Python library's new architecture is well-prepared to handle multimodal output, tool calls, reasoning display, and other modern LLM features. For developers building AI applications on the LLM library, this is an important upgrade worth watching closely.

Key Takeaways

• LLM 0.32a0 introduces message sequence prompts, supporting direct injection of conversation history via user() and assistant() builder functions, solving the previous pain point of being unable to load existing conversations from external sources
• New typed streaming event mechanism (stream_events) supports mixed streaming responses of multiple types including text, tool calls, and reasoning output, adapting to the complex output of modern multimodal models
• Provides a general-purpose serialization/deserialization interface (to_dict/from_dict), letting developers freely choose storage solutions independent of SQLite
• All changes maintain backward compatibility—the existing prompt= parameter and iterative streaming output still work
• Future plans include restructuring the SQLite logging system into a graph structure for efficient conversation expansion and deduplicated storage