LLM 0.32 Refactoring Deep Dive: Message Sequence Input and Streaming Multi-Type Response Design

Overview

Simon Willison has released version 0.32a0 alpha of his Python library LLM, representing a major backward-compatible refactoring. LLM is a Python library and CLI tool for accessing large language models, supporting thousands of different models through a plugin system. The core goal of this refactoring is to enable the abstraction layer to better accommodate the increasingly diverse input and output types of today's frontier models.

Simon Willison is the co-creator of the Django framework and a well-known developer in data journalism and open-source tools. His LLM library adopts a plugin architecture philosophy similar to yt-dlp—the core library provides a unified interface while specific model integrations are implemented through independent plugins. There are already dozens of community plugins including llm-claude-3, llm-gemini, llm-ollama, and others, covering everything from the OpenAI GPT series to locally-run open-source models. This design allows users to switch between underlying model providers without modifying code, significantly reducing vendor lock-in risk.

Why the LLM Library Needed This Refactoring

When the LLM project launched in April 2023, the simple "text in, text out" abstraction was perfectly sufficient. But over two-plus years, the large language model ecosystem has undergone enormous changes:

• Multimodal inputs (images, audio, video) supported via attachments
• Structured JSON output supported via schemas
• Tool calling capabilities added
• Reasoning output emerged
• Models began returning images and even audio clips

The original "one prompt maps to one text response" abstraction could no longer support these complex scenarios. The LLM 0.32 refactoring was designed to address this fundamental architectural bottleneck.

Message Sequences: A New Input Abstraction

Limitations of the Old Model

Previously, LLM managed multi-turn conversations through a conversation object, but this approach could only build conversations from scratch—it couldn't directly inject an existing conversation history. This made tasks like building an OpenAI chat completions API compatibility layer extremely difficult.

OpenAI's Chat Completions API has become the de facto industry standard interface format, with almost all model providers (Anthropic, Google, Mistral, etc.) offering compatibility layers. The core design of this API accepts a messages array, where each message has a role (system/user/assistant) and content field. Previously, LLM's conversation abstraction couldn't directly map to this pattern, requiring extensive adapter code when building proxy servers or middleware. The new messages interface was designed precisely to bridge this gap.

The New Messages Interface Design

Version 0.32a0 introduces llm.user() and llm.assistant() builder functions, allowing developers to pass in message sequences directly:

import llm
from llm import user, assistant

model = llm.get_model("gpt-5.5")
response = model.prompt(messages=[
    user("Capital of France?"),
    assistant("Paris"),
    user("Germany?"),
])
print(response.text())

The existing prompt= parameter still works—LLM internally upgrades it to a single-message array. Additionally, the new response.reply() method provides a more natural way to continue conversations without manually managing conversation objects.

This design lets developers flexibly construct arbitrary conversation contexts, making it ideal for scenarios requiring preset conversation histories, such as building few-shot examples, implementing conversation branching, or forwarding requests through API gateways.

Streaming Typed Parts: Handling Mixed-Type Output

Mixed Output Requirements of Modern LLMs

A single response from a modern LLM may contain multiple content types: reasoning text, body text, tool call requests, tool execution results, and even image and audio clips. Server-side tools like OpenAI's code interpreter and Anthropic's web search make response content even more complex and diverse.

Streaming responses are a key user experience optimization in modern LLM applications. Since large language models generate text token by token, streaming allows users to begin reading content before the complete response is generated, significantly reducing perceived latency. The underlying implementation typically uses the HTTP Server-Sent Events (SSE) protocol, with clients receiving data incrementally through persistent connections. LLM 0.32's stream_events design draws inspiration from LangChain's astream_events interface concept but provides a lighter-weight implementation that stays closer to the underlying model API semantics.

The stream_events Streaming Event Interface

The new version provides typed event streams through response.stream_events() and its async counterpart response.astream_events():

for event in response.stream_events():
    if event.type == "text":
        print(event.chunk, end="", flush=True)
    elif event.type == "tool_call_name":
        print(f"\nTool call: {event.chunk}(", end="", flush=True)
    elif event.type == "tool_call_args":
        print(event.chunk, end="", flush=True)

One direct benefit of this design is that the CLI tool can now display "thinking" text and final response text in different colors, with reasoning text output to stderr so it doesn't interfere with pipeline operations. The new -R/--no-reasoning flag can completely suppress reasoning token output.

Some background on reasoning output: reasoning output originated from the "thinking" mechanism introduced by models like OpenAI's o1/o3 series and DeepSeek-R1. These models output an internal reasoning process (Chain-of-Thought) before generating the final answer, helping improve accuracy on complex tasks. These reasoning tokens typically need to be displayed differently from the final answer—users may want to see the thought process to build trust, but when processing programmatically, they may want only the final result. LLM 0.32's event type system elegantly addresses this need.

Complete Closed-Loop Tool Calling Implementation

Tool calling (Function Calling / Tool Use) has been one of the most important capability extensions in the LLM space since the second half of 2023. Models can no longer only generate text—they can also output structured function call requests, which the application layer executes and returns results to the model. This forms the fundamental AI Agent loop: model thinks → requests tool → gets results → continues reasoning. OpenAI, Anthropic, and Google all support this capability, but each provider's API format differs slightly—OpenAI uses a tool_calls array while Anthropic uses tool_use content blocks—making LLM's unified abstraction particularly valuable here.

After a response completes, developers can call response.execute_tool_calls() to directly execute the requested functions, or use response.reply() to automatically send tool return values back to the model, forming a complete agent loop. This makes building AI Agents with the LLM library much more concise, eliminating the need to manually assemble multi-turn messages and handle format differences across providers.

Serialization and Deserialization Mechanism

Addressing the rigidity of LLM's current SQLite persistence scheme, version 0.32a0 adds a general serialization interface:

serializable = response.to_dict()
# Store wherever you like
response = Response.from_dict(serializable)

The returned dictionary is a TypedDict defined in the llm/serialization.py module, providing developers with flexible storage options beyond SQLite. Whether storing in Redis, PostgreSQL, or writing directly to JSON files, implementation is straightforward.

The background for this design lies in Simon Willison's unique philosophy regarding SQLite. He is a strong advocate for SQLite, and his Datasette project is a data exploration tool built entirely around SQLite. The LLM library uses SQLite to store all conversation logs, aligning with his vision of "personal data infrastructure"—all interaction records are saved in a single local file that can be queried and analyzed at any time. But as response structures have grown increasingly complex (containing tool call chains, reasoning text, multimedia attachments, etc.), the original relational table structure has struggled to elegantly accommodate these nested data types. The new serialization interface preserves the convenience of SQLite logging while opening the door for users who need other storage solutions.

Upcoming Version Plans and Roadmap

Following the alpha release, Simon plans to validate the design in real-world environments for a few days before releasing stable version 0.32. The main remaining task is redesigning the SQLite logging system to more granularly capture the detailed information returned by the new abstractions.

He is also considering modeling conversation storage as a graph structure to support scenarios where conversations in the chat completions API context keep expanding without redundant storage. This feature may be implemented in version 0.32 or 0.33. Graph-structured storage means each message exists as a node with conversation branches represented by edges, so when multiple conversations share prefixes (e.g., the same system prompt with different user questions), duplicate storage of identical message content can be avoided while naturally supporting conversation tree visualization and backtracking.

Design Philosophy of Open-Source Project Evolution

This refactoring embodies the evolutionary wisdom of excellent open-source projects: introducing new abstraction layers to adapt to rapid domain changes while maintaining backward compatibility. From "text in, text out" to "message sequence input, typed streaming output," the LLM library's abstraction layer has finally caught up with the pace of large language model technology itself.

Notably, this incremental refactoring strategy is especially important in the fast-evolving AI tool ecosystem. Unlike LangChain's frequent breaking changes, the LLM library has chosen a more conservative but user-friendly path: new features are exposed through new interfaces while old interfaces continue to work but are internally bridged to the new implementation. This reduces upgrade costs and reflects Simon Willison's emphasis on API stability as a seasoned open-source maintainer.

For developers currently using the LLM library or considering integrating it into their projects, version 0.32 marks this tool's evolution from a simple text interaction layer into a mature framework capable of handling the complex demands of modern AI applications.

Key Takeaways

• LLM 0.32a0 introduces message sequences (messages) as input, allowing direct injection of conversation history rather than only building conversations from scratch
• The new stream_events interface supports streaming typed parts, distinguishing between text, tool calls, reasoning, and other response content types
• The CLI tool can now display reasoning text and final responses in different colors, with reasoning output to stderr that doesn't interfere with pipeline operations
• New to_dict/from_dict serialization mechanism lets developers freely choose storage solutions beyond SQLite
• The refactoring maintains full backward compatibility—the existing prompt= parameter and iterative streaming still work