LLM 0.32 Refactoring Deep Dive: Message Sequence Input and Streaming Multi-Type Response Design

Overview

Simon Willison has released version 0.32a0 alpha of his open-source project LLM, marking a major backward-compatible refactoring. LLM is a Python library and CLI tool that provides unified abstract access to thousands of different large language models through a plugin system. Simon Willison is the co-creator of the Django web framework and the author of the data exploration tool Datasette. The LLM project carries forward his plugin-based architecture philosophy proven in Datasette—the core library defines only abstract interfaces, while specific model adapters (such as llm-anthropic, llm-gemini, llm-ollama, etc.) are independently developed and maintained by the community. This design means developers only need to install the corresponding plugin to call models from OpenAI, Anthropic, Google, local Ollama, and other sources using exactly the same code. The core goal of this refactoring is to enable LLM's abstraction layer to keep pace with the rapid evolution of modern AI models in terms of input/output diversity.

Why the Refactoring Was Needed

The LLM project began in April 2023 with a very simple design: text in, text out.

import llm
model = llm.get_model("gpt-5.5")
response = model.prompt("Capital of France?")
print(response.text())

However, over the past two-plus years, the LLM ecosystem has undergone enormous changes. The LLM library progressively added attachments (handling image, audio, and video inputs), schemas (structured JSON output), tool calling, and other features. Meanwhile, frontier models have continuously evolved, supporting reasoning output, image generation, audio segments, and various other capabilities. The original "text in, text out" abstraction could no longer meet these demands.

Core Change One: Message Sequences as Input

Limitations of the Old Approach

Previously, LLM handled multi-turn conversations through a conversation object, but this approach could only build conversations from scratch—it couldn't directly inject an existing conversation history. This made scenarios like building OpenAI-compatible APIs extremely difficult. While the CLI tool worked around this limitation by persisting sessions in SQLite, this was never part of a stable Python API.

The New Message Sequence API

Version 0.32a0 introduces llm.user() and llm.assistant() builder functions, allowing developers to pass in complete message sequences directly:

import llm
from llm import user, assistant

model = llm.get_model("gpt-5.5")
response = model.prompt(messages=[
    user("Capital of France?"),
    assistant("Paris"),
    user("Germany?"),
])
print(response.text())

This design directly aligns with the OpenAI Chat Completions API message format. OpenAI's Chat Completions API uses a message array as input, where each message contains a role (such as system, user, assistant) and a content field. This format has become the de facto industry standard—Anthropic, Google Gemini, Mistral, and other major model providers all offer compatible or similar interfaces. The core advantage of message sequences is making conversation history explicit: developers can precisely control every message in the context window, enabling advanced operations like history injection, context trimming, and system prompt management. This is critical for building RAG systems, Agent frameworks, and API gateways. This change in LLM 0.32a0 makes building compatible services and injecting historical context natural and concise. The existing prompt= parameter still works—LLM internally upgrades it to a single-message array.

Additionally, the new version supports replying directly to a response as an alternative way to build multi-turn conversations:

response2 = response.reply("How about Hungary?")

Core Change Two: Streaming Multi-Type Responses

Problem Background

Modern model outputs are no longer a single text stream. A single Claude call might sequentially return reasoning text, a formal reply, tool call requests, tool execution results, and other types of content. Server-side tools like OpenAI's Code Interpreter and Anthropic's web search make the type combinations in responses even more complex. Multimodal output models can even intersperse image or audio segments within streaming responses.

Traditional streaming responses refer to models returning results token by token during generation, rather than waiting for complete generation before returning all at once. This pattern is typically implemented via Server-Sent Events (SSE) protocol and can significantly reduce the user-perceived Time to First Token (TTFT). However, as model capabilities expand, a single response may interleave multiple content types: Anthropic Claude's extended thinking feature outputs the reasoning chain before the final answer, OpenAI's Code Interpreter embeds code execution results in responses, and multimodal models may even intersperse generated images within text streams. Traditional plain-text streaming interfaces cannot distinguish between these different types of content fragments, necessitating the introduction of a typed event stream model.

The New stream_events API

The new version models responses as a stream of typed message parts. Developers can process events one by one through response.stream_events() or its async version response.astream_events():

for event in response.stream_events():
    if event.type == "text":
        print(event.chunk, end="", flush=True)
    elif event.type == "tool_call_name":
        print(f"\nTool call: {event.chunk}(", end="", flush=True)
    elif event.type == "tool_call_args":
        print(event.chunk, end="", flush=True)

This design delivers tangible user experience improvements. For example, the CLI tool can now display "thinking" text and final reply text in different colors, with reasoning content output to stderr to avoid interfering with pipeline operations. Combined with the updated llm-anthropic plugin, Claude's reasoning process can be streamed in real-time as gray text.

Complete Tool Calling Loop

After the response concludes, you can call response.execute_tool_calls() to execute the functions the model requested, or use response.reply() to automatically invoke tools and send return values back to the model, forming a complete Agent loop.

Tool calling (Function Calling) is one of the core capabilities of modern LLM applications. During reasoning, a model can decide to invoke predefined external functions (such as search engines, database queries, calculators, etc.). The application layer executes the function and returns the result to the model, which then continues reasoning based on the result. This "model requests → application executes → results returned" loop is the foundational pattern for building AI Agents and the core mechanism of Agent frameworks like LangChain and CrewAI. LLM 0.32a0 builds this loop into the library through execute_tool_calls() and reply() methods, enabling developers to implement lightweight Agent functionality without relying on heavyweight frameworks, lowering the barrier to building autonomous decision-making systems.

Serialization and Deserialization Mechanism

To address the rigidity of SQLite persistence, 0.32a0 provides a universal serialization interface:

serializable = response.to_dict()
# Store anywhere
response = Response.from_dict(serializable)

The returned dictionary is a TypedDict defined in the llm/serialization.py module. TypedDict is a feature of Python's type system (PEP 589) that allows precise type annotations for each dictionary key, providing static type checking support while maintaining dictionary flexibility. LLM's choice of TypedDict over Pydantic models or dataclasses as the serialization format reflects a pursuit of lightweight design and interoperability—what's returned is a plain Python dictionary that can be directly serialized with json.dumps or stored in MongoDB, Redis, the filesystem, or any other backend without introducing additional serialization/deserialization dependencies. Developers can freely choose their storage backend and are no longer tied to SQLite.

Future Plans

This is an alpha release, and Simon Willison plans to validate it in real-world environments for a few days before releasing the stable version 0.32. He is also considering redesigning the SQLite logging system to model conversations as a graph structure, avoiding duplicate data storage in OpenAI-style continuous conversation scenarios.

The LLM project has used a SQLite database to log all conversations from the beginning, consistent with Simon Willison's long-standing advocacy of "SQLite as an application data format." However, the current linear conversation storage model produces significant data duplication when facing branching conversations (e.g., initiating multiple different follow-up questions from the same context)—each branch requires a complete copy of the preceding conversation history. Modeling conversations as a Directed Acyclic Graph (DAG) structure can solve this problem: each message becomes a node in the graph, conversation branches are represented by edges, and shared historical context only needs to be stored once. This design is similar to Git's commit graph model and can efficiently represent complex conversation topologies. This feature may be included in 0.32 or deferred to 0.33.

Design Philosophy Insights

This refactoring exemplifies the evolution strategy of excellent open-source projects: backward-compatible incremental refactoring. The old prompt= interface still works, the old iteration methods still function, but the new abstractions lay a solid foundation for future multimodal, multi-tool, and multi-type response scenarios.

For developers building LLM applications, this "message sequences + typed event streams" pattern is worth learning from—it maintains ease of use for simple scenarios while providing sufficient expressiveness for complex ones.

Key Takeaways

• LLM 0.32a0 introduces message sequences (messages) as input, allowing direct injection of complete conversation history, aligned with the OpenAI Chat Completions API format
• The new stream_events API models responses as a typed event stream, supporting differentiated handling of text, tool calls, reasoning, and other content types
• A universal to_dict/from_dict serialization mechanism frees developers from being tied to SQLite storage
• The CLI tool can now display reasoning text and final replies in different colors, with reasoning content output to stderr
• The overall refactoring maintains backward compatibility—the old prompt= parameter and iteration methods still work