LLM 0.32 Refactoring Deep Dive: Message Sequence Input and Typed Streaming Response Design

Overview

Simon Willison has released LLM 0.32a0 alpha, a major backward-compatible refactoring of his open-source Python library and CLI tool. LLM is a unified abstraction layer that accesses thousands of different models through a plugin system. This update fundamentally redesigns the input/output model to accommodate the increasingly complex multimodal interaction requirements of today's frontier models.

Simon Willison is the co-creator of the Django web framework and the author of the data tool Datasette, with a long history of active involvement in the open-source community and AI application development. His LLM project positions itself as the "Swiss Army knife" of large language models — through a unified command-line interface and Python API, developers don't need to write different integration code for each model provider. Its plugin system is built on Python's pluggy framework, and the community has already developed plugins for dozens of model backends including OpenAI, Anthropic Claude, Google Gemini, and local Ollama. Users can connect to a new model with just llm install llm-claude-4 without modifying any business code. This "write once, run on any model" design philosophy has made LLM a popular tool for rapid prototyping and model evaluation.

Why a Refactoring Was Needed

Limitations of the Old Abstraction

The LLM project began in April 2023 with a very simple initial design: text in, text out. This was perfectly reasonable at the time — GPT-3.5 had just opened its API, and the vast majority of use cases were "give the model some text, get some text back." But over the past two-plus years, the large language model landscape has undergone a sea change.

Over time, the LLM library progressively added attachments (handling image, audio, and video inputs), schemas (structured JSON output), tool calling, and more. Meanwhile, models themselves kept evolving — reasoning capabilities, image generation, multimodal outputs, and other new features emerged one after another. The original "text in, text out" abstraction could no longer support these complex scenarios.

This problem of "leaky abstractions" is a classic issue in software engineering: when underlying capabilities exceed the expressive range of the abstraction layer, developers either bypass the abstraction to call the underlying API directly (losing uniformity) or keep patching the abstraction layer (leading to bloated interfaces). Simon chose a third path — redesigning the core abstraction while maintaining backward compatibility.

Core Change One: Message Sequences as Input

In previous versions, LLM built conversations incrementally through a conversation object, but couldn't inject a complete conversation history from the start. This made scenarios like building OpenAI-compatible APIs extremely difficult.

The new version introduces the messages=[] parameter along with llm.user() and llm.assistant() builder functions:

import llm
from llm import user, assistant

model = llm.get_model("gpt-5.5")
response = model.prompt(messages=[
    user("Capital of France?"),
    assistant("Paris"),
    user("Germany?"),
])
print(response.text())

This design directly aligns with the OpenAI Chat Completions API message format. The Chat Completions API was introduced by OpenAI in March 2023 alongside GPT-3.5-turbo as a conversational interface that models dialogue as an array of messages, each tagged with a role: system (system instructions that set model behavior), user (user input), and assistant (model responses). This format has become the de facto industry standard — Anthropic's Messages API, Google's Gemini API, and open-source model inference servers like vLLM/TGI all adopt similar message array structures. LLM's alignment with this format means developers can easily pass in existing conversation data (e.g., historical sessions loaded from a database, conversation records imported from other systems) directly, without having to replay them one by one via conversation.prompt().

The original prompt= parameter still works — LLM upgrades it to a single-message array behind the scenes.

Additionally, the new version supports replying directly to a response via the response.reply() method as an alternative way to build conversations:

response2 = response.reply("How about Hungary?")

This chaining pattern is particularly well-suited for interactive scenarios and agent loops — each reply() automatically carries the complete prior conversation context, so developers don't need to manually manage message history.

Core Change Two: Typed Streaming Message Parts

This is the more forward-looking design change. Modern large language model responses are no longer a single text stream — Claude might first return reasoning (the "chain of thought," intermediate reasoning steps the model shows before giving a final answer), then output text, then initiate a tool call; OpenAI's code interpreter returns execution results; multimodal models can even mix image and audio fragments within a streaming response.

Traditional streaming responses are implemented via the Server-Sent Events (SSE) protocol: the server pushes data chunks over a persistent HTTP connection, and the client receives and renders them incrementally, producing the "typewriter effect." But when response content extends from pure text to multiple types, consumers need to know the semantics of each data chunk — is this text the final answer or internal reasoning? Is this JSON fragment a tool call argument or structured output?

The new version exposes a typed event stream through the stream_events() and astream_events() (the async version, based on Python's async/await mechanism) methods:

for event in response.stream_events():
    if event.type == "text":
        print(event.chunk, end="", flush=True)
    elif event.type == "tool_call_name":
        print(f"\nTool call: {event.chunk}(", end="", flush=True)
    elif event.type == "tool_call_args":
        print(event.chunk, end="", flush=True)

This means consumers can handle events differently based on type — the CLI tool can now display "thinking" text and final responses in different colors, with reasoning text output to stderr to avoid interfering with pipeline operations (in Unix philosophy, stdout is for a program's formal output while stderr is for auxiliary information, so pipe commands like llm prompt "..." | jq won't be disrupted by reasoning text).

This typed event stream design pattern isn't original to LLM — LangChain's callback system and the Vercel AI SDK's streamText() both use similar approaches. But LLM's implementation is more lightweight: event types are distinguished by simple string enumerations, and plugin developers only need to specify the type when yield-ing, without inheriting complex base classes.

Serialization and Deserialization: Breaking Free from SQLite

Addressing LLM's previous over-reliance on SQLite for conversation storage, 0.32a0 provides a general-purpose serialization mechanism:

serializable = response.to_dict()
# Store wherever you like
response = Response.from_dict(serializable)

The returned dictionary is a TypedDict defined in the llm/serialization.py module. TypedDict is a type annotation feature introduced in Python 3.8 (PEP 589) that allows developers to specify value types for each key in a dictionary, gaining IDE autocompletion and static type checking support while maintaining dictionary flexibility. Compared to Pydantic's BaseModel, TypedDict is more lightweight — it's essentially a plain dictionary that requires no extra conversion step when serializing to JSON.

The significance of this change lies in separation of concerns: the LLM library no longer makes any assumptions about the storage backend. Developers can serialize conversations and store them in PostgreSQL, Redis, S3, or even write directly to the filesystem. For teams building production-grade applications, this removes an important architectural constraint — previously, if you wanted to use the LLM library in a distributed environment, SQLite's single-file locking mechanism would become a concurrency bottleneck.

CLI-Level Changes

Interestingly, this major refactoring introduces only one new flag at the CLI level: -R/--no-reasoning, used to suppress reasoning token output. This reflects the backward-compatible design philosophy — the underlying architecture evolves dramatically while the user interface remains stable.

Reasoning tokens (also called "thinking tokens") represent an important recent trend in model development. Exemplified by OpenAI's o1/o3 series and Anthropic's Claude extended thinking feature, these models perform internal reasoning before generating their final answer. This reasoning content is sometimes valuable for debugging but is usually noise in production pipelines. The -R flag lets users control this behavior on demand.

Future Plans

Simon states this is an alpha release, intended to validate the new design in real-world environments. The stable 0.32 release is expected to be very close to the alpha, unless testing exposes design flaws.

There's also one major remaining task: redesigning the SQLite logging system to store conversations as a graph structure, avoiding duplicate storage of the same conversation history in Chat Completions-style API scenarios. This may be implemented in 0.32 or 0.33.

The "graph structure" concept here is worth elaborating. The current linear storage model treats each conversation as an independent sequence of messages. If a user initiates multiple different follow-up questions based on the same conversation history (i.e., conversation branching), each branch fully duplicates all previous messages. This is similar to code version management before Git — every version is a complete copy. Graph-structured storage is analogous to Git's directed acyclic graph (DAG): shared conversation prefixes are stored only once, with different paths after branch points stored independently. This not only saves storage space but also naturally represents the tree-like branching structure of conversations, which is particularly important in agent scenarios — an agent might need to try multiple different tool-calling strategies at the same conversation node.

API Design Takeaways

This refactoring is a valuable case study in API design evolution. Facing a rapidly changing large language model ecosystem, Simon chose the following strategies:

Backward compatibility: Old code runs without modification. This follows the spirit of Semantic Versioning (SemVer) — although breaking changes are theoretically allowed during the 0.x phase, Simon still chose the more conservative route because LLM already has a large user base and plugin ecosystem.
Progressive enhancement: New capabilities are exposed through new parameters and methods. messages=[] is a new parameter, stream_events() is a new method, and the existing prompt= and response.text() continue to work normally.
Separation of concerns: Serialization is decoupled from storage; streaming events are decoupled from rendering. This is a classic software architecture principle, but in the fast-iterating AI tool ecosystem, many projects skip this step for speed, leading to dramatically rising refactoring costs later.
Future-proofing: The typed event stream reserves extension space for model capabilities that don't yet exist. When future models start returning 3D models, interactive charts, or real-time video streams, only new event types need to be added — the overall structure of consuming code doesn't need to change.

For developers building large language model applications, this evolution from "simple text pipelines" to "typed message streams" is worth watching and learning from. It reflects an industry-wide consensus: large language models are evolving from "text generators" into "universal computation interfaces," and the toolchains around them must upgrade accordingly.

Key Takeaways

LLM 0.32a0 introduces message sequences (messages=[]) as input, allowing developers to directly inject complete conversation history, aligning with the OpenAI Chat Completions API format
A new typed streaming event system (stream_events) supports distinguishing between text, tool calls, reasoning, and other response content types
A general-purpose response.to_dict()/from_dict() serialization mechanism removes the hard dependency on SQLite storage
The overall refactoring maintains backward compatibility, with only the -R/--no-reasoning flag added at the CLI level
Future plans include redesigning the SQLite logging system with a graph structure to optimize conversation storage deduplication