LLM 0.32 Refactor: An Architecture Upgrade from Text Q&A to Multimodal Streaming Interaction

Simon Willison just released LLM 0.32a0 alpha — a major refactor of his LLM Python library and CLI tool. While maintaining backward compatibility, the underlying abstraction model has undergone a fundamental shift — evolving from simple "text in, text out" to a modern architecture supporting message sequence input and multi-type streaming output.

Simon Willison is the co-creator of the Django framework and a well-known developer in the data journalism and open-source tools space. His LLM project is positioned as a unified command-line interface and Python library that, through a plugin architecture, lets users interact with OpenAI, Anthropic Claude, Google Gemini, local Llama models, and thousands of other large language models in a consistent way. The CLI tool's design philosophy follows the Unix tradition — it can be combined with other commands through pipes, e.g., cat file.txt | llm 'summarize this'. This design makes it a Swiss Army knife tool in developers' daily workflows.

Why the LLM Library Needed This Refactor

Looking back to when the LLM project launched in April 2023, LLM interaction patterns were relatively simple: send a piece of text, get a text response back. This abstraction was elegant enough and practical enough.

But two years later, the LLM ecosystem has changed dramatically. The LLM library successively added attachments for image, audio, and video input, then schemas for structured JSON output, and then tools for function calling. Meanwhile, frontier models kept evolving — reasoning capabilities, image generation, audio output, and other multimodal abilities emerged one after another.

The original "text in, text out" abstraction could no longer accommodate this complexity. As an abstraction layer covering thousands of models through its plugin system, LLM had to keep pace with the times.

Message Sequence API: More Flexible Multi-Turn Conversation Management

Limitations of the Old Pattern

Before 0.32, LLM managed multi-turn conversations through a conversation object:

conversation = model.conversation()
r1 = conversation.prompt("Capital of France?")
r2 = conversation.prompt("Germany?")

This approach worked fine when building conversations from scratch, but couldn't inject an existing conversation history from outside. If you wanted to build an OpenAI chat completions API-compatible service on top of the LLM library, this became a thorny obstacle. While the CLI tool implemented conversation persistence through SQLite, this was never part of a stable Python API.

New Message Sequence Usage

0.32a0 introduces user() and assistant() builder functions, allowing direct message sequence input:

from llm import user, assistant

response = model.prompt(messages=[
    user("Capital of France?"),
    assistant("Paris"),
    user("Germany?"),
])

This directly corresponds to the message format used by OpenAI and other major APIs, significantly reducing the complexity of building compatible services. The Chat Completions API introduced by OpenAI in March 2023 established the industry's de facto standard message format: conversations are modeled as an array of messages, each containing a role (system/user/assistant) and content field. This format has been adopted or made compatible by virtually all major LLM providers including Anthropic, Google, and Mistral. Its advantage lies in being stateless — each request carries the complete conversation history, so the server doesn't need to maintain session state, which is critical for horizontal scaling of API services.

The original prompt= parameter still works — LLM internally upgrades it to a single-message array.

Additionally, the new version supports replying directly to a response as an alternative way to build conversations:

response2 = response.reply("How about Hungary?")

Typed Streaming Output: Handling Mixed Multimodal Content

The Challenge of Mixed Content

Modern LLM output is far from plain text. A single Claude call might sequentially return: reasoning process, text content, tool call requests, and then more text. Server-side tools like OpenAI's code interpreter and Anthropic's web search mix tool calls and tool outputs into responses. Multimodal output models can even intersperse image or audio segments within streaming responses.

Streaming is a core feature of modern LLM APIs. Since large language models generate text token by token, streaming allows clients to begin receiving and displaying content during generation rather than waiting for the entire response to complete. This is typically implemented via HTTP Server-Sent Events (SSE). At the user experience level, streaming reduces perceived latency from several seconds to the time-to-first-token (usually a few hundred milliseconds) — this is the technical foundation behind the character-by-character display effect used by products like ChatGPT.

The stream_events Interface in Detail

The new version introduces stream_events() and its async counterpart astream_events(), modeling streaming output as a typed event stream:

for event in response.stream_events():
    if event.type == "text":
        print(event.chunk, end="")
    elif event.type == "tool_call_name":
        print(f"\nTool call: {event.chunk}(", end="")
    elif event.type == "tool_call_args":
        print(event.chunk, end="")

The practical impact of this design is immediate — the CLI tool can now display "thinking" text and final response text in different colors, with thinking content output to stderr so it doesn't interfere with pipe operations. Combined with the updated llm-anthropic plugin, Claude models' reasoning process can be streamed in gray, while the formal response is presented in regular colors.

For tool calling, after a response completes you can call response.execute_tool_calls() to execute the requested functions, or use response.reply() to automatically invoke tools and pass results back to the model. Tool calling (Function Calling) has been one of the most important capability extensions in the LLM space since mid-2023. It works like this: developers declare available function signatures (name, parameters, description) to the model; during reasoning, the model determines whether it needs to call a tool, and if so returns a structured call request (function name and argument JSON); the client executes the actual call and passes the result back to the model to continue reasoning. This forms the foundational paradigm for AI Agents — the model is no longer just a text generator but an intelligent agent capable of planning and executing multi-step tasks.

Serialization and Deserialization: Breaking Free from SQLite

Addressing the rigidity of the current SQLite persistence approach, 0.32a0 adds a general-purpose serialization interface:

serializable = response.to_dict()
# Store anywhere
response = Response.from_dict(serializable)

The returned dictionary is a TypedDict defined in the llm/serialization.py module. Developers can freely choose their storage backend — whether Redis, PostgreSQL, or the filesystem — no longer bound to SQLite. TypedDict is a feature in Python's typing module that allows developers to define precise key-value type constraints for dictionaries. Unlike regular dictionaries, TypedDict provides complete type inference and error detection in static type checking tools (like mypy, pyright). Using TypedDict rather than custom classes in serialization scenarios means the serialized result is a standard Python dictionary that can be directly converted to JSON via json.dumps() without additional serialization logic, while still preserving type safety.

Future Version Plans and Roadmap

This alpha release is intended to validate the new design in real-world environments. Simon expects the stable 0.32 release to be very close to the alpha, unless testing exposes design flaws.

There's also an important TODO item: redesigning the SQLite logging system to better capture the fine-grained information returned by the new abstractions. The ideal approach would model conversations as a graph structure, avoiding redundant storage of the same conversation history in OpenAI-style chat completions scenarios. Traditional conversation storage saves each API call's complete message array as a single record, which leads to massive duplication in multi-turn conversations — the record for turn N contains all content from the previous N-1 turns. A graph structure approach treats each message as an independent node, using edges to represent causal relationships between messages. This not only saves storage space but naturally represents conversation branches (multiple different replies to the same context) and conversation merges — complex topologies that are particularly important in AI Agent multi-path exploration scenarios. This feature may be implemented in 0.32 or 0.33.

Summary: An Architecture Evolution Worth Watching

The LLM 0.32 refactor demonstrates how an excellent open-source project can keep pace with technological evolution while maintaining backward compatibility. The abstraction upgrade from "text Q&A" to "message sequences + typed streaming output" not only solves current pain points but also reserves space for emerging capabilities like multimodal output and server-side tool calling. For developers building applications on the LLM Python library, this is an architecture evolution worth paying close attention to.

Key Takeaways

• LLM 0.32a0 upgrades the input model from single text prompts to message sequences (messages), supporting direct injection of conversation history and aligning with OpenAI and other major API formats
• New stream_events() typed streaming output interface distinguishes between text, tool calls, reasoning process, and other response content types
• Introduces general-purpose to_dict()/from_dict() serialization mechanism, removing the hard dependency on SQLite storage
• Maintains full backward compatibility — the original prompt= parameter and iterative streaming read patterns continue to work
• Future plans include redesigning conversation logs as a graph structure to avoid redundant storage of the same conversation history