Datasette Streaming Export Plugin datasette-arrows Released

Datasette's new datasette-arrows plugin provides Apache Arrow IPC stream export via the streaming export hook.
Simon Willison released the datasette-arrows 0.1a0 plugin, adding Apache Arrow IPC stream export to Datasette. The plugin is based on a new capability in Datasette 1.0a43: the `register_output_renderer()` hook now supports streaming export, letting plugins output data chunk by chunk like core CSV/TSV exports without loading all results into memory. datasette-arrows is the first reference implementation, reflecting the author's principle of always shipping a sample plugin with a new plugin feature. Arrow was chosen because its columnar in-memory format is widely supported by DuckDB, pandas, Polars, and other tools, so exported streams can be read directly without conversion. Installation is just `datasette install datasette-arrows`. The main branch already shows the Arrow option, and grouping adjustments in the official release will arrive in the next alpha.
The Datasette ecosystem welcomes an interesting new member. Simon Willison has released datasette-arrows 0.1a0, which adds the ability to export in Apache Arrow IPC stream format to Datasette. This plugin was born out of an underlying capability upgrade introduced in Datasette 1.0a43: plugins can now implement streaming export formats.

Streaming Export: From Core Capability to Plugin Ecosystem
Datasette core already supported streaming CSV and TSV exports. This means that even if a table contains millions of rows, the export process does not need to load everything into memory at once; instead, it outputs data in chunks as a stream. For scenarios with large amounts of data, this is a very practical capability.
Datasette 1.0a43 opened this capability up to plugin authors. The register_output_renderer() plugin hook now allows third-party plugins to implement the same streaming mode, rather than being limited to the two formats built into core.
datasette-arrows is the first practitioner of this hook. In the release notes, the author mentions a principle: "I never ship a new plugin feature without a companion plugin." This "feature plus example" release approach has real value for ecosystem building—plugin authors have a reference implementation, and users can immediately see what the new capability is for.
register_output_renderer() is one of the core hooks in the Datasette plugin system, allowing plugins to register custom export formats. It receives an object containing parameters such as datasette, columns, rows (or a streaming cursor), and sql, and returns a three-tuple: the response body, the content type, and an optional filename. Before 1.0a43, this hook could only receive a complete list of results and could not handle generators or async iterators, so plugins could not achieve true streaming output. The 1.0a43 change makes the hook support passing in iterable streaming data sources, so plugins can consume data chunk by chunk and write it directly to the response, thereby avoiding loading all results into memory. This upgrade is especially critical for handling large SQLite tables, because Datasette's typical use case is quickly exposing databases that may contain millions of rows.
Why Choose Apache Arrow
According to the release notes, the suggestion to use the Apache Arrow stream format for this plugin came from Claude Fable 5.2, and it completed the plugin build almost in one go.
Apache Arrow's IPC stream format is suitable for this scenario for several reasons. It is a columnar in-memory format that is widely supported in the data analysis toolchain. Arrow streams exported through Datasette can be read directly by tools such as DuckDB, pandas, and Polars, eliminating the intermediate step of format conversion. For users who need to bring data from Datasette into their analysis workflows, this is closer to what downstream tools expect as input than CSV.
Installation is simple: datasette install datasette-arrows. After installation, an Arrow option appears in the export dialog.
Apache Arrow IPC stream format is a cross-language standard for in-memory columnar data representation, with the core design goal of sharing data between different systems with zero copies. The IPC (Inter-Process Communication) stream format serializes data into a continuous byte stream in record batches; each batch contains a fixed number of rows and carries metadata describing the schema. Compared with CSV, Arrow preserves strict data types (such as int64, timestamp, decimal), so there is no need to re-infer types on the reading side, and it avoids floating-point precision loss and date-parsing ambiguity. The columnar layout also allows downstream tools to read only the columns they need and to use vectorized execution engines to accelerate analysis. Tools such as DuckDB, pandas (via pyarrow), and Polars all natively support reading data from Arrow IPC streams, so streams exported by Datasette can be consumed directly by these tools without intermediate conversion.
Current Status and Future Plans
The screenshot shows the result on Datasette's main branch. In the export dialog, an Arrow option has been added next to TSV and CSV, displaying "All 927 rows" and labeled Apache Arrow IPC stream, indicating that it is suitable for tools such as DuckDB, pandas, and Polars, with two buttons: Download and Copy URL.
It should be noted that the most recent official release has not yet grouped streaming formats such as Arrow together with TSV and CSV under "All rows in this table"; this adjustment will come in the next alpha version. In other words, the plugin's functionality is already usable, but the interface grouping will continue to be polished.
A Brief Observation
This release reflects a characteristic of the Datasette plugin system: the core stays lean, capabilities are extended through plugins, and when a new hook launches, it comes with a reference implementation. This approach reduces the maintenance burden on core and also gives plugin authors in the ecosystem a clear paradigm to follow.
For data analysis workflows, the addition of Arrow export makes Datasette more like a data transfer station—streaming directly from SQLite tables to analysis tools, with one less format conversion step in between. For people who frequently move data between Datasette and the Python data stack, this plugin is worth trying.
One-line Summary
Datasette's new plugin datasette-arrows provides Apache Arrow IPC stream format export via the streaming export hook.
Paragraph Summary
Simon Willison has released the datasette-arrows 0.1a0 plugin, adding Apache Arrow IPC stream format export capability to Datasette. The plugin is based on a new capability opened up by Datasette 1.0a43: the register_output_renderer() hook now supports streaming export, so plugins can output data chunk by chunk like core CSV/TSV exports, without loading all results into memory. datasette-arrows is the first reference implementation of this hook, and the author adheres to the principle of "always shipping an example plugin when releasing a new plugin feature." Arrow was chosen because its columnar in-memory format is widely supported by analysis tools such as DuckDB, pandas, and Polars, and the exported stream can be read directly, eliminating format conversion. Installation only requires datasette install datasette-arrows. The main branch currently shows the Arrow option, and the grouping adjustment in the official release will arrive in the next alpha.
Related articles

Incident Response in the Age of AI Attackers: A 6-Month Retrospective and the Four Pillars
AI attackers are making attacks on AI infrastructure more frequent. This article explains MarkTechPost's four-pillar incident response playbook based on a 6-month retrospective, with Fireworks AI and Klue case studies on implementation.

Abnormal Error Rate Spike in Claude Mythos 5.2 and Fable 5.2
Claude Mythos 5.2 and Claude Fable 5.2 show an abnormal spike in request error rates, marked as investigating on the official status page. This article covers the incident overview, practical impact on developers, and recommended responses.

ReAct Runtime Context Trimming: Why Claude Code Acts at 70%
A deep dive into ReAct runtime context trimming: why Claude Code compresses at ~70% usage, why tool calls dominate, why messages must be dropped in pairs, and why codebases don't need RAG.