PicoMQ: A Lightweight Message Streaming Engine Built on Object Storage

Introduction: Simplifying Message Queue Architecture

In modern distributed systems, message queues are virtually indispensable infrastructure. Whether it's Kafka, RabbitMQ, or Pulsar, they all provide powerful capabilities for asynchronous communication, traffic smoothing, and system decoupling. However, these mature solutions often come with significant operational complexity—requiring dedicated Broker clusters, partition management, replica synchronization, and storage capacity planning.

The PicoMQ project, which recently caught attention on Hacker News (the Show HN post received 92 points and 18 comments), proposes a radically different approach: building persistent message streams directly on top of object storage and exposing them via HTTP. This design philosophy has sparked considerable discussion in the developer community.

PicoMQ's Core Design Philosophy

Object Storage as the Persistence Backend

PicoMQ's most notable feature is its choice of object storage (such as Amazon S3, MinIO, or other S3-compatible storage) as the message persistence layer. This stands in stark contrast to traditional message queues that rely on local disks or dedicated storage engines.

Object Storage is a data storage paradigm fundamentally different from traditional file systems and block storage. It manages data as "objects," where each object contains the data itself, variable-length metadata, and a globally unique identifier. Unlike file systems with hierarchical directory structures, object storage uses a flat namespace, giving it exceptional scalability for massive data scenarios. Since Amazon S3 launched in 2006 and became the de facto standard API for object storage, open-source implementations like MinIO and Ceph RGW have adopted S3 protocol compatibility. The core design trade-off of object storage is: sacrificing random read/write and low-latency access in exchange for nearly unlimited horizontal scalability and extremely high data durability.

This architectural choice brings several significant advantages:

• No storage capacity management needed: Object storage inherently provides nearly unlimited elastic capacity, freeing developers from worrying about running out of disk space.
• Durability and reliability guaranteed by cloud providers: Major object storage services typically offer 99.999999999% (11 nines) data durability, far exceeding self-managed storage reliability. This means that when storing 1 billion objects, you'd expect to lose only one object on average every 100 years.
• Cost optimization: Object storage's per-unit cost is typically lower than block storage, making it especially suitable for high-volume, infrequently accessed persistent message scenarios. For reference on AWS, S3 Standard storage costs roughly one-fifth the price of EBS gp3 volumes.

HTTP as the Transport Protocol

PicoMQ uses HTTP as the interface protocol for message production and consumption, rather than the proprietary binary protocols commonly used by traditional message queues (such as Kafka's wire protocol or AMQP).

This choice lowers the barrier to entry—any client capable of making HTTP requests (whether a browser, Serverless function, or IoT device) can easily produce or consume messages without importing specialized SDKs or complex connection management logic. This design is particularly friendly for cloud-native and edge computing scenarios. Notably, HTTP already has an extremely mature ecosystem for security (TLS), authentication (OAuth, API Key), load balancing, and observability. PicoMQ can directly leverage existing API gateways, CDNs, WAFs, and other infrastructure without needing to build separate security and routing layers for message transport.

Why This Architecture Deserves Attention

Alignment with Serverless and Cloud-Native Trends

In recent years, the industry has seen a strong trend toward "complete separation of storage and compute." Projects like KIP-405 (Tiered Storage) in the Kafka ecosystem, as well as WarpStream and AutoMQ, are all exploring the possibility of offloading data to object storage. PicoMQ can be seen as a lightweight implementation of this philosophy.

Apache Kafka's original architecture stored data on Broker local disks. While this tightly coupled design delivered extremely high throughput and low latency, it also made cluster scaling complex—every node addition or removal required large-scale partition data migration. KIP-405 (Tiered Storage) was proposed by the Kafka community in 2019 as an architectural evolution. Its core idea is to divide data into a "hot tier" (local disk) and a "cold tier" (remote object storage), with historical data automatically sinking to cheap storage like S3. WarpStream takes a more radical approach, completely eliminating local disk dependency—all data is written directly to object storage, and Brokers become purely stateless compute nodes. AutoMQ adopts a similar philosophy, achieving second-level scaling through a shared storage layer. Together, these projects represent the industry trend of message systems evolving from "integrated storage-compute" to "separated storage-compute."

For applications running in serverless environments like AWS Lambda or Cloudflare Workers, maintaining a long-lived message Broker cluster is both expensive and contrary to the Serverless philosophy. The core characteristics of Serverless computing are pay-per-invocation, auto-scaling, and stateless execution. Function instances have extremely short lifespans (typically milliseconds to minutes), and different invocations don't share memory or connection state. This fundamentally conflicts with the design assumptions of traditional message queue clients: Kafka consumers need to maintain long-lived connections to Brokers, participate in consumer group coordination, and manage offset commits—all stateful operations. Every Lambda cold start requires re-establishing TCP connections, completing authentication, and joining consumer groups, and these overheads can significantly impact performance and cost in high-concurrency scenarios.

PicoMQ's "stateless compute + object storage" model naturally aligns with the cloud-native philosophy of on-demand scaling and pay-per-use. HTTP's stateless nature means each request completes independently without maintaining connection pools or session state, perfectly fitting the Serverless execution model.

Dramatically Reduced Operational Complexity

The operational burden of traditional message queues is often underestimated. Kafka cluster partition rebalancing, replica synchronization, and ZooKeeper (or KRaft) coordination all require continuous investment from specialized SRE teams. Take partition rebalancing as an example: when new nodes are added to the cluster, partition data must be migrated from existing nodes to new ones. This process can take hours or even days, consuming significant network bandwidth and potentially impacting online service performance. ZooKeeper, the metadata management component in earlier Kafka versions, is itself a distributed system requiring independent operations, and the complexity of its election mechanisms and session management frequently becomes a source of failures. While the KRaft mode introduced in Kafka 3.x removes the ZooKeeper dependency, the overall operational complexity of the cluster remains considerable.

PicoMQ delegates all persistence responsibilities to object storage, theoretically achieving "zero Broker operations"—which holds significant appeal for small teams and startups.

PicoMQ's Trade-offs and Limitations

No architectural choice comes for free. Using object storage as a message backend introduces some trade-offs that must be acknowledged:

Latency Concerns

Object storage access latency typically ranges from tens to hundreds of milliseconds, far higher than local SSDs (microseconds) or memory (nanoseconds). This means PicoMQ is better suited for latency-insensitive async tasks, log aggregation, and event archival rather than applications requiring millisecond-level responses like high-frequency trading or real-time bidding. For reference, Kafka's end-to-end latency on SSDs is typically 2-10 milliseconds, while message delivery via object storage may reach 100-500 milliseconds—a gap of one to two orders of magnitude.

Throughput vs. Batching Trade-offs

To achieve acceptable performance on object storage, small messages typically need to be batch-written into larger object files. While this batching mechanism improves throughput, it further increases end-to-end latency, creating a classic "throughput for latency" trade-off. Specifically, object storage charges a fixed fee per PUT operation (e.g., S3 charges $0.005 per 1,000 PUT requests). Writing each message as an individual object would not only result in high latency but also rapidly accumulate API call costs. Therefore, a reasonable batching window (aggregating by time window or message count) becomes a critical design parameter for such systems.

Challenges with Transactions and Ordering Guarantees

Object storage doesn't provide transaction semantics natively. Implementing strict message ordering, exactly-once delivery, and other advanced features requires additional engineering effort at the application layer. This was one of the technical details that developers focused on in the Hacker News discussion.

Exactly-once delivery semantics is the most difficult guarantee level to implement in distributed messaging systems, requiring that each message is "processed exactly once"—neither lost nor duplicated. Achieving this goal under abnormal scenarios like network partitions and node failures requires complex coordination mechanisms. Kafka approaches exactly-once semantics through Idempotent Producers and Transactional Messaging, relying on sequence number tracking, two-phase commits, and transaction coordinators. The difficulty of implementing similar semantics on top of object storage lies in the fact that S3 and similar services only provide simple PUT/GET primitives without support for atomic compare-and-swap operations or transaction logs. Developers typically need to rely on external coordination services (such as DynamoDB's conditional writes) to achieve deduplication and ordering guarantees, which increases overall system complexity.

PicoMQ's Use Cases and Future Outlook

All things considered, PicoMQ isn't meant to replace heavyweight message systems like Kafka. Instead, it provides a lighter, more hassle-free option for specific scenarios:

• Cost-sensitive small-to-medium projects: No need to pay the high price of operating a full message cluster.
• Serverless and edge computing: Seamless integration via HTTP, highly compatible with stateless compute paradigms.
• Log and event archival: Leveraging object storage's low cost and high durability for long-term data stream retention.

As object storage performance continues to improve (with the emergence of low-latency products like S3 Express One Zone), the applicability of "object-storage-based message streaming" is expected to expand further. S3 Express One Zone is a high-performance storage class AWS introduced at their 2023 re:Invent conference, reducing object storage access latency from the traditional S3's tens of milliseconds to single-digit milliseconds. By constraining data to a single Availability Zone and using SSDs as the underlying medium, it achieves data access speeds 10x faster than standard S3. This product blurs the performance boundary between object storage and block storage, making previously latency-constrained scenarios feasible for object storage. However, single-AZ deployment also means sacrificing the high availability provided by cross-AZ redundancy, requiring users to weigh their business availability requirements.

As a lightweight exploration in this space, PicoMQ is worth continued attention from developers.

Conclusion

PicoMQ represents a pragmatic engineering philosophy: using mature infrastructure (object storage + HTTP) to solve message persistence problems, trading for operational minimalism. It may not be suitable for every scenario, but in today's world where cloud-native and Serverless are mainstream, this "simplicity over complexity" design philosophy undoubtedly offers a fresh perspective on message queue architectural evolution. For teams struggling with message system operations, it's worth giving it a try.