How Cloudflare Hyperdrive Solved MySQL Cache Consistency with a Single Canonical Copy

Cloudflare Hyperdrive is a distributed connection pooler designed for Workers: the Client node runs alongside the Worker, the Endpoint node sits near the database, and the two share a data-center cache so that edge requests can reuse already-established database connections. MySQL support has now reached GA after roughly 18 months in beta, and the real story behind this release is a consistency bug hidden in the details of cache encoding.

What Hyperdrive Does Between the Worker and the Database

Workers run at the edge, while databases typically live in a single region. Cloudflare's own estimate is that establishing a connection between the two ends takes at least five network round trips; if every request opened a new connection, the handshake overhead would exceed the query itself. Hyperdrive's approach is to keep connections hot: the Endpoint node maintains an open connection pool shared by a large number of Worker requests.

In addition to the pool, Hyperdrive stores the results of frequently repeated queries in the CDN cache of each data center. The official numbers: connection pooling makes queries 3x faster, repeated queries up to 100x faster, and cache hits have latency under 5 milliseconds. Migration is lightweight too — just swap the connection string, with supported Node.js drivers including mysql2 and mysql.

Two Nodes, One Cache, and a Divergence in Encoding

The architecture places a cache on each end, and when the Client and Endpoint are in the same data center they share the same cache. That is a sensible default for performance: either node can answer a repeated query without touching the database. But it also means two processes write to the same key space, and that is where the bug lived.

Cloudflare's description of the underlying mechanism is quite specific: cache entries are keyed by a hash of the query's wire message (with some normalization), and the value is the result bytes. For this to hold, the same response must be stored as identical bytes regardless of which node writes it. The implementation inherited an assumption from the Postgres era — that the bytes written back to the Worker are the bytes worth caching.

The MySQL Capability Negotiation Mismatch Behind Empty Results

MySQL makes this assumption unsafe, and the reason lies in the details of capability negotiation in its wire protocol. Capabilities are negotiated when a connection is established, and CLIENT_DEPRECATE_EOF changes how a result set ends: the metadata EOF packet disappears and the final packet becomes an OK packet. MySQL 5.7.5 and later support this capability, but the client must actively request the new behavior.

The popular Node.js driver mysql2 does not request it, while Hyperdrive does. So the Endpoint and the Client use different capability sets, and the pooling layer has to translate each result set into the shape the Worker's driver expects — inserting EOF packets, converting OK packets back. Caching the translated bytes breaks the translation in both directions: when the Client reads a result the Endpoint wrote using Hyperdrive's own encoding, it sends it to the Worker without translation; when the Endpoint reads bytes already translated by the Client, it hands them to another Client to translate again, causing an EOF packet to be emitted early or the connection to hang outright.

The symptoms were narrow but the explanation was mechanical: about 0.001% of queries for one high-traffic customer returned empty results, the problem appeared only on instances with caching enabled, and the investigation traced all the way back to the MySQL protocol documentation to find this assumption.

The MySQL wire protocol uses a client/server capability negotiation mechanism: during the connection handshake, both sides exchange 32-bit capability flags to declare the optional features they support, and only capabilities set by both sides are enabled for the session. Common flags include CLIENT_PROTOCOL_41, CLIENT_SSL, CLIENT_MULTI_STATEMENTS, CLIENT_DEPRECATE_EOF, and so on. Capability negotiation is essentially a set of protocol-level feature switches, and the server must organize its response bytes according to the capability set the client actually requested; otherwise the client will misalign packet boundaries when parsing.

CLIENT_DEPRECATE_EOF (introduced in MySQL 5.7.5) changes how a result set ends: the traditional protocol sends an EOF packet (0xFE) after the column definitions to mark the end of metadata, then sends rows one by one and closes with a final EOF packet; with this capability enabled, no EOF is sent at the end of metadata, and the result set is terminated by an OK packet (starting with 0x00, carrying status information such as affected rows and warning count). This saves a round trip but requires the client and server to have exactly the same interpretation of "when the packet ends." Drivers such as mysql2 do not request this capability by default, so the server falls back to the old EOF delimiters; if a middle layer unilaterally parses according to the new capability, it will mistake a data row for the end of the result set, or treat an OK packet as row data — which is exactly the protocol-level cause of the empty-result and hang symptoms.

The Fix: Store a Single Canonical Copy per Result

Cloudflare's fix is a rule about what gets cached: no matter which node produces a result, a single canonical copy is always stored, encoded with Hyperdrive's own capabilities. The Client translates on the egress path, and the Endpoint reads a format it understands. After rollout, the empty-row reports stopped, and the client connection success rate reached the 99.99% target set by the vendor.

Generalizing from a single vendor's bug is usually cheap, but this conclusion holds up: if a shared cache has two writers, there must be a single serialization convention; and the translation layer must be explicit about which side of the cache it sits on. The same discipline applies to any self-built pooling or caching data path, including those placed in front of inference stacks.

The Small Bugs on the Road to GA

There were twenty fixes from beta to GA, two of which are encoding defects worth calling out separately because they show what the same class of failure looks like in lower-risk scenarios. The startup handshake negotiated utf8 — a deprecated alias for utf8mb3, at most three bytes per code point — so emoji written to the database came back corrupted, and the fix was to negotiate utf8mb4.

The second came from the cleanup step. Hyperdrive uses COM_RESET_CONNECTION to clear session state between checkouts, but that command restores the connection to server defaults rather than the encoding negotiated at startup. When the server default is latin1, the returned connection is latin1, and the pooling layer has to re-set the encoding after every reset. Both bugs fall into the category that test suites covering only the happy path will miss, but that customers with unusual data will hit.

utf8mb3 is MySQL's formal renaming of its legacy three-byte UTF-8 implementation (the utf8 alias), first appearing in documentation and error messages in 8.0.28 to make users aware that it is not full UTF-8. It uses at most 3 bytes per character, covering characters within the Basic Multilingual Plane (BMP), but cannot represent four-byte code points such as emoji, some rare Chinese characters, and musical symbols. utf8mb4 is the full UTF-8 implementation, with up to 4 bytes per character, available since MySQL 5.5.3.

MySQL character sets exist at multiple levels: server default, database default, table default, column default, and connection level (controlled by the three session variables character_set_client, character_set_connection, and character_set_results). The connection-level character set is negotiated during the handshake, but COM_RESET_CONNECTION only restores session state to server defaults and does not replay the handshake negotiation. Therefore, if the server default is latin1, the reset connection degrades to latin1, and four-byte characters are truncated or replaced with ? on write or read. This is also why connection pool implementations generally need to explicitly re-set the encoding after a reset.

How to Read the 99.99% Number

Cloudflare reports the client connection success rate, consistent with the 99.99% production availability target set for validation during the beta. This is the vendor's own number; the article does not cite independent measurements. For teams planning a migration, the product page and developer docs state that Hyperdrive supports any MySQL or Postgres database, including managed services and Postgres-compatible engines, and the docs list the supported drivers.

For applications with a regional database and a read-heavy, globally distributed workload, the operational path is clear: route one read path through the pooler, measure the cache hit rate on real traffic, and watch the two metrics this release actually moved — the proportion of empty or malformed responses, and the connection success rate. That is also how any cache consistency effort should be measured: quantify anomalous responses first, then talk about performance gains.

Three Lessons for Self-Built Cache Paths

• A shared cache must agree on a single serialization format; it must not store data in each writer's own encoding.
• The translation layer must be explicit about which side of the cache it sits on, otherwise bidirectional translation will contaminate itself.
• Encoding negotiation (utf8mb4, resetting after COM_RESET_CONNECTION) belongs in test cases, not in discoveries by customers using unusual data.