Building PlanetScale from Scratch: A Deep Dive into Distributed Database Infrastructure

Introduction: Why Build PlanetScale from Scratch

PlanetScale, as an industry-leading distributed MySQL database platform, is renowned for its Vitess-based horizontal scaling capabilities and zero-downtime schema changes. Understanding how its underlying infrastructure is built not only helps engineers deeply grasp the operational principles of distributed databases but also provides invaluable architectural references for building your own highly available database services.

This article takes "Building PlanetScale from Scratch" as its entry point, focusing on the design philosophy of its infrastructure layer and dissecting the core problems that a modern Database-as-a-Service (DBaaS) platform must solve at its foundation. DBaaS is a product of the cloud-native era—it encapsulates complex operations like database deployment, operations, and scaling into service interfaces, allowing developers to focus on business logic rather than infrastructure management. From AWS RDS to Google Cloud Spanner to PlanetScale, DBaaS has evolved from simple hosting to intelligent orchestration.

Core Components of the Infrastructure

Compute-Storage Separation Architecture

The first task in building a PlanetScale-like platform is establishing a clear compute-storage separation architecture. Traditional single-machine databases tightly couple compute and storage, making independent scaling difficult. Modern DBaaS platforms universally adopt a decoupled design—compute nodes handle query processing and transaction coordination, while the storage layer focuses on data persistence and replica management.

The compute-storage separation concept was first validated in data warehousing (e.g., Snowflake), then introduced to OLTP databases. AWS Aurora pioneered this pattern for relational databases by replacing MySQL's storage layer with a distributed log-structured storage engine. Unlike Aurora, PlanetScale's Vitess-based approach retains the standard MySQL storage engine (InnoDB) and achieves scaling through horizontal sharding rather than storage layer rewriting. This maintains higher compatibility with native MySQL, but it also means that storage scaling within a single shard is still limited by the capacity of the underlying block storage.

This separation brings significant advantages:

• Elastic scaling: Compute resources can scale elastically based on query load, while storage can be expanded independently
• Cost optimization: Avoids being forced to scale expensive compute resources just to add storage
• Fault recovery: Stateless compute nodes make fault recovery much simpler

Vitess as the Sharding Hub

The core technical foundation of PlanetScale is the open-source project Vitess. Vitess was originally developed by YouTube to solve the challenge of large-scale horizontal scaling for MySQL. In the infrastructure layer of a self-built platform, Vitess handles critical responsibilities including sharding routing, connection pool management, and query rewriting.

Vitess was born around 2010 at YouTube, when YouTube faced the bottleneck of a single MySQL instance being unable to handle massive video metadata queries. Vitess's design philosophy is to achieve transparent sharding through a middleware proxy layer without modifying the MySQL kernel. Its core components include: VTGate (stateless query routing layer), VTTablet (a sidecar proxy for each MySQL instance), Topology Service (metadata storage based on etcd or ZooKeeper), and VTCtld (cluster management control plane). Vitess became a CNCF incubating project in 2018, graduated as a top-level project in 2019, and is currently used at scale in production by companies like Slack, GitHub, and Square.

With VTGate as the query entry point, the application layer doesn't need to be aware of the underlying data shard distribution—Vitess automatically routes queries to the correct shard. VTGate implements MySQL protocol compatibility, meaning any MySQL client library can connect directly to VTGate without code modifications. Internally, VTGate maintains a set of VSchema (Vitess Schema) metadata that describes each table's sharding key (Vindex) and sharding strategy, enabling it to immediately determine which shards a SQL query involves upon receipt and dispatch sub-queries in parallel to the corresponding VTTablets. This abstraction layer allows databases to complete horizontal scaling without any business-level awareness.

High Availability and Disaster Recovery Design

Multi-Replica and Automatic Failover

The infrastructure layer must solve the single point of failure problem. The typical approach is to deploy a primary-replica replication topology for each shard, usually consisting of one primary node and multiple replica nodes. When the primary node fails, the system needs to complete automatic failover within seconds, promoting a replica to become the new primary.

This process involves the following key steps:

1. Leader election: Determining the best candidate replica
2. Replication state verification: Ensuring data consistency
3. Connection redirection: Switching traffic to the new primary

The most challenging problem in automatic failover is avoiding "split-brain"—where two nodes simultaneously believe they are the primary and accept writes, causing data divergence. Vitess implements fault detection and automatic switching through its Orchestrator component (which later evolved into VTOrc). VTOrc uses semi-synchronous replication to ensure at least one replica has the latest data, combined with GTID (Global Transaction ID) to precisely locate each replica's replication position. During failover, the system selects the replica with the GTID closest to the original primary as the new primary, and atomically updates routing information through the topology service, ensuring VTGate can complete traffic switching within milliseconds.

Vitess's built-in orchestration component monitors the health status of each node and triggers automatic switching when anomalies are detected, minimizing service interruption time.

Cross-Availability Zone Deployment Strategy

To handle datacenter-level failures, replicas typically need to be distributed across multiple Availability Zones (AZs). This way, even if an entire availability zone goes down, replicas in other zones can still guarantee service continuity.

Cross-AZ deployment directly touches on the CAP theorem constraints of distributed systems. The CAP theorem states that when a network partition occurs, a distributed system can only choose between Consistency and Availability. Network latency between availability zones within the same cloud region is typically 1-3 milliseconds, making the performance impact on synchronous replication relatively manageable. In practice, PlanetScale typically uses semi-synchronous replication—the primary node waits for at least one cross-AZ replica to acknowledge receipt of the transaction log before returning success to the client. This strikes a balance between zero RPO (Recovery Point Objective) and write latency. For cross-Region deployments, where network latency can reach tens of milliseconds, the approach typically degrades to asynchronous replication, with Vitess's Cell concept enabling proximity-based routing for read traffic.

This design places higher demands on network latency and consistency protocols, requiring trade-offs between availability and performance.

Implementation Principles of Zero-Downtime Schema Changes

One of PlanetScale's most popular features among developers is its support for online, lock-free schema changes. In traditional MySQL, executing ALTER TABLE on large tables often causes prolonged table locks, bringing business operations to a halt. This problem becomes particularly severe when table data reaches hundreds of millions of rows—a simple column addition might take hours or even days, during which the table is completely unwritable.

At the infrastructure layer, implementing this capability relies on the shadow table mechanism:

1. The system creates a shadow table with the new structure in the background
2. Data from the old table is gradually migrated to the new table
3. Incremental changes are captured through triggers or binlog
4. Finally, the table switch is completed atomically

Zero-downtime schema change technology has gone through multiple generations of evolution. The early pt-online-schema-change (from the Percona toolkit) implemented incremental synchronization through triggers, but triggers introduce additional write overhead and can easily trigger metadata lock contention under high concurrency. GitHub's gh-ost innovatively replaced triggers with binlog parsing, capturing incremental changes by simulating a replica reading the binlog, significantly reducing performance impact on the primary. Online DDL in PlanetScale/Vitess integrates both approaches, supporting gh-ost and pt-osc as backend strategies, and after Vitess 13 introduced the native VReplication approach, which leverages Vitess's own streaming replication infrastructure for data migration, eliminating external tool dependencies. The VReplication approach's advantage is that it natively understands Vitess's sharding topology and can coordinate schema change progress across shards.

This process transforms database changes from "high-risk operations tasks" into "routine development workflows," drastically reducing the risk of production environment changes. PlanetScale also built a Git-like branching workflow on top of this—developers can create database branches for schema experimentation, submit changes via Deploy Requests (similar to Pull Requests), merge to the production branch after review, all without requiring manual DBA intervention.

Practical Considerations for Self-Built Platforms

Operational Complexity Should Not Be Underestimated

Although building such an infrastructure from scratch is entirely feasible technically, one must clearly recognize the operational costs behind it. Monitoring, backups, recovery drills, capacity planning, and other aspects of distributed systems all require significant engineering investment. PlanetScale can charge for its managed service precisely because it shields users from this complexity.

Specifically, operating a Vitess cluster requires handling daily tasks including: high-availability maintenance of the Topology Service (etcd cluster), data migration orchestration during resharding, backup strategy formulation and recovery verification, slow query analysis and VSchema tuning, and rolling releases for MySQL version upgrades. Each task requires deep understanding of Vitess internals, with extremely limited margin for error in production environments. According to the PlanetScale team, they have dedicated SRE teams responsible for this operational automation work, with engineering investment rivaling product feature development.

The Value of the Open-Source Ecosystem

It's worth emphasizing that PlanetScale's core, Vitess, is fully open-source (Apache 2.0 license). This means capable teams can build similar platforms based on Vitess, maintaining complete control over the technology stack. For scenarios with high data sovereignty requirements or those seeking deep customization, the self-built route holds unique appeal.

A rich ecosystem has formed around Vitess: the Kubernetes Operator (vitess-operator) can automate Vitess cluster deployment and management on K8s; VReplication provides cross-cluster data synchronization capabilities; and Vitess offers complete monitoring solutions integrated with Prometheus and Grafana. Community activity continues to grow, with major contributors including engineers from PlanetScale, Slack, Nozzle, and other companies.

Conclusion

The value of the "Building PlanetScale from Scratch" proposition lies not in actually replicating a commercial product, but in understanding the engineering wisdom behind modern distributed databases through dissecting its infrastructure—compute-storage separation, Vitess-based shard routing, multi-replica high availability, and zero-downtime changes.

For every engineer interested in database technology, these design patterns represent universal assets for building scalable systems. Whether you ultimately choose a managed service or a self-built solution, deeply understanding the underlying principles will make your technical decisions more confident.