Let's Build PlanetScale from Scratch: Infrastructure

I worked for a database tools company many years ago and was lucky enough to build a few database cloning tools before PlanetScale was cool.

Those tools were never destined to be as successful as PlanetScale, but they had their uses.

The idea behind those tools was simple: isolate the storage layer from compute and use the best storage technology for cloning the database files.

This idea came back to me a while ago while I was tweeting about an interview I had with an F1 team that was looking for a Rust dev with hands-on Ceph experience. Mentioning Ceph reminded me of a failed product attempt I was part of, and I jokingly told a friend that I was going to build Homescale, PlanetScale at home.

Homescale

I am building Homescale at github.com/homescale-dev/homescale.

Homescale creates writable database instances and point-in-time branches from immutable snapshots without copying the full database.

I borrowed Docker’s image and container model to represent the database state: A database image is an immutable starting point. A database container is a writable clone created from that image. A branch is another container created from the current state of an existing container.

The CLI I have in mind looks like this:

homescale image create --engine postgres postgres-base
homescale container create --image postgres-base dev-db
homescale container connect dev-db

homescale branch create --container dev-db feature-login
homescale container connect feature-login

The examples in this series use Postgres, but Homescale’s storage model is database agnostic. The image, container, and branch model can apply to any database engine that keeps its durable state on a filesystem backed by a block device and can be prepared for a recoverable snapshot. Postgres is the first engine I plan to support and gives me something concrete to use while building the storage and orchestration layers.

Engine-specific behavior belongs behind an adapter. That adapter initializes an image, starts the database process, exposes connection details, and prepares the database for a snapshot when necessary. Homescale handles the lifecycle around it: volumes, immutable states, writable clones, workloads, and lineage.

The word branch describes the relationship between the containers. After running these commands, dev-db and feature-login are both writable database containers. feature-login simply started from the state of dev-db at the moment the branch was created.

Underneath, the lineage looks like this:

flowchart LR
    Image[/postgres-base<br/>image/]
    Dev[dev-db<br/>writable]
    State[/read-only<br/>state/]
    Feature[feature-login<br/>writable]

    Image -->|clone| Dev
    Dev -->|snapshot| State
    State -->|clone| Feature

The image is already immutable, so Homescale can create dev-db from it directly. Branching from a writable container needs an intermediate state. Homescale first captures dev-db at a point in time, then creates feature-login from that state.

Creating feature-login cannot mean copying dev-db byte for byte. A 100 GB database would require another 100 GB of storage before the branch could start.

The branch should initially share its data with the captured state of dev-db. Only the data that changes afterward should require additional storage.

That requires branching to happen below the database process, in the storage layer.

Separating storage from compute

Separating database storage from compute is not a new idea. Standard Amazon RDS engines store database and log files on EBS volumes. Google Cloud SQL runs the database process in a VM with attached network block storage, using Persistent Disk or Hyperdisk depending on the machine series. The database still sees a normal block device, but its durable data is not tied to the compute host’s local disk.

Aurora, AlloyDB, and Azure SQL Hyperscale take this further. Their compute instances connect to shared or distributed storage systems designed specifically for the database. Compute instances can be replaced or scaled without creating a complete copy of the data for each instance.

Homescale is closer to the first model. A database process reads and writes a filesystem on what looks like a block device. I only need the storage behind that device to have a lifecycle independent of the process using it.

flowchart TD
    Database[Database process]
    Filesystem[Filesystem]
    Device[Block device]
    Storage[(Persistent storage)]

    Database -->|file reads and writes| Filesystem
    Filesystem -->|block I/O| Device
    Device --> Storage

That boundary lets Homescale create the storage before starting a database process and keep it after the process stops. More importantly, it lets branching happen in the storage layer without requiring the database engine to implement branching itself.

The two Postgres processes know nothing about their shared history. Each sees its own writable block device. Another supported engine would see the same storage boundary. The engine adapter would change, but the snapshot, clone, and lineage model would not.

Homescale still needs a way to create a writable block device that shares unchanged data with its parent.

COW is the secret sauce

Copy-on-write (COW) allows a writable clone to share unchanged data with the immutable state it was created from.

When Homescale creates dev-db, the new container initially reads its data from postgres-base. Writes made by dev-db are stored in the container without changing the image.

Creating feature-login repeats the same process. Homescale captures the current state of dev-db, then creates a new writable container from it.

The captured state, dev-db@feature-login, is read-only. It preserves the point where the two containers separated. dev-db can continue changing, while feature-login stores its own changes on top of that captured state.

When feature-login reads data it has not changed, the storage layer follows the chain back to its parent. This can continue through several generations until it finds the data. On the first write to shared data, the storage layer copies the relevant allocation unit into the writable container and applies the change there.

A branch of a 100 GB database therefore appears as a complete 100 GB database without requiring another 100 GB copy up front. Its initial cost is mostly metadata. Storage usage grows as dev-db and feature-login diverge, although a small database write may cause a larger storage allocation underneath.

This model also creates dependencies. feature-login relies on dev-db@feature-login for data it has not copied into its own container. Homescale must not remove that intermediate state while the branch still depends on it.

Homescale therefore needs persistent block devices, immutable snapshots, and writable COW clones.

Ceph

Ceph provides those operations through RBD. The database only sees an ordinary block device.

Ceph provides object, file, and block storage on top of the same underlying system. I only care about the block interface, called RADOS Block Device or RBD. An RBD image is a virtual block device that can be mapped to a machine, formatted with a filesystem, and mounted like a normal disk.

In the earlier storage stack, the persistent storage at the bottom is a Ceph RBD image. The database still reads and writes through the same filesystem and block device interface.

Underneath that RBD image, Ceph splits the device into objects and stores them across its object store. The Ceph documentation describes RBD images as block devices striped over objects in RADOS. The default object size is 4 MB, although it is configurable.

This is why a small database write does not necessarily create an equally small allocation in a clone. When a clone first writes to data it still shares with its parent, Ceph may need to copy the corresponding RADOS object into the child before applying the write.

An RBD snapshot is a read-only view of an image at a point in time. A clone is a writable RBD image that refers back to that snapshot for data it does not yet own. Ceph calls this snapshot layering.

Ceph can snapshot the volume without knowing which database is using it. Whether the database can safely start from that snapshot is a separate problem.

One direct RBD sequence for that operation is:

rbd snap create homescale/dev-db@feature-login
rbd snap protect homescale/dev-db@feature-login
rbd clone \
  homescale/dev-db@feature-login \
  homescale/feature-login

Homescale represents that sequence with one command, although Kubernetes and Ceph CSI will perform the backend storage operations:

homescale branch create --container dev-db feature-login

dev-db@feature-login is the read-only state between the two containers. Ceph will not delete it while it is protected. feature-login is a regular writable RBD image that can be mounted, snapshotted, and branched again.

RADOS, OSDs and RBDs

RBD sits on top of RADOS, which stores its objects through OSDs:

flowchart TB
    RBD[RBD image]
    Objects[RADOS objects]
    PG[Placement group]
    OSDs[Acting OSD set]
    Disks[(Storage devices)]
    Pool[Pool policy<br/>replicas and CRUSH]

    RBD -->|split into| Objects
    Objects -->|hash to| PG
    Pool -.->|controls placement| PG
    PG -->|maps to| OSDs
    OSDs -->|persist on| Disks

RADOS is Ceph’s distributed object store. RBD provides the block interface on top of it. When a database writes to an RBD-backed filesystem, RBD turns those block operations into reads and writes against RADOS objects.

Those objects live in a pool. A pool defines how its data is stored, including the number of replicas and the CRUSH rule used for placement.

Ceph does not map every object directly to a disk. It first maps the object to a placement group, then maps that placement group to one or more OSDs. This allows Ceph to move data when OSDs are added, removed, or fail without keeping a central table of every object’s location. The Ceph architecture documentation describes the same path: objects to placement groups, then placement groups to OSDs.

An OSD stores RADOS objects on a storage device and handles their reads, writes, and replication. Ceph can split one device between multiple OSDs, but the documented layout is one storage drive per OSD. That is the layout I will use.

Homescale will not talk to RADOS directly. Its storage boundary is RBD: create a volume, take a snapshot, and clone it. The operator will request those operations through Kubernetes PVCs and VolumeSnapshot resources.

Kubernetes as the control plane

I do not want Homescale to manage database processes, volume mounts, and network endpoints directly. It will describe the desired database and storage state through Kubernetes resources, then rely on controllers to make that state real.

The CLI talks to the Homescale API. Homescale creates database workloads, PVCs, and VolumeSnapshot resources. Ceph CSI translates the storage resources into RBD operations, while Rook operates the Ceph cluster itself.

flowchart TB
    CLI[Homescale CLI] --> Control

    subgraph Kubernetes
        Control["Homescale"]
        Databases
        StorageAPI["Storage API<br/>PVCs, snapshots, CSI"]
        Ceph["Ceph storage<br/>RBD and OSDs"]

        Control -->|reconciles| Databases
        Control -->|creates| StorageAPI
        StorageAPI --> Ceph
        Databases --> Ceph
    end

    Ceph --> Disks[(Storage devices)]

Kubernetes controllers reconcile these resources toward their desired state. If a database pod disappears, its controller replaces it. If a PVC requests dynamic provisioning through a compatible StorageClass, the CSI driver provisions its storage. Homescale can use that machinery instead of implementing its own process supervision and storage attachment logic.

The application-specific resources and the controllers that reconcile them belong to the next part. The important boundary here is the Kubernetes storage API: Homescale requests PVCs and VolumeSnapshot resources, and Ceph CSI translates those requests into RBD operations.

Local and replicated deployments

For now, Homescale will run locally on macOS in a single-node Kubernetes cluster, probably using Colima or Lima. This local profile has one Kubernetes node, one Ceph OSD, and one storage device.

I may also deploy Homescale to the private Talos cluster I run on Hetzner Cloud. The Terraform for that cluster already supports separate worker pools, so I can add a Ceph node pool without putting OSDs beside the existing application workloads.

A replicated pool with size: 3 and failureDomain: host can place each object’s replicas across those nodes. If a disk or node fails, Ceph can continue serving the volume from the surviving replicas.

Locally, there is one copy of the data. On Hetzner, I can run the same Homescale operator against a replicated Ceph pool spread across the storage nodes.

Rook connects Kubernetes to Ceph

A CephCluster resource tells Rook which Ceph version to run, where it can place monitors and managers, and which devices it can consume for OSDs. A CephBlockPool describes the RBD pool and its replication and failure-domain settings. The Rook operator watches those resources and creates or updates the corresponding Ceph daemons and configuration.

The split looks like this:

flowchart TB
    Pod[Database pod] -->|mounts| PVC[PVC]
    Snapshot[VolumeSnapshot] -->|source for| NewPVC[New PVC]
    PVC --> CSI[Ceph CSI]
    NewPVC --> CSI
    CSI -->|provisions in| Ceph[Ceph cluster]

    CRs[Ceph resources] --> Rook[Rook operator]
    Rook -->|reconciles| Ceph

Rook configures the Ceph CSI drivers used to provision and mount storage. Homescale can use PVCs and VolumeSnapshot resources without running rbd commands or carrying Ceph credentials itself. The Rook snapshot documentation describes the same restore flow: create a VolumeSnapshot from a PVC, then use that snapshot as the data source for another PVC.

This keeps Homescale independent of the Ceph deployment shape. The local cluster can use one OSD and a pool with one copy. The Hetzner cluster can use three OSDs and replication across hosts. As long as both expose compatible StorageClass and VolumeSnapshotClass resources, the Homescale operator follows the same reconciliation flow.

What is next?

The next part will cover the Homescale services: the API used by the CLI, the controllers that manage database resources, and the Postgres adapter.

I will start with the image and container path. An image request needs to initialize a Postgres volume and capture it as an immutable VolumeSnapshot. A container request needs to clone that snapshot into a new PVC, start Postgres, wait for it to become ready, and return connection details.

Once that path works, I can add branching from a running container. That service flow must prepare the source database for a snapshot, create the intermediate state, record its lineage, and reconcile another writable container from it.