Learning database engineering from open source
Read a database from the disk upwards. Find how a row or a key is laid out in a page or an SST file before you read anything about queries, because the storage layout explains almost every decision made above it — why the index looks like that, why the write path batches, why that operation is fast and that other one is not. After the layout, read the write path, and then crash recovery, which is where the real design is.
Why reading this kind of code is worth the time
- Database code is the clearest available demonstration that data structures have consequences. A B-tree and an LSM tree lead to entirely different systems, and the difference is visible in every layer above them.
- Crash recovery is the hardest part of any storage system and is almost never taught. Reading a write-ahead log implementation is the fastest way to understand durability as an engineering problem rather than a checkbox.
- Query planning is a beautiful, self-contained optimisation problem that you can read and understand in an afternoon, and it changes how you write SQL forever.
A reading sequence
Ordered by difficulty, not importance. Each stage assumes the one before it.
Start
An embedded key-value store — small, single-node, no network layer.
Look for: The on-disk format. Find the code that serialises a record and the code that reads it back.
Next
The write path of that same store.
Look for: What is buffered, when it is flushed, and what is durable at each point. Follow one `put` all the way to `fsync`.
Then
Crash recovery — the write-ahead log and the code that replays it.
Look for: What the recovery code assumes about a half-written record. That assumption is the durability guarantee.
Deeper
A query planner, in a database or an embedded engine.
Look for: The cost model. Find the constants, and ask what they assume about the hardware.
Concepts you will keep meeting
- On-disk layout and pages
- B-trees and LSM trees
- Write-ahead logging
- Crash recovery
- Indexing
- Query planning and cost models
- MVCC and isolation levels
Questions to ask of any codebase in this area
Reading with a question in mind is the difference between studying code and scrolling through it.
- 1.How is a single record laid out on disk, byte by byte?
- 2.Follow one write to the point where it survives a power cut. How many steps?
- 3.What does recovery do with a record that was half written?
- 4.What does the cost model assume, and would those assumptions hold on an SSD?
Common questions
Get a matching repository every day, with a reading plan
Repo Dive does what this guide describes, daily and automatically: it finds a repository matching what you want to learn, explains why it chose that one, names the files to read first and asks three questions about it. Free, and it tracks what you have already studied so the picks keep moving forward.
Start with GitHubRelated guides
- Learning backend architecture from production code
How to read backend codebases for their architecture: request flow, boundaries, caching, background work, and the questions that reveal how a service is really put together.
- Learning distributed systems from real implementations
Which distributed systems codebases to read in which order, and how to find the failure-handling code that the papers leave out.
- Learning Rust by reading real Rust
Which kinds of Rust projects to read in which order, what ownership and error handling look like in production code, and the questions to ask of any Rust codebase.