Database scaling and sharding
Play this lessonThis lesson comes with an interactive animation you can break, a live-code challenge and a chapter quiz.There are two ways to scale a database:
- Vertical: a more powerful machine. Amazon RDS offers servers with 24 TB of RAM, and in 2013 Stack Overflow ran 10 million monthly visitors on one master database. But there are hardware limits, a SPOF risk, and high cost.
- Horizontal = sharding: split a large database into smaller shards. Every shard has the same schema but holds different data.
A sharding key (partition key) decides where data goes, for example shard = user_id % 4. Choose a key that spreads data evenly.
New challenges
- Resharding: needed when a shard fills up or data is uneven (shard exhaustion). With
% N, changing N moves almost everything. Consistent hashing (Chapter 5) fixes this. - Celebrity problem (hotspot key): if Katy Perry, Justin Bieber and Lady Gaga all land on one shard, it gets overloaded. You may need a dedicated shard per celebrity, or even further partitioning.
- Joins and de-normalization: joins across shards are hard. A common workaround is to de-normalize the data so queries can be answered from a single table.
Key takeaways
- Shard = same schema, different rows.
- A good sharding key spreads data evenly.
- Watch for resharding, hotspots and cross-shard joins.