How Sharding Works

What is sharding?

An illustrated example of vertical and horizontal partitioning
# Example of vertical partitioning
fetch_user_data(user_id) -> db[“USER”].fetch(user_id)
fetch_photo(photo_id) -> db[“PHOTO”].fetch(photo_id)
# Example of horizontal partitioning
fetch_user_data(user_id) -> user_db[user_id % 2].fetch(user_id)

Before you start: you may not need to shard!

Driving Principles

Common Definitions

Case 1 — Algorithmic Sharding

An algorithmically sharded database, with a simple sharding function

Case 2— Dynamic Sharding

A dynamic sharding scheme using range based partitioning.

Case 3 — Entity Groups

Entity Groups partitions all related tables together

Case 4 — Hierarchical keys &
Column-Oriented Databases

Column-oriented databases partition its data by row keys.

Understanding the pitfalls

Illustration of a hotspot at the end of partition range even after numerous shard splits.

Concluding Remarks

--

--

Read, Write, Execute

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store