Stage 03 · Scaling data
Hotspots & Dynamic Resharding
You picked a shard key that preserves locality — range, geography, tenant — because your queries needed it (Sharding and Partitioning). Now some of your shards are on fire and others are idle.
This is not a mistake you made. It’s the bill for locality, and it always arrives.
Why it’s unavoidable
Hash sharding distributes evenly because hash functions destroy any structure in your keys. Any key that keeps structure — so that related things sit together — also keeps whatever skew the real world has.
And the real world is extremely skewed:
Popularity. One celebrity account, one viral product, one enterprise customer who is 40% of your traffic.
Density. A city is 78 km² and does more business than a 90,000 km² rural region.
Sequential keys. Timestamps and auto-increment ids mean every new write goes to the last range. The other shards are archives.
Time. This one is the real problem. A stadium is empty at 3pm and saturated at 10pm. Load doesn’t just sit unevenly — it moves. Any static partitioning is wrong within hours of being right.
The fixes, cheapest first
Vary the granularity
Don’t use one partition size everywhere. Let a shard in a sparse region own a large slice of keyspace, and a shard in a dense region own a tiny one. Shards end up equal in load instead of equal in keyspace, which is what you wanted in the first place.
This is exactly what a quadtree does with depth — subdivide more finely where the data is dense — lifted from the index layer up to the shard layer.
Split and merge at runtime
Keep the live partition set in a small directory table. Splitting is then one transaction:
BEGIN;
DELETE FROM partitions WHERE id = :hot;
INSERT INTO partitions SELECT unnest(children_of(:hot));
COMMIT;The children hash to different machines, so load spreads immediately. Merging is the same statement reversed. This is the mechanism behind HBase region splits and DynamoDB partition splits.
Automate it
A control loop watching per-shard QPS or CPU, splitting above a threshold and merging below one — with hysteresis, so a shard hovering at the boundary doesn’t split and merge every five minutes.
Manual splitting is a perfectly honest starting answer. Automation is the obvious follow-up, and volunteering it is a good look.
Sub-shard one hot key
When a single key exceeds a machine, append a suffix: key#0 through key#n. Reads have to fan out across those sub-shards, which is fine because the fan-out is small and bounded.
Use this last. It breaks locality within that key, which is the thing you were protecting.
The part people forget: moving the data
Splitting the map is a one-line transaction. Moving the data is the hard part — unless you’ve arranged not to have to.
| Your data | How you migrate |
|---|---|
| Disposable, TTL’d, regenerated by its source | Don’t. Old rows expire, new writes land in the new place. The shard moves by attrition. |
| Durable and small | Copy it, then atomically flip the directory |
| Durable and large | Dual-write, backfill, verify, cut over — a project, not an operation |
That first row is worth designing for in advance. If you know a dataset will be resharded often, make its rows expire. You turn the hardest part of resharding into a no-op before you’ve written a line of migration code. → TTL as a Coordination Primitive
Staleness during a split
Your servers cache the directory, because reading it on every request would be absurd. So for one refresh interval after a split, some of them route to the old partition.
Whether that matters depends entirely on which code path is asking. The useful question is not “is this consistent?” but “what breaks if this particular caller reads a stale map?”
| Path | What a misroute costs | So it reads |
|---|---|---|
| High-volume disposable writes | one row that expires anyway | the cached copy |
| Low-volume correctness-critical reads | a wrong or empty answer | the leader, or a quorum |
The asymmetry is usually what makes the expensive answer affordable: adding a database round-trip to a path that runs 350 times a second costs nothing, while adding it to one that runs a million times a second would be fatal. → Replication and Read Consistency
In an interview
If you choose a locality-preserving key — range, geo, tenant — raise the hotspot yourself before the interviewer does. It shows you understand the cost of the key you picked.
The strong fix to describe is a small directory table of live partitions, so splitting a hot one is a single transaction. Then say how the data moves — and if it’s regenerated or TTL’d data, point out that it doesn’t need to.
The trap: proposing to fix a hotspot by changing the shard key. That trades a solvable balance problem for losing the locality you sharded for.
Seen in
- 09 - H3 and Geo Sharding — a fixed cell size puts an entire city on one shard
- 10 - Dynamic Resharding and the Shard Index — splitting as one transaction, with TTL absorbing the migration
Related
Sharding and Partitioning · TTL as a Coordination Primitive · Caching Strategies · Replication and Read Consistency