What is sharding in databases?

Sharding is a way of splitting a large database into several smaller parts (shards) that are stored on different servers.

The main idea is to distribute data and load across multiple servers instead of keeping everything on a single one.

What problem does it solve?

Imagine we are building a social network used by millions of people. The database stores user profiles, their posts, and other information. Over time, the number of users grows so much that a single server can no longer handle the load.

We can buy a more powerful server, but sooner or later we will hit its limits.

Instead of storing all users on one server, we can distribute them across several:

1. Shard 1 — users with user_id from 1 to 1,000,000.
2. Shard 2 — users with user_id from 1,000,001 to 2,000,000.
3. Shard 3 — users with user_id from 2,000,001 to 3,000,000.

Now, when we need to fetch information about the user with user_id = 1500000, the system determines that it lives on the second server and queries it directly.

This way, we can scale the database horizontally by adding new servers as the load grows.

How do we decide which server stores the data?

This is done with a Shard Key — a field the system uses to determine which shard a record belongs to.

There are several popular approaches:

1. Range-based Sharding — distribution by value ranges, for example user_id from 1 to 1,000,000.
2. Hash-based Sharding — distribution based on the hash of a particular field, for example user_id.
3. Geographic Sharding — distribution by geographic location, for example users from Europe and the US on different servers.

How does sharding differ from partitioning?

Both approaches involve splitting data into smaller parts.

But there is an important difference:

1. Partitioning — splits a single table into parts within one database server.
2. Sharding — distributes data across multiple independent database servers.

Does sharding have any drawbacks?

Yes. Although sharding lets you scale a system, it makes its architecture significantly more complex.

For example:

1. You need to determine which shard holds the data you need.
2. Queries that need data from several shards become more complicated.
3. Transactions across different shards are harder to implement.
4. Uneven data distribution can overload individual servers.