What Is Database Sharding?

Have you ever found yourself constantly hitting refresh on a page hoping that it will sense your impatience and increase its response time?

It's not a very encouraging feeling. But not to worry!
In this article, we will look at how to improve database performance via DATABASE SHARDING.

Don’t worry if you’ve never heard of it — we’ll look at everything you need to know about sharding and how it can help you scale your databases and improve performance.

In this article, we will look at:

• Why database scalability is a problem?
• What is database sharding?
• How does it work?
• What is a sharding key?
• How to choose an appropriate sharding key
• Why we should adopt sharding (benefits)
• What issues can arise from sharding? (challenges)
• Final advice for using sharding as a solution to database scalability

Why is Database Scalability a Problem?

Our database is scalable if it can handle increasing amounts of data and user traffic without experiencing performance issues or downtime.
This is not an easy task🥲.

Most of the applications we interact with process insane amounts of data per minute. As user traffic for an application increases, so does the data volume that its database has to handle.

Eventually, we get to a point when the database reaches its limits in terms of capacity and performance, and can no longer readily accommodate the growing needs of our users.

That's when we can start experiencing increased response times, data inconsistencies, and even system crashes. In some cases, a database may become completely unusable due to overload or capacity constraints.
Sounds scary right?

Over the years, various solutions have been developed, including partitioning, caching, and load balancing. These solutions are designed to distribute the load and improve the performance of our database, enabling it to handle increasing amounts of data and user traffic.

Let’s look at one type of solution — Database sharding.

What is Sharding?

Database Sharding is a design strategy where we break up a large database into smaller, more manageable pieces called shards. Each shard contains a subset of data and is stored on a separate server or set of servers.

The main idea is to reduce the amount of data that needs to be processed on a single server. When our database becomes too large to fit on a single server, it will become slow leading to degraded performance and user experience.

Sharding allows our database to scale horizontally by distributing the load across multiple servers, instead of relying on a single server to handle all the data.

How Does Database Sharding Work?

Next, let's have a step-by-step view of how database sharding works.

Step one: The first step in sharding is to partition our database into smaller pieces (shards). We can do this by selecting a partition key that evenly divides the data into subsets. Examples of partition keys include user ID, age, location, or date range.

Step two: The next step is distribution. We basically distribute each shard to a separate server or set of servers. This can be done manually or using an automated process, depending on the complexity of our database and the resources we have available.

Step Three: When a user requests data from our database, the request is first routed to a shard based on the partition key. The shard processes the request and returns the results to the user.

What if the data being requested needs to come from multiple shards🤔?

In this case, a query router is used to route the query to the appropriate shards, and then combine the results into a single response.

Cool, now let's really understand the concept of a sharding key.

What is a Sharding Key?

A sharding key is a field or combination of fields that we can use to divide the data in our database into smaller, more manageable pieces.

Let's look at a hypothetical example where we are building a voting management system for 7 thousand of accredited voters.

Our partition key in this case could be the voter ids. We could have 7 shards separated such that each shard serves a thousand voters i.e
- shard one serves users with voter ids 1–1,000
- shard two serves users with voter ids 1,001–2,000
- shard three serves users with voter ids 2,001–3,000 etc.

We need to pay close attention when we are selecting a sharding key because it will have a serious impact on the performance and scalability of our sharded database.
Let’s look at some factors we should consider when choosing a sharding key.

Factors we should Consider when Selecting a Sharding Key

Here are four important factors we should consider when selecting a sharding key.

• Our sharding key should have high cardinality. This simply means that it should have many unique values. In our voter's example, each voter would have their own unique ID that distinguishes them from every other voter in the system. A high-cardinality key ensures that our data is distributed evenly across shards, preventing a situation where a particular area of our database is heavily accessed or updated by many users at the same time.
• Our sharding key should distribute the data evenly across shards. We must ensure that there are no shards that have significantly more data than other shards.
  For example, imagine our voter’s database is sharded based on the first letter of the voter’s last name. If we have more voters with last names that start with the letter “A” than any other letter, our shard that contains voters with last names starting with “A” will end up with a much larger volume of data than other shards.
  This will cause that shard to become overloaded and will eventually lead to performance issues.
• Our sharding key should align with the most common queries required by the business logic of our application. The idea here is to minimize the need to check multiple shards for a single result (cross-shard querying).
• A sharding key is delicate and we must have one that will remain stable as our data grows. Changing it is not easy and may require a database migration. If this is not done properly, it could lead to data loss. So we have to be intentional and choose carefully.

Note: this is not an exhaustive list.

Selecting a sharding key sounds exhausting, so why should we consider database sharding?

What are the Advantages of Sharding For Database Scalability

Database sharding is indeed a very powerful technique that we can use to scale large databases.

• For starters, it solves the ancient problem of database scalability. Sharding allows us to horizontally scale our database. This means that as our data size increases, we can add more servers to handle the increased load and provide linear scalability.
• Another very important point comes up when we talk about performance. By distributing our data across many servers, we can drastically improve query performance by reducing the load on any one server and minimizing the need for cross-shard queries.
• That's not all! Sharding can also help us save money. How?
  When we distribute the load on our servers, we also indirectly reduce the need for expensive hardware upgrades or database licenses — lowering overall costs.
• Very importantly, with sharding, we remove the existence of a single point of failure. If one of our servers fails, users can still access data from other servers, this can be called Partition tolerance in the CAP theorem.
• Finally, it provides us with flexibility in database design. We can dynamically change a database schema, and add or remove shards as needed.
  At this point, I sound like a sharding evengelist😅

Just like many beautiful things in this world, sharding comes with its own problems.

Potential Issues That Could Arise From Sharding

Let's look at some issues that database sharding might throw our way.

The first issue which we have seen is how difficult it can be to select a suitable sharding key. What we use as the sharding key will affect the effectiveness of our resultant system.

If our sharding key is not evenly distributed or does not align with the query patterns of our app, some of our shards may be heavily loaded while others are underutilized. This can lead to uneven performance and potentially create bottlenecks.

Another major issue is the additional complexity that sharding brings to our database architecture. We need to manage each shard separately and writing queries that span multiple shards can be complex to optimize. This can make our database infrastructure more difficult to maintain.

Like other distributed systems, a sharded system may experience issues relating to data consistency. Queries that span multiple shards can return inconsistent results if the data is not synchronized correctly.

To ensure consistency, updates to the database are either propagated to all shards in real-time or batched and distributed to all shards periodically.

While it’s true that sharding can reduce cost by removing the need to scale our hardware, we should have it in mind that each shard requires its own server and resources, which can increase hardware and maintenance costs.

Also, it can introduce additional failure modes and recovery processes. If one of our shards fails, depending on our system architecture, it may impact the availability of the entire system.

For us to recover the failed shard, we have to attempt to restore the system without impacting any other shards. This can be a complex process.

Sharding is indeed a two-edged sword but that should not discourage you!

Final thoughts🙂

It's obvious that sharding is a powerful tool for scaling large databases and improving system performance. However, we have seen that it can also introduce new challenges, such as data consistency and increased complexity in our application development and maintenance.

When we are considering sharding as a solution to scale our database, we must plan carefully and consider various factors. It is important for us to choose an appropriate sharding key, implement a data consistency management strategy, and consider the overall impact on our application’s development and maintenance.

If the data system to be sharded belongs to an organization, then the Organization should evaluate its needs and resources carefully before embarking on the sharding journey.

Will you use database sharding? Drop your thoughts in the comment section.

Thanks for reading. Stay Jiggy!