Sharding

What is Sharding?

Sharding is a scaling solution that works by partitioning the database into smaller pieces called shards.

The Long Definition

Sharding is a database partitioning technique that splits a network into shards. Each shard contains and processes its own unique subset of data. This happens simultaneously with other shards. That way, the overall speed and responsiveness of the database are improved.

Blockchain networks store and process transaction data. This makes them databases in their own right. As a result, sharding is seriously considered a solution to the scalability problem. Ethereum, through the Ethereum foundation, has been exploring and testing ways to implement sharding.

History of Sharding

The earliest concept of sharding was developed in the 80s. This was the System for Highly Available Replicated Data, coincidentally named SHARD. SHARD was an architecture that split the database across multiple servers. Its primary goal was to improve the availability of data and the overall fault tolerance of the system.

In the early 2000s, rapidly growing companies like Google and Amazon needed ways to handle large amounts of data. They would soon realize that dividing a database into smaller parts can be used to improve the scalability and performance of a system. Scalability refers to the ability of a network to handle a growing demand for system resources.

During this period, sharding techniques became much more sophisticated than the SHARD architecture. Google built its own Bigtable and Amazon had Dynamo. Both systems were built with sharding as a core component.

A few years later in 2007, Facebook was facing the same problem. Their database had grown to over 100 million users. In response, the company designed Cassandra, a database architecture that utilized sharding to ensure excellent system performance. Since then, sharding has become a widely used technique in database architecture.

Sharding and Blockchain

Blockchains are distributed systems that process and store transaction data. In recent years, blockchains like Ethereum and Bitcoin have become quite popular. The Ethereum protocol, for instance, has millions of active users.

This is a good thing in regard to the adoption of blockchain technology. However, the design of these protocols is currently unable to handle that many users. Ethereum can process a maximum of 15 to 20 transactions per second (TPS). Bitcoin, on the other hand, processes 7 TPS.

As a result, the protocols tend to struggle during periods of peak usage. Transactions take a long time to process. They also become expensive. In other words, the protocols are not scalable. They can not efficiently process a high number of transactions as demand arises.

Blockchain experts have processed several solutions to this problem. One of these is sharding. A blockchain is, after all, a database. Therefore, it can be theoretically improved by dividing it into shards. The Ethereum blockchain is at the forefront of exploring how this will increase its throughput and scalability.

How Does Sharding Work?

Sharding works by dividing a database into shards. Each shard contains its unique subset of data. To understand how this works in blockchain requires an understanding of how blockchains currently work.

A blockchain protocol consists of nodes. A node is a computer on the network. Being decentralized, each node independently runs the protocol’s client software. It also processes every single transaction on the network and stores the full history of the blockchain. The nodes then check against each other through a consensus mechanism.

With every node processing and recording every transaction, the network ensures security. However, this model comes at a cost. It slows down transaction processing speeds. This is a problem because blockchain proponents hope that the technology will someday be used to process millions of transactions.

And this is where sharding comes in. By splitting the blockchain into smaller parts called shard chains, the technique can reduce the workload on nodes. Each node no longer has to process all of the network’s workload. Instead, it handles a smaller partition, making transaction processing faster and more efficient.

However, each shard can still be shared among other shards. That way, it is accessible to every user in the network. This allows users to view the whole ledger of transactions.

Benefits of Sharding

The biggest benefit of sharding is improved scalability. There is another benefit– it allows a protocol to scale while maintaining decentralization. How?

As a protocol grows bigger, so does its existing database. Under current architecture, this increases the technical requirements. Nodes have a bigger load to handle. Therefore, they’ll have to be powerful computers that consume a lot of electricity.

This increases the cost of running a node, raising the barriers to entry. High barriers to entry mean only a select group of people can run nodes. And herein lies the danger. As fewer and fewer people run nodes, centralization starts to creep into the protocol.

This won’t happen with sharding. Nodes don’t have to store all of the network’s state data. Therefore, as the database scales, the system requirement will not increase. This will make running a node more accessible.

Ideally, sharding can reach a point where it’s possible to run a node on a mobile phone. Such a low barrier of entry will attract more node operators. This will lead to a more decentralized protocol.

Risks of Sharding

Security is the biggest risk of sharding. Shards are separate, with each processing its own data. However, each shard can be shared among other shards. This has created a security concern over cross-shard attacks. Hackers can attack one shard and use it to corrupt another shard(s). This can cause data loss, harming the whole network.

Ethereum and Sharding

Ethereum considers sharding as a potential solution to its scalability issues. The Merge, an upgrade that transitioned the protocol to a Proof of Stake consensus mechanism, set the stage for sharding. Ethereum is currently testing the technique with plans to implement it in the future.