shards
A library that adds transparent sharding and partitioning to ETS tables in Erlang/Elixir for improved concurrency and scalability.
What is shards?
Shards is an Erlang/Elixir library that provides sharding and partitioning capabilities for ETS (Erlang Term Storage) tables. It addresses lock contention issues, particularly for write-heavy workloads, by distributing data across multiple shards to enable higher concurrency and scalability. The library is fully compatible with the standard ETS API, acting as a drop-in replacement that requires minimal code changes.
Erlang and Elixir developers working with ETS tables in high-concurrency, write-heavy applications who need to overcome lock contention bottlenecks. It is also suitable for developers building distributed caching systems or in-memory key/value stores that require scalable ETS backends.
Developers choose Shards because it offers transparent sharding without altering the existing ETS API, making integration seamless. Its unique selling point is enabling ETS tables to scale horizontally and handle higher concurrency by reducing write-lock contention, a common performance issue in standard ETS usage.
Overview
Partitioned ETS tables for Erlang and Elixir
Use Cases
Best For
- Scaling ETS tables for write-heavy workloads where lock contention is a bottleneck.
- Building distributed caching frameworks that require partitioned in-memory storage, such as Nebulex or Cacherl.
- Creating high-concurrency in-memory key/value stores, like KVX, that need sharded ETS backends.
- Implementing transparent sharding for existing ETS-based applications with minimal code changes due to full API compatibility.
- Enabling horizontal scalability for ETS tables in Erlang/Elixir systems to support increased throughput.
- Reducing performance degradation from write-locks in ETS tables within real-time or high-performance applications.
Not Ideal For
- Projects requiring distributed ETS tables out-of-the-box without additional libraries, as distributed support is handled separately by shards_dist.
- Applications with predominantly read-heavy workloads where ETS lock contention is not a bottleneck, as sharding adds overhead for read operations.
- Systems with strict memory constraints, since maintaining multiple ETS tables for shards increases memory usage compared to a single table.
- Teams relying on niche ETS features or optimizations that might not be fully supported in the sharded wrapper.
Pros & Cons
Pros
Fully compatible with the standard ETS API, acting as a drop-in replacement that requires minimal code changes, as highlighted in the README's key features.
Reduces write-lock contention in ETS tables, enabling higher concurrency for write-heavy workloads, which is the primary motivation stated in the project description.
Implements partitioning without altering the existing ETS API, allowing seamless integration and scaling out, as emphasized in the transparent sharding feature.
Used in popular projects like Nebulex and KVX, demonstrating real-world adoption and support for building distributed caching and key/value stores.
Cons
Distributed functionality is not built-in and requires an additional library (shards_dist), adding complexity for deployments needing distribution across nodes.
Sharding can slow down read operations due to the need to query multiple shards, whereas standard ETS offers faster single-table reads for non-contentious scenarios.
Optimizing sharding parameters (e.g., number of shards, hashing strategies) adds configuration overhead beyond the simplicity of standard ETS setup.
Frequently Asked Questions
Related Projects
ectoA toolkit for data mapping and language integrated query.
postgrexPostgreSQL driver for Elixir
eventstoreEvent store using PostgreSQL for persistence
redixFast, pipelined, resilient Redis driver for Elixir. 🛍
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.