A library that adds transparent sharding and partitioning to ETS tables in Erlang/Elixir for improved concurrency and scalability.
Shards is an Erlang/Elixir library that provides sharding and partitioning capabilities for ETS (Erlang Term Storage) tables. It addresses lock contention issues, particularly for write-heavy workloads, by distributing data across multiple shards to enable higher concurrency and scalability. The library is fully compatible with the standard ETS API, acting as a drop-in replacement that requires minimal code changes.
Erlang and Elixir developers working with ETS tables in high-concurrency, write-heavy applications who need to overcome lock contention bottlenecks. It is also suitable for developers building distributed caching systems or in-memory key/value stores that require scalable ETS backends.
Developers choose Shards because it offers transparent sharding without altering the existing ETS API, making integration seamless. Its unique selling point is enabling ETS tables to scale horizontally and handle higher concurrency by reducing write-lock contention, a common performance issue in standard ETS usage.
Partitioned ETS tables for Erlang and Elixir
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Fully compatible with the standard ETS API, acting as a drop-in replacement that requires minimal code changes, as highlighted in the README's key features.
Reduces write-lock contention in ETS tables, enabling higher concurrency for write-heavy workloads, which is the primary motivation stated in the project description.
Implements partitioning without altering the existing ETS API, allowing seamless integration and scaling out, as emphasized in the transparent sharding feature.
Used in popular projects like Nebulex and KVX, demonstrating real-world adoption and support for building distributed caching and key/value stores.
Distributed functionality is not built-in and requires an additional library (shards_dist), adding complexity for deployments needing distribution across nodes.
Sharding can slow down read operations due to the need to query multiple shards, whereas standard ETS offers faster single-table reads for non-contentious scenarios.
Optimizing sharding parameters (e.g., number of shards, hashing strategies) adds configuration overhead beyond the simplicity of standard ETS setup.