How does Roshi handle network partitions?

Roshi uses CRDT semantics and replication across farms to remain available during partitions, with read-repair mechanisms to converge data once connectivity is restored, prioritizing availability over consistency.

Roshi vs Redis for timestamped event storage?

Roshi builds on Redis sorted sets but adds distributed sharding, replication, and CRDT-based consistency, making it better for large-scale, fault-tolerant event indexing, whereas vanilla Redis is more general-purpose but lacks built-in distribution.

How to set up read strategies in Roshi?

Configure read strategies via the farm package, which allows choices like querying multiple clusters for consistency; details are in the package documentation, affecting how read-repair is triggered.

What happens if a Redis instance fails in Roshi?

Roshi can tolerate failures through replication, but recovering lost instances requires manual processes like keyspace walking or cloning, with risks of data loss during the gap, as noted in the fault tolerance section.

Can Roshi be used for real-time analytics?

It's optimized for high-throughput event indexing and querying, such as activity streams, but lacks built-in aggregation or complex querying, so it's best as a backend index rather than a full analytics engine.

How does the read-repair mechanism work?

During reads, if discrepancies are detected across replicas, Roshi unions the sets, returns the result, and triggers background repairs to synchronize data lazily, ensuring eventual consistency.

Open-Awesome

roshi

BSD-2-ClauseGo

A distributed, eventually consistent CRDT set implementation for timestamped events, built on Redis.

GitHub

3.2k stars155 forks0 contributors

What is roshi?

Roshi is a distributed system that implements a Last-Writer-Wins element set CRDT for storing timestamped events. It provides a scalable, eventually consistent index for time-series data, designed to handle large datasets across multiple Redis instances without requiring consensus protocols. It solves the problem of high-performance event storage in critical request paths, such as activity streams.

Target Audience

Developers and architects building large-scale, fault-tolerant systems that need to index and query timestamped events, particularly those requiring high availability and partition tolerance.

Value Proposition

Developers choose Roshi for its simplicity, leveraging CRDT semantics to avoid complex consensus algorithms, and its performance as a distributed layer on Redis. Its stateless design and built-in read-repair make it robust for scalable event indexing.

Overview

Roshi is a large-scale CRDT set implementation for timestamped events.

Use Cases

Best For

Building high-performance activity streams or timelines
Indexing timestamped events in distributed applications
Implementing eventually consistent sets without consensus overhead
Storing time-series data that requires horizontal scaling
Creating fault-tolerant event storage layers
Handling large-scale event data where availability is critical

Not Ideal For

Systems requiring strong consistency or ACID compliance for critical data
Applications needing dynamic resharding or elastic scaling during runtime
Projects where data loss is unacceptable, as Roshi does not guarantee perfect durability
Teams looking for built-in authentication, authorization, or data validation

Pros & Cons

Pros

CRDT-Based Eventual Consistency

Uses LWW-element-set CRDT semantics to ensure data convergence without complex consensus protocols, making operations associative, commutative, and idempotent as described in the theory section.

Distributed Fault Tolerance

Shards data across Redis clusters and replicates it in farms, providing partition tolerance and high availability with stateless, 12-factor compliant servers for easy horizontal scaling.

High-Performance Event Index

Designed to sit in critical request paths, offering fast Insert/Delete/Select API for timestamped events, optimized for scalability and low latency in use cases like activity streams.

Lazy Read-Repair Mechanism

Detects and repairs inconsistencies across replicas during read operations, promoting data convergence without active consensus, as explained in the replication and farm packages.

Cons

Static Sharding Limits Flexibility

Does not support elasticity; sharding configuration is static and cannot be changed during operations, hindering dynamic scalability adjustments.

No Guaranteed Data Durability

Explicitly admits potential data loss, especially on unread keys, and warns against using it as a source of truth, making it risky for critical storage.

Client Timestamp Dependency

Relies on correct client timestamps; incorrect timestamps can cause data to never appear or always override other values, leading to consistency issues.

Frequently Asked Questions

Related Projects

influxdb

Scalable datastore for metrics, events, and real-time analytics

An embedded key/value database for Go.

Stars14,631

Forks1,544

Last commit8 years ago

groupcache

groupcache is a caching and cache-filling library, intended as a replacement for memcached in many cases.

Stars13,339

Forks1,395

Last commit1 year ago

go-cache

An in-memory key:value store/cache (similar to Memcached) library for Go, suitable for single-machine applications.

Stars8,816

Forks905

Last commit2 years ago

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

Submit a project Star on GitHub