A lightweight, thread-safe, append-only in-memory log data structure inspired by Kafka, for Go applications.
memlog is an in-memory, append-only log data structure for Go, designed as a simple building block for streaming data scenarios. It provides an ordered, immutable record stream with concurrent read/write access, bridging legacy systems or APIs to modern streaming interfaces without the complexity of full-scale messaging platforms.
Go developers building applications that need to expose a durable data source (like a legacy system, REST API, or database) as a modern streaming API, or those requiring a lightweight, thread-safe, in-memory ordered cache for real-time data.
Developers choose memlog for its minimal API using Go standard library primitives, thread-safe operations without data races, and its role as a lightweight abstraction that avoids the over-engineering of full messaging platforms like Kafka or Redis Streams for simpler use cases.
A Kafka log inspired in-memory and append-only data structure
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Supports concurrent reads and writes without data races, which is essential for real-time data streaming in Go applications, as highlighted in the thread-safe operations feature.
Uses Go standard library primitives for ease of integration, reducing the learning curve and avoiding over-engineering, as emphasized in the motivation section.
Offers Log.Stream() for continuous consumption instead of manual polling, simplifying client code for asynchronous, in-order data access, as demonstrated in the streaming API examples.
Divides the log into active and history segments with size-based purging, allowing control over memory usage and data retention, as detailed in the configurable segments feature.
As an in-memory log, all data is lost on process crash, requiring external checkpointing mechanisms and a durable source, which the README explicitly admits as a limitation.
Performance and capacity are limited by available RAM, and the segment-based design may not optimize for growable slices, as the author notes uncertainty in benchmark discussions.
Lacks built-in persistence, replication, or advanced features like those in Kafka or Redis Streams, making it unsuitable for complex distributed systems without additional tooling.