A high-performance, disk-backed queue library using memory-mapped files for fast, persistent, and thread-safe data processing.
Big Queue is a Java-based persistent queue library that uses memory-mapped files to provide fast, disk-backed message storage. It solves the problem of handling large volumes of data with high throughput while ensuring crash resistance and minimal memory usage. The library is designed for applications that need reliable, thread-safe queue operations without the overhead of traditional database or messaging systems.
Java developers building high-throughput data processing systems, log collectors, or messaging backends that require persistent, scalable queue semantics. It's particularly useful for applications dealing with big data on commodity hardware.
Developers choose Big Queue for its exceptional performance close to direct memory access, disk-based persistence that scales with available storage, and lightweight design without external dependencies. Its memory-mapped file approach provides reliability through OS-managed persistence while maintaining simplicity.
A big, fast and persistent queue based on memory mapped file.
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Offers near O(1) memory access for enqueue and dequeue, with throughput up to 333MB/s in sequential tests, as highlighted in performance reports.
Queue size is limited only by available disk space, and data is OS-persisted via memory-mapped files, ensuring crash resistance without memory constraints.
Uses automatic paging and swapping to keep only recently accessed data in memory, minimizing footprint for large queues as described in the feature list.
Supports concurrent enqueue and dequeue by multiple threads without data corruption, a key feature emphasized in the documentation.
The library is standalone; for distributed messaging, users must refer to Luxun, indicating it's not designed for clustered environments and lacks features like replication.
Last stable release was in 2013, so it may lack support for newer Java versions, modern best practices, and could have unpatched issues or security vulnerabilities.
Requires log4j for logging, adding an extra dependency that might conflict with or be redundant in projects using other logging frameworks like SLF4J or Logback.