samza-luwak
An experimental integration of Apache Samza and Luwak for scalable real-time document matching against large query sets.
What is samza-luwak?
Samza-Luwak is an experimental integration that combines Apache Samza's distributed stream processing capabilities with Luwak's optimized Lucene-based query matching. It enables scalable real-time matching of streaming documents against large sets of stored search queries, addressing use cases like media monitoring or alert systems. The project explores partitioning and pipeline architectures to handle high throughput and query volumes efficiently.
Developers and engineers building scalable real-time search or alerting systems, particularly those dealing with high-volume document streams and large query sets. It's also relevant for researchers or teams experimenting with distributed stream processing and search technologies.
It offers a distributed, fault-tolerant architecture for real-time document-query matching that can scale to handle both high document throughput and large numbers of complex queries. Unlike some existing solutions, it focuses on performance optimizations through partitioning strategies and leverages the strengths of Samza for stream processing.
Overview
Integration of Samza and Luwak
Use Cases
Best For
- Building scalable media monitoring or alerting systems
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
