A Python stream processing library that ports Kafka Streams concepts to Python, enabling scalable event processing and in-memory durable key/value stores.
Faust is a Python stream processing library that ports the ideas from Kafka Streams to Python. It enables developers to build scalable event processing applications and in-memory durable key/value stores as a library, leveraging asyncio and static typing. It is designed to handle real-time data pipelines processing billions of events daily.
Python developers and data engineers building distributed systems, real-time data pipelines, or event-driven applications that require stream processing with high availability and scalability.
Developers choose Faust because it allows them to use familiar Python syntax and libraries for stream processing without learning a new DSL, while providing Kafka Streams-like reliability, state management, and horizontal scalability.
Python Stream Processing. A Faust fork
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Uses plain Python without a DSL, enabling seamless integration with libraries like NumPy, Django, and Flask for stream processing, as highlighted in the README.
Automatically recovers from node failures with standby nodes and Kafka changelog replication, ensuring fault tolerance in distributed systems.
Leverages RocksDB for fast, durable in-memory key/value storage, capable of processing tens of thousands of events per second per core, as noted in the performance section.
Built with mypy compatibility for static type checks, promoting robust and maintainable code in stream applications.
Requires a Kafka cluster for operation, adding significant infrastructure complexity and maintenance overhead compared to standalone stream processors.
Heavily relies on asyncio and async/await, which can be challenging for developers unfamiliar with asynchronous Python, despite the README's reassurance.
As a fork due to release issues in the original project, there may be concerns about long-term community support, stability, and alignment with upstream developments.