A Python toolkit for developing, testing, and managing Apache Storm streaming data processing topologies.
Petrel is a Python toolkit for creating and managing Apache Storm streaming data processing topologies. It allows developers to write, test, and deploy Storm topologies entirely in Python, eliminating the need for Java or Clojure. The tool handles packaging, dependency management, and provides utilities for debugging and monitoring topologies in both local and cluster environments.
Python developers and data engineers building real-time stream processing applications with Apache Storm who want to avoid Java/Clojure development overhead.
Petrel offers a pure Python development experience for Storm with automated environment setup, built-in testing capabilities, and enhanced logging that simplifies the entire topology lifecycle from development to production monitoring.
Tools for writing, submitting, debugging, and monitoring Storm topologies in pure Python
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Allows writing complete Storm spouts and bolts in Python without Java or Clojure, as shown in the wordcount example, making Storm accessible to Python developers.
Creates virtual environments and handles Python package installation per topology via setup.sh scripts, simplifying deployment across Storm clusters.
Provides a mock module for unit testing individual components or simple topology chains without a Storm cluster, improving development efficiency.
Sets up dedicated logging for each component and captures stack traces on errors, with logs stored separately from Storm's output for easier troubleshooting.
Requires installation of system packages like libyaml and thrift, plus tools like Maven for source builds, adding setup overhead compared to pure Python solutions.
Only supports Python 2.7 or 3.5, not compatible with Python 2.6 which Storm's base package supports, potentially causing version conflicts in legacy environments.
Exclusively designed for Storm with no flexibility for other streaming platforms, creating vendor lock-in and limiting future migration options.