A Go library for building data processing workflows and pipelines with functional operations, cycles, and fan-out capabilities.
Machine is a Go library for creating data workflows and pipelines. It provides a functional interface to build data processing streams that can handle transformations, conditional logic, cycles, and fan-out operations. It solves the problem of orchestrating complex data flows in a maintainable and composable way.
Go developers building data processing applications, ETL pipelines, or stream processing systems who need a lightweight, functional workflow library.
Developers choose Machine for its functional design, support for complex patterns like cycles and recursion, and built-in OpenTelemetry integration, offering a robust alternative to rolling custom pipeline logic.
Machine is a workflow/pipeline library for processing data
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Uses Monad and Filter types for transformations and branching, enabling clear, maintainable pipeline logic as described in the philosophy of composable design.
Supports cycles (While), recursion (Recurse, Memoize), and conditional splits (If, Or, And), allowing for retry mechanisms and complex data flows as highlighted in the features.
Integrates OpenTelemetry via a slog wrapper, providing metrics and traces out-of-the-box for monitoring data processing, as shown in the telemetry integration section.
Offers options like FIFO order, buffer sizing, and graceful flushing via Option helpers, giving fine-grained control over processing behavior.
Includes Distribute method with Edge interface for pub/sub or parallel processing patterns, enabling scalable data distribution as mentioned in the fan-out feature.
The Transform method cannot be used inside loops due to type safety issues, restricting type changes in cyclical workflows, as admitted in the code comments.
Only basic Edge implementations for Pub/Sub and HTTP are provided, requiring custom development for integration with other data sources or sinks, indicating a smaller ecosystem.
Requires understanding of functional programming concepts like Monads and Y Combinators, which might be challenging for developers not versed in these paradigms, despite the detailed README.