Peloton
A unified resource scheduler for co-scheduling batch, stateless, and stateful workloads in a single cluster to maximize resource utilization.
What is Peloton?
Peloton is a unified resource scheduler that co-schedules mixed workloads—such as batch, stateless, and stateful jobs—in a single compute cluster. It solves the problem of low resource utilization in large-scale environments by enabling elastic resource sharing, overcommitment, and workload preemption. Designed for web-scale operations, it handles millions of containers and tens of thousands of nodes efficiently.
Platform engineers, SREs, and infrastructure teams at web-scale companies managing large, heterogeneous clusters with diverse workload types. It is particularly relevant for organizations running big data and machine learning workloads alongside traditional services.
Developers choose Peloton for its ability to maximize cluster utilization through advanced features like hierarchical resource pools, GPU/gang scheduling, and cloud-agnostic deployment. Its scalability and support for mixed workloads make it a robust alternative to simpler schedulers in complex, high-demand environments.
Overview
Unified Resource Scheduler to co-schedule mixed types of workloads such as batch, stateless and stateful jobs in a single cluster for better resource utilization.
Use Cases
Best For
- Managing large-scale clusters with millions of containers
- Co-scheduling batch, stateless, and stateful workloads together
- Improving resource utilization in heterogeneous compute environments
- Running GPU-intensive machine learning jobs like TensorFlow with gang scheduling
- Implementing hierarchical resource sharing across multiple teams
- Deploying a cloud-agnostic scheduler for on-premise or cloud datacenters
Not Ideal For
- Small to medium-sized clusters with under a thousand nodes, where complexity outweighs benefits
- Teams already standardized on Kubernetes seeking seamless container orchestration without Mesos integration
- Organizations lacking dedicated infrastructure teams to manage dependencies like Zookeeper and Cassandra
- Projects requiring quick, out-of-the-box scheduling with minimal configuration and setup time
Pros & Cons
Pros
Uses hierarchical resource pools to dynamically allocate resources among teams, enabling better utilization in large clusters as highlighted in the README's feature list.
Supports GPU and gang scheduling for TensorFlow and dynamic resource allocation for Spark, making it ideal for big data and machine learning workloads, per the README's optimization features.
Designed to scale to millions of containers and tens of thousands of nodes, proven in web-scale environments like Uber, as stated in the README's scalability claims.
Can run in on-premise datacenters or public clouds, providing deployment versatility without vendor lock-in, emphasized in the project philosophy.
Cons
Requires managing four separate daemons and dependencies on Zookeeper for service discovery and Mesos for resource abstraction, increasing operational overhead and setup complexity.
Demands expertise in distributed systems and specific components like Cassandra for storage, which may deter teams without dedicated infrastructure resources.
Built on Mesos with a niche focus, Peloton has a smaller community and tooling compared to more popular schedulers like Kubernetes, potentially limiting support and integrations.
Frequently Asked Questions
Related Projects
ZipkinZipkin is a distributed tracing system
Netflix: StaashA language-agnostic as well as storage-agnostic web interface for storing data into persistent storage systems, the metadata layer abstracts a lot of storage details and the pattern automation APIs take care of automating common data access patterns.
JanuesGraph-UtilsDevelop a graph database app using JanusGraph
SSTable ToolsTools for parsing, creating and doing other fun stuff with sstables
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.