A highly scalable and available P2P Docker registry designed for fast distribution of large images in hybrid cloud environments.
Kraken is a peer-to-peer (P2P) powered Docker registry designed for scalable and highly available Docker image management, replication, and distribution in hybrid cloud environments. It integrates as a distribution layer into existing Docker registry setups with pluggable backend support, handling massive-scale workloads like distributing over 1 million blobs daily. The architecture uses agents on each host, dedicated origin seeders, and a tracker to form a pseudo-random regular graph for efficient P2P transfers.
DevOps engineers and platform teams managing large-scale Docker deployments in hybrid or multi-cloud environments, particularly those needing to optimize image distribution across thousands of hosts. It is also suitable for organizations with existing Docker registry setups seeking to enhance scalability and availability without replacing their storage backends.
Developers choose Kraken for its ability to distribute Docker images at over 50% of the max download speed per host while supporting clusters of at least 15,000 hosts and blobs up to 20GB, with no single point of failure. Its unique selling point is the P2P-driven architecture that reduces global max download time and communication overhead, outperforming alternatives like Dragonfly in scalability for large blobs and offering built-in cross-cluster replication and pluggable storage integration.
P2P Docker registry capable of distributing TBs of data in seconds
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Supports clusters of at least 15,000 hosts and distributes blobs up to 20GB at >50% of max download speed per host, with minimal performance degradation as size or scale increases, as benchmarked in production at Uber.
Architecture has no single point of failure, with components like Origin forming a self-healing hash ring and multiple trackers, ensuring reliability for large-scale deployments.
Pluggable backend support allows easy integration with reliable blob storage like S3, GCS, or ECR through a simple interface, fitting into existing registry setups.
Provides rule-based asynchronous replication between clusters with lossless data transfer, essential for hybrid cloud environments as highlighted in the features.
Mutating tags (e.g., updating 'latest') suffers from Nginx caching delays and replication failures, as admitted in the Limitations section, requiring workarounds like reducing cache intervals.
Ultra-large blobs (100GB+) are not recommended for production; the README advises chunking them into <10GB parts, which adds overhead for massive layer distributions.
Involves multiple components (Agent, Origin, Tracker, etc.) with non-trivial configuration, as seen in the Helm and devcluster setup instructions, increasing initial setup effort.