s3git
A CLI tool that applies Git-like version control to cloud storage, enabling distributed, decentralized, and deduplicated data repositories.
What is s3git?
s3git is a command-line tool that applies Git-like version control to cloud storage, enabling distributed and decentralized data repositories. It solves the problem of managing huge datasets (up to petabytes) with versioning, deduplication, and the ability to work locally on SSDs while storing data in S3-compatible backends. Unlike Git, it's optimized for large binary files and scales to hundreds of millions of objects.
Developers, DevOps engineers, and data engineers who need to version, manage, and collaborate on large binary datasets, such as build artifacts, multimedia files, analytics data, or release binaries, using cloud storage backends.
Developers choose s3git for its seamless integration with cloud storage, infinite scalability, and familiar Git workflow, making it the go‑open-source solution for versioning massive datasets without server overhead. Its deduplication and directory snapshotting provide efficient storage and reliable rollbacks.
Overview
s3git: git for Cloud Storage. Distributed Version Control for Data. Create decentralized and versioned repos that scale infinitely to 100s of millions of files. Clone huge PB-scale repos on your local SSD to make changes, commit and push back. Oh yeah, it dedupes too and offers directory versioning.
Use Cases
Best For
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
