s5cmd
A parallel S3 and local filesystem execution tool optimized for high-speed object storage operations.
What is s5cmd?
s5cmd is a parallel S3 and local filesystem execution tool built for high-performance object storage operations. It enables fast uploads, downloads, deletions, and synchronization of files between local storage and cloud services like Amazon S3 and Google Cloud Storage. The tool solves the problem of slow bulk file transfers by leveraging concurrent processing and efficient batching.
Developers, data engineers, and system administrators who regularly work with large volumes of data in S3 or compatible object storage and need fast, scriptable command-line tools for data management.
Developers choose s5cmd for its exceptional speed—up to 32x faster than s3cmd for uploads—and its parallel execution model that saturates high-bandwidth links. It offers a rich feature set including wildcard operations, batch command execution, and sync capabilities, all while being lightweight and easy to integrate into automated workflows.
Overview
Parallel S3 and local filesystem execution tool.
Use Cases
Best For
- Bulk uploading or downloading thousands of files to/from S3
- Synchronizing large directories between local storage and object storage
- Running batch operations on S3 objects using wildcard patterns
- High-speed data migration between S3-compatible storage services
- Filtering and extracting JSON data from S3 objects with SQL queries
- Automating S3 file management in CI/CD pipelines or scripts
Not Ideal For
- Copying single S3 objects larger than 5GB between S3 buckets (explicitly unsupported per the README)
- Direct synchronization between two remote storage endpoints without using local disk as an intermediate step
- Interactive use cases requiring detailed, real-time progress bars or GUI-based management
- Projects needing full AWS CLI command parity for advanced S3 features like object locking or lifecycle policies
Pros & Cons
Pros
Uses concurrent worker pools to achieve up to 32x faster uploads than s3cmd and can saturate 40Gbps links for downloads, as documented in benchmarks.
Supports multi-level wildcards for all operations and allows running thousands of commands in parallel from a file via the 'run' command, enabling high-throughput workflows.
Offers configurable sync strategies (size-only or size and time) and automatically verifies uploads with MD5 and SHA256 checksums to prevent data corruption.
Works with Amazon S3, Google Cloud Storage, MinIO, and other S3-compatible services via custom endpoints, as shown in the GCS support section.
Cons
Does not support copying objects larger than 5GB from S3 to S3, with an open ticket acknowledging this limitation, which can hinder data migration workflows.
Syncing between two cloud storage services requires downloading to local disk first, then uploading—a cumbersome workaround that adds overhead and complexity.
Intentionally diverges from AWS CLI syntax and feature set, requiring users to learn new commands and potentially adapt existing scripts.
Open Source Alternative To
s5cmd is an open-source alternative to the following products:
s3cmd is a command-line tool for managing Amazon S3 and CloudFront services, allowing users to upload, download, and manage objects in S3 buckets.
The AWS Command Line Interface (AWS CLI) is a unified tool to manage your AWS services, allowing you to control multiple AWS services from the command line and automate them through scripts.
Frequently Asked Questions
Related Projects
KubernetesProduction-Grade Container Scheduling and Management
MobyThe Moby Project - a collaborative project for the container ecosystem to assemble container-based systems
traefikThe Cloud Native Application Proxy
minioMinIO is a high-performance, S3 compatible object store, open sourced under GNU AGPLv3 license.
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.