Fast, parallelized, streaming access to Amazon S3 with a CLI tool and Go library.
s3gof3r is a Go library and command-line interface (`gof3r`) that provides fast, concurrent, and streaming access to Amazon S3. It is optimized for high-speed transfer of large objects by leveraging parallelism and pipelining, solving the problem of slow or unreliable S3 transfers for bulk data operations. The tool supports streaming via Unix pipes, enabling real-time data processing during uploads and downloads.
Developers and system administrators who need to transfer large volumes of data to or from Amazon S3 efficiently, particularly those working in cloud environments like AWS EC2 where network bandwidth is critical.
Developers choose s3gof3r for its exceptional speed and reliability in S3 transfers, with built-in retry logic, integrity checking, and memory efficiency. Its streaming capability and CLI integration with Unix pipes offer flexibility for data transformation workflows without intermediate storage.
Fast, concurrent, streaming access to Amazon S3, including gof3r, a CLI. http://godoc.org/github.com/rlmcpherson/s3gof3r
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Benchmarks in the README show speeds over 1 Gbps on EC2 instances for large objects, leveraging parallelism to saturate bandwidth.
Supports real-time data transformation during transfers, as demonstrated with tar commands piped directly into gof3r put/get operations.
Implements parallel MD5 hashing, content-MD5 headers, and ETag verification, ensuring end-to-end data consistency per the S3 API.
Retries all HTTP requests and individual parts on timeouts or failures, crucial for completing large transfers under high load conditions.
Recycles memory buffers and allows tuning of concurrency and part size, with default usage under 300 MB for 10 concurrent parts.
Only supports Amazon S3, making it incompatible with other cloud storage providers or S3-compatible services without modifications.
Requires manual tuning of parameters like concurrency and part size for optimal performance, which can be cumbersome for casual users.
Benchmarks are EC2-centric; performance may degrade on non-AWS networks or with slower internet connections, limiting portability.
Focuses on core transfer operations; lacks built-in support for advanced S3 features like bucket policies or detailed logging out of the box.