How fast is s3gof3r compared to the AWS CLI?

s3gof3r often outperforms AWS CLI for large objects due to parallelism, achieving over 1 Gbps on EC2, but AWS CLI may be simpler for small files or basic operations.

How to install gof3r on Ubuntu without installing Go?

Download the statically-linked binary from GitHub Releases, make it executable, and add it to your PATH. Alternatively, use the go get method with environment variables if Go is available.

Does s3gof3r support S3 server-side encryption?

Yes, you can set encryption headers like x-amz-server-side-encryption using the -m flag in commands, as shown in the README example for AES256 encryption.

Can I use s3gof3r with MinIO or Backblaze B2?

No, s3gof3r is designed specifically for Amazon S3 and may not work with S3-compatible services due to API differences and optimizations for AWS infrastructure.

How to handle network errors in s3gof3r?

s3gof3r automatically retries failed requests and parts. For custom error handling in Go code, consult the godoc for retry configurations and error types.

What's the memory usage for transferring a 50 GB file?

With default settings, memory usage stays under 300 MB. Adjust concurrency and part size via flags to reduce footprint, but this may impact transfer speed.

s3gof3r

MITGov0.5.0

Fast, parallelized, streaming access to Amazon S3 with a CLI tool and Go library.

GitHub

1.1k stars180 forks0 contributors

What is s3gof3r?

s3gof3r is a Go library and command-line interface (`gof3r`) that provides fast, concurrent, and streaming access to Amazon S3. It is optimized for high-speed transfer of large objects by leveraging parallelism and pipelining, solving the problem of slow or unreliable S3 transfers for bulk data operations. The tool supports streaming via Unix pipes, enabling real-time data processing during uploads and downloads.

Target Audience

Developers and system administrators who need to transfer large volumes of data to or from Amazon S3 efficiently, particularly those working in cloud environments like AWS EC2 where network bandwidth is critical.

Value Proposition

Developers choose s3gof3r for its exceptional speed and reliability in S3 transfers, with built-in retry logic, integrity checking, and memory efficiency. Its streaming capability and CLI integration with Unix pipes offer flexibility for data transformation workflows without intermediate storage.

Overview

Fast, concurrent, streaming access to Amazon S3, including gof3r, a CLI. http://godoc.org/github.com/rlmcpherson/s3gof3r

Use Cases

Best For

Transferring multi-gigabyte files to/from S3 with maximum bandwidth utilization

Related Projects

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

Submit a project Star on GitHub

Streaming data directly from tar/compression pipelines into S3 without temporary files

Ensuring data integrity during high-volume S3 uploads and downloads

Automating large-scale data backups or migrations to Amazon S3

Handling S3 transfers in environments with frequent network timeouts

Integrating S3 operations into Go applications requiring concurrent data access

Not Ideal For

Projects handling primarily small files (e.g., <100 MB) where parallelism overhead negates speed benefits
Teams using S3-compatible services like MinIO or Google Cloud Storage, as it's optimized for AWS S3 only
Applications requiring simple, GUI-based S3 uploads without command-line or scripting expertise
Environments with extremely strict memory constraints (<100 MB available) despite its efficiency

Pros & Cons

Pros

Exceptional Transfer Speeds

Benchmarks in the README show speeds over 1 Gbps on EC2 instances for large objects, leveraging parallelism to saturate bandwidth.

Streaming via Unix Pipes

Supports real-time data transformation during transfers, as demonstrated with tar commands piped directly into gof3r put/get operations.

Comprehensive Integrity Checking

Implements parallel MD5 hashing, content-MD5 headers, and ETag verification, ensuring end-to-end data consistency per the S3 API.

Reliable Retry Logic

Retries all HTTP requests and individual parts on timeouts or failures, crucial for completing large transfers under high load conditions.

Memory-Efficient Design

Recycles memory buffers and allows tuning of concurrency and part size, with default usage under 300 MB for 10 concurrent parts.

Cons

AWS S3 Exclusive

Only supports Amazon S3, making it incompatible with other cloud storage providers or S3-compatible services without modifications.

Configuration Complexity

Requires manual tuning of parameters like concurrency and part size for optimal performance, which can be cumbersome for casual users.

Performance Tied to AWS

Benchmarks are EC2-centric; performance may degrade on non-AWS networks or with slower internet connections, limiting portability.

Limited High-Level Features

Focuses on core transfer operations; lacks built-in support for advanced S3 features like bucket policies or detailed logging out of the box.

Frequently Asked Questions

Home

Amazon Web Services

Kubernetes

Production-Grade Container Scheduling and Management

The Moby Project - a collaborative project for the container ecosystem to assemble container-based systems

The Cloud Native Application Proxy

MinIO is a high-performance, S3 compatible object store, open sourced under GNU AGPLv3 license.

Stars61,226

Forks7,560

Last commit1 month ago

Amazon Web Services14.0k