A Go library providing a unified API for file operations across local, cloud, and network storage systems.
afs (Abstract File Storage) is a Go library that provides a unified interface for performing file operations across diverse storage systems, including local filesystems, cloud storage (like AWS S3 and Google Cloud Storage), network protocols (SCP), and archives (ZIP, TAR). It solves the problem of writing different code for each storage backend by abstracting common operations into a single API.
Go developers building applications that need to interact with multiple storage backends, such as data pipelines, backup tools, or cloud-native services requiring portable file handling.
Developers choose afs for its clean abstraction that reduces boilerplate code, its support for a wide range of storage systems out of the box, and its built-in testing capabilities that make it easier to validate storage-related error handling.
Abstract File Storage
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Provides a single API for CRUD operations across local files, S3, GCS, SCP, and archives, reducing boilerplate code when switching backends, as shown in the usage examples with consistent List, Upload, and Copy methods.
Supports streaming large files with configurable chunk sizes via option.NewStream, enabling efficient handling of big data without loading entire files into memory, demonstrated in the streaming data section.
Includes a faker storage (afs.NewFaker()) and error injection options like UploadError, making it easy to simulate storage failures and validate error handling in unit tests, as highlighted in the testing mode section.
Allows auth configuration per base URL using the Init method, reducing repetitive credential passing in code, as exemplified in the SCP and GCS authentication examples.
Focuses on basic file operations and may not expose advanced cloud storage features like metadata management, versioning, or ACLs that are available in native SDKs, limiting use in complex cloud-native applications.
Configuring authentication and options for different storages can be verbose and error-prone, as seen in examples requiring separate auth methods for SCP, S3, and GCS, adding initial setup overhead.
For cloud backends like S3 and GCS, it relies on separate packages (e.g., afsc), which adds dependencies and might lag behind official SDK updates, potentially missing new features or bug fixes.