A Perl-based multithreaded client for Amazon Glacier, enabling efficient multipart uploads, downloads, and synchronization with local journal tracking.
mt-aws-glacier is a command-line client for Amazon Glacier, designed to handle large-scale archival and backup operations on Unix-like systems. It provides a robust, feature-rich alternative to AWS's native tools, with a focus on performance and reliability for long-term data storage.
System administrators and developers managing large-scale data archival and backup operations on Unix-like systems (Linux, *BSD, Solaris, macOS) who need a reliable, high-performance client for Amazon Glacier.
Developers choose mt-aws-glacier for its multithreaded multipart upload/download capabilities, local journal tracking for integrity and recovery, and flexible file synchronization options, offering more control and performance than AWS's native tools.
Perl Multithreaded Multipart sync to Amazon Glacier
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Utilizes parallel threads for multipart uploads and downloads, accelerating large-scale data transfers significantly compared to single-threaded approaches, as highlighted in the multithreaded multipart upload/download feature.
Maintains an append-only journal file to track all operations, enabling integrity checks and recovery without relying on cloud metadata, which is central to the journal concept for fault tolerance.
Supports sync modes for new, modified, and deleted files with configurable detection methods like treehash and mtime, offering precise control over backup updates as detailed in the sync command specifications.
Allows journal restoration from Amazon Glacier metadata if lost, ensuring backup continuity even in failure scenarios, as explained in the restoring journal section with retrieve-inventory and download-inventory commands.
Requires manual setup of Perl dependencies with different steps for various Unix distributions, and lacks a straightforward universal installer, as detailed in the lengthy installation/system requirements section.
Only supports multipart uploads, not plain uploads, which can be inefficient for small files or in scenarios where multipart isn't optimal, as admitted in the important bugs/missing features list.
Heavily relies on a local journal file for operations; if lost without backup, recovery involves slow inventory retrieval from Glacier (taking 4+ hours), adding risk and operational delay.
Designed exclusively for Unix-like systems with no Windows support, and has specific issues like treating Mac OS X filesystem as case-sensitive, plus manual encoding configuration needed for non-UTF-8 systems, limiting cross-platform use.