A high-performance embedded library and CLI tool for multi-TCP data transfer between systems at near-hardware speeds.
WDT (Warp speed Data Transfer) is an embeddable C++ library and command-line tool for high-speed data transfer between systems over multiple TCP connections. It solves the problem of slow data movement across networks by parallelizing transfers to achieve near-hardware-limited speeds, significantly outperforming traditional HTTP-based solutions.
System engineers, DevOps teams, and developers who need to transfer large datasets (like database snapshots or bulk files) between servers with minimal latency and maximum throughput, especially in data center or cross-region scenarios.
Developers choose WDT for its ability to saturate high-bandwidth network links (e.g., 40 Gbit/s) with minimal CPU overhead, its support for transfer resumption, and its embeddable library design that avoids heavy dependencies while delivering 3x faster speeds than optimized HTTP alternatives.
Warp speed Data Transfer (WDT) is an embeddedable library (and command line tool) aiming to transfer data between 2 systems as fast as possible over multiple TCP paths.
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Uses multiple TCP connections to maximize throughput and reduce latency impact, achieving speeds up to 4 GB/s on 40 Gbit/s NICs, as shown in performance tests.
Employs blocking thread I/O to minimize system calls and kernel switches, ensuring low CPU and memory usage while keeping subsystems busy, which enhances efficiency.
Maintains transfer history to resume interrupted transfers without data loss, improving reliability for long-distance or large-scale data migration scenarios.
Built with portability in mind, requiring only a C++11 compiler and optional logging, making it easy to integrate into custom applications without heavy dependencies.
Optimized for fast I/O like flash or memory, so throughput on traditional disk storage is not as good, as the README explicitly states, limiting its use in some environments.
While extensible, the current implementation lacks integrated compression or encryption units, with the README noting plans for future restructuring, which may delay feature adoption.
Requires CMake and dependencies like Facebook's Folly library and OpenSSL for encryption, complicating setup and integration compared to simpler, single-binary tools.