Distributed Processing

3 projects

Showing 3 of 3 projects

A high-performance PHP import framework for distributed data processing with optimized memory consumption.

Java-based web archive deduplication tool that identifies duplicates and converts them to reference records in WARC files.

A splitable Hadoop InputFormat for processing concatenated GZIP files and web archive (*.warc.gz) data efficiently in distributed systems.

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.