HadoopConcatGz

Java

A splitable Hadoop InputFormat for processing concatenated GZIP files and web archive (*.warc.gz) data efficiently in distributed systems.

GitHub

9 stars2 forks0 contributors

Overview

A Splitable Hadoop InputFormat for Concatenated GZIP Files and *.(w)arc.gz

Related Projects

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

Submit a project Star on GitHub

8 years ago

CreatedSince 2016

Built With

Included in

Web Archiving2.5k

Auto-fetched 10 hours ago

warctools

Command line tools and libraries for handling and manipulating WARC files (and HTTP contents)

Stars175

Forks33

Last commit10 months ago

FastWARC

A robust web archive analytics toolkit

Stars142

Forks18

Last commit16 days ago

node-warc

Parse And Create Web ARChive (WARC) files with node.js

Stars104

Forks22

Last commit1 year ago

jwarc

Java library for reading and writing WARC files with a typed API

Stars60