Sparkling

MITScalalatest-SNAPSHOT

A Scala/Spark library for efficient processing, extraction, and derivation of web archive data (CDX/WARC).

GitHub

17 stars2 forks0 contributors

Overview

Internet Archive's Sparkling Data Processing Library

Related Projects

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

Submit a project Star on GitHub

1 month ago

CreatedSince 2022

Built With

sbt

Included in

Web Archiving2.5k

Auto-fetched 1 day ago

warctools

Command line tools and libraries for handling and manipulating WARC files (and HTTP contents)

Stars175

Forks33

Last commit10 months ago

FastWARC

A robust web archive analytics toolkit

Stars142

Forks18

Last commit15 days ago

node-warc

Parse And Create Web ARChive (WARC) files with node.js

Stars104

Forks22

Last commit1 year ago

warc

:gear: A Rust library for reading and writing WARC files

Stars60

Forks19