Squidwarc
A high-fidelity, user-scriptable archival web crawler using Chrome/Chromium to preserve JavaScript-rendered content.
What is Squidwarc?
Squidwarc is a high-fidelity archival web crawler that uses Chrome or Chromium to capture JavaScript-rendered web content. It solves the problem of traditional crawlers failing to preserve modern, dynamic websites by executing JavaScript and saving content in WARC format. The project is designed to be user-friendly and scriptable, making it suitable for both personal and research archiving.
Digital archivists, researchers, and individuals who need to preserve JavaScript-heavy websites with accuracy and ease. It is particularly valuable for those who find traditional tools like Heritrix too complex or limited for modern web content.
Developers choose Squidwarc for its ability to handle JavaScript execution out-of-the-box, its scriptable nature for custom archiving needs, and its straightforward setup compared to enterprise-grade alternatives. It bridges the gap between high-fidelity archiving and accessibility.
Overview
Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head
Use Cases
Best For
- Archiving JavaScript-heavy websites like social media platforms
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
