How does Browsertrix Crawler compare to wget for web archiving?

Browsertrix Crawler uses real browsers to handle JavaScript and dynamic content, making it superior for modern websites, while wget is faster and simpler for static HTML scraping. Choose Browsertrix for accuracy, wget for speed on basic sites.

Can I use Chrome instead of Brave with Browsertrix Crawler?

The README specifies Brave Browser, but since Puppeteer supports Chromium, modifications might allow Chrome. Out-of-the-box, it's optimized for Brave, so switching may require custom configuration.

What are the system requirements for running Browsertrix Crawler?

It requires Docker and sufficient resources—typically at least 2GB RAM per browser instance. For parallel crawls, a system with 8+ GB RAM and multiple CPU cores is recommended to avoid performance issues.

How to configure parallel crawling in Browsertrix?

Parallel crawling is set by adjusting the number of browser windows in the crawl configuration file. Refer to the documentation for parameters like 'parallel' or 'workers' to control concurrency based on your needs.

Is Browsertrix Crawler suitable for archiving social media sites?

Yes, its high-fidelity capture handles dynamic content well, but you'll need to configure authentication and respect rate limits. Custom scripts or settings may be required for login pages and interactive elements.

Browsertrix Crawler vs ArchiveBox: which is better?

Browsertrix excels at customizable, high-fidelity crawls with Docker for precise archiving, while ArchiveBox offers a simpler, all-in-one solution with less setup. Use Browsertrix for complex projects; ArchiveBox for straightforward, automated archives.

Open-Awesome

Browsertrix Crawler

AGPL-3.0TypeScriptv1.12.4

A standalone Docker container for high-fidelity, browser-based web archiving crawls using Puppeteer and Brave.

Visit Website GitHub

1.0k stars136 forks0 contributors

What is Browsertrix Crawler?

Browsertrix Crawler is a standalone browser-based high-fidelity crawling system that runs in a single Docker container. It automates web archiving by controlling Brave Browser windows with Puppeteer to capture web content exactly as rendered, solving the problem of inaccurate or incomplete archival crawls. It is designed for complex, customizable archiving tasks where fidelity to the original web experience is critical.

Target Audience

Digital archivists, researchers, librarians, and developers who need to preserve web content with high accuracy for projects like digital libraries, compliance, or historical records.

Value Proposition

Developers choose Browsertrix Crawler because it offers containerized simplicity with browser-based precision, avoiding the limitations of traditional crawlers. Its use of real browsers via Puppeteer ensures superior fidelity, making it a reliable open-source tool for professional-grade web archiving.

Overview

Run a high-fidelity browser-based web archiving crawler in a single Docker container

Use Cases

Best For

Creating high-fidelity archives of dynamic websites with JavaScript-heavy content
Running parallel crawls for large-scale web archiving projects
Digital preservation initiatives requiring exact visual and functional capture
Research projects that need reproducible, browser-accurate web snapshots
Self-hosted web archiving solutions with Docker deployment
Archiving complex web applications that rely on real browser rendering

Not Ideal For

Scraping static websites without JavaScript rendering
Projects requiring real-time data extraction with low latency
Environments with limited computational resources (e.g., low RAM or CPU)
Teams needing flexibility to use non-Chromium browsers like Firefox or Safari

Pros & Cons

Pros

High-Fidelity Capture

Uses Chrome Devtools Protocol with real Brave Browser instances to capture web content exactly as rendered, ensuring accuracy for dynamic sites as highlighted in the README.

Parallel Crawling Efficiency

Manages multiple browser windows simultaneously via Puppeteer, enabling scalable archiving for large projects as described in the key features.

Docker Simplicity

Runs as a single Docker container for easy deployment and isolation, simplifying setup and reproducibility across environments.

Customizable Configurations

Supports complex crawl setups tailored to specific archiving needs, allowing fine-tuning for different website structures.

Cons

Resource Intensive

Running multiple browser instances consumes significant memory and CPU, making it unsuitable for low-end hardware or cloud deployments with tight budgets.

Browser Limitation

Tied to Brave Browser through Puppeteer, limiting compatibility for projects that require other browsers or specific browser versions.

Complex Setup and Learning

Requires expertise in Docker, Puppeteer, and archiving parameters, which can be daunting for users without technical backgrounds or for quick, simple tasks.

Frequently Asked Questions

Related Projects

ArchiveBox

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

⬛️ CLI tool and library for saving complete web pages as a single HTML file

Stars15,130

Forks454

Last commit6 days ago

DiskerNet

💾 dn - offline full-text search and archiving for your Chromium-based browser.

Stars3,904

Forks148

Last commit2 months ago

Heritrix

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

Stars3,228

Forks789