grab-site

The built-in dashboard provides real-time visibility into all active crawls, showing queued URLs, progress, and status, as highlighted in the usage section where starting gs-server enables monitoring via a web interface.

Users can add or modify ignore rules during a crawl by editing the DIR/ignores file, allowing real-time skipping of junk URLs to prevent infinite traps, as described in the 'Changing ignores during the crawl' section.

Includes extensively tested default ignore sets for common site types like forums, Reddit, and MediaWiki, reducing manual configuration effort, with sets available in the libgrabsite/ignore_sets directory.

Stores the URL queue on disk instead of memory, enabling crawls of sites with up to ~10 million pages, as noted in the README's key features and philosophy for handling large archives.