Crawly

Uses a familiar callback model similar to Scrapy, making it intuitive for Elixir developers to define crawls with URL generation and parsing logic, as shown in the quickstart example.

Configurable to fetch pages with JavaScript rendering via tools like Splash or Chrome, essential for scraping dynamic content from modern websites, as documented in the browser rendering guide.

Offers pluggable components for customizing request handling and item processing, demonstrated in config examples with DomainFilter, UniqueRequest, and WriteToFile pipelines.

Enables running spiders via Docker with YAML or module definitions, simplifying deployment without full Elixir project setup, as covered in the standalone documentation.

Provides a built-in web interface on localhost:4001 for starting/stopping spiders and viewing items, with options to disable or integrate as a plug in existing apps.