Question 1

How to use SiteOne Crawler for SEO audits?

Accepted Answer

Run the crawler with the target URL and use options like --extra-columns for SEO data or select the SEO Analysis preset in the interactive wizard. It automatically scores factors like title uniqueness and meta descriptions, generating detailed HTML or JSON reports.

Question 2

SiteOne Crawler vs Screaming Frog: which is better for website analysis?

Accepted Answer

SiteOne Crawler is open-source, written in Rust for performance, and offers unique features like offline exports and Markdown conversion. Screaming Frog is proprietary with more advanced SEO tools. Choose SiteOne for cost-effectiveness and export capabilities, or Screaming Frog for deeper SEO analysis.

Question 3

Can SiteOne Crawler handle JavaScript-heavy sites?

Accepted Answer

Yes, it can crawl JavaScript-rendered sites, but for offline exports, disabling JavaScript with --disable-javascript might be necessary for optimal results, as noted in the README. It respects modern frameworks but may require adjustments.

Question 4

How to integrate SiteOne Crawler into a CI/CD pipeline?

Accepted Answer

Use the --ci flag with quality thresholds (e.g., --ci-min-score) in your pipeline script. The crawler exits with code 10 on failure, allowing automated deployment blocking. Examples for GitHub Actions and GitLab CI are provided in the documentation.

Question 5

Is SiteOne Crawler good for large e-commerce sites?

Accepted Answer

Yes, it supports high-throughput crawling with configurable workers and memory limits, but for very large sites, use --result-storage=file to manage memory. However, it's not distributed, so single-server limits may apply.

Question 6

How to convert a website to Markdown with SiteOne Crawler?

Accepted Answer

Use the --markdown-export-dir option or select the Markdown Export preset in the wizard. It exports to multi-file or single-file Markdown, with smart features like accordion collapsing for links, ideal for AI model ingestion.

siteone-crawler

What is siteone-crawler?

Overview

Use Cases

Best For

Related Projects

Found a gem we're missing?

Not Ideal For

Pros & Cons

Pros

Cons

Frequently Asked Questions