How to use cloudflare-scrape with a proxy?

Pass the proxies keyword argument to functions like get_tokens() or create_scraper(), as shown in the README examples with proxies configuration. This ensures that the challenge-solving IP matches the visitor IP for Cloudflare compliance.

Does cloudflare-scrape work with reCAPTCHA?

No, it only bypasses Cloudflare's JavaScript challenge page (IUAM) and does not handle reCAPTCHA. The README clearly states this limitation, so for sites with reCAPTCHA, you'll need alternative methods.

Cloudflare-scrape vs selenium for bypassing Cloudflare?

Cloudflare-scrape is lighter and faster since it uses Node.js for JavaScript without a full browser, making it ideal for Python scraping pipelines. Selenium can handle more complex interactions like reCAPTCHA but is heavier and slower for simple bypass tasks.

How to update cloudflare-scrape to the latest version?

Run 'pip install -U cfscrape' to upgrade. The README recommends this before filing issues, as many problems stem from outdated versions due to Cloudflare's frequent changes.

Why is cloudflare-scrape not working on a site?

First, ensure you have the latest version installed and check if the site uses reCAPTCHA. If issues persist, provide details like version number and URL in a GitHub issue, as the README suggests, since Cloudflare updates can break functionality.

Can I use cloudflare-scrape without installing Node.js?

No, Node.js is required to solve the JavaScript challenges. The README specifies that it's necessary for interpreting Cloudflare's obfuscated code, so you must have Node.js version 10 or above installed.

Open-Awesome

Scrape

MITPython2.1.1

A Python module to bypass Cloudflare's anti-bot page by solving JavaScript challenges using Node.js.

GitHub

3.5k stars451 forks0 contributors

What is Scrape?

Cloudflare-scrape is a Python module that bypasses Cloudflare's anti-bot protection page to enable web scraping and crawling of protected websites. It solves JavaScript challenges automatically using Node.js, allowing scripts to mimic legitimate browser traffic without manual intervention. The module handles Cloudflare's 'I'm Under Attack Mode' (IUAM) but does not bypass reCAPTCHA challenges.

Target Audience

Developers and researchers who need to scrape or crawl websites protected by Cloudflare's anti-bot measures, such as data miners, automation script writers, and security testers.

Value Proposition

It provides a reliable, up-to-date solution that automatically adapts to Cloudflare's changing techniques, with a simple API that integrates seamlessly with the popular Requests library, reducing development overhead.

Overview

A Python module to bypass Cloudflare's anti-bot page.

Use Cases

Best For

Scraping data from Cloudflare-protected websites for research or analysis
Automating interactions with sites that use Cloudflare's IUAM mode
Integrating Cloudflare bypass capabilities into existing Python scraping pipelines
Crawling websites that block standard HTTP clients with anti-bot pages
Testing website accessibility behind Cloudflare protections
Building tools that require consistent access to Cloudflare-secured content

Not Ideal For

Projects that need to bypass reCAPTCHA challenges, as it only handles JavaScript-based Cloudflare protection
Environments where Node.js is not available or cannot be installed, such as restricted server setups
Applications requiring high-speed scraping with minimal delays, due to mandatory waiting periods enforced by Cloudflare
Teams seeking a pure-Python solution without external runtime dependencies

Pros & Cons

Pros

Automatic JavaScript Solving

Uses Node.js to interpret and solve Cloudflare's obfuscated JavaScript challenges automatically, mimicking a browser without manual deobfuscation, as stated in the README.

Requests-Compatible API

Provides a CloudflareScraper class that works identically to a Requests Session object, making integration seamless for developers familiar with Requests, with usage examples in the README.

Easy Token Extraction

Offers helper functions like get_tokens() and get_cookie_string() to retrieve Cloudflare cookies for external tools, simplifying integration with other applications, as demonstrated in the README.

Configurable Delay Options

Allows overriding the default 5-second delay required by Cloudflare, with the README providing guidance on when and how to adjust delays for better reliability.

Cons

No reCAPTCHA Bypass

Explicitly does not work with reCAPTCHA challenges, limiting its effectiveness on sites that use more advanced bot protection, as admitted in the README.

Node.js Dependency

Requires Node.js version 10 or above to run, adding an external dependency that complicates setup and may not be feasible in all environments, as noted in the installation section.

Maintenance Sensitivity

Relies on frequent updates to keep up with Cloudflare's changing techniques, so it can break if not maintained, with the README warning about this and urging users to upgrade regularly.

Frequently Asked Questions

Related Projects

cloudflared

Cloudflare Tunnel client

Stars13,929

Forks1,262

Last commit8 days ago

CloudFlair

🔎 Find origin servers of websites behind CloudFlare by using Internet-wide scan data from Censys.

Stars2,946

Forks383

Last commit11 months ago

Th3inspector

Th3Inspector 🕵️ Best Tool For Information Gathering 🔎

Stars2,618

Forks518

Last commit1 year ago

Lexicon

Manipulate DNS records on various DNS providers in a standardized way.

Stars1,525

Forks300