How do I change the number of Tor instances in the rotating proxy?

Set the 'tors' environment variable when running the Docker container, for example, use '--env tors=50' to increase to 50 Tor instances, as shown in the usage example in the README.

Rotating Tor proxy vs paid residential proxies: which is better for scraping?

This project uses free Tor networks, offering anonymity but with slower speeds and less reliability; paid residential proxies provide faster, more stable IPs better for high-volume scraping, though at a cost.

How can I monitor the proxy performance in real-time?

Access the built-in HAProxy stats page by navigating to http://127.0.0.1:4444/haproxy?stats after starting the container, as detailed in the README's monitoring section.

Does this proxy support HTTPS scraping or only HTTP?

It supports HTTPS through the HTTP proxy protocol; clients can use it for HTTPS requests, but performance may be impacted by Tor's encryption layers, as implied by the use of Polipo for translation.

Can I use this with Python requests library for web scraping?

Yes, configure the Python requests library to use the HTTP proxy at 127.0.0.1:5566, but note that Tor's slow speeds might require timeouts or retry logic for reliable scraping.

Is there a way to add authentication to the proxy endpoints?

The README does not mention authentication features; users would need to modify the HAProxy configuration manually, which adds complexity and is not covered in the documentation.

How often does the IP address rotate in this setup?

IP rotation depends on HAProxy's load-balancing across Tor instances; each request can get a different IP, but the README doesn't specify rotation intervals, so it's request-based rather than time-based.

rotating-proxy — Docker Rotating Tor Proxy

What is rotating-proxy?

Rotating TOR proxy is a Docker container that sets up a rotating proxy service using multiple Tor circuits. It solves the problem of IP-based blocking or rate limiting by providing a pool of different IP addresses through a single endpoint, making it ideal for anonymous web scraping or testing.

Target Audience

Developers and researchers who need to rotate IP addresses for web scraping, data collection, or testing applications without being blocked by IP-based restrictions.

Value Proposition

It offers an easy-to-deploy, self-contained solution with built-in load balancing and monitoring, eliminating the need to manually manage multiple Tor instances while ensuring reliable IP rotation.

Rotating TOR proxy with Docker

Use Cases

Best For

Web scraping websites with IP-based rate limiting
Testing applications from multiple geographic locations
Anonymous data collection without revealing your real IP
Bypassing IP bans or restrictions on online services
Load testing with distributed IP addresses
Research projects requiring anonymous web access

Not Ideal For

Applications requiring high-speed or low-latency data transfer, such as real-time web scraping or API calls
Projects needing persistent IP addresses for session-based authentication or whitelisting
Environments where Docker containerization is not feasible or allowed, like some strict production servers
Use cases demanding direct SOCKS proxy support without HTTP translation overhead

Pros & Cons

Pros

Easy Docker Deployment

Provides a ready-to-use Docker container with simple build and run commands, as shown in the README with 'docker run -d -p 5566:5566 -p 4444:4444', abstracting the complexity of managing multiple Tor instances.

Single Endpoint Simplicity

HAProxy offers one endpoint (port 5566) for clients to connect to, load-balancing across all Tor proxies, simplifying client configuration as illustrated in the architecture diagram.

Built-in Monitoring Interface

Includes a web-based HAProxy stats page accessible via port 4444, allowing real-time performance monitoring without additional setup, mentioned in the 'monitor' section.

Automatic IP Rotation

Rotates through multiple Tor circuits to provide different IP addresses per request, ideal for bypassing IP-based rate limits, as highlighted in the key features.

Cons

Inherent Tor Latency

Relies on the Tor network, which is known for high latency and slow speeds, making it unsuitable for time-sensitive or high-throughput applications like fast web scraping.

Depends on Outdated Polipo

Uses Polipo for HTTP-to-SOCKS translation, which is an older, less-maintained project, potentially leading to compatibility issues or security concerns as noted in the README.

Limited to HTTP Proxy

Only supports HTTP proxy protocol via HAProxy and Polipo, not direct SOCKS, which may restrict clients that require SOCKS proxies or advanced proxy features.

Basic Configuration Options

Offers minimal configuration through environment variables (e.g., tors=25), with no documentation for advanced HAProxy or Tor settings, limiting customization for complex use cases.

Frequently Asked Questions

What is rotating-proxy?

Target Audience

Developers and researchers who need to rotate IP addresses for web scraping, data collection, or testing applications without being blocked by IP-based restrictions.

Value Proposition

It offers an easy-to-deploy, self-contained solution with built-in load balancing and monitoring, eliminating the need to manually manage multiple Tor instances while ensuring reliable IP rotation.

Use Cases

Best For

Web scraping websites with IP-based rate limiting
Testing applications from multiple geographic locations
Anonymous data collection without revealing your real IP
Bypassing IP bans or restrictions on online services
Load testing with distributed IP addresses
Research projects requiring anonymous web access

Not Ideal For

Applications requiring high-speed or low-latency data transfer, such as real-time web scraping or API calls
Projects needing persistent IP addresses for session-based authentication or whitelisting
Environments where Docker containerization is not feasible or allowed, like some strict production servers
Use cases demanding direct SOCKS proxy support without HTTP translation overhead

Pros & Cons

Pros

Easy Docker Deployment

Single Endpoint Simplicity

HAProxy offers one endpoint (port 5566) for clients to connect to, load-balancing across all Tor proxies, simplifying client configuration as illustrated in the architecture diagram.

Built-in Monitoring Interface

Includes a web-based HAProxy stats page accessible via port 4444, allowing real-time performance monitoring without additional setup, mentioned in the 'monitor' section.

Automatic IP Rotation

Rotates through multiple Tor circuits to provide different IP addresses per request, ideal for bypassing IP-based rate limits, as highlighted in the key features.

Cons

Inherent Tor Latency

Relies on the Tor network, which is known for high latency and slow speeds, making it unsuitable for time-sensitive or high-throughput applications like fast web scraping.

Depends on Outdated Polipo

Uses Polipo for HTTP-to-SOCKS translation, which is an older, less-maintained project, potentially leading to compatibility issues or security concerns as noted in the README.

Limited to HTTP Proxy

Only supports HTTP proxy protocol via HAProxy and Polipo, not direct SOCKS, which may restrict clients that require SOCKS proxies or advanced proxy features.

Basic Configuration Options

Offers minimal configuration through environment variables (e.g., tors=25), with no documentation for advanced HAProxy or Tor settings, limiting customization for complex use cases.

Frequently Asked Questions

rotating-proxy

What is rotating-proxy?

Overview

Use Cases

Best For

Not Ideal For

Pros & Cons

Pros

Cons

Frequently Asked Questions

Related Projects

Found a gem we're missing?

rotating-proxy

What is rotating-proxy?

Overview

Use Cases

Best For

Not Ideal For

Pros & Cons

Pros

Cons

Frequently Asked Questions

Related Projects

Found a gem we're missing?