Intégration proxy Scrapy

Scrapy ships with HttpProxyMiddleware enabled by default, so a spider only needs to attach the proxy to each request meta. Putting the credentials in the proxy URL keeps it to a single field, and because every request carries the setting, it slots cleanly into large concurrent crawls.

spider.pypython
import scrapy

# Residential gateway with your dashboard credentials,
# or "http://USER:PASS@IP:PORT" for one of your ISP proxies.
PROXY = "http://USERNAME:PASSWORD@gw-eu.murphyproxies.com:8888"

class IpSpider(scrapy.Spider):
    name = "ip"

    def start_requests(self):
        yield scrapy.Request(
            "https://httpbin.org/ip",
            meta={"proxy": PROXY},   # HttpProxyMiddleware reads this
            callback=self.parse,
        )

    def parse(self, response):
        self.logger.info("Exit IP: %s", response.text)

# Or set it once for the whole crawl: HttpProxyMiddleware also reads the
# standard environment variables, so drop the meta and run
#   export http_proxy="http://USERNAME:PASSWORD@gw-eu.murphyproxies.com:8888"
#   export https_proxy="$http_proxy"
#   scrapy crawl ip

Configuration en 5 étapes

  1. 1Install the client with pip install scrapy.
  2. 2Build the proxy URL as http://user:pass@host:port: gw-eu.murphyproxies.com:8888 for residential (port 7777 on Private 2), or one of your ISP lines.
  3. 3Attach it to each request through meta={"proxy": PROXY}; the built-in middleware handles the rest.
  4. 4Log response.text from an IP echo endpoint to confirm requests leave through the proxy.
  5. 5To set it once for the whole crawl, export http_proxy and https_proxy before scrapy crawl instead.

Questions sur les proxies Scrapy

Tout ce qu'il faut pour faire passer Scrapy par Murphy, de l'authentification à la rotation.

Récupérez vos identifiants pour Scrapy

Choisissez le réseau selon votre cible : les proxies ISP pour des sessions rapides et stables, les résidentiels pour multiplier les IP face aux anti-bots les plus stricts.