Интеграция прокси с Scrapy

Scrapy ships with HttpProxyMiddleware enabled by default, so a spider only needs to attach the proxy to each request meta. Putting the credentials in the proxy URL keeps it to a single field, and because every request carries the setting, it slots cleanly into large concurrent crawls.

spider.pypython
import scrapy

# Residential gateway with your dashboard credentials,
# or "http://USER:PASS@IP:PORT" for one of your ISP proxies.
PROXY = "http://USERNAME:PASSWORD@gw-eu.murphyproxies.com:8888"

class IpSpider(scrapy.Spider):
    name = "ip"

    def start_requests(self):
        yield scrapy.Request(
            "https://httpbin.org/ip",
            meta={"proxy": PROXY},   # HttpProxyMiddleware reads this
            callback=self.parse,
        )

    def parse(self, response):
        self.logger.info("Exit IP: %s", response.text)

# Or set it once for the whole crawl: HttpProxyMiddleware also reads the
# standard environment variables, so drop the meta and run
#   export http_proxy="http://USERNAME:PASSWORD@gw-eu.murphyproxies.com:8888"
#   export https_proxy="$http_proxy"
#   scrapy crawl ip

Настройка за 5 шагов

  1. 1Install the client with pip install scrapy.
  2. 2Build the proxy URL as http://user:pass@host:port: gw-eu.murphyproxies.com:8888 for residential (port 7777 on Private 2), or one of your ISP lines.
  3. 3Attach it to each request through meta={"proxy": PROXY}; the built-in middleware handles the rest.
  4. 4Log response.text from an IP echo endpoint to confirm requests leave through the proxy.
  5. 5To set it once for the whole crawl, export http_proxy and https_proxy before scrapy crawl instead.

Вопросы о прокси для Scrapy

Всё, что нужно, чтобы пустить Scrapy через Murphy, от авторизации до ротации.

Получите доступы для Scrapy

Выберите сеть под свою задачу. ISP-прокси для быстрых и стабильных сессий, резидентные, когда против жёстких антиботов нужно много разных IP.

Другие интеграции