Scrapyのプロキシ連携

Scrapy ships with HttpProxyMiddleware enabled by default, so a spider only needs to attach the proxy to each request meta. Putting the credentials in the proxy URL keeps it to a single field, and because every request carries the setting, it slots cleanly into large concurrent crawls.

spider.pypython
import scrapy

# Residential gateway with your dashboard credentials,
# or "http://USER:PASS@IP:PORT" for one of your ISP proxies.
PROXY = "http://USERNAME:PASSWORD@gw-eu.murphyproxies.com:8888"

class IpSpider(scrapy.Spider):
    name = "ip"

    def start_requests(self):
        yield scrapy.Request(
            "https://httpbin.org/ip",
            meta={"proxy": PROXY},   # HttpProxyMiddleware reads this
            callback=self.parse,
        )

    def parse(self, response):
        self.logger.info("Exit IP: %s", response.text)

# Or set it once for the whole crawl: HttpProxyMiddleware also reads the
# standard environment variables, so drop the meta and run
#   export http_proxy="http://USERNAME:PASSWORD@gw-eu.murphyproxies.com:8888"
#   export https_proxy="$http_proxy"
#   scrapy crawl ip

5ステップで設定

  1. 1Install the client with pip install scrapy.
  2. 2Build the proxy URL as http://user:pass@host:port: gw-eu.murphyproxies.com:8888 for residential (port 7777 on Private 2), or one of your ISP lines.
  3. 3Attach it to each request through meta={"proxy": PROXY}; the built-in middleware handles the rest.
  4. 4Log response.text from an IP echo endpoint to confirm requests leave through the proxy.
  5. 5To set it once for the whole crawl, export http_proxy and https_proxy before scrapy crawl instead.

Scrapyのプロキシに関する質問

認証からローテーションまで、ScrapyをMurphy経由で使うためのポイントをまとめました。

Scrapy用の認証情報を取得

ターゲットに合ったネットワークを選びましょう。高速で安定したセッションにはISPプロキシ、厳しいボット対策には幅広いIPを持つレジデンシャルプロキシ。