Scrapyのプロキシ連携
Scrapy ships with HttpProxyMiddleware enabled by default, so a spider only needs to attach the proxy to each request meta. Putting the credentials in the proxy URL keeps it to a single field, and because every request carries the setting, it slots cleanly into large concurrent crawls.
import scrapy
# Residential gateway with your dashboard credentials,
# or "http://USER:PASS@IP:PORT" for one of your ISP proxies.
PROXY = "http://USERNAME:PASSWORD@gw-eu.murphyproxies.com:8888"
class IpSpider(scrapy.Spider):
name = "ip"
def start_requests(self):
yield scrapy.Request(
"https://httpbin.org/ip",
meta={"proxy": PROXY}, # HttpProxyMiddleware reads this
callback=self.parse,
)
def parse(self, response):
self.logger.info("Exit IP: %s", response.text)
# Or set it once for the whole crawl: HttpProxyMiddleware also reads the
# standard environment variables, so drop the meta and run
# export http_proxy="http://USERNAME:PASSWORD@gw-eu.murphyproxies.com:8888"
# export https_proxy="$http_proxy"
# scrapy crawl ip5ステップで設定
- 1Install the client with pip install scrapy.
- 2Build the proxy URL as http://user:pass@host:port: gw-eu.murphyproxies.com:8888 for residential (port 7777 on Private 2), or one of your ISP lines.
- 3Attach it to each request through meta={"proxy": PROXY}; the built-in middleware handles the rest.
- 4Log response.text from an IP echo endpoint to confirm requests leave through the proxy.
- 5To set it once for the whole crawl, export http_proxy and https_proxy before scrapy crawl instead.
Scrapyのプロキシに関する質問
認証からローテーションまで、ScrapyをMurphy経由で使うためのポイントをまとめました。
No. Scrapy enables HttpProxyMiddleware by default, so setting meta["proxy"] on a request is enough, and the same middleware picks up the http_proxy and https_proxy environment variables. A custom downloader middleware that sets request.meta["proxy"] only helps when you want to choose or rotate the proxy centrally, for example across a list of ISP IPs.
On residential proxies, generate rotating credentials in the dashboard for a new IP on each request, or sticky ones to hold an IP for up to 48 hours; the targeting and session are built into the line you copy. ISP proxies are static, so rotating means cycling through your list of dedicated IPs, which the scrapy-rotating-proxies package can do for you.
Not out of the box: the built-in middleware handles HTTP and HTTPS proxies only. Keep the http:// scheme with your Murphy proxy, which carries HTTPS sites as well, and no extra dependency is needed.
Scrapy用の認証情報を取得
ターゲットに合ったネットワークを選びましょう。高速で安定したセッションにはISPプロキシ、厳しいボット対策には幅広いIPを持つレジデンシャルプロキシ。