Tích hợp proxy với Scrapy
Scrapy ships with HttpProxyMiddleware enabled by default, so a spider only needs to attach the proxy to each request meta. Putting the credentials in the proxy URL keeps it to a single field, and because every request carries the setting, it slots cleanly into large concurrent crawls.
import scrapy
# Residential gateway with your dashboard credentials,
# or "http://USER:PASS@IP:PORT" for one of your ISP proxies.
PROXY = "http://USERNAME:PASSWORD@gw-eu.murphyproxies.com:8888"
class IpSpider(scrapy.Spider):
name = "ip"
def start_requests(self):
yield scrapy.Request(
"https://httpbin.org/ip",
meta={"proxy": PROXY}, # HttpProxyMiddleware reads this
callback=self.parse,
)
def parse(self, response):
self.logger.info("Exit IP: %s", response.text)
# Or set it once for the whole crawl: HttpProxyMiddleware also reads the
# standard environment variables, so drop the meta and run
# export http_proxy="http://USERNAME:PASSWORD@gw-eu.murphyproxies.com:8888"
# export https_proxy="$http_proxy"
# scrapy crawl ipCài đặt trong 5 bước
- 1Install the client with pip install scrapy.
- 2Build the proxy URL as http://user:pass@host:port: gw-eu.murphyproxies.com:8888 for residential (port 7777 on Private 2), or one of your ISP lines.
- 3Attach it to each request through meta={"proxy": PROXY}; the built-in middleware handles the rest.
- 4Log response.text from an IP echo endpoint to confirm requests leave through the proxy.
- 5To set it once for the whole crawl, export http_proxy and https_proxy before scrapy crawl instead.
Câu hỏi về proxy cho Scrapy
Mọi điều bạn cần để định tuyến Scrapy qua Murphy, từ xác thực đến xoay vòng IP.
No. Scrapy enables HttpProxyMiddleware by default, so setting meta["proxy"] on a request is enough, and the same middleware picks up the http_proxy and https_proxy environment variables. A custom downloader middleware that sets request.meta["proxy"] only helps when you want to choose or rotate the proxy centrally, for example across a list of ISP IPs.
On residential proxies, generate rotating credentials in the dashboard for a new IP on each request, or sticky ones to hold an IP for up to 48 hours; the targeting and session are built into the line you copy. ISP proxies are static, so rotating means cycling through your list of dedicated IPs, which the scrapy-rotating-proxies package can do for you.
Not out of the box: the built-in middleware handles HTTP and HTTPS proxies only. Keep the http:// scheme with your Murphy proxy, which carries HTTPS sites as well, and no extra dependency is needed.
Nhận thông tin đăng nhập cho Scrapy
Chọn mạng lưới hợp với mục tiêu của bạn. Proxy ISP cho phiên nhanh và ổn định, proxy dân cư cho nguồn IP đa dạng trước các hệ thống chống bot khắt khe.