Scrapy proxy integration
Scrapy ships with HttpProxyMiddleware enabled by default, so a spider only needs to attach the proxy to each request meta. Putting the credentials in the proxy URL keeps it to a single field, and because every request carries the setting, it slots cleanly into large concurrent crawls.
import scrapy
PROXY = "http://USERNAME:PASSWORD@gateway.murphyproxies.com:8000"
class IpSpider(scrapy.Spider):
name = "ip"
def start_requests(self):
yield scrapy.Request(
"https://httpbin.org/ip",
meta={"proxy": PROXY}, # HttpProxyMiddleware reads this
callback=self.parse,
)
def parse(self, response):
self.logger.info("Exit IP: %s", response.text)
# Or set it once for the whole spider:
# custom_settings = {
# "HTTPPROXY_ENABLED": True,
# "DEFAULT_REQUEST_HEADERS": {},
# }Setup in 5 steps
- 1Install the client with pip install scrapy.
- 2Build the proxy URL as user:pass@gateway.murphyproxies.com:8000.
- 3Attach it to each request through meta={"proxy": PROXY}; the built-in middleware handles the rest.
- 4Log response.text from an IP echo endpoint to confirm requests leave through the proxy.
- 5For rotation, vary the username per request (or add scrapy-rotating-proxies) to draw fresh IPs.
Scrapy proxy questions
Everything you need to route Scrapy through Murphy, from authentication to rotation.
No. Scrapy enables HttpProxyMiddleware by default, so setting meta["proxy"] on a request is enough. A custom middleware only helps when you want to inject or rotate the proxy centrally rather than per request.
Rotation is driven by the credentials you send to the gateway. Use your base username for a fresh IP per request, or vary a session token to control stickiness. For a turnkey option, the scrapy-rotating-proxies package cycles a proxy list for you.
The built-in middleware handles HTTP and HTTPS. For SOCKS5, set the proxy URL scheme to socks5 and install a SOCKS-aware download handler, or keep the HTTP scheme on the Murphy gateway, which covers the same traffic without extra dependencies.
Get credentials for Scrapy
Pick the network that fits your target. ISP proxies for fast, stable sessions, residential proxies for broad IP diversity against strict anti-bot systems.