Do I need a custom middleware for Scrapy proxies?

No. Scrapy enables HttpProxyMiddleware by default, so setting meta["proxy"] on a request is enough. A custom middleware only helps when you want to inject or rotate the proxy centrally rather than per request.

How do I rotate IPs across a crawl?

Rotation is driven by the credentials you send to the gateway. Use your base username for a fresh IP per request, or vary a session token to control stickiness. For a turnkey option, the scrapy-rotating-proxies package cycles a proxy list for you.

Does Scrapy support SOCKS5 proxies?

The built-in middleware handles HTTP and HTTPS. For SOCKS5, set the proxy URL scheme to socks5 and install a SOCKS-aware download handler, or keep the HTTP scheme on the Murphy gateway, which covers the same traffic without extra dependencies.

Scrapy proxy integration

Scrapy ships with HttpProxyMiddleware enabled by default, so a spider only needs to attach the proxy to each request meta. Putting the credentials in the proxy URL keeps it to a single field, and because every request carries the setting, it slots cleanly into large concurrent crawls.

spider.pypython

import scrapy

PROXY = "http://USERNAME:PASSWORD@gateway.murphyproxies.com:8000"

class IpSpider(scrapy.Spider):
    name = "ip"

    def start_requests(self):
        yield scrapy.Request(
            "https://httpbin.org/ip",
            meta={"proxy": PROXY},   # HttpProxyMiddleware reads this
            callback=self.parse,
        )

    def parse(self, response):
        self.logger.info("Exit IP: %s", response.text)

# Or set it once for the whole spider:
# custom_settings = {
#     "HTTPPROXY_ENABLED": True,
#     "DEFAULT_REQUEST_HEADERS": {},
# }

Setup in 5 steps

1Install the client with pip install scrapy.
2Build the proxy URL as user:pass@gateway.murphyproxies.com:8000.
3Attach it to each request through meta={"proxy": PROXY}; the built-in middleware handles the rest.
4Log response.text from an IP echo endpoint to confirm requests leave through the proxy.
5For rotation, vary the username per request (or add scrapy-rotating-proxies) to draw fresh IPs.

Scrapy proxy questions

Everything you need to route Scrapy through Murphy, from authentication to rotation.

Get credentials for Scrapy

Pick the network that fits your target. ISP proxies for fast, stable sessions, residential proxies for broad IP diversity against strict anti-bot systems.

ISP Proxies Residential Proxies

Other integrations

Python requests cURL Selenium Puppeteer Playwright

import scrapy PROXY = "http://USERNAME:PASSWORD@gateway.murphyproxies.com:8000" class IpSpider(scrapy.Spider): name = "ip" def start_requests(self): yield scrapy.Request( "https://httpbin.org/ip", meta={"proxy": PROXY}, # HttpProxyMiddleware reads this callback=self.parse, ) def parse(self, response): self.logger.info("Exit IP: %s", response.text) # Or set it once for the whole spider: # custom_settings = { # "HTTPPROXY_ENABLED": True, # "DEFAULT_REQUEST_HEADERS": {}, # }

Setup in 5 steps

1Install the client with pip install scrapy.

2Build the proxy URL as user:pass@gateway.murphyproxies.com:8000.

3Attach it to each request through meta={"proxy": PROXY}; the built-in middleware handles the rest.

4Log response.text from an IP echo endpoint to confirm requests leave through the proxy.

5For rotation, vary the username per request (or add scrapy-rotating-proxies) to draw fresh IPs.