Scrapy : How to write a HttpProxyMiddleware? - scrapy

Question

The docs of Scrapy says about HttpProxyMiddleware is like this:
This middleware sets the HTTP proxy to use for requests, by setting the proxy meta value for Request objects.
Like the Python standard library modules urllib and urllib2, it obeys the following environment variables:
http_proxy
https_proxy
no_proxy
You can also set the meta key proxy per-request, to a value like http://some_proxy_server:port or http://username:password#some_proxy_server:port. Keep in mind this value will take precedence over http_proxy/https_proxy environment variables, and it will also ignore no_proxy environment variable.
docs:https://doc.scrapy.org/en/latest/topics/downloader-middleware.html?highlight=Proxy#module-scrapy.downloadermiddlewares.httpproxy
But there are no examples in the docs.
I have no ideas how to write a HttpProxyMiddleware.
Are there any suggestions?

Answer 1

In settings.py just do this.
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 100
}
And then while yielding each request do this
yield Request(meta={'proxy': "http://%s"%(random.choice(["IP:PORT", "IP:PORT"]))})
That's it!

Answer 2

You don't need to write one. HttpProxyMiddleware already exists in Scrapy.
As docs state, there are two ways of letting Scrapy know you need your requests to go through a proxy:
Setting environment variables
(e.g. from the command line)
export http_proxy="http://username:password#host:port"
You can also set the meta key "proxy" per-request, to a value like http://some_proxy_server:port or http://username:password#some_proxy_server:port.
Keep in mind this value will take precedence over http_proxy/https_proxy environment variables, and it will also ignore no_proxy environment variable
e.g.
yield Request("http://google.com", meta={'proxy':'http://username:password#some_proxy_server:port'}, callback=self.some_method)