How can I use a random user agent whenever I send a request? - scrapy

Question

i know how to use random (fake) user agent in scrapy. but after i run scrapy. i could see only one random user agent on terminal. so i guessed maybe 'settings.py' run only one time when i run scrapy. if scrapy work really like this and send 1000 request to some web page to collect 1000 data, scrapy will just send same user agent. Surely it can be easy to get ban i think.
can you tell me how can i send random user agent when scrapy send request to some website?
i used this lib(?) in my scrapy project.
after i set faker in user-agent in settings.py
https://pypi.org/project/Faker/
from faker import Faker
fake = Faker()
Faker.seed(fake.random_number())
fake_user_agent = fake.chrome()
USER_AGENT = fake_user_agent
in settings.py i wrote like this. can it work well ??

Answer 1

If you are setting USER_AGENT in your settings.py like in your question then you will just get a single (random) user agent for your entire crawl.
You have a few options if you want to set a fake user agent for each request.
Option 1: Explicitly set User-Agent per request
This approach involves setting the user-agent in the headers of your Request directly. In your spider code you can import Faker like you do above but then call e.g. fake.chrome() on every Request. For example
# At the top of your file
from faker import Faker
# This can be a global or class variable
fake = Faker()
...
# When you make a Request
yield Request(url, headers={"User-Agent": fake.chrome()})
Option 2: Write a middleware to do this automatically
I won't go into this because you might as well use one that already exists
Option 3: Use an existing middleware to do this automatically (such as scrapy-fake-useragent)
If you have lots of requests in your code option 1 isn't so nice, so you can use a Middleware to do this for you. Once you've installed scrapy-fake-useragent you can set it up in your settings file as described on the webpage
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
'scrapy.downloadermiddlewares.retry.RetryMiddleware': None,
'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
'scrapy_fake_useragent.middleware.RetryUserAgentMiddleware': 401,
}
FAKEUSERAGENT_PROVIDERS = [
'scrapy_fake_useragent.providers.FakeUserAgentProvider',
'scrapy_fake_useragent.providers.FakerProvider',
'scrapy_fake_useragent.providers.FixedUserAgentProvider',
]
Using this you'll get a new user-agent per Request and if a Request fails you'll also get a new random user-agent. One of the key parts of setting this up is FAKEUSERAGENT_PROVIDERS. This tells us where to get the User-Agent from. They are tried in the order they are defined, so the second will be tried if the first one fails for some reason (if getting the user-agent fails, not if the Request fails). Note that if you want to use Faker as the primary provider, then you should put that one first in the list
FAKEUSERAGENT_PROVIDERS = [
'scrapy_fake_useragent.providers.FakerProvider',
'scrapy_fake_useragent.providers.FakeUserAgentProvider',
'scrapy_fake_useragent.providers.FixedUserAgentProvider',
]
There are other configuration options (such as using a random chrome-like user-agent, listed in the scrapy-fake-useragent docs.
Example spider
Here is an example spider. For convenience I set the settings inside the spider, but you can put these into your settings.py file.
# fake_user_agents.py
from scrapy import Spider
class FakesSpider(Spider):
name = "fakes"
start_urls = ["http://quotes.toscrape.com/"]
custom_settings = dict(
DOWNLOADER_MIDDLEWARES={
"scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": None,
"scrapy.downloadermiddlewares.retry.RetryMiddleware": None,
"scrapy_fake_useragent.middleware.RandomUserAgentMiddleware": 400,
"scrapy_fake_useragent.middleware.RetryUserAgentMiddleware": 401,
},
FAKEUSERAGENT_PROVIDERS=[
"scrapy_fake_useragent.providers.FakerProvider",
"scrapy_fake_useragent.providers.FakeUserAgentProvider",
"scrapy_fake_useragent.providers.FixedUserAgentProvider",
],
)
def parse(self, response):
# Print out the user-agent of the request to check they are random
print(response.request.headers.get("User-Agent"))
next_page = response.css("li.next a::attr(href)").get()
if next_page is not None:
yield response.follow(next_page, self.parse)
Then if I run this with scrapy runspider fake_user_agents.py --nolog the output is
b'Mozilla/5.0 (Macintosh; PPC Mac OS X 10 11_0) AppleWebKit/533.1 (KHTML, like Gecko) Chrome/59.0.811.0 Safari/533.1'
b'Opera/8.18.(Windows NT 6.2; tt-RU) Presto/2.9.169 Version/11.00'
b'Opera/8.40.(X11; Linux i686; ka-GE) Presto/2.9.176 Version/11.00'
b'Opera/9.42.(X11; Linux x86_64; sw-KE) Presto/2.9.180 Version/12.00'
b'Mozilla/5.0 (Macintosh; PPC Mac OS X 10 5_1 rv:6.0; cy-GB) AppleWebKit/533.45.2 (KHTML, like Gecko) Version/5.0.3 Safari/533.45.2'
b'Opera/8.17.(X11; Linux x86_64; crh-UA) Presto/2.9.161 Version/11.00'
b'Mozilla/5.0 (compatible; MSIE 5.0; Windows NT 5.1; Trident/3.1)'
b'Mozilla/5.0 (Android 3.1; Mobile; rv:55.0) Gecko/55.0 Firefox/55.0'
b'Mozilla/5.0 (compatible; MSIE 9.0; Windows CE; Trident/5.0)'
b'Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10 11_9; rv:1.9.4.20) Gecko/2019-07-26 10:00:35 Firefox/9.0'