Scrapy: How to add parameters to each request with start_requests? - scrapy

Question

I am using scrapy 2.1 where I scrapy content from category pages which are paginated. The default result set is 20 and I want to increase this to 1000 in order to request viewer pages.
e.g.
/category1
/category1?VIEW_SIZE=20&VIEW_INDEX=1
wanted:
/category1?VIEW_SIZE=1000&VIEW_INDEX=1
Rules:
rules = (
# parse all index pages
Rule(
LinkExtractor(
allow = ['^.*\/[a-z-0-9]+(\?VIEW_SIZE=20&VIEW_INDEX=\d{0,3}&filterBy=default&sortBy=default)?(?<![\d])$'],
# deny=[''],
restrict_xpaths=(['//aside[#id="aside-st"]', # navigation bar
'//ul[#class="page-lst"]/li[position()<last()]']), # include pagination
),
follow = True,
callback= 'parse_item'
),
)
Overriding start_requests to add parameters to each URL requested:
def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(url, meta={'VIEW_SIZE': 1000})
This does not work. How can I add to each URL I want to scrape the given parameters within the request?

Answer 1

This solved my problem of adding custom parameters to each request:
Add callback with the help of process_links to Rule:
rules = (
Rule(
LinkExtractor(
allow=['^.*\/c\/.*'],
# deny=[''],
restrict_xpaths=(['//li[contains(#class, "navigation--entry")]',
]),
),
process_links='link_filtering',
callback='parse_item',
follow=True
),
)
Add parameters to each URL
# gett max amount of results per category and add n=x results to url
def link_filtering(self, links):
for link in links:
link.url = "%s?p=1&n=1000" % link.url
return links