Scrapy finishing process before all pages are scraped - scrapy

Question

I sat up a test Scrapy scraper which looks like this:
import scrapy
class testSpider(scrapy.Spider):
name = 'test'
start_urls = ['https://www.realestate.com.kh/buy/']
def parse(self, response):
nr_pages = response.xpath('//div[#class="desktop-buttons"]/a[#class="css-1en2dru"]//text()').getall()
for nr in range(1, 40):
req_url = f'?page={nr}'
self.item = {}
self.item['page'] = nr
yield scrapy.Request(url=response.urljoin(req_url), callback=self.parse_page, meta={'item': self.item})
def parse_page(self, response):
page = response.meta['item']['page']
ads = response.xpath('//*[#class="featured css-ineky e1jqslr40"]//a/#href')
for url in ads:
absolute_url = response.urljoin(url.extract())
self.item = {}
self.item['page'] = page
yield scrapy.Request(absolute_url, callback=self.parse_ad, meta={'item': self.item})
def parse_ad(self, response):
page = response.meta['item']['page']
# DO THINGS
yield {
'page': page
}
It goes though loads each https://www.realestate.com.kh/buy/?page=NR, where nr is all numbers between 1 and 40
On each of these pages, it get all ads
On each ad of each page, it scraps things and yield them.
It work fine for the first 26 items (two first pages, and 2 or 3 items from the 3rd one, out of 40) and then finished the scraping without an error.
Here are the stats :
{'downloader/request_bytes': 23163,
'downloader/request_count': 66,
'downloader/request_method_count/GET': 66,
'downloader/response_bytes': 3801022,
'downloader/response_count': 66,
'downloader/response_status_count/200': 66,
'elapsed_time_seconds': 5.420036,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2020, 10, 22, 19, 48, 37, 549853),
'item_scraped_count': 26,
'log_count/INFO': 9,
'memusage/max': 49963008,
'memusage/startup': 49963008,
'request_depth_max': 2,
'response_received_count': 66,
'scheduler/dequeued': 66,
'scheduler/dequeued/memory': 66,
'scheduler/enqueued': 66,
'scheduler/enqueued/memory': 66,
'start_time': datetime.datetime(2020, 10, 22, 19, 48, 32, 129817)}
What could be ending the scraping so early?

Answer 1

Your spider is actually going through all the pages, the problem is that in parse_page the selector for ads only works in the earlier pages, in later pages the class name changes. The class name seems to be dynamically generated, so you need an XPath that won't select by class.
This XPath '//div/header/parent::div' would return the same div element that '//*[#class="featured css-ineky e1jqslr40"]' so replacing this line should allow you to select all ads from all pages:
ads = response.xpath('//div/header/parent::div/article/a/#href')
Unrelated note:
This isn't causing any problems yet, but it's a recipe for future problems.
for url in ads:
absolute_url = response.urljoin(url.extract())
self.item = {}
self.item['page'] = page
yield scrapy.Request(absolute_url, callback=self.parse_ad, meta={'item': self.item})
Scrapy works in an asynchronous way, so most of the time using an instance variable ( like self.item) gives the wrong intuition, as you don't really control the order in which the requests are parsed. That's why when you need to pass information between methods you use meta (or cb_kwargs) and not just store it in an instance variable.

Answer 2

You are working on the wrong way to get pages, there are 50 pages on the site right now. You should walk around by the next page. Look this code:
import scrapy
from scrapy.shell import inspect_response
class testSpider(scrapy.Spider):
name = 'test'
start_urls = ['https://www.realestate.com.kh/buy/']
def parse(self, response):
page = response.xpath('//div[#class="list"]//div[#class="desktop-buttons"]/a[#class="css-owq2hj"]/span/text()').get()
ads = response.xpath('//div[#class="list"]/div/header/a/#href').getall()
for url in ads:
yield scrapy.Request(response.urljoin(url), callback=self.parse_ad, meta={'page': page})
# next page
url = response.xpath('//div[#class="list"]//div[#class="desktop-buttons"]/a[#class="css-owq2hj"]/following-sibling::a[1]/#href').get()
if url:
yield scrapy.Request(response.urljoin(url), callback=self.parse)
def parse_ad(self, response):
page = response.meta['page']
# DO THINGS
yield {
'page': page
}