Scrapy behavior changes when download delay is set - scrapy

Question

Here is my scrapy spider
class Spider(scrapy.Spider):
name = "name"
start_urls = ["https://www.aurl"]
def parse(self, response):
links_page_urls = response.css("a.dynamic-linkset::attr(href)").extract()
for url in contract_page_urls:
yield response.follow(url, callback=self.parse_list_page)
next_data_cursor = response.css("li.next").css("a::attr(href)").extract_first()
if next_data_cursor:
self.log("going to next page - {}".format(next_data_cursor))
yield response.follow(next_data_cursor, callback=self.parse)
def parse_list_page(self, response):
list = response.css("div.row div.list-group a.list-group-item").css("a::attr(href)").extract()
for url in list:
self.log("url - {}".format(url))
yield scrapy.Request(url=self.base_url + url, callback=self.parse_page)
def parse_page(self, response):
#Lots of code for parsing elements from a page
# build an item and return it
My observations is that on my own machine at home and with no download delay set, the actual pages are visited in rapid succession and saved to mongo. When I move this code to an EC2 instance and set the download delay to 60, what I now notice is that the web pages aren't visited for scraping and instead the first page is visited, a next data token is scraped and it's visited. Then I see a lot of print outs related to scraping the list pages rather than each individual page.
The desired behavior is to visit the initial URL, get the page list, and then visit each page and scrape it, then move onto the next data cursor and repeat this process.

Answer 1

You could try to set a much lower DOWNLOAD_DELAY (probably= 2) and also set CONCURRENT_REQUESTS = 1 assigning a higher priority to the requests from the page list, something like:
yield response.follow(url, callback=self.parse_list_page, priority=1)