Scraping a list of urls after log in - scrapy

Question

The site to scrape has multiple projects with multiple pages and requires a log in. I tried:
def start_request(self):
return [scrapy.FormRequest(, callback=self.logged_in)]
def logged_in(self, response):
with open(...) as f:
for url in f.readlines():
return scrapy.Request(url, callback=self.parse)
def parse(self, response):
... do some scraping ...
... determine the url of the next page ...
return scrapy.Request(... next page ..., self.parse)
This results in scraping all pages of one project (login is successful), but then it stops.
If return scrapy.Request() in function logged_in() is replaced by yield scrapy.Request() than it reads the first page of all projects.
I played around with the returns and yields, but I can't get it to scrape all pages of all projects.
BTW I tried to create an array start_uls, but that doesn't work because it first needs to log into the site.

Answer 1

return will always return once, so don't expect more than one Request to be returned there, if you wish to return more than once, use yield
Scrapy Requests has a parameter called dont_filter that probably filter your calls to parse function
dont_filter (boolean) – indicates that this request should not be filtered by the scheduler. This is used when you want to perform an identical request multiple times, to ignore the duplicates filter. Use it with care, or you will get into crawling loops. Default to False.

Answer 2

#Guy Gavriely makes a good point about return. I would add that you need another parse method to filter all the pages you want:
def start_request(self):
return [scrapy.FormRequest(, callback=self.logged_in)]
def logged_in(self, response):
with open(...) as f:
for url in f.readlines():
yield scrapy.Request(url, callback=self.parse)
def parse(self, response):
... do some scraping ...
yield scrapy.Request(... next page ..., self.parse_user)
def parse_user(self, response):
... do some scraping ....
yield items
You're not done yet! The final parse method may need to be iterated for example:
def parse(self, response):
... do some scraping ...
for sel in (some_list_of_users_by_an_xpath):
user_profile_tag = response.xpath('xpath_to_user_profile_urls')
user_profile_url_clean = user_profile_tag(might need to clean prior to Request ie 'domain' + user_profile_tag or .split or .replace, etc.)
yield scrapy.Request(user_profile_url_clean, self.parse_user)
In this case the parse function will parse a user every time in that list of users. Then parse_user will do most of the actual digging and scraping. Once it's done it will return to the original parse method and just go to the next one from the list.
Good luck!