Scrapy only scrapes first 4 starting urls - scrapy

Question

Here is my spider
class Spider(scrapy.Spider):
name = "spider"
start_urls = []
with open("clause/clauses.txt") as f:
for line in f:
start_urls(line)
base_url = "<url>"
start_urls = [base_url + "-".join(url.split()) for url in start_url]
def start_requests(self):
self.log("start_urls - {}".format(self.start_urls))
for url in self.start_urls:
yield scrapy.Request(url, dont_filter=True, priority=2, callback=self.parse)
def parse(self, response):
text_items = response.css("some css").extract()
for text in text_items:
if text == "\n":
continue
yield Item({"text" : text})
yield response.follow(response.css("a::attr(href)").extract_first(), callback=self.parse)
There are 20 start urls, yet Im noticing that only the first 4 urls are actually being called and the rest aren't ever executed. The ideal behavior would be for scrappy to first call all 20 start urls, and then from each continue to the next.

Answer 1

Looks like you have a typo:
start_urls = [base_url + "-".join(url.split()) for url in start_url]
should probably be:
start_urls = [base_url + "-".join(url.split()) for url in start_urls]
Notice the missing s in start_urls.
And I suspect this:
with open("clause/clauses.txt") as f:
for line in f:
start_urls(line)
should be:
with open("clause/clauses.txt") as f:
for line in f:
start_urls.append(line)