Scrapy not crawling pages allowed by LinkExtractor - scrapy

Question

I'm quite stuck and cannot figure out why my CrawlSpider cannot pickup and process relative links in HTML code.
Below is a cawlspider to which I can pass some params from the command line, like so:
scrapy crawl domain_spider -a url="https://djangosnippets.org/snippets/?page=15" -s ROBOTSTXT_OBEY=0 -s AUTOTHROTTLE_ENABLED=0
The spider runs and absolutely refuses to crawl the paginated pages in the list. The HTML looks like this:
< Previous 20
If I pass in params from the commans like -a allowed="page=" then it will pickup two more pages, but it still refuses to continue.
Can anyone spot the problem in my code below?
My CrawlSpider:
def __init__(self, url=None, category='default', allowed=None, denied=None, single_page=False, **kwargs):
self.category = category
if allowed == '':
allowed = None
if denied == '':
denied = None
if single_page is not False and single_page != '':
denied = '.*'
self.start_urls = ['{}'.format(url)]
self.allowed_domains = [urlparse(url).netloc]
self.domain = urlparse(url).netloc
self.rules = (
Rule( LinkExtractor(allow=allowed, deny=denied, unique=True), callback='parse_page' ),
)
super(DomainSpider, self).__init__(**kwargs)

Answer 1

OK, so I'm glad I was able to answer this so quickly for myself.
If using CrawlSpider with Rules and you define a callback (as I have), then you must add follow=True to the Rule if you want it to continue to spider pages. Otherwise you will need to yield requests from your callback. Therefore, below is the before/after of the fix which will make CrawlSpider follow links.
Before
Rule(
LinkExtractor(allow=allowed, deny=denied, unique=True),
callback='parse_page'
),
After
Rule(
LinkExtractor(allow=allowed, deny=denied, unique=True),
callback='parse_page',
follow=True
),