CrawlSpider: Ignore URL before request - scrapy

Question

I have a CrawlSpider derived spider. When the url has a certain format, it does a callback to a function named parse_item.
rules = (
Rule(
LinkExtractor(
allow=('/whatever/', )
)
),
Rule(
LinkExtractor(
allow=('/whatever/detailpage/1234/')
),
callback='parse_item'
),
)
I have a status only_new=True for my spider. When this state is enabled, I don`t want to crawl urls which are already in my database.
I would like to check the url BEFORE the request is done, because when I have 5 new detailpages I want to crawl but 1000 detailpages I don`t want to crawl, i want to send 5 requests instead of 1000.
But in the callback funtion, the request has already been done. I would like to do something like following:
rules = (
(...)
Rule(
LinkExtractor(
allow=('/whatever/detailpage/1234/')
),
callback_before_request='check_if_request_is_nessesary'
),
)
def check_if_request_is_nessesary(spider, url):
if spider.only_new and url_exists_in_database():
raise IgnoreRequest
else:
do_request_and_call_parse_item(url)
Is this possible with a middleware or something?

Answer 1

You're looking for the process_links attribute for the Rule -- it allows you to specify a callable or a method name to be used for filtering the list of Link objects returned by the LinkExtractor.
Your code would look something like this:
rules = (
(...)
Rule(
LinkExtractor(
allow=('/whatever/detailpage/1234/')
),
process_links='filter_links_already_seen'
),
)
def filter_links_already_seen(self, links):
for link in links:
if self.only_new and url_exists_in_database(link.url):
continue
else:
yield link