Easier way to follow links with Scrapy - scrapy

Question

I have the following code in a scrapy spider:
class ContactSpider(Spider):
name = "contact"
# allowed_domains = ["http://www.domain.com/"]
start_urls = [
"http://web.domain.com/DECORATION"
]
BASE_URL = "http://web.domain.com"
def parse(self, response):
links = response.selector.xpath('//*[contains(#class,"MAIN")]/a/#href').extract()
for link in links:
absolute_url = self.BASE_URL + link
yield Request(absolute_url, headers= headers, callback=self.second)
I'm surprised there is not a simpler way in scrapy to follow links rather than build each absolute_url. Is there a a better way to do this?

Answer 1

For absolute urls you can use urlparse.urljoin, Response already has a shortcut for that via response.urljoin(link). So your code could easily be replaced by:
def parse(self, response):
links = response.selector.xpath('//*[contains(#class,"MAIN")]/a/#href').extract()
for link in links:
yield Request(response.urljoin(link), headers=headers, callback=self.second)
You can also use scrapy LinkExtractors which extract links according to some rules and manages all of the joining automatically.
from scrapy.linkextractors import LinkExtractor
def parse(self, response):
le = LinkExtractor(restrict_xpaths='//*[contains(#class,"MAIN")]/a/#href')
links = le.extract_links(response)
for link in links:
yield Request(link.url, headers= headers, callback=self.second)
Regarding more automated crawling experience - scrapy has CrawlSpider which uses set of rules to extract and follow links on each page. You can read about it more here: http://doc.scrapy.org/en/latest/topics/spiders.html#crawlspider
The docs have some examples of it as well.