Scrappy: Extracting data from sub pages - scrapy

Question

I'm trying to extract price details from such web links using scrapy. When I select each color browser sends a new ajax request to the server. eg for color Vert cèdre.
import scrapy
class TestSpider(scrapy.Spider):
name = "Test"
def start_requests(self):
urls = [
'https://www.alinea.com/fr-fr/p/vence-canape-1.5-places-fixe-en-lin-vert-cedre-26943589.html',
'... other URL's',
]
for url in urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
# extract each color and color url
# hit each color page url to get pricing details
Scrappy gives me contents of the main page in the parse function. My question is how I can hit the sub page links (for colors) and extract the contents from it in the parse method so I can get the pricing detail for each color in a single object.
eg
{
'url': 'https://www.alinea.com/fr-fr/p/vence-canape-1.5-places-fixe-en-lin-vert-cedre-26943589.html'
'pricing': [{
'color': 'Beige roucas',
'price': '699,00 €'
},{
'color': 'Blanc capelan',
'price': '699,00 €'
}
....... other colors
]
}
If I yield color page url's from parse method as new requests, how can I merge pricing to get above structure.

Answer 1

After going over some links, doesn't seem like the prices changes with color (if I'm wrong then, you'd probably use CrawlSpider with rules or Splash together with Scrapy for a more robust spider).
But for now, for the color, their respective links etc., you could try the parse function below. Edit accordingly.
def parse(self, response):
url = response.url
price = response.xpath('//*[#class="product-price product-pricing"]/div/span/text()').get().strip()
prices = []
# Selector for the color options
color_list = response.xpath('//*/li[#class="attribute attr-color "]/div[#class="value full-line"]/select/option')
# Check if that selector exists &
# Cycle through colors adding the data
if color_list:
for color_data in color_list:
prices.append({
'color': color_data.xpath('#data-title').get().rsplit(':: ')[1],
'price': price,
'link': color_data.xpath('#data-link').get()
})
yield {
'url': url,
'pricing': prices,
}