How to merge items with same identifier from different urls - scrapy

Question

I am currently crawling multiple product-sites from an Excel-document.
The document looks basically like this:
ID URL1 URL2 URL3
01 abc.com/1 def.com/1 ghi.com/1
02 abc.com/2 def.com/2 ghi.com/2
03 abc.com/3 def.com/3 ghi.com/3
My spider now takes the ID and the url and, since the sites are different, yields a request to the corresponding parse-function and passes the ID to each function.
The problem now is that my csv-output lists each ID multiple times, like this:
ID PriceURL1 PriceURL2 PriceURL3
01 xx.xx
01 xx.xx
01 xx.xx
02 xx.xx
02 xx.xx
...
Is there a way to merge the items, so that each ID has the prices gathered from all the different parse-functions?
So far I think my options would be:
Implement a scraping order, so that the spider crawls URL1 first, URL2 second and so on, yielding a request in each parse_urlX-function, but I don't know if this is really an elegant and resource-friendly way, as I'd have to read the urls from the document each time.
Crawl it like it is, and merge the scraped items in the output-csv-file after the spider is done scraping.
What I think might be the best way, work with an item pipeline, so that it adds the ID to a set and subsequently checks if the currently scraped ID has already been scraped and if yes, adds the current price to the item.
I tried 3. but until now, I can't figure out how to access previously scraped items and update them.
What would be the best way to achieve this?

Answer 1

In my opinion the best approach is to process the items afterwards. The item pipeline is not designed to hold items and merge items, you can filter out but not merge them in an elegant way.
Aggregating the output is straightforward:
import json
from collections import defaultdict
groups = defaultdict(list)
for line in open("items.jl"):
item = json.loads(line)
key = get_key(item) # a function that returns a key for an item
groups[key].append(item)
for items in groups.values():
merged = merge_items(items) # a function that returns a single item
# store the merged item somewhere.
...
If your data is large enough that doesn't fit memory, I'd suggest to post a new question and you will get plenty of answers.