Scrapy output only the last incrementally updated item - scrapy

Question

Can someone help me with this please, I have been searching for this information for 2 days, no luck.
I have an item with 1 field as a list of another items. The spider works fine, but in the output file I get all the lines of this item.
For example, I need json to be printed as:
{"id": "AAAA", "details": [
{"date" : "2013-01-10", type="A"},
{"date" : "2013-02-10", type="B"},
{"date" : "2013-03-10", type="C"},
{"date" : "2013-04-10"}, type="D"]}
but I get:
{"id": "AAAA", "details": [
{"date" : "2013-01-10", type="A"}]}
{"id": "AAAA", "details": [
{"date" : "2013-01-10", type="A"},
{"date" : "2013-02-10", type="B"}]}
{"id": "AAAA", "details": [
{"date" : "2013-01-10", type="A"},
{"date" : "2013-02-10", type="B"},
{"date" : "2013-03-10", type="C"}
]}
{"id": "AAAA", "details": [
{"date" : "2013-01-10", type="A"},
{"date" : "2013-02-10", type="B"},
{"date" : "2013-03-10", type="C"},
{"date" : "2013-04-10"}, type="D"]}
I use a function to update my parent item:
def rePackIt(parent, item):
if 'details' in parent:
items = parent.get('details')
else:
items = []
items.append(dict(item))
parent['details'] = items
return parent
In parse function I do:
parent = ParentItem()
parent['id'] = self.param # actually I parse a text file with many IDs
parent['details'] = []
yield FormRequest.from_response(response,
formname='...',
formdata={'...':'...', '...': parent['id'],
'...':''},
meta = {'parent': parent, 'dont_merge_cookies': True},
callback=self.parse1)
def parse1(self, response):
parent = response.meta['parent']
sel = HtmlXPathSelector(response)
records = sel.select('//ul[#class="...."]')
for record in records:
item = DetailItem()
item['type'] = record.select('child...')
doc_link = record.select('child.../a/#href').extract()
yield Request(doc_link,
callback=self.parse2,
method='GET',
headers={...},
meta={'dont_merge_cookies': True, 'cookiejar': cookieJar, 'item' : item, 'parent' : parent}
)
def parse2(self, response):
item = response.meta['item']
parent = response.meta['parent']
sel = HtmlXPathSelector(response)
# some other parsing code
item['date'] = cell.select('span[1]/text()[1]').extact()
rePackIt(parent, item)
return parent

Answer 1

The page you are trying to scrap and output as json has this structure
MainItem 1 {some information }
Detail Item 1
Detail Item 2
Main Item 2
Detail Item 1
Detail Item 2
You are returning the parent object for each of the detail item scrapped. While your intention is to return the parent object only once, after it is "complete". Meaning your parent is populated with all the detailed item 1..n. The problem is you don't have a nicer way to say when you finished building the parent item.
One of way to handle this would be writing the pipeline(http://doc.scrapy.org/en/latest/topics/item-pipeline.html). This might sound complicated but its not.
Basically, there is three steps in the pipeline
open_spider
you create your global object of the form
itemlist = []
process_item
if item is parent then
add the item to the list
if item is child then
find the parentitem from the itemlist
parentitem["detail"].add(childitem)
close_spider
Write your json serialise and write to the desired file. One caveat with this is, if you are scrapping huge data, all the scraped item will live in memory, until you write them to the file in this method, as you won't be able to stream write your json items.
Let me know if this works or did you find any better solution.