Scrapy - Extract data from table - scrapy

Question

I am trying to fetch data from a table into separate fields of a CSV file.
The table in the website looks like this:
And (part of) the source of the webpage looks like this:
<div id="right">
<div id="rightwrap">
<h1>Krimpen aan den IJssel</h1>
<div class="tools">
print
terug
</div>
<h2 class="lijst">Krimpen aan den IJssel</h2>
<div class="dotted"> </div>
<div class="zoekres">
<h4>Aantal kindplaatsen</h4>
<div class="paratable">
<table cellpadding="0" cellspacing="0">
<tr>
<th> </th>
<th>2006</th>
<th>2008</th>
<th>2009</th>
<th>2010</th>
<th>2011</th>
</tr>
<tr>
<th>KDV</th>
<td>144</td>
<td>144</td>
<td>174</td>
<td>243</td>
<td>-</td>
</tr>
<tr>
<th>BSO</th>
<td>135</td>
<td>265</td>
<td>315</td>
<td>365</td>
<td>-</td>
</tr>
<tr>
<th>Totaal</th>
<td>279</td>
<td>409</td>
<td>489</td>
<td>608</td>
<td>-</td>
</tr>
</table>
</div>
</div>
</div>
</div>
<div class="brtotal"> </div>
</div>
I managed to retrieve the name of the place "Krimpen aan den IJssel" using this code:
def parse(self, response):
item = OrderedDict()
for col in self.cols:
item[col] = 'None'
item['Gemeente'] = response.css('h2.lijst::text').get('')
yield item
But I am unable to retrieve the values displayed in the table of this website. The standard approach for table using:
response.xpath('//*[#class="table paratable"]
doesn't seem to work or I am not experienced enough to set the parameters right.
Can anyone provide me with some lines of code that will bring the
values from this table into the following columns of my CSV-file
KDV_2006 KDV_2008 KDV_2009 KDV_2010 KDV_2011 BSO_2006 BSO_2008
BSO_2009 BSO_2010 BSO_2011

Answer 1

One possible way:
result = {}
years = response.xpath('//div[#class="paratable"]/table/tr[1]/th[position() > 1]/text()').getall()
for row in response.xpath('//div[#class="paratable"]/table/tr[position() > 1][position() < last()]'):
field_name = row.xpath('./th/text()').get()
values = row.xpath('./td/text()').getall()
for year, value in zip(years, values):
result[f'{field_name}_{year}'] = value