How to customize URI in Scrapy with non built in storage URI parameters - scrapy

Question

I want to customize the Scrapy feed URI to s3 to include the dimensions of the uploaded file. Currently I have the following in settings.py file:
FEEDS = {
's3://path-to-file/file_to_have_dimensions.csv': {
'format': 'csv',
'encoding': 'utf8',
'store_empty': False,
'indent': 4,
}
}
But would like to have something like the following:
NUMBER_OF_ROWS_IN_CSV = file.height()
FEEDS = {
f's3://path-to-files/file_to_have_dimensions_{NUMBER_OF_ROWS_IN_CSV}.csv': {
'format': 'csv',
'encoding': 'utf8',
'store_empty': False,
'indent': 4,
}
}
Note that I would like the number of rows to be inserted automatically.
Is this possible to do this solely through changing settings.py, or is it required to change other parts of the scrapy code?

Answer 1

The feed file is created when the spider starts running at which point the number of items is not yet know. However, when the spider finishes running, it calls a method named closed from which you can access the spider stats, settings and also you can perform any other tasks that you want to run after the spider has finished scraping and saving items.
In the case below i renamed the feed file from intial_file.csv to final_file_{item_count}.csv.
As you cannot rename files in s3,I use the boto3 library to copy the initial_file to a new file and name it with the item_count value included in the file name and then delete the initial file.
import scrapy
import boto3
class SampleSpider(scrapy.Spider):
name = 'sample'
start_urls = [
'http://quotes.toscrape.com/',
]
custom_settings = {
'FEEDS': {
's3://path-to-file/initial_file.csv': {
'format': 'csv',
'encoding': 'utf8',
'store_empty': False,
'indent': 4,
}
}
}
def parse(self, response):
for quote in response.xpath('//div[#class="quote"]'):
yield {
'text': quote.xpath('./span[#class="text"]/text()').extract_first(),
'author': quote.xpath('.//small[#class="author"]/text()').extract_first(),
'tags': quote.xpath('.//div[#class="tags"]/a[#class="tag"]/text()').extract()
}
def closed(self, reason):
item_count = self.crawler.stats.get_value('item_scraped_count')
try:
session = boto3.Session(aws_access_key_id = 'awsAccessKey', aws_secret_access_key = 'awsSecretAccessKey')
s3 = session.resource('s3')
s3.Object('my_bucket', f'path-to-file/final_file_{item_count}.csv').copy_from(CopySource = 'my_bucket/path-to-file/initial_file.csv')
s3.Object('my_bucket', 'path-to-file/initial_file.csv').delete()
except:
self.logger.info("unable to rename s3 file")