Problems while trying to crawl with scrapy/splash? - scrapy

Question

I am interested in crawl a website, and at first instance I would like to get the links from this section of the website:
<div id="lo-mas-visto">
<div class="item seccion-nacion">
<span class="fecha">04:21</span>
<span class="title">
Tras videoescándalo cae candidata de Morena
</span>
</div>
<div class="item seccion-nacion">
<span class="fecha">02:01</span>
<span class="title">
</div>
<div class="item seccion-mundo">
<span class="fecha">03:20</span>
<span class="title">
Trump se rinde; no incluye muro en presupuesto
</span>
</div>
<div class="item seccion-nacion">
<span class="fecha">02:02</span>
<span class="title">
AMLO y el dinero sucio
</span>
</div>
<div class="item seccion-nacion">
<span class="fecha">02:06</span>
<span class="title">
Morena y los nuevos videoescándalos
</span>
</div>
<div class="item seccion-metropoli">
<span class="fecha">01:09</span>
<span class="title">
Delfina Gómez y su equipo se premian con bonos
</span>
</div>
<div class="item seccion-metropoli">
<span class="fecha">01:06</span>
<span class="title">
Van tras mujer que prostituye a niñas de secundaria y prepa
</span>
Hence, I tried to implement a spider for pulling such info:
# -*- coding: utf-8 -*-
import scrapy
from scrapy_splash import SplashRequest
class OpinionsSpider(scrapy.Spider):
name = "news"
allowed_domains = ["http://www.eluniversal.com.mx/"]
start_urls = ['http://www.eluniversal.com.mx/']
def start_requests(self):
for url in self.start_urls:
yield SplashRequest(url, self.parse, args={'wait': 0.5})
def parse(self, response):
#print(response.body)
item = {
'url' : response.xpath(".//*[#id='lo-mas-visto']//div//span//a")
}
yield item
The problem is that I am not getting the urls from the above object:
user#MacBook-Pro-de-User-3:~/PycharmProjects$ scrapy runspider opinions.py
2017-04-27 17:09:28 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: scrapybot)
2017-04-27 17:09:28 [scrapy.utils.log] INFO: Overridden settings: {'SPIDER_LOADER_WARN_ONLY': True}
2017-04-27 17:09:28 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
2017-04-27 17:09:29 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-04-27 17:09:29 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-04-27 17:09:29 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-04-27 17:09:29 [scrapy.core.engine] INFO: Spider opened
2017-04-27 17:09:29 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-04-27 17:09:29 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-04-27 17:09:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.eluniversal.com.mx/> (referer: None)
2017-04-27 17:09:29 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.eluniversal.com.mx/>
{'url': []}
2017-04-27 17:09:29 [scrapy.core.engine] INFO: Closing spider (finished)
2017-04-27 17:09:29 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 220,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 53389,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2017, 4, 27, 22, 9, 29, 901099),
'item_scraped_count': 1,
'log_count/DEBUG': 3,
'log_count/INFO': 7,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2017, 4, 27, 22, 9, 29, 143023)}
2017-04-27 17:09:29 [scrapy.core.engine] INFO: Spider closed (finished)
How can I fix the spider in order to get the links, and start a pagination scheme?...

Answer 1

From the comments, I think your problem is that you don't have a scrapy project and splash is not correctly configured. Your spider does not render the JavaScript, so you can't acces the links you want.
Also your xpath is missing /#href at the end if you want to extract links.
Here is a very usefull tool if you want to test your xpaths: videlibri.
You first need to make a scrapy project (startproject).
Then you need to configure correctly splash so you can render the JavaScript from the website and get the links you want. Here is the documentation you need: scrapy_splash
Hope this helps.