How to handle error 302 in the scrapy shell - scrapy

Question

I' trying to scrape a page I'm getting redirected, I try setting an user agent but it didn't work either.
I saw this in other question:
meta = {'dont_redirect': True,'handle_httpstatus_list': [302]}
How can I test that in the scrapy shell?

Answer 1

When using scrapy shell, the simplest way is probably to just disable RedirectMiddleware with a REDIRECT_ENABLED=0 setting on the command line.
Compare this, disabling redirections altogether:
$ scrapy shell -s REDIRECT_ENABLED=0
2016-02-09 10:16:27 [scrapy] INFO: Scrapy 1.0.4 started (bot: scrapybot)
2016-02-09 10:16:27 [scrapy] INFO: Optional features available: ssl, http11
2016-02-09 10:16:27 [scrapy] INFO: Overridden settings: {'REDIRECT_ENABLED': '0', 'LOGSTATS_INTERVAL': 0, 'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter'}
2016-02-09 10:16:30 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, CoreStats, SpiderState
2016-02-09 10:16:32 [scrapy] INFO: Enabled downloader middlewares:
HttpAuthMiddleware,
DownloadTimeoutMiddleware,
UserAgentMiddleware,
RetryMiddleware,
DefaultHeadersMiddleware,
MetaRefreshMiddleware,
HttpCompressionMiddleware,
CookiesMiddleware,
ChunkedTransferMiddleware,
DownloaderStats
2016-02-09 10:16:33 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-02-09 10:16:33 [scrapy] INFO: Enabled item pipelines:
2016-02-09 10:16:33 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-02-09 10:16:39 [root] DEBUG: Using default logger
(you can notice that RedirectMiddleware is not in the list of "Enabled downloader middlewares")
with the default :
$ scrapy shell
2016-02-09 10:17:18 [scrapy] INFO: Scrapy 1.0.4 started (bot: scrapybot)
2016-02-09 10:17:18 [scrapy] INFO: Optional features available: ssl, http11
2016-02-09 10:17:18 [scrapy] INFO: Overridden settings: {'LOGSTATS_INTERVAL': 0, 'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter'}
2016-02-09 10:17:19 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, CoreStats, SpiderState
2016-02-09 10:17:19 [scrapy] INFO: Enabled downloader middlewares:
HttpAuthMiddleware,
DownloadTimeoutMiddleware,
UserAgentMiddleware,
RetryMiddleware,
DefaultHeadersMiddleware,
MetaRefreshMiddleware,
HttpCompressionMiddleware,
RedirectMiddleware,
CookiesMiddleware,
ChunkedTransferMiddleware,
DownloaderStats
2016-02-09 10:17:19 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-02-09 10:17:19 [scrapy] INFO: Enabled item pipelines:
2016-02-09 10:17:19 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-02-09 10:17:19 [root] DEBUG: Using default logger