Scrapy finished instantly after start - scrapy

Question

I dont know why, but my scrapy is most times instantly stop crawling after it starts.
Sometimes he work with crawling, but most times he start, get 200 Page and instant stopped.
2019-11-08 08:15:41 [scrapy.core.engine] INFO: Spider opened
2019-11-08 08:15:41 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-11-08 08:15:41 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6024
2019-11-08 08:15:41 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 0, unchecked: 1011, reanimated: 0, mean backoff time: 0s)
2019-11-08 08:15:41 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36
2019-11-08 08:15:41 [rotating_proxies.expire] INFO: Getting an unchecked proxy: http://xxxx:xxxxxx#xxxxxxxxxxxx:8080
2019-11-08 08:15:42 [rotating_proxies.expire] DEBUG: Proxy <http://xxxx:xxxx#xxxxxx:8080> is GOOD
2019-11-08 08:15:42 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.de/Produkte-303000031/s?me=A8KICS1PHF7ZO&rh=n%3A303001031> (referer: None)
2019-11-08 08:15:42 [scrapy.core.engine] INFO: Closing spider (finished)
2019-11-08 08:15:42 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 398,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 2812,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'elapsed_time_seconds': 1.356492,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2019, 11, 8, 7, 15, 42, 637300),
'log_count/DEBUG': 3,
'log_count/INFO': 12,
'log_count/WARNING': 42,
'memusage/max': 59146240,
'memusage/startup': 59146240,
'proxies/good': 1,
'proxies/mean_backoff': 0.0,
'proxies/reanimated': 0,
'proxies/unchecked': 1011,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2019, 11, 8, 7, 15, 41, 280808)}
2019-11-08 08:15:42 [scrapy.core.engine] INFO: Spider closed (finished)
I dont see any error or something.
And as i say, sometimes it is working.
If i do that a few times, it will work on some try
That the code is working, i can see on the trys when he is not stopping instantly...
I have the same crawler for another site, that has not the Problem. Or is that a problem on amazon?

Answer 1

Can you share start_requests/parser functions here?
One reason for such behavior could be that target website knows you are a crawler and the server sent a blank/wrong page with 200 return code.
Trying importing:
from scrapy.utils.response import open_in_browser
and use:
def myfunction(self, response):
open_in_browser(response)
In order to see what the crawler see.
or try random user agents with other random headers.

Answer 2

That is just error logs. We need to see a little bit of python code to better help you.
or maybe its the command you are running if you are running your spiders.
Looking at your error logs there doesn't seem to be any syntax issues in python.
But your spider crawled at least 1 page in amazon.
Also try running
scrapy shell http://URL
Then you can debug your code. line by line in there.