Traceback (most recent call last):
File "C:\Users\xin\Desktop\spider_thief\venv\lib\site-packages\scrapy\utils\defer.py", line 102, in iter_errback
yield next(it)
GeneratorExit
Exception ignored in:
RuntimeError: generator ignored GeneratorExit
Unhandled error in Deferred:
2018-07-17 17:50:00 [twisted] CRITICAL: Unhandled error in Deferred:
2018-07-17 17:50:00 [twisted] CRITICAL:
Traceback (most recent call last):
File "C:\Users\xin\Desktop\spider_thief\venv\lib\site-packages\twisted\internet\task.py", line 517, in _oneWorkUnit
result = next(self._iterator)
File "C:\Users\xin\Desktop\spider_thief\venv\lib\site-packages\scrapy\utils\defer.py", line 63, in
work = (callable(elem, *args, **named) for elem in iterable)
File "C:\Users\xin\Desktop\spider_thief\venv\lib\site-packages\scrapy\core\scraper.py", line 183, in _process_spidermw_output
self.crawler.engine.crawl(request=output, spider=spider)
File "C:\Users\xin\Desktop\spider_thief\venv\lib\site-packages\scrapy\core\engine.py", line 210, in crawl
self.schedule(request, spider)
File "C:\Users\xin\Desktop\spider_thief\venv\lib\site-packages\scrapy\core\engine.py", line 216, in schedule
if not self.slot.scheduler.enqueue_request(request):
File "C:\Users\xin\Desktop\spider_thief\venv\lib\site-packages\scrapy_redis\scheduler.py", line 167, in enqueue_request
self.queue.push(request)
File "C:\Users\xin\Desktop\spider_thief\venv\lib\site-packages\scrapy_redis\queue.py", line 99, in push
data = self._encode_request(request)
File "C:\Users\xin\Desktop\spider_thief\venv\lib\site-packages\scrapy_redis\queue.py", line 43, in _encode_request
return self.serializer.dumps(obj)
File "C:\Users\xin\Desktop\spider_thief\venv\lib\site-packages\scrapy_redis\picklecompat.py", line 14, in dumps
return pickle.dumps(obj, protocol=-1)
RecursionError: maximum recursion depth exceeded while calling a Python object
maximum recursion depth exceeded
你的递归调用层数太多了，超过了python预定的限制，所以异常了。
你要设置一下你的爬虫递归调用的层数，超过一定层数就返回，不要继续递归了
看下你的遍历有没有循环，比如页面a指向页面b，页面b又有链接指向了a，那么，就会造成无限递归调用了。
你可以每次爬取以前，先判断下这个页面有没有爬过。