2021-07-07 09:50:44 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.qiushibaike.com/robots.txt>
(failed 3 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
2021-07-07 09:50:44 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading <GET https://www.qiushibaike.com/robots
.txt>: [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
Traceback (most recent call last):
File "c:\users\qyc\appdata\local\programs\python\python38-32\lib\site-packages\scrapy\core\downloader\middleware.py", line
44, in process_request
return (yield download_func(request=request, spider=spider))
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connect
ion was closed cleanly.>]
2021-07-07 09:50:45 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.qiushibaike.com/text/> (fai
led 3 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
2021-07-07 09:50:45 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.qiushibaike.com/text/>
Traceback (most recent call last):
File "c:\users\qyc\appdata\local\programs\python\python38-32\lib\site-packages\scrapy\core\downloader\middleware.py", line
44, in process_request
return (yield download_func(request=request, spider=spider))
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connect
ion was closed cleanly.>]
源码是对糗事百科文章的作者和内容的爬取:
蜘蛛也得张网,才能捕到虫子——民谚