2021-03-28 22:10:19 [scrapy.utils.log] INFO: Scrapy 2.4.1 started (bot: demo1)
2021-03-28 22:10:19 [scrapy.utils.log] INFO: Versions: lxml 4.6.2.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.7.3 (default, Nov 27 2020, 09:27:44) - [GCC 8.3.0], pyOpenSSL 20.0.1 (OpenSSL 1.1.1d 10 Sep 2019), cryptography 2.6.1, Platform Linux-5.4.70-amd64-desktop-x86_64-with-Deepin-20.1-apricot
2021-03-28 22:10:19 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor
2021-03-28 22:10:19 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'demo1',
'DOWNLOAD_DELAY': 3,
'NEWSPIDER_MODULE': 'demo1.spiders',
'SPIDER_MODULES': ['demo1.spiders'],
'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, '
'like Gecko) Chrome/88.0.4324.182 Safari/537.36'}
2021-03-28 22:10:19 [scrapy.extensions.telnet] INFO: Telnet Password: 49b4c0e84af998cf
2021-03-28 22:10:19 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.logstats.LogStats']
2021-03-28 22:10:19 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2021-03-28 22:10:19 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2021-03-28 22:10:19 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2021-03-28 22:10:19 [scrapy.core.engine] INFO: Spider opened
2021-03-28 22:10:19 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2021-03-28 22:10:19 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2021-03-28 22:10:19 [scrapy.core.scraper] ERROR: Error downloading <GET http://python123.io/ws/demo.html>
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/usr/local/lib/python3.7/dist-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/home/ppddwm/.local/lib/python3.7/site-packages/scrapy/core/downloader/middleware.py", line 45, in process_request
return (yield download_func(request=request, spider=spider))
File "/home/ppddwm/.local/lib/python3.7/site-packages/scrapy/utils/defer.py", line 55, in mustbe_deferred
result = f(*args, **kw)
File "/home/ppddwm/.local/lib/python3.7/site-packages/scrapy/core/downloader/handlers/__init__.py", line 75, in download_request
return handler.download_request(request, spider)
File "/home/ppddwm/.local/lib/python3.7/site-packages/scrapy/core/downloader/handlers/http11.py", line 88, in download_request
return agent.download_request(request)
File "/home/ppddwm/.local/lib/python3.7/site-packages/scrapy/core/downloader/handlers/http11.py", line 355, in download_request
d = agent.request(method, to_bytes(url, encoding='ascii'), headers, bodyproducer)
File "/usr/local/lib/python3.7/dist-packages/twisted/web/client.py", line 1744, in request
parsedURI.originForm)
File "/usr/local/lib/python3.7/dist-packages/twisted/web/client.py", line 690, in originForm
(b'', b'', self.path, self.params, self.query, b''))
File "/usr/local/lib/python3.7/dist-packages/twisted/web/client.py", line 22, in urlunparse
result = _urlunparse(tuple([p.decode("charmap") for p in parts]))
File "/usr/local/lib/python3.7/dist-packages/twisted/web/client.py", line 22, in <listcomp>
result = _urlunparse(tuple([p.decode("charmap") for p in parts]))
LookupError: unknown encoding: charmap
2021-03-28 22:10:19 [scrapy.core.engine] INFO: Closing spider (finished)
2021-03-28 22:10:19 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 1,
'downloader/exception_type_count/builtins.LookupError': 1,
'downloader/request_bytes': 295,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'elapsed_time_seconds': 0.31267,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2021, 3, 28, 14, 10, 19, 789381),
'log_count/ERROR': 1,
'log_count/INFO': 10,
'memusage/max': 52699136,
'memusage/startup': 52699136,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2021, 3, 28, 14, 10, 19, 476711)}
2021-03-28 22:10:19 [scrapy.core.engine] INFO: Spider closed (finished)
#这是我的代码
import scrapy
class BaiduSpider(scrapy.Spider):
name = 'baidu'
allowed_domains = ['baidu.com']
start_urls = ['https://www.baidu.com/']
def parse(self, response):
print(response.text)# Scrapy settings for demo1 project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# https://docs.scrapy.org/en/latest/topics/settings.html
# https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html
BOT_NAME = 'demo1'
SPIDER_MODULES = ['demo1.spiders']
NEWSPIDER_MODULE = 'demo1.spiders'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36'
# Obey robots.txt rules
ROBOTSTXT_OBEY =False
# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32
# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16
# Disable cookies (enabled by default)
#COOKIES_ENABLED = False
# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False
# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
# 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
# 'Accept-Language': 'en',
#}
# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
# 'demo1.middlewares.Demo1SpiderMiddleware': 543,
#}
# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
# 'demo1.middlewares.Demo1DownloaderMiddleware': 543,
#}
# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
# 'scrapy.extensions.telnet.TelnetConsole': None,
#}
# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
#ITEM_PIPELINES = {
# 'demo1.pipelines.Demo1Pipeline': 300,
#}
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False
# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
基本没改，按照视频教程一步一步来的，是不是环境问题？
我跑一下看看先
你这个scrapy 代码不完整，完整发一下看看。另外，我这边有一个写好的scrapy代码你，可以参考一下。
链接：https://pan.baidu.com/s/1233q5vcCOKC_M7rPjjybiw
提取码：hpfu
你好，我是有问必答小助手。为了技术专家团更好地为您解答问题，烦请您补充下（1）问题背景详情，（2）您想解决的具体问题，（3）问题相关图片。便于技术专家团更好地理解问题，并给出解决方案。
您可以点击问题下方的【编辑】，进行补充修改问题。