2021-04-19 09:02:17 [scrapy.utils.log] INFO: Scrapy 2.5.0 started (bot: maoyan)
2021-04-19 09:02:17 [scrapy.utils.log] INFO: Versions: lxml 4.6.3.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.7.8 (tags/v3.7.8:4b47a5b6ba,
Jun 28 2020, 08:53:46) [MSC v.1916 64 bit (AMD64)], pyOpenSSL 20.0.1 (OpenSSL 1.1.1k 25 Mar 2021), cryptography 3.4.7, Platform Windows-10-10.0.19041-SP0
2021-04-19 09:02:17 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2021-04-19 09:02:17 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'maoyan',
'NEWSPIDER_MODULE': 'maoyan.spiders',
'SPIDER_MODULES': ['maoyan.spiders']}
2021-04-19 09:02:17 [scrapy.extensions.telnet] INFO: Telnet Password: fdd9a3885297bf3a
2021-04-19 09:02:17 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
2021-04-19 09:02:17 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2021-04-19 09:02:17 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2021-04-19 09:02:17 [scrapy.middleware] INFO: Enabled item pipelines:
['maoyan.pipelines.MaoyanPipeline']
2021-04-19 09:02:17 [scrapy.core.engine] INFO: Spider opened
2021-04-19 09:02:17 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2021-04-19 09:02:17 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2021-04-19 09:02:17 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://verify.meituan.com/v2/web/general_page?action=spiderindefence&requestCode=ad75fb7f7
3f04814a785cf69dc0d008a&platform=1000&adaptor=auto&succCallbackUrl=https%3A%2F%2Foptimus-mtsi.meituan.com%2Foptimus%2FverifyResult%3ForiginUrl%3Dhttp%253A%252F%252Fm.maoyan.com%252Fmmd
b%252Fcomments%252Fmovie%252F248906.json%253F_v_%253Dyes%2526offset%253D0%2526startTime%253D2019-02-21%25252018%253A01%253A22> from <GET http://m.maoyan.com/mmdb/comments/movie/248906.
json?_v_=yes&offset=0&startTime=2019-02-21%2018:01:22>
2021-04-19 09:02:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://verify.meituan.com/v2/web/general_page?action=spiderindefence&requestCode=ad75fb7f73f04814a785cf69dc0d008a&pl
atform=1000&adaptor=auto&succCallbackUrl=https%3A%2F%2Foptimus-mtsi.meituan.com%2Foptimus%2FverifyResult%3ForiginUrl%3Dhttp%253A%252F%252Fm.maoyan.com%252Fmmdb%252Fcomments%252Fmovie%2
52F248906.json%253F_v_%253Dyes%2526offset%253D0%2526startTime%253D2019-02-21%25252018%253A01%253A22> (referer: None)
2021-04-19 09:02:18 [scrapy.core.scraper] ERROR: Spider error processing <GET https://verify.meituan.com/v2/web/general_page?action=spiderindefence&requestCode=ad75fb7f73f04814a785cf69
dc0d008a&platform=1000&adaptor=auto&succCallbackUrl=https%3A%2F%2Foptimus-mtsi.meituan.com%2Foptimus%2FverifyResult%3ForiginUrl%3Dhttp%253A%252F%252Fm.maoyan.com%252Fmmdb%252Fcomments%
252Fmovie%252F248906.json%253F_v_%253Dyes%2526offset%253D0%2526startTime%253D2019-02-21%25252018%253A01%253A22> (referer: None)
Traceback (most recent call last):
File "e:\learning-software\python\lib\site-packages\scrapy\utils\defer.py", line 120, in iter_errback
yield next(it)
File "e:\learning-software\python\lib\site-packages\scrapy\utils\python.py", line 353, in __next__
return next(self.data)
File "e:\learning-software\python\lib\site-packages\scrapy\utils\python.py", line 353, in __next__
return next(self.data)
File "e:\learning-software\python\lib\site-packages\scrapy\core\spidermw.py", line 56, in _evaluate_iterable
for r in iterable:
File "e:\learning-software\python\lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 29, in process_spider_output
for x in result:
File "e:\learning-software\python\lib\site-packages\scrapy\core\spidermw.py", line 56, in _evaluate_iterable
for r in iterable:
File "e:\learning-software\python\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 342, in <genexpr>
return (_set_referer(r) for r in result or ())
File "e:\learning-software\python\lib\site-packages\scrapy\core\spidermw.py", line 56, in _evaluate_iterable
for r in iterable:
File "e:\learning-software\python\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 40, in <genexpr>
return (r for r in result or () if _filter(r))
File "e:\learning-software\python\lib\site-packages\scrapy\core\spidermw.py", line 56, in _evaluate_iterable
for r in iterable:
File "e:\learning-software\python\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 58, in <genexpr>
return (r for r in result or () if _filter(r))
File "e:\learning-software\python\lib\site-packages\scrapy\core\spidermw.py", line 56, in _evaluate_iterable
for r in iterable:
File "D:\maoyan\maoyan\spiders\comment.py", line 52, in parse
json_data = json.loads(data)['cmts']
File "e:\learning-software\python\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "e:\learning-software\python\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "e:\learning-software\python\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2021-04-19 09:02:18 [scrapy.core.engine] INFO: Closing spider (finished)
2021-04-19 09:02:18 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 1015,
'downloader/request_count': 2,
'downloader/request_method_count/GET': 2,
'downloader/response_bytes': 8119,
'downloader/response_count': 2,
'downloader/response_status_count/200': 1,
'downloader/response_status_count/302': 1,
'elapsed_time_seconds': 0.699377,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2021, 4, 19, 1, 2, 18, 482625),
'httpcompression/response_bytes': 21862,
'httpcompression/response_count': 1,
'log_count/DEBUG': 2,
'log_count/ERROR': 1,
'log_count/INFO': 10,
'response_received_count': 1,
'scheduler/dequeued': 2,
'scheduler/dequeued/memory': 2,
'scheduler/enqueued': 2,
'scheduler/enqueued/memory': 2,
'spider_exceptions/JSONDecodeError': 1,
'start_time': datetime.datetime(2021, 4, 19, 1, 2, 17, 783248)}
2021-04-19 09:02:18 [scrapy.core.engine] INFO: Spider closed (finished)
JSONDecodeError，JSON格式数据没办法解析。
下面是代码不知道哪里出问题了
# -*- coding: utf-8 -*-
import scrapy
import random
from scrapy.http import Request
import datetime
import json
from maoyan.items import MaoyanItem
class CommentSpider(scrapy.Spider):
name = 'comment'
allowed_domains = ['maoyan.com']
uapools = [
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.163 Safari/535.1',
'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0) Gecko/20100101 Firefox/6.0',
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50',
'Opera/9.80 (Windows NT 6.1; U; zh-cn) Presto/2.9.168 Version/11.50',
'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; InfoPath.3)',
'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB7.0)',
'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)',
'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)',
'Mozilla/5.0 (Windows; U; Windows NT 6.1; ) AppleWebKit/534.12 (KHTML, like Gecko) Maxthon/3.0 Safari/534.12',
'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; .NET4.0C; .NET4.0E)',
'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; .NET4.0C; .NET4.0E; SE 2.X MetaSr 1.0)',
'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.33 Safari/534.3 SE 2.X MetaSr 1.0',
'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; .NET4.0C; .NET4.0E)',
'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.41 Safari/535.1 QQBrowser/6.9.11079.201',
'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; .NET4.0C; .NET4.0E) QQBrowser/6.9.11079.201',
'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)',
'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0'
]
thisua = random.choice(uapools)
header = {'User-Agent': thisua}
# current_time = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
current_time = '2019-02-21 18:01:22'
end_time = '2019-02-05 00:00:00' #电影上映时间
url = 'http://m.maoyan.com/mmdb/comments/movie/248906.json?_v_=yes&offset=0&startTime=' + current_time.replace(' ', '%20')
def start_requests(self):
current_t = str(self.current_time)
if current_t > self.end_time:
try:
yield Request(self.url, headers=self.header, callback=self.parse)
except Exception as error:
print('请求1出错-----' + str(error))
else:
print('全部有关信息已经搜索完毕')
def parse(self, response):
item = MaoyanItem()
data = response.body.decode('utf-8', 'ignore')
json_data = json.loads(data)['cmts']
count = 0
for item1 in json_data:
if 'cityName' and 'nickName' and 'userId' and 'content' and 'score' and 'startTime' and 'userLevel' and 'gender' in item1:
try:
city = item1['cityName']
comment = item1['content']
user_id = item1['userId']
nick_name = item1['nickName']
score = item1['score']
time = item1['startTime']
user_level = item1['userLevel']
gender = item1['gender']
item['city'] = city
item['content'] = comment
item['user_id'] = user_id
item['nick_name'] = nick_name
item['score'] = score
item['time'] = time
item['user_level'] = user_level
item['gender'] = gender
yield item
count += 1
if count >= 15:
temp_time = item['time']
current_t = datetime.datetime.strptime(temp_time, '%Y-%m-%d %H:%M:%S') + datetime.timedelta(seconds = -1)
current_t = str(current_t)
if current_t > self.end_time:
url1 = 'http://m.maoyan.com/mmdb/comments/movie/248906.json?_v_=yes&offset=0&startTime=' + current_t.replace(' ', '%20')
yield Request(url1, headers=self.header, callback=self.parse)
else:
print('全部有关信息已经搜索完毕')
except Exception as error:
print('提取信息出错1-----' + str(error))
elif 'cityName' and 'nickName' and 'userId' and 'content' and 'score' and 'startTime' and 'userLevel' and 'gender' not in item1:
try:
city = item1['cityName']
comment = item1['content']
user_id = item1['userId']
nick_name = item1['nickName']
score = item1['score']
time = item1['startTime']
user_level = item1['userLevel']
item['city'] = city
item['content'] = comment
item['user_id'] = user_id
item['nick_name'] = nick_name
item['score'] = score
item['time'] = time
item['user_level'] = user_level
item['gender'] = '2' #2代表无性别
yield item
count += 1
if count >= 15:
temp_time = item['time']
current_t = datetime.datetime.strptime(temp_time, '%Y-%m-%d %H:%M:%S') + datetime.timedelta(seconds = -1)
current_t = str(current_t)
if current_t > self.end_time:
url1 = 'http://m.maoyan.com/mmdb/comments/movie/248906.json?_v_=yes&offset=0&startTime=' + current_t.replace(' ', '%20')
yield Request(url1, headers=self.header, callback=self.parse)
else:
print('全部有关信息已经搜索完毕')
except Exception as error:
print('提取信息出错2-----' + str(error))
else:
print('部分信息不全,已滤除')
您好，我是有问必答小助手，你的问题已经有小伙伴为您解答了问题，您看下是否解决了您的问题，可以追评进行沟通哦~
如果有您比较满意的答案 / 帮您提供解决思路的答案，可以点击【采纳】按钮，给回答的小伙伴一些鼓励哦～～
ps:问答VIP仅需29元，即可享受5次/月 有问必答服务，了解详情>>>https://vip.csdn.net/askvip?utm_source=1146287632
非常感谢您使用有问必答服务，为了后续更快速的帮您解决问题，现诚邀您参与有问必答体验反馈。您的建议将会运用到我们的产品优化中，希望能得到您的支持与协助！
速戳参与调研>>>https://t.csdnimg.cn/Kf0y