问题描述:
最近在学习mongodb,发现mongodb并发查询比较慢,集合的数据大小2G左右,总共文档6万+,其中每个文档包含1000左右的子文档,单次查询响应时间在400ms左右,10个线程并发查询响应时间在1800ms左右,并发线程数增加,对应的查询响应时间也在增加
spring-boot-starter-data-mongodb版本
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-mongodb</artifactId>
<version>2.2.1.RELEASE</version>
</dependency>
spring boot 配置
spring:
data:
mongodb:
host: 127.0.0.1
port: 27017
authentication-database: admin
database: xxx
connections-per-host: 200
min-connections-per-host: 20
max-connection-idel-time: 0
threadsAllowedToBlockForConnectionMultiplier: 30
java 代码
// 查询条件字段用 xxx_1、2、3 代替
Criteria criteria = new Criteria();
criteria.and("xxx_1").is("xxx");
criteria.and("xxx_2").in(list);
DateFormat format = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
criteria.andOperator(Criteria.where("xxx_3").gte(format.parse("2020-08-05 00:00:00")),
Criteria.where("xxx_3").lte(format.parse("2021-08-05 23:59:59")));
Query query = new Query(criteria);
// 省略查询字段,不包含子文档
query.fields(...);
long startTime = System.currentTimeMillis();
List<TestBO> queryList = mongoTemplate.find(query, TestBO.class);
System.out.println("查询耗时:" + (System.currentTimeMillis() - startTime));
System.out.println(queryList.size());
mongodb 脚本执行计划
查询条件字段 xxx_1、xxx_2、xxx_3 为联合索引
{
plannerVersion: NumberInt("1"),
namespace: "xxx.xxx",
indexFilterSet: false,
parsedQuery: {
$and: [
{
xxx_1: {
$eq: "xxx"
}
},
{
xxx_3: {
$lte: ISODate("2021-07-29T23:59:59.000Z")
}
},
{
xxx_2: {
$in: [
...
]
}
}
]
},
queryHash: "C16527B7",
planCacheKey: "4538160D",
winningPlan: {
stage: "PROJECTION_SIMPLE",
transformBy: {
/*查询字段*/
},
inputStage: {
stage: "FETCH",
inputStage: {
stage: "IXSCAN",
keyPattern: {
xxx_1: NumberInt("1"),
xxx_2: NumberInt("1"),
xxx_3: NumberInt("1")
},
indexName: "index_name",
isMultiKey: false,
multiKeyPaths: {
xxx_1: [ ],
xxx_2: [ ],
xxx_3: [ ]
},
isUnique: false,
isSparse: false,
isPartial: false,
indexVersion: NumberInt("2"),
direction: "forward",
indexBounds: {
xxx_1: [
...
],
xxx_2: [
...
],
xxx_3: [
...
]
}
}
}
},
rejectedPlans: [
{
stage: "PROJECTION_SIMPLE",
transformBy: {
/*查询字段*/
},
inputStage: {
stage: "FETCH",
filter: {
$and: [
{
xxx_1: {
$eq: "xxx"
}
},
{
xxx_3: {
$lte: ISODate("2021-07-29T23:59:59.000Z")
}
}
]
},
inputStage: {
stage: "IXSCAN",
keyPattern: {
xxx_2: NumberInt("1")
},
indexName: "r_index",
isMultiKey: false,
multiKeyPaths: {
xxx_2: [ ]
},
isUnique: false,
isSparse: false,
isPartial: false,
indexVersion: NumberInt("2"),
direction: "forward",
indexBounds: {
xxx_2: [
...
]
}
}
}
},
{
stage: "PROJECTION_SIMPLE",
transformBy: {
/*查询字段*/
},
inputStage: {
stage: "FETCH",
filter: {
$and: [
{
xxx_1: {
$eq: "xxx"
}
},
{
xxx_2: {
$in: [
...
]
}
}
]
},
inputStage: {
stage: "IXSCAN",
keyPattern: {
xxx_3: NumberInt("1")
},
indexName: "xxx_3",
isMultiKey: false,
multiKeyPaths: {
xxx_3: [ ]
},
isUnique: false,
isSparse: false,
isPartial: false,
indexVersion: NumberInt("2"),
direction: "forward",
indexBounds: {
xxx_3: [
"(true, new Date(1627603199000)]"
]
}
}
}
}
]
}
问题算是解决了,没有找到特别好的解决方式,最后用的是非映射对象的方式解决的,相同的查询条件,单个请求查询平均在200ms内,10到30个线程并发查询循环10次,查询时间差不多也在1000ms内响应
Bson eq = Filters.eq("xxx_1", "xxx_1");
Bson in = Filters.in("xxx_2", xxx_2);
// 查询条件
Bson and = Filters.and(eq, in);
BasicDBObject fieldsObject = new BasicDBObject();
// 指定返回字段
fieldsObject.put("xxx_1", 1);
...
FindIterable<Document> quota = mongoTemplate.getCollection("Test").find(and).projection(Document.parse(fieldsObject.toString()));
List<TestBO> qlist = new ArrayList<>();
for(Document document : quota) {
TestBO q = new TestBO();
// document.get("") ;
// 通过逐个对象取值然后在赋值给具体的对象,试过用Fastjson的方式转换,但是效果更慢
q.setxxx(document.getString("xxx"));
qlist.add(q);
}