I'm running a tool that runs on a mongo collection with 7,766,558 documents in following template.
{
"_id" : ObjectId("53f602685a38d0bf5e8c6df1"),
"ids" : "5667",
"h" : [
{
"s" : "Briefly, 36 h following transfection, the cells were harvested and lysed in 500 µL HTE buffer containing calcium or calcium chelating reagents according to the experimental condition (150 mM NaCl, 20 mM HEPES (pH 7.4), 50 mM NaF, 1 mM Na3VO4, 1% Triton X-100 and 5 mM EDTA + 5 mM EGTA or 100 µM CaCl2 and complete protease inhibitor cocktail (Roche Applied Science)).",
"p" : "roche"
}
],
"ct" : 1445951888.0000000000000000
}
What the query does is filters the documents on "ids" (supplier id) and "ct" (created date) > [given date] and then run a mongo regex match on "h.s". Three instances of this job runs on 3 threads on a given time which each process difference suppliers. At the start of these threads, all 3 runs their queries quite fast. But the longer they run (approximately after 12 hours or so) the querying tend to get slower and slower.
The query that i'm running is
db.collection.find({'ids':4296, 'ct':{"$gte":1455062400}, 'h.s':{ '$regex': "/[^a-zA-Z]" . $catalogId . "[^0-9a-zA-Z]/i" }})
the solution doesn't throw any errors. The problem i'm facing is that the time that the query takes to perform drastically drops based on time. If the tool query 10 times per second at start. after like 10-12 hours its only around 5 times per second.
Any ideas into any performance updates i could do to over come this?
Thanks