I'm new to elasticsearch and I have some technical difficulties. Currently I have docs that are stored in hourly indexes and they are time series data. What I'm trying to figure out is how to efficiently extracting only the key field values, which is defined as "key": { "type": "long" }. I tried initially the naive method, which is scrolling through all indices and extract the field, but apparently that doesn't finish very quickly, each hourly index has about 10M docs and scrolling 3 indexes already takes forever.
Then I came to terms aggregations, tried to make key field as the aggregation term:
"aggregations": {
"test_group": {
"terms": {
"field": "key",
"size": 100000
}
}
}
That gives me better performance but still not sufficient as a real-time system as users try to search for the history, because key is a high cardinality field. Some rough benchmarks told me that:
size = 50k, indices = 4, time range = 3hrs: 7.1s
size = 100k, indices = 4, time range = 3hrs: 7.669s
size = 1m, indices = 4, time range = 2hrs: 12.669s
size = 1m, indices = 4, time range = 3hrs: 14.669s
This is not the end of it, because I'm using elastic search go library to parse the output and do some processing, which adds non-trivial time to the overall response.
My question is: is this already the best ES can do? Is there any other ways that I'm missing? I'm currently on ES 5.6 and 3 nodes for the cluster, all using Amazon i3-4xl instances. Thanks.
If I understand your question correctly you are trying to retrieve a specific field from your document called 'key' and I assume you have other fields in your documents that are being returned that you don't care about?
If so, try this:
GET /_search
{
"_source": {
"includes": ["key"]
}
}
I am not exactly sure what you are trying to achieve but retrieve one field from your document usually required store parameter as true, so this fields doesn't need to be parsed from _source field.
Check doc: https://www.elastic.co/guide/en/elasticsearch/reference/current/number.html#number-params