My question is specific to the "gopkg.in/olivere/elastic.v2" package I am using.
I am trying to return all documents that match my query:
termQuery := elastic.NewTermQuery("item_id", item_id)
searchResult, err := es.client.Search().
Index(index).
Type(SegmentsType). // search segments type
Query(termQuery). // specify the query
Sort("time", true). // sort by "user" field, ascending
From(0).Size(9).
Pretty(true). // pretty print request and response JSON
Do() // execute
if err != nil {
// Handle error
return timeline, err
}
The problem is I get an internal server error if I increase the size to something large. If I eliminate the line that states:
From(0).Size(9).
then the default is used (10 documents). How may I return all documents?
Using a scroller to retrieve all results is just a bit different and in the interest of brevity I'm not including much error handling that you might need.
Basically you just need to slightly change your code from Search
to Scroller
and then loop with the Scroller
calling Do
and handling pages of results.
termQuery := elastic.NewTermQuery("item_id", item_id)
scroller := es.client.Scroller().
Index(index).
Type(SegmentsType).
Query(termQuery).
Sort("time", true).
Size(1)
docs := 0
for {
res, err := scroller.Do(context.TODO())
if err == io.EOF {
// No remaining documents matching the search so break out of the 'forever' loop
break
}
for _, hit := range res.Hits.Hits {
// JSON parse or do whatever with each document retrieved from your index
item := make(map[string]interface{})
err := json.Unmarshal(*hit.Source, &item)
docs++
}
}