Is there a way to loop through datastore results via a cursor, until there are no results?
I need to pull every single record and iterate through it. But it's like 4 million records. So I wanted to work in chunks of 1000 to start with.
I know the below won't work. But it's just to show the sort of logic I'm hoping for.
q := datastore.NewQuery("Foo").Limit(1000)
while t, err := models.Client.GetAll(models.Ctx, q, &results) {
...
myCursor, err := t.Cursor()
q.Start(cursor)
}
EDIT: Here is the only way I found how to do it. I'm sure there is a more elegant way.
limit := 1000
offset := 0
resultCount := 1
for resultCount > 0 {
q := datastore.NewQuery("Foo").Limit(limit).Offset(offset)
_, err := client.GetAll(ctx, q, &foo)
checkErr(err)
resultCount = len(foo)
processPaths(foo)
offset += resultCount
}
It doesn't actually use the cursor.