I have a golang AppEngine application which runs a Monte Carlo- style simulation in parallel using the task queue, and makes heavy uses of memcache to store intermediate results. Everything works fine for small numbers of processes/tasks (< 100), but I'm finding that at large numbers (> 1000) I get a lot of failures to read or update intermediate data from memcache, with the error 'Canceled: Deadline exceeded'. This seems to happen 7-10 seconds after initialising a task. Particular offenders seem to be memcache.JSON.Get and memcache.IncrementExisting. Is it possible I'm overloading memcache somehow ?
Issue 6519 might be relevant
If you have appestats enabled, and the memcache logging exceeds 1mb, it gets a "out of heap" error and then all subsequent memcache sets/gets/admin calls fail. Instance upload "fixes" the problem temporarily. I disabled appestats to fix the problem more permanently.
Seems this was related to an underlying AppEngine infrastructure problem (now resolved)