使用Go将文件写入Google App Engine存储空间似乎有10MB的限制？

I am using http://godoc.org/google.golang.org/cloud/storage#Writer to write files to GAE Storage. Using io.Copy returns successful results as long as the size of the file I am trying to copy is less than 10MB. If the file is greater than 10MB, my io.Copy exits successfully with number of bytes written but the file is nowhere to be found in my bucket.

I can't find any pointers in the docs so would appreciate some community feedback on how they're doing this. Here is my code snippet:

func (s *GoogleStorage) StoreReader(r io.ReadCloser) error {
  defer r.Close()

  wc := storage.NewWriter(s.Context, s.Bucket, s.BackupName)
  wc.ContentType = "text/plain"
  wc.ACL = []storage.ACLRule{{storage.AllUsers, storage.RoleReader}}
  defer wc.Close()

  count, err := io.Copy(wc, r)
  if err != nil {
    s.app.Infof("Error doing copy:", err)
  }
  s.app.Infof("Bytes written: %d", count)
  return err
}

I updated the above code to return more information:

func (s *GoogleStorage) StoreReader(r io.ReadCloser) error {
  defer r.Close()

  wc := storage.NewWriter(s.Context, s.Bucket, s.BackupName)
  wc.ContentType = "text/plain"
  wc.ACL = []storage.ACLRule{{storage.AllUsers, storage.RoleReader}}

  count, err := io.Copy(wc, r)
  if err != nil {
    s.app.Infof("Error doing copy:", err)
  }
  if err := wc.Close(); err != nil {
    s.app.Infof("ERROR ", err)
  }
  s.app.Infof("updated object:", wc.Object())
  s.app.Infof("Bytes written: %d", count)
  return err
}

And this is the error I get after the wc.Close()

    I 16:31:19.729 ERROR %!(EXTRA *url.Error=Post https://www.googleapis.com/upload/storage/v1/b/maxr-test/o?alt=json&uploadType=multipart: Call error 5: )
I 16:31:19.729 updated object:%!(EXTRA *storage.Object=<nil>)

One of the solutions may be to use Cloud Storage Java Client Library to upload/write the file in Cloud Storage. The advantage of using that it automatically support resumable/chunk uploads so that you will not fall under that URL Fetch request size exceed error. Here is the Go example.

It's due to limit on Google Storage 10MB. In my logs:

com.google.appengine.api.urlfetch.RequestPayloadTooLargeException: The request to https://www.googleapis.com/upload/storage/v1/b/XXXXXXXX exceeded the 10 MiB limit.

This is what I've found and I'm using Java lib: (com.google.api.services.storage.Storage)

Here I found example by google however my team mate is still investigating this.

Maybe that would help you. I would add my results when finished.

Ok. Solved. MediaHttpUploader.DEFAULT_CHUNK_SIZE is around 10MB and this is already too much. Everything below 10MB would be accepted.

    Storage client = getService();
    Storage.Objects.Insert insertRequest = client.objects().insert(
            BUCKET_NAME, objectMetadata, contentStream);

    //done by adding following two lines:
    //  - chunk is set to <10MB (~5MB in this case)
    //  - directed upload is disabled.
    insertRequest.getMediaHttpUploader().setChunkSize(MediaHttpUploader.DEFAULT_CHUNK_SIZE/2); //set to 5MB
    insertRequest.getMediaHttpUploader().setDirectUploadEnabled(false);

    final StorageObject storageObject = insertRequest.execute();

Maybe this would lead you by analogy to solution in your GO problem

Cheers!