使用AWS开发工具包Go的完整URI从S3下载文件

The examples I've seen for downloading a file from S3 using the AWS SDK for Go are of the form:

downloader := s3manager.NewDownloader(session, /* other args */)

s3object := &s3.GetObjectInput{
    Bucket: aws.String(myBucket),
    Key: aws.String(myKey),
}

bytesDownloaded, err := downloader.Download(myFile, s3object)

That is, one uses the bucket and key to specify the file. But what if I already have the full URI of the file on S3? E.g.:

https://s3.us-west-2.amazonaws.com/myBucket/myKey

Is there a way using the SDK to specify the file to download using the URL directly?

No, the bucket is not public. In the SDK, I'm also setting the access and secret keys (elided from the example code).

Lastly, if it's simply not possible to do what I'm asking via the SDK, that's an acceptable (though not desired) answer.

There is no way to do what you want. The only ways to get a private object are:

1. Use the bucket and key to download the file
2. Generate a presigned URL an do an hhtp.Get on that URL (to generate the presigned URL you need the bucket and key)

If you really want to use the URL I recommend you create a wrapper around s3.GetObjectInput that accepts the URL as an argument, parses it and then create the s3.GetObjectInput object.

Also, I don't know if you could do this it depends on your use case, but if you are storing the URLs somewhere and that's why you want to use them, you can consider storing instead the presigned url with a long expiration time, and then when you want to fetch the object you can just use that URL.

Here you first need to enable public access on the object that you are trying to download from S3. You could achieve this by either of following approach.

• Make object publicly readable. But remember, whoever got access on the resource url can download your object from S3.
• Grant an anonymous access to your S3 bucket through bucket policies. Refer here to read about this approach with policy samples. Remember to verify what you expose when you go in this route.
• Use a PreSignedUrl that permit to download the object in time-limited manner. This is what I would prefer from aforementioned approaches.

Now last step would be to use go's http client to make the call to your pre-signed url and download the required object.

You could parse the URL to get the bucket and key:

package main

import (
    "fmt"
    "log"
    "net/url"
    "strings"
)

func main() {
    u, err := url.Parse("https://s3.us-west-2.amazonaws.com/myBucket/myKey/morekey")
    if err != nil {
        log.Fatal(err)
    }

    path := strings.SplitN(u.Path, "/", 3)
    bucket := path[1]
    key := path[2]
    fmt.Println(bucket)
    fmt.Println(key)
}

Runnable: https://play.golang.org/p/G3Mxmm4f4qM

Created new S3URLtoURI parser for this.

func S3URLtoURI(s3Url string) (map[string]string, error) {
    m := make(map[string]string)
    u, err := url.Parse(s3Url)
    if err != nil {
        return m, err
    }

    if u.Scheme == "s3" {
        //s3: //bucket/key
        m["bucket"] = u.Host
        m["key"] = strings.TrimLeft(u.Path, "/")
    } else if u.Scheme == "https" {
        host := strings.SplitN(u.Host, ".", 2)
        if host[0] == "s3" {
            // No bucket name in the host;
            path := strings.SplitN(u.Path, "/", 3)
            m["bucket"] = path[1]
            m["key"] = path[2]
        } else { //bucket name in host
            m["bucket"] = host[0]
            m["key"] = strings.TrimLeft(u.Path, "/")
        }

    }
    return m, err
}

Runnable: https://play.golang.org/p/vkPJ7E4Fyno