I have a function to download a file from S3. It works but does not recognize the IfModifiedSince option. The below function looks for the same named file locally, and if present sets a time.Time object to the Modification date and time. That is then used in the request to S3 to only download the file if it has been modified since then.
func DownloadS3Media(filename string, mediaDirectory string, bucket string, c *configuration.Configuration) (dest string, bytes int64, err error) {
os.Setenv("AWS_ACCESS_KEY_ID", c.AWS_ACCESS_KEY_ID)
os.Setenv("AWS_SECRET_ACCESS_KEY", c.AWS_SECRET_ACCESS_KEY)
// Create the directories in the path
finalFilePath := filepath.Join(mediaDirectory, filename)
if err1 := os.MkdirAll(filepath.Dir(finalFilePath), 0775); err1 != nil {
return "", 0, err1
}
//Create the blank file to push the data into
var t *time.Time
if fi, err2 := os.Stat(finalFilePath); err2 == nil {
t1 := fi.ModTime()
t = &t1
}
tempFilePath := filepath.Join(mediaDirectory, filename + ".tmp")
tempFile, err3 := os.Create(tempFilePath)
if err3 != nil {
log.Println("Failed to create file", err3)
return finalFilePath, 0, err3
}
defer tempFile.Close()
logLevel := aws.LogDebug
downloader := s3manager.NewDownloader(session.New(&aws.Config{Region: aws.String("us-east-1"), LogLevel:&logLevel}))
params := &s3.GetObjectInput{Bucket: aws.String(bucket),
Key: aws.String(filename),
IfModifiedSince: t}
numBytes, err4 := downloader.Download(tempFile, params)
if err4 != nil {
log.Println("Failed to download file", err4)
return finalFilePath, 0, err4
}
//if downloader pulled down the file, rename the tmp file to original.
if _, err5 := os.Stat(tempFilePath); err5 == nil {
os.Rename(tempFilePath, finalFilePath)
log.Println("Renamed ", tempFilePath, "to", finalFilePath)
}
log.Println("Downloaded file", filename, "to", finalFilePath, numBytes, "bytes")
return finalFilePath, numBytes, nil
}
When debugging, I see the proper value for &t which is something like sec: 63606119179
. By the time AWS sends the request, the header is present but is set to 1/1/0001.
---[ REQUEST POST-SIGN ]-----------------------------
GET http://*****.s3.amazonaws.com/domenic-test.png HTTP/1.1
Host: *****.s3.amazonaws.com
User-Agent: aws-sdk-go/1.3.1 (go1.6.3; darwin; amd64) S3Manager
Authorization: AWS4-HMAC-SHA256 Credential= *****/20160806/us-east-1/s3/aws4_request, SignedHeaders=host;if-modified-since;range;x-amz-content-sha256;x-amz-date, Signature= *****
If-Modified-Since: Mon, 1 Jan 0001 00:00:00 GMT
Range: bytes=0-5242879
X-Amz-Content-Sha256: *****
X-Amz-Date: 20160806T223302Z
Try this simplification:
var t time.Time
if fi, err2 := os.Stat(finalFilePath); err2 == nil {
t := fi.ModTime()
}
and later:
params := &s3.GetObjectInput{Bucket: aws.String(bucket),
Key: aws.String(filename),
IfModifiedSince: aws.Time(t)}
...as suggested in the example code here: https://github.com/aws/aws-sdk-go/blob/f1e26250235022af782521266389e4b2ae2945e4/service/s3/examples_test.go#L975
At first I thought it was an erasure issue, but after testing, I find that the time.Time
value that t
was pointing to escapes the block (which honestly surprised me). Maybe there's some other bug happening in between the two, but I would assume that since their tests are passing, this should work.