I am trying to write a Go struct in a Parquet file and upload to S3. What format and type do I specify for timestamp parameter in the struct so that athena displays correct timestamp when reading from the parquet file.
type example struct {
ID int64 `parquet:"name=id, type=INT64"`
CreatedAt int64 `parquet:"name=created_at,type=TIMESTAMP_MILLIS"`
}
ex := example{}
ex.ID = int64(10)
ex.CreatedAt = time.Now().Unix()
fw, err := ParquetFile.NewLocalFileWriter("new.parquet")
pw, err := ParquetWriter.NewParquetWriter(fw, new(example), 1)
pw.Write(ex)
Upload the file new.parquet to S3
Reference - https://github.com/xitongsys/parquet-go. I created a table in Athena with int and timestamp field for the same and trying querying the table. The date is showing something like - 1970-01-18 21:54:23.751. which no where matches the current timestamp.
For example,
package main
import (
"fmt"
"time"
)
func main() {
type example struct {
CreatedAt int64 `parquet:"name=created_at,type=TIMESTAMP_MILLIS"`
}
ex := example{}
ex.CreatedAt = time.Now().UnixNano() / int64(time.Millisecond)
fmt.Println(ex.CreatedAt)
}
Playground: https://play.golang.org/p/ePOlUKiT6fD
Output:
1257894000000