I'm creating a microservice in Go, using protocol buffers and gRPC. It interacts with a third-party API (Snooth) and I'm trying to unmarshal the JSON response into a protobuf struct I've created, using the proto package.
Unmarshalling returns an unexpected EOF error.
json.newDecoder instead of json.unmarshaljsonpb.Unmarshal instead of proto.Unmarshal (returns a bad value in StructValue for key error)io.LimitReaderI've also read something about prefixing the proto types with a size tag or something? But I'm not sure what that is or if it's relevant. Here's the repo on Github.
What is causing this unexpected EOF error and how do I fix it, so that the API response is successfully unmarshalled into the proto struct?
Side note: I am new to Go and would also appreciate any feedback/improvements on the following code. Thanks!
Proto
message Response {
message Meta {
int32 results = 1;
int32 returned = 2;
string errmsg = 3;
int32 status = 4;
}
Meta meta = 1;
repeated google.protobuf.Struct wines = 2;
repeated google.protobuf.Struct actions = 3;
}
main.go
func fetchFromSnooth(e string, qs string, c chan response) {
defer func() {
if r := recover(); r != nil {
log.Printf("Error fetching from Snooth: %s", r)
errmsg := fmt.Sprint(r)
c <- response{nil, snoothApiError{errmsg}}
}
}()
v := url.Values{"akey": {os.Getenv("SNOOTH_API_KEY")}}
requestUrl := fmt.Sprintf("%s%s/?%s%s", snoothRoot, e, v.Encode(), qs)
log.Printf("Fetching: %s", requestUrl)
res, err := httpClient.Get(requestUrl)
if err != nil {
panic(err)
}
defer res.Body.Close()
body, err := ioutil.ReadAll(res.Body)
if err != nil {
panic(err)
}
result := &pb.Response{}
if err := proto.Unmarshal(body, result); err != nil {
panic(err)
}
c <- response{result, snoothApiError{""}}
}
func (s *snoothApiService) searchWines(params *pb.Parameters_WineSearch) response {
c := make(chan response)
defer close(c)
v := url.Values{}
go fetchFromSnooth("wines", v.Encode(), c)
return <-c
}
func main() {
snooth := snoothApiService{}
resp := snooth.searchWines(nil)
fmt.Println(resp)
}
Here's an example of the type of API response I'm trying to unmarshal:
{
"meta": {
"results": 1489442,
"returned": 5,
"errmsg": "",
"status": 1
},
"wines": [
{
"name": "Conway Deep Sea Chardonnay la Costa Wine Co",
"code": "conway-deep-sea-chardonnay-la-costa-wine-co-2008-1",
"region": "USA > California > Central Coast",
"winery": "Conway Family Wines",
"winery_id": "conway-family-wines",
"varietal": "Chardonnay",
"price": "21.99",
"vintage": "2008",
"type": "White Wine",
"link": "http:\/\/www.snooth.com\/wine\/conway-deep-sea-chardonnay-la-costa-wine-co-2008-1\/",
"tags": "",
"image": "https:\/\/ei.isnooth.com\/multimedia\/0\/2\/8\/image_787698_square.jpeg",
"snoothrank": 3,
"available": 0,
"num_merchants": 0,
"num_reviews": 10
},
{
"name": "Olmaia Cabernet di Toscana",
"code": "olmaia-cabernet-di-toscana",
"region": "Italy > Tuscany > Toscana Igt",
"winery": "Col D Orcia",
"winery_id": "col-d-orcia",
"varietal": "Cabernet Sauvignon",
"price": "0.00",
"vintage": "",
"type": "Red Wine",
"link": "http:\/\/www.snooth.com\/wine\/olmaia-cabernet-di-toscana\/",
"tags": "",
"image": "https:\/\/ei.isnooth.com\/multimedia\/d\/e\/e\/image_790198_square.jpeg",
"snoothrank": 3.5,
"available": 0,
"num_merchants": 0,
"num_reviews": 25
},
{
"name": "Dominio Dostares Prieto Picudo Vino de la Tierra de Castilla Y León Cumal",
"code": "dominio-dostares-prieto-picudo-vino-de-la-tierra-de-castilla-y-leon-cumal-2006",
"region": "Spain > Castilla y León > Vino de la Tierra de Castilla y León",
"winery": "Bischöfliches Priesterseminar Trier",
"winery_id": "bischofliches-priesterseminar-trier",
"varietal": "Prieto Picudo",
"price": "15.89",
"vintage": "2006",
"type": "Red Wine",
"link": "http:\/\/www.snooth.com\/wine\/dominio-dostares-prieto-picudo-vino-de-la-tierra-de-castilla-y-leon-cumal-2006\/",
"tags": "",
"image": "https:\/\/ei.isnooth.com\/multimedia\/d\/0\/4\/image_336595_square.jpeg",
"snoothrank": "n\/a",
"available": 0,
"num_merchants": 0,
"num_reviews": 1
},
{
"name": "Dominio Dostares Prieto Picudo Vino de la Tierra de Castilla Y León Cumal",
"code": "dominio-dostares-prieto-picudo-vino-de-la-tierra-de-castilla-y-leon-cumal-2005",
"region": "Spain > Castilla y León > Vino de la Tierra de Castilla y León",
"winery": "Bischöfliches Priesterseminar Trier",
"winery_id": "bischofliches-priesterseminar-trier",
"varietal": "Prieto Picudo",
"price": "38.99",
"vintage": "2005",
"type": "Red Wine",
"link": "http:\/\/www.snooth.com\/wine\/dominio-dostares-prieto-picudo-vino-de-la-tierra-de-castilla-y-leon-cumal-2005\/",
"tags": "",
"image": "https:\/\/ei.isnooth.com\/multimedia\/1\/d\/a\/image_336596_square.jpeg",
"snoothrank": "n\/a",
"available": 0,
"num_merchants": 0,
"num_reviews": 1
},
{
"name": "The Little Penguin Chardonnay Premier",
"code": "the-little-penguin-chardonnay-premier-2010",
"region": "South East Australia",
"winery": "The Little Penguin",
"winery_id": "the-little-penguin",
"varietal": "Chardonnay",
"price": "11.99",
"vintage": "2010",
"type": "White Wine",
"link": "http:\/\/www.snooth.com\/wine\/the-little-penguin-chardonnay-premier-2010\/",
"tags": "",
"image": "https:\/\/ei.isnooth.com\/multimedia\/2\/c\/4\/image_826282_square.jpeg",
"snoothrank": "n\/a",
"available": 0,
"num_merchants": 0,
"num_reviews": 7
}
]
}
I've now got the response unmarshalling as desired, using the jsonpb.Unmarshal method. To do so though I had to unmarshal and marshal the response with the regular json library first, in order to get around some escaped values in the response (I was receiving a 'bad value in Struct' error:
resJson, err := ioutil.ReadAll(res.Body)
j := make(map[string]interface{})
if err := json.Unmarshal(resJson, &j); err != nil {
panic(err)
}
jbytes, err := json.Marshal(j)
result := &pb.Response{}
r := strings.NewReader(string(jbytes))
if err := jsonpb.Unmarshal(r, result); err != nil {
panic(err)
}
It DOES parse with then encoding/json package: https://play.golang.org/p/IQzMm2tDI7w
The protoc-generated code's Unmarshal parses a Protocol Buffers encoded byte stream, NOT JSON!