I'm trying to decode a JSON stream from a file. The file is newline separated JSON objects. But it's not formatted with array brackets surrounding the objects, so I can't figure out how to use encoding/json
stream decoder.
I get the error not at beginning of value
because there is no opening bracket.
package main
import (
"encoding/json"
"fmt"
"log"
"strings"
)
func main() {
// My data is missing opening and closing brackets around the jsonStream string.
// [
const jsonStream = `
{"Name": "Ed", "Text": "Knock knock."},
{"Name": "Sam", "Text": "Who's there?"},
{"Name": "Ed", "Text": "Go fmt."},
{"Name": "Sam", "Text": "Go fmt who?"},
{"Name": "Ed", "Text": "Go fmt yourself!"}
`
// ]
type Message struct {
Name, Text string
}
dec := json.NewDecoder(strings.NewReader(jsonStream))
// read open bracket -- this fails (no bracket)
t, err := dec.Token()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%T: %v
", t, t)
// while the array contains values
for dec.More() {
var m Message
// decode an array value (Message)
err := dec.Decode(&m)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%v: %v
", m.Name, m.Text)
}
// read closing bracket -- this also would fail (no bracket)
t, err = dec.Token()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%T: %v
", t, t)
}
Removing the dec.Token() error checks doesn't work either. I guess the parser doesn't expect to see objects before it has seen an array open bracket?
Is there some workaround?
One suggestion is to use a pipe, and prepend and append the []
.
reader := strings.NewReader(jsonStream)
pr, pw := io.Pipe()
go func() {
pw.Write([]byte(`[`))
io.Copy(pw, reader)
pw.Write([]byte(`]`))
}()
dec := json.NewDecoder(pr)
You should add some error handling.
See https://play.golang.org/p/Oogu_NdAUyR for the whole thing.
I ended up doing this:
f, err := os.Open(fname)
if err != nil {
// handle error
}
d := json.NewDecoder(f)
for {
var v ValueTypeToUnmarshalTo
if err := d.Decode(&v); err == io.EOF {
break // done decoding file
} else if err != nil {
// handle error
}
// do something with v
}
The decoder appears to ignore newlines so there's not even a need to use the stream decoder. That's only if you actually have a JSON array.
Taken from part of this answer: https://stackoverflow.com/a/34388102
Thanks for the help!