I am using Go to read CSV file and save the records in a MS SQL database using go-odbc. It works great, but I have an issue where some records (about 10 records) do not get stored. This a random problem, sometimes 3 do not get saved, other times 2, etc. The only time where all the records are saved is when I put fmt.Printf(" ")
at the end of the for loop. Notice it has to print a blank space it cannot just be fmt.Printf("")
. I am not sure what am I do not wrong. Any suggestions is appreciated. Also, no errors are produced, the program terminates normally.
I included the code with the related problem, let me know if you need me to post the entire code.
Go version: go1.1 windows/amd64
for {
record, err := c.Read()
if err == io.EOF {
break
} else if err != nil {
fmt.Printf("Error while reading %s: %s
", filename, err)
} else {
//replace the single quote at the beginning and end of string
re, err := regexp.Compile("^'|'$")
params := make([]interface{}, 0, numElements)
valueHolders := make([]string, 0, numElements)
tmpFields := make([]string, 0, numElements)
count := 0
for i:=1;i<=numElements;i++ {
tmp := re.ReplaceAllString(record[i],"")
//insert only non-empty values
if len(tmp) > 0 {
params = params[0:count+1]
params[count] = tmp
valueHolders = valueHolders[0:count+1]
valueHolders[count] = "?"
tmpFields = tmpFields[0:count+1]
tmpFields[count] = fieldNames[i-1]
count++
}
}
query := "insert into [l2test].[dbo]."+tablename+" (" + strings.Join(tmpFields, ",") + ") values (" + strings.Join(valueHolders, ",") + ")"
stmt, err := dest.Prepare(query)
if stmt == nil {
fmt.Printf("Error preparing statment: %s
Query: %s
%v
", err, query, params)
} else {
stmt.Execute(params...)
stmt.Close()
}
}
fmt.Printf(" ")
}
When you drop a few elements at the end, it's usually because you dropped them from the end of the input or didn't write/flush/commit/close them on output. For example, on input you may be ignoring the last slice at EOF. I would write your EOF and error tests differently.
The error handling for your database access needs improvement too. For example, it should check for errors.
Once we find database access errors, add some diagnostic information, such as the query
and the record
, to the returned error (err
).
func insertRecord(conn *odbc.Connection, query string, params []interface{}) error {
stmt, err := conn.Prepare(query)
defer func() {
if stmt != nil {
stmt.Close()
}
}()
if err != nil {
return err
}
err = stmt.Execute(params...)
if err != nil {
return err
}
return nil
}
for {
record, err := c.Read()
if err != nil {
if err != io.EOF {
fmt.Printf("Error while reading %s: %s
", filename, err)
break
}
if len(record) == 0 {
break
}
}
// do things with a record
query := "insert into [l2test].[dbo]." + tablename +
" (" + strings.Join(tmpFields, ",") + ")" +
" values (" + strings.Join(valueHolders, ",") + ")"
err = insertRecord(dest, query, params)
if err != nil {
err = fmt.Errorf("%s
%s
%v
%s", err, query, params, strings.Join(record, "||"))
fmt.Println(err)
continue
}
}