I have written the following function for executing the snt2cooc command (one of the preprocessing steps for running GIZA++. For our purposes I think we can consider the snt2cooc script to be a black box):
func SNTToCOOC(srcVocab, tgtVocab, sntPath, outpath string) error {
// open the out file for writing
outfile, err := os.Create(outpath)
if err != nil {
return err
}
defer outfile.Close()
cmdStr := "snt2cooc"
args := []string{srcVocab, tgtVocab, sntPath}
cmd := exec.Command(cmdStr, args...)
cmd.Stdout = outfile
if err = cmd.Run(); err != nil {
return err
}
cmd.Wait()
return err
}
When running, the function executes without an error, but the output file is empty. This same code works for other similar commands, but not for this specific snt2cooc command, and I noticed that when I run this command in the shell directly:
snt2cooc file1.vcb file2.vcb file3.snt
I get the following output:
END.
0 2
0 3
0 4
0 5
0 6
(truncated for brevity)
And if I send the output of the command to a file directly from the shell:
snt2cooc file1.vcb file2.vcb file3.snt > out.txt
The contents of out.txt are as expected:
0 2
0 3
0 4
0 5
0 6
Notice how in the first case, the line END. is output to stdout first, and only then is the real output of the command sent to stdout. I therefore think there is a race condition going on, where the Go code finishes executing before the command's final output is written to file. This despite calling cmd.Wait(). I'm not too sure what exactly the snt2cooc command is doing internally. Could someone provide a hint on how to solve this?
Edit 1:
It seems like the following code, with the sleep of 500ms included, consistently writes output to the file for the snt2cooc command:
cmdStr := "snt2cooc"
args := []string{srcVocab, tgtVocab, sntPath}
cmd := exec.Command(cmdStr, args...)
stdout, err := cmd.StdoutPipe()
time.Sleep(500 * time.Millisecond)
if err != nil {
return err
}
err = cmd.Start()
if err != nil {
return err
}
out := bufio.NewScanner(stdout)
for out.Scan() {
outfile.Write(out.Bytes())
outfile.WriteString("
")
}
if err := out.Err(); err != nil {
return err
}
This proves to me that there is some race condition going on, with the Go program exiting before all output is written to file. I added a bounty to this question, with the hope that someone can 1) explain why this is happening and 2) provide a non-hacky way (i.e. 500ms sleep) to fix it.
First, clean up your code.
cmd.Stderr = os.DevNull, so you ignore stderr. Stdout and Stderr specify the process's standard output and error. If either is nil, Run connects the corresponding file descriptor to the null device (os.DevNull).
cmd.Wait() returns error, you ignore it. func (c *Cmd) Wait() error.
Wait waits for the command to exit. It must have been started by Start. You use Run, not Start.
What output do you get when you run this code?
failure.go:
package main
import (
"fmt"
"os"
"os/exec"
)
func main() {
err := SNTToCOOC("file1.vcb", "file2.vcb", "file3.snt", "out.txt")
if err != nil {
fmt.Println(err)
}
}
func SNTToCOOC(srcVocab, tgtVocab, sntPath, outpath string) error {
outfile, err := os.Create(outpath)
if err != nil {
return err
}
defer outfile.Close()
cmdStr := "snt2cooc"
args := []string{srcVocab, tgtVocab, sntPath}
cmd := exec.Command(cmdStr, args...)
cmd.Stdout = outfile
cmd.Stderr = os.Stderr
err = cmd.Run()
if err != nil {
return err
}
return err
}
Run:
$ rm -f out.txt && go run failure.go && cat out.txt
Also, what output do you get when you run this code with cmd.Stdout = os.Stdout substituting for cmd.Stdout = outfile.
The problem isn't with the innard of SNTtoCooc but how you write to the file itself using cmd.Stdout:
func anyWrite(args []string, outpath string) error {
outfile, err := os.Create(outpath)
if err != nil {
return err
}
defer outfile.Close()
// I use simple "echo" here
cmd := exec.Command("echo", args...)
stdout, err := cmd.Output()
if err != nil {
return err
}
// Use this instead of cmd.Stdout seems to solve the problem
outfile.Write(stdout)
return nil
}
func main() {
args := []string{"Line 1", "Line 2", "Line 3"}
if err := anyWrite(args, "./outfile.txt"); err != nil {
panic(err)
}
}
As per the comments in os/exec
Stdout and Stderr specify the process's standard output and error. If either is nil, Run connects the corresponding file descriptor to the null device (os.DevNull).