I'm currently trying to read in a file with 200+ columns and 1000+ rows. I use the following code:
var result []string
file, err := os.Open("t8.txt")
if (err != nil) {
fmt.Println(err)
}
defer file.Close()
scan := bufio.NewScanner(file)
for scan.Scan() {
result = append(result, scan.Text())
}
fmt.Println(scan.Err()) //token too long
However when I print out the results, all I get is the first line because it says the token is too long. When I try it on smaller files, it works fine. Is there a way in Golang that I could scan in large files?
As already pointed out by @Dave C in the comments you are running into MaxScanTokenSize = 64 * 1024
To get around that limitation, use bufio.Reader which has a ReadString(delim byte) method which seems appropriate for your case.
From the Scanner go doc (specifically the last sentence):
Scanner provides a convenient interface for reading data such as a file of newline-delimited lines of text. Successive calls to the Scan method will step through the 'tokens' of a file, skipping the bytes between the tokens. The specification of a token is defined by a split function of type SplitFunc; the default split function breaks the input into lines with line termination stripped. Split functions are defined in this package for scanning a file into lines, bytes, UTF-8-encoded runes, and space-delimited words. The client may instead provide a custom split function.
Scanning stops unrecoverably at EOF, the first I/O error, or a token too large to fit in the buffer. When a scan stops, the reader may have advanced arbitrarily far past the last token. Programs that need more control over error handling or large tokens, or must run sequential scans on a reader, should use bufio.Reader instead.