I am trying to read a CSV file in reverse. I can do that by loading the rows of CSV file in a slice and reversing the slice but this is a wrong approach for large file sizes. CSV reader returns a cursor which does not load the whole file in memory.
But I could not find anything which implements something like a reverseCSVReader.
How to read the CSV file without loading the whole file in memory in reverse?
How to read the CSV file without loading the whole file in memory in reverse?
Basically not at all. CSV is not a format with a fixed record size and thus you cannot access individual lines directly. You always must read from the beginning. But of course there is no need to keep in memory what you have read and are not interested in: E.g. the encoding/csv doesn't do what you want to avoid.
A few general suggestions: apologies I don't know Go or its CSV library well enough to know if these are practical or not here.
Perform a pass of the CSV file forwards without processing the data, saving the file offset every line or every 100 lines (say). You can then work backwards through your array of file offsets, seeking to each individual line to process or read in a block of 100 lines in one go and then work through the block in reverse in your code.
Implement your own backwards file reader: read in the last 16K of the file into a binary buffer and then work backwards through it to find a line break to identify that you've found a line. Then process the line and seek the next one; if you run out of data then load the next 16K buffer to find the start of that line.
Hopefully you can borrow logic for this from the Go runtime library. This does however become complicated if you have linebreaks inside quoted blocks in your file, i.e. a linebreak might not represent a new CSV line, but hopefully you'll know whether your data does or not.
Find some external utility to reverse your file for you, e.g. if each line starts with a timestamp or record ID then you can probably just use your OS's built in sort (give or take the column header line). You can then just process the already-reversed file in your app.