I was wondering how I can remove the first N lines of a specific file efficiently.
Of course, we can load the complete file into memory, then remove the first N lines and write the complete changed contents to the file.
But, I was wondering if there is a way to make this more effective compared to just loading all contents in memory.
Filesystems doesn't support deleting from arbitrary locations of a file, so you at least need to re-write the entire file yourself. The common way to implement what you need is to
Are you trying to do something akin to log truncation and rotation?
A common pattern that might help you would be to split logs across multiple files and overwrite or delete files in the order they were created.
For example you might have 2018-09-09.log.txt or log.1.txt and rotate to a new log file every hour. That way older files could be deleted to free up space.
Other than that, I don't think there is a specific way to remove early lines of a file in Go without rewriting the remainder of the file.
The most efficient way to do this is read the file backwards.
/*
This package implements funcationality similar to the UNIX command tail.
It prints out the last "x" many lines of a file, usually 10.
The number of lines in a file is determined by counting newline (
) characters.
This can be use either from CLI or imported into other projects.
*/
package main
import (
"errors"
"flag"
"fmt"
"io"
"os"
)
var (
//DEFINE FLAG DEFAULTS
filename = ""
numLines = 10
//ErrNoFilename is thrown when the path the the file to tail was not given
ErrNoFilename = errors.New("You must provide the path to a file in the \"-file\" flag.")
//ErrInvalidLineCount is thrown when the user provided 0 (zero) as the value for number of lines to tail
ErrInvalidLineCount = errors.New("You cannot tail zero lines.")
)
//GET FLAGS FROM CLI
func init() {
flag.StringVar(&filename, "file", "", "Filename to get last lines of text from.")
flag.IntVar(&numLines, "n", numLines, "Number of lines to get from end of file.")
flag.Parse()
}
//MAIN FUNCTIONALITY OF APP
//make sure filename (path to file) was given
//run it through the tailing function
//print the output to stdout
func main() {
//TAIL
text, err := GoTail(filename, numLines)
if err != nil {
fmt.Println(err)
return
}
//DONE
fmt.Print(text)
return
}
//GoTail IS THE FUNCTION THAT ACTUALLY DOES THE "TAILING"
//this can be used this package is imported into other golang projects
func GoTail(filename string, numLines int) (string, error) {
//MAKE SURE FILENAME IS GIVEN
//actually, a path to the file
if len(filename) == 0 {
return "", ErrNoFilename
}
//MAKE SURE USER WANTS TO GET AT LEAST ONE LINE
if numLines == 0 {
return "", ErrInvalidLineCount
}
//OPEN FILE
file, err := os.Open(filename)
if err != nil {
return "", err
}
defer file.Close()
//SEEK BACKWARD CHARACTER BY CHARACTER ADDING UP NEW LINES
//offset must start at "-1" otherwise we are already at the EOF
//"-1" from numLines since we ignore "last" newline in a file
numNewLines := 0
var offset int64 = -1
var finalReadStartPos int64
for numNewLines <= numLines-1 {
//seek to new position in file
startPos, err := file.Seek(offset, 2)
if err != nil {
return "", err
}
//make sure start position can never be less than 0
//aka, you cannot read from before the file starts
if startPos == 0 {
//set to -1 since we +1 to this below
//the position will then start from the first character
finalReadStartPos = -1
break
}
//read the character at this position
b := make([]byte, 1)
_, err = file.ReadAt(b, startPos)
if err != nil {
return "", err
}
//ignore if first character being read is a newline
if offset == int64(-1) && string(b) == "
" {
offset--
continue
}
//if the character is a newline
//add this to the number of lines read
//and remember position in case we have reached our target number of lines
if string(b) == "
" {
numNewLines++
finalReadStartPos = startPos
}
//decrease offset for reading next character
//remember, we are reading backward!
offset--
}
//READ TO END OF FILE
//add "1" here to move offset from the newline position to first character in line of text
//this position should be the first character in the "first" line of data we want
endPos, err := file.Seek(int64(-1), 2)
if err != nil {
return "", err
}
b := make([]byte, (endPos+1)-finalReadStartPos)
_, err = file.ReadAt(b, finalReadStartPos+1)
if err == io.EOF {
return string(b), nil
} else if err != nil {
return "", err
}
//special case
//if text is read, then err == io.EOF should hit
//there should *never* not be an error above
//so this line should never return
return "**No error but no text read.**", nil
}
In the console you must type : gotail -file <filename> -n=<number of lines> So with this you can read the last number of lines from the end of the file. If you want all lines except the first 10 lines, then i will alter this code.
You can find the total number of lines in the file using wc -l <fileName> let it be x, and suppose u want to read all lines except first 1000 then use tail -n<x-1000> <fileName>.
An efficient way would be to use the sed. See this answer