I have a task written in GO to read .gz files from AWS S3, size of each .gz file will be 20M.
Each goroutine will download a .gz file from s3 to local disk, then read its content line by line via gzip.NewReader.
When task(goroutine) count exceeds 70, 67 goroutine will finish their operations successfully. But the remain goroutine then will be paused for several minutes.
At the moment it paused, I saw CPU is 100%, and then, CPU will low down to 0.2%(4CPU, 16G Memory) to stay for several minutes.
Question: I'm confused that, why goroutines doing nothing at that time with CPU quite low? What may cause this?
Test results:
60 goroutines, it will finish successfully in 27 seconds;
70 goroutines, it will finish successfully in 30s;
80 goroutines, 77 of them will finish successfully in 30s, but the remaining 3 will **pause 4 minutes**, then restart and finish in 10s
Code as below:
Main goroutine:
func main() {
runtime.GOMAXPROCS(runtime.NumCPU()*2)
chs := make([]chan string, 70)
for i:=min; i<max; i++ {
chs[i] = make(chan string, 1)
go readobj.Reads3obj(i, chs[i])
}
for _, ch := range chs {
fmt.Println(<-ch)
}
}
ReadObject:
func Reads3obj(s3KeyName, ch chan string) {
sess, err := session.NewSession(&aws.Config{
Region: aws.String("x"),
Credentials: credentials.NewStaticCredentials("x", "x", "")},)
downloader := s3manager.NewDownloader(sess)
//create a zipFile to download from S3 to local
zipFile, err := os.Create(zipname)
//download .gz file from S3
n, err := downloader.Download(zipFile, &s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(s3KeyName),
})
//create a file reader
fileReader, err := os.Open(zipFile)
//create a gzip reader
gzipReader, err := gzip.NewReader(fileReader)
//create a buffered reader
buf := bufio.NewReader(gzipReader)
//read zip file line by line
for line, isPrefix, err := []byte{0}, false, error(nil); len(line) > 0 && err == nil; {
line, isPrefix, err = buf.ReadLine()
//insert line to ES
}
}
EDITED: When it paused, opened file count is quite small, so I think the opened files didn't exceeds most and it won't matters. Because:
The output of ll /proc/PID/fd is
zc@ip-xxx:/proc/18059/fd$ ll
total 0
dr-x------ 2 zc zc 0 Dec 26 06:50 ./
dr-xr-xr-x 9 zc zc 0 Dec 26 06:50 ../
lrwx------ 1 zc zc 64 Dec 26 06:50 0 -> /dev/pts/0
lrwx------ 1 zc zc 64 Dec 26 06:50 1 -> /dev/pts/0
lrwx------ 1 zc zc 64 Dec 26 06:50 12 ->
/home/zc/75.gz
lrwx------ 1 zc zc 64 Dec 26 06:50 2 -> /dev/pts/0
lrwx------ 1 zc zc 64 Dec 26 06:50 21 ->
/home/zc/78.gz
lrwx------ 1 zc zc 64 Dec 26 06:50 253 -> socket:[76054]
lrwx------ 1 zc zc 64 Dec 26 06:50 280 -> socket:[77064]
lrwx------ 1 zc zc 64 Dec 26 06:50 47 ->
/home/zc/58.gz
lrwx------ 1 zc zc 64 Dec 26 06:50 65 -> anon_inode:[eventpoll]
lrwx------ 1 zc zc 64 Dec 26 06:50 93 -> socket:[75984]
Golang gzip library read file in one goroutine,
So big file occupy the cpu for long time on 100%. Use pzip instead.
Use as a replacement for gzip, exchange
import "compress/gzip"
with
import gzip "github.com/klauspost/pgzip".