os.Create“打开的文件太多”

I have around 220'000 image files (.png) to create. I run into this error message when trying to create the 1'081th file: panic: open /media/Snaps/pics/image1081_0.png: too many open files

I've added the defer w.Close() line but it did not change the error.

    i := 1
    for i <= 223129 {

    (some other code to prepare the data and create the chart)

    img := vgimg.New(450, 600)
    dc := draw.New(img)

    canvases := table.Align(plots, dc)
    plots[0][0].Draw(canvases[0][0])
    plots[1][0].Draw(canvases[1][0])
    plots[2][0].Draw(canvases[2][0])


    testFile := "/media/Snaps/pics/image"+strconv.Itoa(i+60)+"_"+gain_loss+".png"

        w, err := os.Create(testFile)
        if err != nil {
            panic(err)
        }
        defer w.Close()

        png := vgimg.PngCanvas{Canvas: img}
        if _, err := png.WriteTo(w); err != nil {
            panic(err)
        }   

        //move to next image
        i = i + 1
     }

Surely this limit can be worked around ? Maybe I'm not closing the files properly ?

ok, i got it, changed defer w.Close() to w.Close() and moved it after

png := vgimg.PngCanvas{Canvas: img}
if _, err := png.WriteTo(w); err != nil {
      panic(err)
}

I'm now above 10'000 images and running...

The Go Programming Language Specification

Defer statements

A "defer" statement invokes a function whose execution is deferred to the moment the surrounding function returns, either because the surrounding function executed a return statement, reached the end of its function body, or because the corresponding goroutine is panicking.

DeferStmt = "defer" Expression .

The expression must be a function or method call; it cannot be parenthesized. Calls of built-in functions are restricted as for expression statements.

Each time a "defer" statement executes, the function value and parameters to the call are evaluated as usual and saved anew but the actual function is not invoked. Instead, deferred functions are invoked immediately before the surrounding function returns, in the reverse order they were deferred. If a deferred function value evaluates to nil, execution panics when the function is invoked, not when the "defer" statement is executed.

In other words, if you are processing files in a loop, put the processing for a single file in a separate function to pair the Open with the defer Close(). This avoids the "too many open files" error.

For example, use a file processing structure like this to guarantee each file is closed immediately after use.

package main

import (
    "fmt"
    "io/ioutil"
    "os"
)

// process single file
func processFile(name string) error {
    f, err := os.Open(name)
    if err != nil {
        return err
    }
    defer f.Close()
    fi, err := f.Stat()
    if err != nil {
        return err
    }
    fmt.Println(fi.Name(), fi.Size())
    return nil
}

func main() {
    wd, err := os.Getwd()
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        return
    }
    fis, err := ioutil.ReadDir(wd)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        return
    }
    // process all files
    for _, fi := range fis {
        processFile(fi.Name())
        if err != nil {
            fmt.Fprintln(os.Stderr, err)
        }
    }
}

Playground: https://play.golang.org/p/FrBWqlMOzaS

Output:

dev 1644
etc 1644
tmp 548
usr 822

Deferred statements are not executed until the surrounding function returns, that is why your files stay open until after the for-loop.

To fix this you can simply insert an anonymous function call inside the loop:

for ... {
    func() {
        w, err := os.Create(testFile)
        if err != nil {
            panic(err)
        }
        defer w.Close()
        ...
    }()
}

That way, after each iteration of the loop, the current file is closed.