使用goroutines和aranGO达到限制

I'm trying to parse ~40k files into JSON and save them to ArangoDB (this is a part of my 4fun project). This can be simplified as:

func walkRepository(task *RepositoryProcessingTask) {
    filepath.Walk(task.Repository.Path, func(path string, info os.FileInfo, err error) error {
        // ...
        go parseFile(task, file)
        // ...
        return nil
    })
}

func parseFile(task *RepositoryProcessingTask, file *RepositoryFile) {
    <- task.routinesChannel
    defer finishProcessingFile(task)

    // Saves to ArangoDB (opens http connection)
    err = db.Col("File").Save(file)

    // ...
}

So, basically, I'm spawning 40k routines. Since each routine opens file and http socket to ArangoDB (I'm using arangGO.v2 ), I'm hitting 1k open file limit pretty easily.

I've tried to limit number of routines running in parallel via channel-semaphore, but with no luck:

const (
    maxRunningRoutines = 256
)

// on task creation
task.routinesChannel = make(chan int32, maxRunningRoutines)

// on task start
for i := int32(0) ; i < maxRunningRoutines ; i += 1 {
    task.routinesChannel <- i
}

// when parsing completes
task.routinesChannel <- processedFiles

Finally, I have increased open-file limit, but it causes program to crash:

runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7fcddd12c5f7 m=2

goroutine 0 [idle]:

...

goroutine 81348 [chan receive]:
net/http.(*Transport).getConn.func2.1(0xcc732b2120, 0xc8200ec0c0, 0x959c08)
        /usr/lib/golang/src/net/http/transport.go:698 +0x48
created by net/http.(*Transport).getConn.func2
        /usr/lib/golang/src/net/http/transport.go:702 +0x70

My guess is that HTTP connections spawned by DB client are not closing ASAP which causes trouble. But how could I fix it and prove it that this is real cause?