I'm making a Golang program which needs to download N files and then perform some task on each one. In particular there is a pipeline of operation to do on each single SWF file:
Doing it sequentially could be very inefficient. Doing it with N goroutines could also be non-optimal.
How do I choose / limit the number of goroutines?
A goroutine is lightweight by design:
A goroutine has a simple model: it is a function executing concurrently with other goroutines in the same address space. It is lightweight, costing little more than the allocation of stack space. And the stacks start small, so they are cheap, and grow by allocating (and freeing) heap storage as required.
The limit is not in the number of goroutines you generate, but in what they do.
In your case, opening files (to write a downloaded stream) concurrently could meet the limit of number of opened files per process or in general