I'm experiencing some weird behaviors using goroutine and please provide advise or comments on things I might do wrong. Here is my process:
I am using goroutines to perform a simulation concurrently, but for some reason the time I spent increases with number of goroutines I put in which makes my simulation not possible to finish in a reasonable time. Here are the things I notice:
For every 10k goroutines initiated, the calculation time increases by 5 seconds
I don't see a shortage in CPU or memory. However CPU usage increases only a little bit for every 10k goroutines. For example when I put 200k goroutines, CPU usage is around 70%
I'm not using disks
I ran the simulation without triggering the goroutines and it finishes very fast, so the slowness is inside or due to goroutines
I tried to use additional goroutines in 1 occasion inside each goroutine to run some workload in parallel. CPU usage is boosted to 100% but the overall speed decreased by 50%...
I am passing some large structs to goroutines using pointers. All goroutines use the same data.
Does anyone have a clue on things I might need to optimize, or suggest me any test I can perform? Thanks!
Start few "worker" goroutines and send "jobs" to its via channel. You save many memory allocations that consume a lot of CPU.