I was looking up knowledge on how to perform a lot of HTTP requests efficiently, and I came across this answer: https://stackoverflow.com/a/23319730/749851 with this code:
package main
import (
"flag"
"fmt"
"log"
"net/http"
"runtime"
"time"
)
var (
reqs int
max int
)
func init() {
flag.IntVar(&reqs, "reqs", 1000000, "Total requests")
flag.IntVar(&max, "concurrent", 200, "Maximum concurrent requests")
}
type Response struct {
*http.Response
err error
}
// Dispatcher
func dispatcher(reqChan chan *http.Request) {
defer close(reqChan)
for i := 0; i < reqs; i++ {
req, err := http.NewRequest("GET", "http://localhost/", nil)
if err != nil {
log.Println(err)
}
reqChan <- req
}
}
// Worker Pool
func workerPool(reqChan chan *http.Request, respChan chan Response) {
t := &http.Transport{}
for i := 0; i < max; i++ {
go worker(t, reqChan, respChan)
}
}
// Worker
func worker(t *http.Transport, reqChan chan *http.Request, respChan chan Response) {
for req := range reqChan {
resp, err := t.RoundTrip(req)
r := Response{resp, err}
respChan <- r
}
}
// Consumer
func consumer(respChan chan Response) (int64, int64) {
var (
conns int64
size int64
)
for conns < int64(reqs) {
select {
case r, ok := <-respChan:
if ok {
if r.err != nil {
log.Println(r.err)
} else {
size += r.ContentLength
if err := r.Body.Close(); err != nil {
log.Println(r.err)
}
}
conns++
}
}
}
return conns, size
}
func main() {
flag.Parse()
runtime.GOMAXPROCS(runtime.NumCPU())
reqChan := make(chan *http.Request)
respChan := make(chan Response)
start := time.Now()
go dispatcher(reqChan)
go workerPool(reqChan, respChan)
conns, size := consumer(respChan)
took := time.Since(start)
ns := took.Nanoseconds()
av := ns / conns
average, err := time.ParseDuration(fmt.Sprintf("%d", av) + "ns")
if err != nil {
log.Println(err)
}
fmt.Printf("Connections:\t%d
Concurrent:\t%d
Total size:\t%d bytes
Total time:\t%s
Average time:\t%s
", conns, max, size, took, average)
}
I'm coming from node so I don't really understand this "go" code.
What part of it is limiting it to 500 HTTP actions at a time? And is it operating in chunks of 500, waiting until that chunk of 500 is finished then starting a new 500, OR is it just always chugging along adding 1 more once it hits 499, etc.
I see that the "workerPool" func goes over a loop only as many times as the maximum amount of concurrent requests, calling "worker" 500 times, but how does it do the next 500 or even the whole 1 million eventually?
It isn’t 500, it is 200, and the magic line is:
for i := 0; i < max; i++ {
go worker(t, reqChan, respChan)
}
max defaults to 200; and may be over ridden with a command line switch. Each of these “go routines”, which are akin to ridiculously lightweight threads, initializes itself then waits for channel input. This is where the magic occurs - when a request comes in, it results in sending on a channel. There are max (200) go-routines receiving from this channel, and the channel is unbuffered, therefore up to 200 requests can be inflight. The 201th would cause the sender to wait until one of the workers completed and invoked the receive (<-) op.
The subtlety of Go’s message passing deserves far better, and a bit of googling around will uncover well written essays, tutorials and examples of concurrency in Go.
Good luck with Go; I think it is a wonderful language. It is elegant, expressive and concise. You might never be able to stomach c++ or java again....