I started to deal with Colly. I am trying to set a proxy for every request. But all requests do not go through a proxy
It should work https://github.com/gocolly/colly/blob/master/request.go#L52
I have added this to my code. But requests go without a proxy as before
c.OnRequest(func(r *colly.Request) {
r.ProxyURL = "..."
}
Maybe I didn’t understand correctly how this should work
Instead of setting ProxyURL
in the callback function, have you tried setting it on the collector ?
// SetProxy sets a proxy for the collector. This method overrides the previously
// used http.Transport if the type of the transport is not http.RoundTripper.
// The proxy type is determined by the URL scheme. "http"
// and "socks5" are supported. If the scheme is empty,
// "http" is assumed.
func (c *Collector) SetProxy(proxyURL string) error {
proxyParsed, err := url.Parse(proxyURL)
if err != nil {
return err
}
c.SetProxyFunc(http.ProxyURL(proxyParsed))
return nil
}