sergiotapia at Macbook-Air in ~/Work/go/src/github.com/sergiotapia/gophers on master [!]
$ go build && go install && gophers -github_url=https://github.com/search?utf8=%E2%9C%93&q=location%3A%22San+Fransisco%22+location%3ACA+followers%3A%3E100&type=Users&ref=advsearch&l=
[1] 51873
[2] 51874
[3] 51875
[4] 51877
[2] Done q=location%3A%22San+Fransisco%22+location%3ACA+followers%3A%3E100
[3] Done type=Users
[4]+ Done ref=advsearch
I'm trying to use the long github url as a parameter in my code for Gophers. It works fine for all other url types such as organisations or stargazers. However when I try to use the search results page I get the strange output above.
package main
import (
"flag"
"log"
"strings"
"github.com/PuerkitoBio/goquery"
)
type user struct {
name string
email string
url string
username string
}
func main() {
url := flag.String("github_url", "", "github url you want to scrape")
flag.Parse()
githubURL := *url
doc, err := goquery.NewDocument(githubURL)
if err != nil {
log.Fatal(err)
}
if strings.Contains(githubURL, "/orgs/") {
scrapeOrganization(doc, githubURL)
} else if strings.Contains(githubURL, "/search?") {
scrapeSearch(doc, githubURL)
} else if strings.Contains(githubURL, "/stargazers") {
scrapeStarGazers(doc, githubURL)
} else {
scrapeProfile(doc)
}
}
It's a bash command line (or whatever the mac uses). & and ? are shell metacharacters that you MUST escape. The shell has absolutely no idea what a URL is, nor should it ever have to.
go 'http://....'
^-----------^
Adding quotes will prevent the shell from parsing the metacharacters. The alternative is to manually escape each and ever metachar yourself:
go http://example.com/script.php\?foo=bar\&baz=qux
^--------^
which quickly gets tedious, and error prone.