I am trying to list all the tokens found in a web page. The core is in the function
func find_links(httpBody io.Reader) []string {
links := make([]string, 0)
page := html.NewTokenizer(httpBody)
for {
tokenType := page.Next()
if tokenType == html.ErrorToken {
return links
}
token := page.Token()
fmt.Println("Now token is ", token)
}
}
When I print the output I obtain something like
Now token is <body>
Now token is
Now token is <header>
I don't understand what the second token is and why it is printing an extra blank line.
Full code of a working example here, even if it can't run on playground because of the missing http package
The second token is a TextToken containing a newline.
Change the print to
fmt.Printf("Now token is %T %v
", token, token)
to see the types of the tokens.