As stated in the title. I am needing to return all of the content within the body tags of an html document, including any subsequent html tags, etc. Im curious to know what the best way to go about this is. I had a working solution with the Gokogiri package, however I am trying to stay away from any packages that depend on C libraries. Is there a way to accomplish this with the go standard library? or with a package that is 100% go?
Since posting my original question I have attempted to use the following packages that have yielded no resolution. (Neither of which seem to return subsequent children or nested tags from inside the body. For example:
<!DOCTYPE html>
<html>
<head>
<title>
Title of the document
</title>
</head>
<body>
body content
<p>more content</p>
</body>
</html>
will return body content, ignoring the subsequent <p> tags and the text they wrap):
The over all goal would be to obtain a string or content that would look like:
<body>
body content
<p>more content</p>
</body>
It can be done using the standard encoding/xml package. But it's a bit cumbersome. And one caveat in this example is that it will not include the enclosing body tag, but it will contain all of it's children.
package main
import (
"bytes"
"encoding/xml"
"fmt"
)
type html struct {
Body body `xml:"body"`
}
type body struct {
Content string `xml:",innerxml"`
}
func main() {
b := []byte(`<!DOCTYPE html>
<html>
<head>
<title>
Title of the document
</title>
</head>
<body>
body content
<p>more content</p>
</body>
</html>`)
h := html{}
err := xml.NewDecoder(bytes.NewBuffer(b)).Decode(&h)
if err != nil {
fmt.Println("error", err)
return
}
fmt.Println(h.Body.Content)
}
Runnable example:
http://play.golang.org/p/ZH5iKyjRQp
You could also do this purely with strings:
func main() {
r := strings.NewReader(`
<!DOCTYPE html>
<html>
<head>
<title>
Title of the document
</title>
</head>
<body>
body content
<p>more content</p>
</body>
</html>
`)
str := NewSkipTillReader(r, []byte("<body>"))
rtr := NewReadTillReader(str, []byte("</body>"))
bs, err := ioutil.ReadAll(rtr)
fmt.Println(string(bs), err)
}
The definitions for SkipTillReader and ReadTillReader are here: https://play.golang.org/p/6THLhRgLOa. (But basically skip until you see the delimiter and then read until you see the delimiter)
This won't work for case insensitivity (though that wouldn't be hard to change).
Since you didn't show the source code of your attempt with the html package, I'll have to guess what you were doing, but I suspect you were using the tokenizer rather than the parser. Here is a program that uses the parser and does what you were looking for:
package main
import (
"log"
"os"
"strings"
"github.com/andybalholm/cascadia"
"golang.org/x/net/html"
)
func main() {
r := strings.NewReader(`<!DOCTYPE html>
<html>
<head>
<title>
Title of the document
</title>
</head>
<body>
body content
<p>more content</p>
</body>
</html>`)
doc, err := html.Parse(r)
if err != nil {
log.Fatal(err)
}
body := cascadia.MustCompile("body").MatchFirst(doc)
html.Render(os.Stdout, body)
}
This can be solved by recursively finding the body node, using the html package, and subsequently render the html, starting from that node.
package main
import (
"bytes"
"errors"
"fmt"
"golang.org/x/net/html"
"io"
"strings"
)
func getBody(doc *html.Node) (*html.Node, error) {
var b *html.Node
var f func(*html.Node)
f = func(n *html.Node) {
if n.Type == html.ElementNode && n.Data == "body" {
b = n
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
f(c)
}
}
f(doc)
if b != nil {
return b, nil
}
return nil, errors.New("Missing <body> in the node tree")
}
func renderNode(n *html.Node) string {
var buf bytes.Buffer
w := io.Writer(&buf)
html.Render(w, n)
return buf.String()
}
func main() {
doc, _ := html.Parse(strings.NewReader(htm))
bn, err := getBody(doc)
if err != nil {
return
}
body := renderNode(bn)
fmt.Println(body)
}
const htm = `<!DOCTYPE html>
<html>
<head>
<title></title>
</head>
<body>
body content
<p>more content</p>
</body>
</html>`