I'm currently trying to get search list from RottenTomatoes for some researching purposes. I have integrated goquery
in an easy way for different sites. But the situation for RottenTomatoes was not the same. I can't get the search list even though I have tried different comminutions with queries.
Testing URL: https://www.rottentomatoes.com/search/?search=test
The text I want to get: Testament
doc.Text()
Output: https://pastebin.com/SsWHYXTH
Here is what i do:
func ParseSearchMovies(doc *goquery.Document) *models.SearchResponse {
result := new(models.SearchResponse)
finder0 := doc.Find("section > section > ul")
finder1 := doc.Find("section.movieSection > ul")
finder2 := doc.Find("section.movieSection > ul > li")
finder3 := doc.Find("movieSection")
finder4 := doc.Find("section.movieSection")
finder5 := doc.Find("section.movieSection ul.results_ul")
finder6 := doc.Find("div.Search.Results li.bottom_divider.clearfix")
finder7 := doc.Find("div.search-results-root span.bold")
finder8 := doc.Find("div.search-results-root #details")
finder9 := doc.Find("div.search-results-root li.bottom_divider.clearfix")
finder10 := doc.Find("#movieSection > ul > li:nth-child(1)")
finder11 := doc.Find("#movieSection > ul")
finder12 := doc.Find("#movieSection > ul .bottom_divider.clearfix")
finder13 := doc.Find("#movieSection > ul > li")
finder14 := doc.Find("section.Search.Results .movieSection")
finder15 := doc.Find("#search-results-root section")
finder16 := doc.Find("#search-results-root .section")
finder17 := doc.Find("#search-results-root .section .section")
finder18 := doc.Find("#search-results-root > section > section")
finder19 := doc.Find("#search-results-root > section .section")
//From hlscalon's answer
finder20 := doc.Find("#movieSection > .results_ul > li > div.details")
finder21 := doc.Find("#movieSection > .results_ul > li")
finder22 := doc.Find("#movieSection > .results_ul")
finder23 := doc.Find("#movieSection")
fmt.Printf("%d", len(finder0.Nodes))
fmt.Printf("%d", len(finder1.Nodes))
fmt.Printf("%d", len(finder2.Nodes))
fmt.Printf("%d", len(finder3.Nodes))
fmt.Printf("%d", len(finder4.Nodes))
fmt.Printf("%d", len(finder5.Nodes))
fmt.Printf("%d", len(finder6.Nodes))
fmt.Printf("%d", len(finder7.Nodes))
fmt.Printf("%d", len(finder8.Nodes))
fmt.Printf("%d", len(finder9.Nodes))
fmt.Printf("%d", len(finder10.Nodes))
fmt.Printf("%d", len(finder11.Nodes))
fmt.Printf("%d", len(finder12.Nodes))
fmt.Printf("%d", len(finder13.Nodes))
fmt.Printf("%d", len(finder14.Nodes))
fmt.Printf("%d", len(finder15.Nodes))
fmt.Printf("%d", len(finder16.Nodes))
fmt.Printf("%d", len(finder17.Nodes))
fmt.Printf("%d", len(finder18.Nodes))
fmt.Printf("%d", len(finder19.Nodes))
fmt.Printf("%d", len(finder20.Nodes))
fmt.Printf("%d", len(finder21.Nodes))
fmt.Printf("%d", len(finder22.Nodes))
fmt.Printf("%d", len(finder23.Nodes))
}
Expected:
9 (Movie count in <section id="movieSection">
)
Actual:
All the results are returns 0. Output: 000000000000000000000000
P.S: When I print the result to the console via doc.Text()
, whole the page comes correctly.
To print all movie names in order:
if len(myFinder.Nodes) > 0 {
doc.Find("MY QUERY").Each(func(i int, s *goquery.Selection) {
name := s.Find("a")
fmt.Println(name)
})
}
There are some (many) alternatives but if you want a list of all links of films this should work:
#movieSection > .results_ul > li > div.details > span.bold > a.articleLink
This will lead you directly to the links (a
), so maybe you'll have to change your function to print names or stop one step before (span
)
Edit
Actually, I think can't use goquery to scrap that page. It uses react
, so when you download the page (ie curl "https://www.rottentomatoes.com/search/?search=test"
), you will see this:
<div id="main_container" class="container ">
<div class="col col-left-center col-full-xs">
<div id="search-results-root"></div>
<script>
require(['jquery', 'globals', 'search-results', 'bootstrap'], function($, RT, mount) {
mount($("#search-results-root").get(0), RT.PrivateApiV2FrontendHost, 'test', {"actorCount"....
When goquery is expecting html only