GET请求的输出与查看源不同

I'm trying to extract match data from whoscored.com. When I view the source on firefox, I find on line 816 a big json string with the data I want for that matchid. My goal is to eventually get this json.

In doing this, I've tried to download every page of https://www.whoscored.com/Matches/ID/Live where ID is the id of the match. I wrote a little Go program to GET request each ID up to a certain point:

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
    "os"
)

// http://www.whoscored.com/Matches/614052/Live is the match for
// Eveton vs Manchester
const match_address = "http://www.whoscored.com/Matches/"

// the max id we get
const max_id = 300
const num_workers = 10

// function that get the bytes of the match id from the website
func match_fetch(matchid int) {
    url := fmt.Sprintf("%s%d/Live", match_address, matchid)

    resp, err := http.Get(url)
    if err != nil {
        fmt.Println(err)
        return
    }

    // if we sucessfully got a response, store the
    // body in memory
    defer resp.Body.Close()
    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        fmt.Println(err)
        return
    }

    // write the body to memory
    pwd, _ := os.Getwd()
    filepath := fmt.Sprintf("%s/match_data/%d", pwd, matchid)
    err = ioutil.WriteFile(filepath, body, 0644)
    if err != nil {
        fmt.Println(err)
        return
    }
}

// data type to send to the workers,
// last means this job is the last one
// matchid is the match id to be fetched
// a matchid of -1 means don't fetch a match
type job struct {
    last    bool
    matchid int
}

func create_worker(jobs chan job) {
    for {
        next_job := <-jobs
        if next_job.matchid != -1 {
            match_fetch(next_job.matchid)
        }
        if next_job.last {
            return
        }
    }
}

func main() {
    // do the eveton match as a reference
    match_fetch(614052)

    var joblist [num_workers]chan job
    var v int

    for i := 0; i < num_workers; i++ {
        job_chan := make(chan job)
        joblist[i] = job_chan
        go create_worker(job_chan)
    }
    for i := 0; i < max_id; i = i + num_workers {
        for index, c := range joblist {
            if i+index < max_id {
                v = i + index
            } else {
                v = -1
            }
            c <- job{false, v}
        }
    }
    for _, c := range joblist {
        c <- job{true, -1}
    }
}

The code seems to work in that it fills a directory called match_data with html. The problem is that this html is completely different to what I get in the browser! Here is the section which I think does this: (from the body of the GET request of http://www.whoscored.com/Matches/614052/Live.

(function() { 

var z="";var b="7472797B766172207868723B76617220743D6E6577204461746528292E67657454696D6528293B766172207374617475733D227374617274223B7661722074696D696E673D6E65772041727261792833293B77696E646F772E6F6E756E6C6F61643D66756E6374696F6E28297B74696D696E675B325D3D22723A222B286E6577204461746528292E67657454696D6528292D74293B646F63756D656E742E637265617465456C656D656E742822696D6722292E7372633D222F5F496E63617073756C615F5265736F757263653F4553324C555243543D363726743D373826643D222B656E636F6465555249436F6D706F6E656E74287374617475732B222028222B74696D696E672E6A6F696E28292B222922297D3B69662877696E646F772E584D4C4874747052657175657374297B7868723D6E657720584D4C48747470526571756573747D656C73657B7868723D6E657720416374697665584F626A65637428224D6963726F736F66742E584D4C4854545022297D7868722E6F6E726561647973746174656368616E67653D66756E6374696F6E28297B737769746368287868722E72656164795374617465297B6361736520303A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2072657175657374206E6F7420696E697469616C697A656420223B627265616B3B6361736520313A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2073657276657220636F6E6E656374696F6E2065737461626C6973686564223B627265616B3B6361736520323A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2072657175657374207265636569766564223B627265616B3B6361736520333A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2070726F63657373696E672072657175657374223B627265616B3B6361736520343A7374617475733D22636F6D706C657465223B74696D696E675B315D3D22633A222B286E6577204461746528292E67657454696D6528292D74293B6966287868722E7374617475733D3D323030297B706172656E742E6C6F636174696F6E2E72656C6F616428297D627265616B7D7D3B74696D696E675B305D3D22733A222B286E6577204461746528292E67657454696D6528292D74293B7868722E6F70656E2822474554222C222F5F496E63617073756C615F5265736F757263653F535748414E45444C3D313536343032333530343538313538333938362C31373139363833393832313930303534313833392C31333935303737313737393531363432383234342C3132363636222C66616C7365293B7868722E73656E64286E756C6C297D63617463682863297B7374617475732B3D6E6577204461746528292E67657454696D6528292D742B2220696E6361705F6578633A20222B633B646F63756D656E742E637265617465456C656D656E742822696D6722292E7372633D222F5F496E63617073756C615F5265736F757263653F4553324C555243543D363726743D373826643D222B656E636F6465555249436F6D706F6E656E74287374617475732B222028222B74696D696E672E6A6F696E28292B222922297D3B";for (var i=0;i<b.length;i+=2){z=z+parseInt(b.substring(i, i+2), 16)+",";}z = z.substring(0,z.length-1); eval(eval('String.fromCharCode('+z+')'));})();

The reason I think this is the case is that the javascript in the page fetches and edits the DOM to what I see on view source. How can I get golang to run the javascript? Is there are library to do this? Better still, could I directly grab the JSON from the servers?

This can be done with https://godoc.org/github.com/sourcegraph/webloop#View.EvaluateJavaScript Read their main example https://github.com/sourcegraph/webloop

What you need is a "headless browser" in general.

In general it is better to use an Web API vs. scraping. For example, whoscored themselves use OPTA which you should be able to access directly.

http://www.jokecamp.com/blog/guide-to-football-and-soccer-data-and-apis/#opta