golang是从json字符串获取密钥的有效方法/库，并使用此密钥进行geoip，然后将geoip信息添加到json

just as title,I want to use golang to do geoip,the msg is json format like below

{"type":"big_platform","xrealip":"8.8.8.8","scheme":"http","log_time":"24/Feb/2017:15:36:10 +0800","http_method":"GET","hostname":"XXX.com","url":"/v126330.apk","http_code":"206","send_byte":20972063,"user_agent":"63(android;KOOMII-K8;6.0;480x854;WIFI;1118.24)","@timestamp":"2017-02-24T15:36:10+08:00","process_time":59463,"cache_status":"TCP_HIT","refer":"-"}

I want use Go to parse this message and get xrealip,use xrealip to do geoip.then apped geoip info to this msg.

now I do it like blew

//parse json string ,add geoip info and return new msg
func geoInfoAdd(msg string) ([]byte, error) {
        data := make(map[string]interface{})
        err := json.Unmarshal([]byte(msg), &data)
        if err != nil {
                return nil, err
        }
        //type assert value
        var ok bool
        if ipValue, ok = data[ipKey].(string); !ok {
                // ask_price is not a string
                return nil, errors.New("ipvalue not string")
        }

        //get ip location
        loc, err := ip17mon.Find(ipValue)
        if err != nil {
                return nil, err
        }
        iplocation := map[string]string{"country": loc.Country, "region": loc.Region, "isp": loc.Isp}

        //newdata := data
        data["iplocation"] = iplocation
        newmsg, err := json.Marshal(data)
        if err != nil {
                return nil, err
        }

        return newmsg, nil
}

but thats realy slow,cost lots of cpu,i need to do this action like 70000/second. and i really need better way to do this.

thanks

Have you measured the json serialization to show it's the problem? Alternatives could be;

• write a straight string parser
• use custom struct with json:"arg_name"

However I'd expect most of you time to be here;

loc, err := ip17mon.Find(ipValue)

Given you are using go, goroutines and channels would be an obvious first step to improve throughput performance for me.

1. push each new ip to process to a ipChannel
2. read from ipChannel in 'lookup' goroutine do lookup and process result
3. use error channel for things that go wrong

have as many 'lookup' channels as you need.

If you truly need just the xrealip from this json, just do the work you need by hand. This is probably a lot faster (you'd need to benchmark), as Adam said below your time on the network might be more significant than time spent parsing this json:

// Find the key
key := `xrealip":"`
i := strings.Index(data,key) + len(key)
if i == -1 {
  return errors.New("no ip")
}

// Extract ip up to max ip len (IPV4)
ip := data[i:i+15]

// Cut to just ip 
q := strings.Index(ip,`"`)
if q > -1 {
  ip = ip[:q]
}

Your functions should probably work with bytes, not strings, and scan through rather than cutting by finding " but the above is based on your function.

There are other json packages (https://github.com/pquerna/ffjson), but the main problem is processing all this data when you only want one key.