I am using cURL in PHP to scrap a webpage. Some of the words that I need to get are Japanese characters. I'm also using the Simple DOM Parser to help me parse the source code easily. I am having a little trouble figuring out how to properly obtain the Japanese characters. Everytime I run the following script on my page I get that there is no data received. I'm thinking that I need to somehow convert the characters to UTF-8 standard, but I'm not entirely sure how to do that. It grabs all of the English characters just fine though, so I know that the script does work. It just doesn't work for other characters. Does anyone think they can help me out? I have included a sample of what the source code I'm scrapping looks like as well.
CODE:
$base = '{website url}';
$curl = curl_init();
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($curl, CURLOPT_HEADER, array('User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:19.0) Gecko/20100101 Firefox/19.0'));
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_URL, $base);
curl_setopt($curl, CURLOPT_REFERER, $base);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
$str = curl_exec($curl);
curl_close($curl);
// Create a DOM object
$html = new simple_html_dom();
// Load HTML from a string
$html->load($str);
foreach($html->find('div.holder') as $element){
if($element->find('div.img-small', 0)){
$title = '';
$image = '';
foreach($element->find('a[href]') as $tempElement){
if($tempElement->find('img')){
$image = $tempElement->find('img')->src;
} else {
$title = $tempElement->innertext;
}
}
echo $title.'<br/>';
}
}
DATA:
<div class="holder">
<div class="img-small">
<a href="/link/abcd"><img src="/images/image.png"></a>
</div>
<div>
<div>
<img title="This is a title" class="valign" src="/images/image.png"><b>
<a href="/link/1234abcd">{Japanese characters}</a>
</div>
</div>
</div>
Put this on top of the page
<head>
<META http-equiv="Content-Type" Content="text/html; charset=euc-jp">
</head>