I am using Simple HTML DOM Class for web-page scrapping. Issue is it generates weird characters against unicode character.
हंगामा है कà¥à¤¯à¥‚ठबरपा / अकबर इलाहाबादी
against hindi unicode character.
लेकिन इतना तो हुआ कुछ लोग
Its my Hindi text.
When I print screen output it output in same weird characters.
function getDomContent($data) {
$html = new simple_html_dom();
$html->load($data);
foreach ($html->find('table[id=content] li') as $element) {
$content[] = $element->plaintext;
}
return $content;
}
My Curl function
function getContent($url) {
$timeout = 5;
$ch = curl_init();
$user_agent = 'Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20100101 Firefox/8.0';
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 120);
curl_setopt($ch, CURLOPT_TIMEOUT, 120);
curl_setopt($ch, CURLOPT_MAXREDIRS, 10);
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$data = getContent($url);
$content = getDomContent($data);
echo '<pre>Array Content: ' . '<br/>';
print_r($content);
die($query);
I solved it by adding header to my page...
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
It solved all issues.
It seems problem with encodification. Try use iconv PHP function.
$text = iconv("current text codification", "UTF-8", $text)
But if you don't know current encoding try set it like global configuration with iconv_set_encoding.
iconv_set_encoding("internal_encoding", "UTF-8");