I want to create an API that returns the title and description of a given URL.
I try the solution provided here: https://stackoverflow.com/a/3711554/5618358 with current improvement:
Unfortunately, it doesn't work when you pass yahoo and google URL. But work for other urls like a charm for example Github.com
I try to dump parameters of code step by step and I understand Yahoo return ugly code that can't be processed, and Google's HTML hasn't description meta tag.
How other sites similar:
works?
Please help me solve this problem. I am Apologize for my bad English.
//In routes/api.php
Route::get('/links/helper/meta-tag-extractor', function(Request $request){
$url = $request->get('url');
$result = [];
function file_get_contents_curl($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$html = file_get_contents_curl($url);
//parsing begins here:
$doc = new DOMDocument();
@$doc->loadHTML($html);
$nodes = $doc->getElementsByTagName('title');
//get and display what you need:
//This part issue error for url Yahoo.com:
$result['title'] = $nodes->item(0)->nodeValue;
$metas = $doc->getElementsByTagName('meta');
for ($i = 0; $i < $metas->length; $i++)
{
$meta = $metas->item($i);
if($meta->getAttribute('name') == 'description') {
$result['description'] = $meta->getAttribute('content');
}
//property="og:description"
//<meta property="og:description"
// content="Sean Connery found fame and fortune as the
// suave, sophisticated British agent, James Bond." />
if($meta->getAttribute('property') == 'og:description') {
$result['og:description'] = $meta->getAttribute('property');
}
}
// We haven't 'description' or 'og:description' in result for url: Google.com
// But for url Github.com works like a charm with result:
// {
// "title": "The world’s leading software development platform · GitHub",
// "description": "GitHub brings together the world’s largest community of developers to discover, share, and build better software. From open source projects to private team repositories, we’re your all-in-one platform for collaborative development.",
// "og:description": "og:description"
// }
return $result;
});
After a lot of effort, I had solved part of problem.
I solve problem with yahoo, so I can get it's information.
But for google's URL doesn't work.
Google hasn't description or og:description in it's source when I get it by server.
Code result for Yahoo is:
{
"title": "Yahoo",
"description": "News, email and search are just the beginning. Discover more every day. Find your yodel.",
"og:title": "Yahoo",
"og:type": "website",
"og:url": "http://www.yahoo.com",
"og:description": "News, email and search are just the beginning. Discover more
every day. Find your yodel.",
"og:image": "https://s.yimg.com/dh/ap/default/130909/y_200_a.png",
"og:site_name": "Yahoo"
}
But code result for google is:
{
"title": "Google"
}
Please Help me about Google....
The new source code is:
//In routes/api.php
Route::get('/links/helper/meta-tag-extractor', function(Request $request){
$url = $request->get('url');
$result = [];
function file_get_contents_curl($url)
{
$ch = curl_init();
$timeout = 10;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST,false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER,false);
curl_setopt($ch, CURLOPT_MAXREDIRS, 10);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT_MS, 3000);
curl_setopt($ch,CURLOPT_ENCODING , "gzip");
curl_setopt($ch, CURLOPT_HEADER, 0);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$html = file_get_contents_curl($url);
$doc = new DOMDocument();
@$doc->loadHTML($html);
// echo $html;
$nodes = $doc->getElementsByTagName('title');
if ($nodes->count()) {
$result['title'] = $nodes->item(0)->nodeValue;
}
$metas = $doc->getElementsByTagName('meta');
for ($i = 0; $i < $metas->length; $i++)
{
$meta = $metas->item($i);
if($meta->getAttribute('name') == 'description') {
$result['description'] = $meta->getAttribute('content');
}
if(substr( $meta->getAttribute('property'), 0, 3 ) === 'og:') {
$result[$meta->getAttribute('property')] = $meta->getAttribute('content');
}
}
return $result;
});