I'm new for elasticsearch. So far I've managed to create index, mapping and indexing documents.
With PHP I can add documents from mysql. By looping through all the mysql rows I add the documents one by one.
foreach($items as $item){
$ch = curl_init();
$json_request = '{
"title": "'.$item['title'].'",
"description": "'.$item['description'].'",
"price": "'.$item['price'].'"
}';
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_PORT, 9200);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'PUT');
curl_setopt($ch, CURLOPT_POSTFIELDS, $json_request);
$result = curl_exec($ch);
curl_close($ch);
}
It works for small amount of data. But when I use it for 500 000 items, it took days indexing.
Could some please tell me how I can do better? Thanks.
Two things I'd do:
First, if you want to keep Elasticsearch continually updated with the data from mysql I'd take a look at the Elasticsearch MySQL River: https://github.com/jprante/elasticsearch-river-jdbc
Second, I'd look at the Elasticsearch Bulk API: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-bulk.html This will let you make a single call to insert large numbers of documents with a single API call and is far more efficient when loading large numbers of documents.