I'm uploading a CSV file of records to add to a MySQL database. With each row in the CSV file I need to check another MySQL table in the same database to see that the data we're entering is valid. This check is just a simple SELECT x FROM blah WHERE primery_key = value
but with 5-10,000 records in a CSV file, it takes a long time to run through them all.
I'd like to try and make this faster, but don't know what the best options is, as none of them seem too great so far...
SELECT/INSERT FROM database WHERE IN (array created from CSV file row values)
The CSV file will probably be 5000 records tops but may be as much as 10,000 in future. The table the records need to be checked against is at around 250,000 records.
Edit to add some code:
function processCSV($file)
{
if(empty($file)){
$this->default_error[] = 'No data found';
}
$lines=file($file);
foreach($lines as $line){
$row_count++;
$pieces = str_getcsv($line, ",");
simple SELECT id1 FROM my_table WHERE id2 = . $pieces[0]
if(row exists){
puts value into array to batch process inserts at the end using array_chunk()
}
}
}
It's not actual code as it's called from different classes, but I'm guessing all needed is there? The main issue is a MySQL check on a huge for loop, but let me know if you want more specifics.