I have a recursive function to iterate over 11M database records, 1000 at a time. As it approached 9M it stopped. My assumption of a memory problem was confirmed when I displayed get_memory_usage() after every 1000 records.
The function works something like this:
<?
get_data_block();
function get_data_block($id=0);
{
//open a csv file for writing
$packages_sorted_csv=fopen("./csv/packages_sorted.csv", "a");
//get 1000 records and process them
//$unsorted = array of 1000 records from database
foreach($unsorted as $row);
{
$ct++;
$id++;
//$packages_sorted = array of processed data
//write output
fputcsv($packages_sorted_csv, $packages_sorted);
}
fclose($packages_sorted_csv);
if($ct==1000)
{
unset($unsorted);
echo 'Mem usage: '.memory_get_usage();
get_data_block($id); //call the function again
}else{
//finished
}
}
?>
Does anyone have a tip about how to release all resources with recursive functions? ...or is there a way to call the same function again so it's not called by itself?
Notes:
From the code you posted, recursion doesn't seem like the right approach.
Build a second function and rearrange that one so that you can do something like (pseudo-code):
while (!$done) {
$id = get_data_block($id);
$done = // determine if finished or not
}
i.e. return $id from get_data_block, giving it an invalid value if no more processing needs to be done. You won't have to worry about the stack frames piling up this way.
A good programming language should recognize tail recursion and turn it into an implicit loop.
PHP apparently doesn't, so at some point you'll get a stack overflow. Try rewriting your code to use a loop instead, or e.g. a processing queue to iterate over.