I am getting errors (too many connections) when I cURL from a parent to multiple children and execute mysqli queries
I have tried using persistent connections in just the parent and in the parent and children. When I cURL from the parent to 25 children as a test, roughly 20 will process and then I get a "too many connections" error. Logs confirm that not all the child processes are executing.
Here's the code for the parent:
<!-- parent start -->
$link = mysqli_connect('p:host', 'user', 'password', 'database');
$sql = mysqli_query($link, "SELECT * FROM t1");
$rowcount = mysqli_num_rows($sql);
if ($rowcount > '0') {
$mh = curl_multi_init();
$active = null;
}
while ($row = mysqli_fetch_assoc($sql)) {
$sql2 = mysqli_query($link, "SELECT id FROM t2 WHERE field1 = 'Waiting'");
$row2 = mysqli_fetch_assoc($sql2);
extract($row2);
// spawn child process
$ch = '$ch' . $id;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'loadtest.child.php');
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_USERAGENT, 'loadtest.parent.php');
curl_setopt($ch, CURLOPT_TIMEOUT, 1);
$curl_data = 'jid=' . $id;
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $curl_data);
curl_multi_add_handle($mh,$ch);
}
do {
$mrc = curl_multi_exec($mh, $active);
} while ($active > 0);
curl_multi_remove_handle($mh, $ch);
curl_multi_close($mh);
?>
<!-- parent end -->
And the child:
<!-- child start -->
$link = mysqli_connect('p:host', 'user', 'password', 'database');
$sql = mysqli_query($link, "SELECT * FROM t3 WHERE field1 ='" . $jid . "' LIMIT 1");
$row = mysqli_fetch_assoc($sql);
extract($row);
$sql2 = mysqli_query($link, "SELECT * FROM t4 WHERE field1='" . $jid . "' LIMIT 1");
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'loadtest.script.php);
$result = curl_exec($ch);
curl_close($ch);
$sql_update = mysqli_query($link, "UPDATE t1 SET status = 'Processed' WHERE field1 = '$jid'");
<!-- child end -->
I have removed some code from both parent and child that writes to logs, and the reference in child to loadtest.script.php is to a placeholder that does nothing more than write one line to a text file.
Right now I am on a shared server and have a maximum of 240 connections that I cannot change. I will be going to a dedicated server in the near future.
Any suggestions on how to get the children to run on the same persistent connection and avoid the "too many connections" mysql error?
Also any other suggestions would be appreciated.
Thanks.