I am working on the solution where I have to upload 33,000 or more csv files to a server.
My script is working fine for upto 25,000 files but then it breaks sometimes after 26,000 files upload or sometimes 27,000 files upload etc. with the following PHP Warning i.e.
PHP Warning: ftp_put(): bind() failed: Address already in use (98)
I have researched this error over the internet but didn't find the solution, therefore, is posting here.
I checked the directory no. of files limit on the Ubuntu 14.0.4 LTS Server with ext4 filesystem and couldn't find anything that restricts 33,000 files upload to the single directory.
I tried to upload the same 33,000 files to upload to two different directories i.e. 20,000 to DirA and rest 13,000 to DirB but after 5,000 successful files transferred to DirB, it stops and gives the above error.
Here is the basic code that I am using to upload the files as reference i.e.
$dpath = "/var/www/data/";
$sourceDir = $dpath .'test1';
$remoteDir = 'test2';
$serverIp = "192.168.0.1";
$user = "dataexport";
$pass = "abc";
$connHandle = ftp_connect($serverIp);
if($connHandle)
{
$loginFlag = ftp_login($connHandle, $user, $pass) or die('couldnt connect to $serverIp');
if($loginFlag)
{
if(is_dir($sourceDir))
{
$dirHandle = opendir($sourceDir);
if($dirHandle)
{
while(($file = readdir($dirHandle)) !== false)
{
if($file != "." && $file != "..")
{
if(is_dir($sourceDir.'/'.$file))
{
// code for copying directory.. I am not using it.
}
else
{
$srcFilePath = $sourceDir.'/'.$file;
$desFilePath = $remoteDir.'/'.$file;
if(ftp_put($connHandle, $desFilePath, $srcFilePath, FTP_ASCII))
{
echo "Copied Successfully to path: $desFilePath
";
}
else
{
echo "Copying failed
"; die();
}
}
}
}
closedir($dirHandle);
}
}
else
{
echo "Not a valid directory <br />
";
}
}
else
{
echo "Wrong Credentials for the server with IP = $loginFlag <br />
";
}
}
else
{
echo "Unable to connect to server with IP = $serverIp <br />
";
}
ftp_close($connHandle);
}
NOTE: I modified my real code to make it simple and didn't test it may be it will require some testing.... But I am getting the error reported at the following code line after 25,000 files successfully transferred. i.e.
ftp_put($connHandle, $desFilePath, $srcFilePath, FTP_ASCII)
I tried the file upload with "FTP_BINARY" option as well but didn't work.
Looking for your any good solution advice. Thanks in advance.
Cheers
What is probably happening is that for each file transferred, the client uses a different local port on the VPS. After you upload that many thousand files, all ports on the system are in use.
You can run netstat -anop on the VPS to see the ports used. You may need to adjust the value of net.ipv4.ip_local_port_range to increase the number of ports available and decrease the net.ipv4.tcp_fin_timeout value to free up ports sooner.
If possible you could use SFTP which uses SSH and will not have the same port usage limitations.
The error message is pretty explanatory:
bind() failed: Address already in use (98)
This takes the number of files out of the question, also it has nothing to do with the transfer mode.
The reason is buried deep into the internals of the FTP protocol. The FTP is an old protocol that uses a connection for commands and a new connection for each file it transfers. Each TCP connection is bound to a port; the port number is a 16-bits unsigned integer. This leads to maximum 64k ports available.
The first 1024 port numbers (0..1023) are reserved for the well-known protocols (80 for HTTP, 25 for SMTP, 22 for FTP command connection and so on). The range 1024..49151 contains the "registered" ports and the port numbers between 49152 and 65535 contains the "dynamic" ports. They are available for the OS to bind short-lived connections to them. There are 16k port numbers available in the "dynamic" range.
Some operating systems use a larger range for dynamic ports. As noted on the Wikipedia page):
Many Linux kernels use the port range 32768 to 61000.
There are about 28k ports available in this range. Take into account that there are other programs running on the computer during the execution of your script; many of them create TCP connections and those connections use ports too. This reduces the number of port numbers available to an application.
I don't know what happens inside the PHP's implementation of FTP but it looks like it doesn't close the data connections it uses for file transfers and after several thousands files, the entire stock of free ports that can be used by the applications is consumed and the system call bind() fails.
Since apparently there is no way to configure the behaviour of the FTP functions in PHP, you can try to close the FTP connection and reopen it. Hopefully, PHP will release all the associated resources, including the many ports it used for file transfers.
I didn't check this, I don't know if it works!
In case it doesn't work you can try to use the ssh/sftp functions provided by PHP. SSH/SCP is a protocol designed to avoid many of the flaws of older protocols (like FTP). You can find a basic usage example on the documentation page of function ssh2_scp_send().