What is the proper way to download like 50+ PDF files using PHP curl?
Below is the code I am using:
$fp = fopen("test.pdf", 'w');
$ch = curl_init();
curl_setopt($ch,CURLOPT_PROXY,"http://test:123");
curl_setopt($ch,CURLOPT_PROXYPORT,123);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, 0);
curl_setopt ($ch, CURLOPT_TIMEOUT, 500);
curl_setopt($ch,CURLOPT_URL,"http://download/test.pdf");
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_exec($ch);
curl_close($ch);
fclose($fp);
The problem is sometimes, 2-3 PDF files are not properly downloaded (0-1 bytes in size). Any idea why it is unstable? Must I include anything else in the code above?
Thanks In Advance
Web requests in general can be pretty dodgy even from a browser and a good connection. There could be a few things going wrong here from the remote site refusing your connection, to simultaneous execution of your php file from multiple locations. Have a look at:
http://www.php.net/manual/en/function.curl-error.php
to try to print out error messages somewhere for analysis on curl_exec.
If it's something like a connection issue, you may need to introduce a retry mechanism, Not sure if your script is likely to be executed in parallel from multiple locations, but if it is, you should have a look at http://php.net/manual/en/function.uniqid.php as a means to generate a reasonably unique filename first, attempt download, then only move the file to the intended destination file if no error is returned, otherwise, retry up to X times.
With respect to the stability of Curl, I use it frequently and it hasn't proven unstable for me, but I'm not pulling down binary data. I'm hoping some error detection and output from the error messages may shed some more light on what the root cause is here.