I have some code that copies a file to a temporary location where it is later included in a zip file.
I already have the source files stored in a local cache directory, and have also stored the SHA1 hash of the original files. The files in question are .png images, ranging from a few kb to around 500kb.
My problem is that at high server loads, the copy intermittently fails. Upon examining my logs, I see that even though a healthy file exists in the source location, the destination contains a file with zero bytes.
So, to try and figure out what was going on and to increase reliability, I implemented a SHA1 check of the destination file, so that if it fails, I can retry the copy using the shell.
99.9% of the time, the files copy with no issue. Occasionally, the first copy fails but then the second attempt succeeds. In a few number of cases (around 1 in 2,500; and always at high server load), both copies will fail. In nearly all these cases SHA1 of the destination file is da39a3ee5e6b4b0d3255bfef95601890afd80709
which is consistent with an empty file.
In all occasions, the script continues, and the created zip includes an empty image file. There is nothing in the Nginx, PHP or PHP-FPM error logs that indicates any problem. The script will copy the same file successfully when retried.
My stack is Debian Squeeze with the .deb PHP 5.4/PHP 5.4 FPM packages and Nginx 1.2.6 on an Amazon EBS backed AMI. The file system is XFS and I am not using APC or other caching. The problem is consistent and replicable at server loads >500 hits per second.
I cannot find any documentation of known issues that would explain this behaviour. Can anyone provide any insight into what may be causing this issue, or provide suggestions on how I can more reliably copy an image file to a temporary location for zipping?
For reference, here is an extract of the code used to copy / recopy the files.
$copy = copy ($cacheFile, $outputFile);
if ($copy && file_exists($outputFile) && sha1_file($outputFile) !== $storedHash) {
// Custom function to log debug messages
dbug(array($cacheFile, sha1_file($cacheFile),
$storedHash, $outputFile,
file_exists($outputFile), filesize($outputFile)),
'Corrupt Image File Copy from Cache 1 (native)');
// Try with exec
exec ("cp " . $cacheFile . " " . $outputFile);
if (file_exists($outputFile) && sha1_file($outputFile) !== $storedHash)
dbug(array($cacheFile, sha1_file($cacheFile),
$storedHash, $outputFile,
file_exists($outputFile), filesize($outputFile)),
'Corrupt Image File Copy from Cache 2 (shell exec)');
}