I'm using Perl script to parse and create JSON output for use in PHP.
Basically I'm parsing large text files(5-20MB) using Perl, with data as following:
XXXXXXYYYYYYYYYYYYYZZZZZZZZZYYYYYYYYYPPPPPPPPPPP
XXXXXXYYYYYYYYYYYYYZZZZZZZZZYYYYYYYYYPPPPPPPPPPP
XXXXXXYYYYYYYYYYYYYZZZZZZZZZYYYYYYYYYPPPPPPPPPPP
XXXXXXYYYYYYYYYYYYYZZZZZZZZZYYYYYYYYYPPPPPPPPPPP
The script prints JSON output of each line of data from file:
{ "1" : "XXXXXXYYYYYYYYYYYYYZZZZZZZZZYYYYYYYYYPPPPPPPPPPP", "2": ... }
When ran from terminal, it goes over 100,000 lines of such file in 1.5 seconds, successfully creating JSON.
But when I run the script from within PHP using
exec("/usr/bin/perl script.pl input_file.txt",$output);
print_r(json_decode($output[0],true);
it crashed. Though, if I give it file with 2000-3000 lines, it works well. Not using json_decode and only doing var_dump of $output variable works as well.
I assume it has to do with json_decode.
Is there other way to do this? Suggestions? Solutions?
p.s. I've increased the PHP memory_limit to 128MB.
Thank you.
EDIT:
Perl Script:
#!/usr/bin/perl -w
use strict;
use warnings;
my $file = $ARGV[0];
my $id = 0;
open my $info, $file or die "Could not open $file: $!";
print "{";
while( my $line = <$info>) {
print "\"$id\" : " . "\"" . trim($line) . "\"";
print ',' unless eof;
$id++;
}
print "}";
sub trim {
(my $s = $_[0]) =~ s/^\s+|\s+$//g;
return $s;
}
close $info;
The error does seem to be in your php.ini file. This controls a host of configurable options for your PHP installation. The issue for you is most likely the following:
; Maximum amount of memory a script may consume (128MB)
; http://php.net/memory-limit
memory_limit = 1M
That is just an example. But it seems to be the case that your memory limit is way to small. Try increasing it to something much larger to see if that fixes the problem. Like the following:
; Maximum amount of memory a script may consume (128MB)
; http://php.net/memory-limit
memory_limit = 56M
Also note that anytime you edit your ini file, you'll have to restart your PHP server to have the ini changed be applied. This is an easy step to forget and can lead to serious frustration.
If it's running from console the error lies in your php-config. This could be all sort of configuration error from script-execution-timeout to memory issues or script execution rights. In your case I'd vouch for some timeout occuring or script execution rights, depending on how long it takes to crash. Check your php-error-log for details ...
I think this can be done in pure PHP but i would just concentrate on the error you are having
The Reason you got Server Error" HTTP 500. is because you did not enable error reporting. trying switching on error reporting you would find out that $output returns empty array so $output[0] is not valid see PHP call class in class returns error:500
When working with exec ensure you use full path
$buid = 'FULL PATH TO /script.pl' ;
$input = 'FULL PATH TO /input_file.txt' ;
exec("$buid $input",$output);
print_r($output);
Output
Array
(
[0] => Ok{"0" : "XXXXXXYYYYYYYYYYYYYZZZZZZZZZYYYYYYYYYPPPPPPPPPPP","1" : "XXXXXXYYYYYYYYYYYYYZZZZZZZZZYYYYYYYYYPPPPPPPPPPP","2" : "XXXXXXYYYYYYYYYYYYYZZZZZZZZZYYYYYYYYYPPPPPPPPPPP","3" : "XXXXXXYYYYYYYYYYYYYZZZZZZZZZYYYYYYYYYPPPPPPPPPPP"}
)
You don't need /usr/bin/perl in your exec because your script already starts with #!/usr/bin/perl -w