I'm having an issue with a script I've written. I'm doing some basic screen scraping and saving it to a WordPress database. The script functions as it should but when I'm trying to do multiple pages (like around 20 pages), it gives me a 500 error after running a big chunk of the script. On the script page I included this:
set_time_limit(0);
ignore_user_abort(1);
I have also included max_input_time = 0 in the php.ini file in the root of my server which in theory should prevent it from timing out but I am connecting to two other URLs so I'm curious if it may have something to do with one of those connections. Basically, it goes like this:
Script in folder on server -> Queries info from a URL and pulls it back into my server -> Analyzes the data -> Sends it to WordPress database on my server but in a different directory
Is there any way to determine where the timing out is happening? My assumption is that it's not happening on the script page but with a generic 500 error, I'm not sure how to troubleshoot deeper to identify what's happening.
Thanks for your help!
EDIT
I turned on error reporting and here's the error I received:
[Thu Jul 03 21:29:12 2014] [8808155] [fcgid:warn] (104)Connection reset by peer: [client 50.184.161.228:51983] mod_fcgid: error reading data from FastCGI server
[Thu Jul 03 21:29:12 2014] [8808155] [fcgid:warn] (104)Connection reset by peer: [client 50.184.161.228:51983] mod_fcgid: ap_pass_brigade failed in handle_request_ipc function
The IP address in the error is not my IP address nor is the IP address of the site that I'm scraping from so I'm not sure where that came in.
I'm not sure if this is related to server configuration which obviously I would be limited to updating since I'm on a GoDaddy shared server.
I found this article:
Which I suppose answers my question. I guess I can't update the settings. Now with that said, is there a way that I can have the script crawl five pages, rest, and crawl five more, etc. I've used sleep() but I assume that that still is part of running the 'script' so it's not like it's stopping and restarting, it's just continuing the script but pausing for a second?
You might want to look at phpinfo() to see if running scripts in background is even allowed. ignore_user_abort might be disabled there.
After that - are you really sure that you have access permissions to the 'database in another directory'? I am not sure about the access permissions from users of one site to users of another site, unless in your setup you have the same user or have user, which was granted access privilleges to both databases.
About IP address - some hosting providers hide their entire network behind the set of public IP addresses. Look into $_SERVER variable and see if you have X_HTTP_FORWARDED with that strange IP address to check for this bit of info. If present, it means that outgoing connection from your site goes through some proxy server, while locally your website has another IP address. And, connection from the website on that server to another website on that server might be using this local IP address and not external one.