I am trying to scrape some info from a site. The URLs to the pages which contain the data i need are simple to get to, but not all the values i input for the "Key" in the URL bring me to a valid page.
For example:
a_website.com/Location.aspx?KeyID=3804
Brings me to a valid page with the data i need.
a_website.com/Location.aspx?KeyID=19
Redirects to "a_website.com/net/search/Default.aspx" and says a default error message on the page such as "you made an invalid selection".
Is there a way that i can tell the script to stop and move to the next URL if it gets bounced to the error page instead of a valid page that contains the data i need?
Code snippit below:
function getCommand($key_num) {
$cmd = <<<EOD
curl "http://www.a_website.com/Location.aspx?KeyID={$key_num}
" --location -c "" > output/{$key_num}.txt &
EOD;
return $cmd;
}
If you turn off the --location flag to disable redirects, and then turn on the --fail flag, you should get back the "22" error code for missing pages.