I have an application that downloads favicon files. Recently I switched from using file_get_contents to curlExec because it has a higher success rate.
However when I try and download from www.prisonexp.org , instead of sending the actual file data they send the text Forbidden. Normally, I would see a bunch of binary data converted to ASCII in this field in the test script.
I find this strange because I can just browse to the file in the browser and download it manually.
Is this valid? Or am I missing something? How are they preventing a download one way but not another. To see the test script in action go here.
As a side, how can I detect when instead of sending binary data, a server sends a message as text? I could just check for "Forbidden" but I'm not sure if this is a standard response.
Research / Update
Things that can be done programmatically have to be throttled back or servers could become overwhelmed.
Reading the user-agent is one way this is done.
Wget might have more access as it is commonly used as a command line tool. Similarly images delivered to your client you will have access to directly.