I'm trying to scrape using Google "I'm Feeling Lucky" button.
For small query like 'iteminfo.ca' it works, because it redirects me to iteminfo.ca.
This is the query url:
http://www.google.com/search?hl=en&source=hp&q=iteminfo.ca&btnI=I%27m+Feeling+Lucky
But for the query like '061754020164 site:iteminfo.ca' it doesn't work :
http://www.google.com/search?hl=en&q=061754020164+site%3Aiteminfo.ca&btnI=I%27m+Feeling+Lucky
Because it is not redirecting to http://www.iteminfo.ca/Product.aspx?id=1011972978 as it does when you use that query and hit 'I'm Feeling Lucky' on google homepage.
Any ideas what I'm doing wrong.
Try using the Google Search API rather than screen-scraping.
There's more information here, including PHP example code:
You'll get lucky more often by using the new style URLs: