In the past I have used my own crawler, but now I need something more robust and I was wondering if there were any good free php crawlers?
http://php-crawler.sourceforge.net/