怎样实现多个爬取进程同时进行，并且实现进程通信，防止重复爬取
scrapy本身就支持分布式https://doc.scrapy.org/en/latest/topics/practices.html#distributed-crawls
http://stackoverflow.com/questions/5715217/mix-python-twisted-with-multiprocessing
参考：http://blog.csdn.net/u011497904/article/details/44288771