有没有用过scrapy的啊，来指导下。

[jazdelu]

start_urls=['http://dsb.66wz.com/','http://wzwb.66wz.com/','http://wzed.66wz.com/','http://wzrb.66wz.com/',]

allowed_domains=['dsb.66wz.com','wzed.66wz.com','wzrb.66wz.com','wzwb.66wz.com']

rules = (Rule(SgmlLinkExtractor(allow=r'html/\d+-\d+/\d+/content_\d+.htm'),callback='parse_news'),)


以上是我爬虫的部分代码
为什么1这个网页我不爬不到但是2就能爬到？
是不是这新闻网设置了什么robot.txt？



1.http://wzed.66wz.com/html/2012-12/12/content_1356694.htm


2.http://wzed.66wz.com/html/2013-02/05/content_1391565.htm