@
Chalice 如果 你有一个页面 , 已经在搜索引擎里了, 后来你决定不让搜索引擎 索引该页面,
如果你用 robots.txt 禁止,那么搜索引擎是无法访问该页面的,也就是说它并不知道你是否让它索引,即使你在该 页面里加入 noindex 也是没用的。
https://moz.com/learn/seo/robotstxtBlock with Robots.txt
This tells the engines not to crawl the given URL, but that they may keep the page in the index and display it in in results. (See image of Google results page below.)
Block with Meta NoIndex
This tells engines they can visit, but are not allowed to display the URL in results. This is the recommended method.