SES: Yahoo Search Product Manager Priyank Garg

Robots.txt serves as a tool to guide obedient search engine spiders to the content a webmaster wants indexed. It also blocks those spiders from reaching content as the webmaster desires.
It’s a relatively simple concept to implement, and it began out of a cooperative discussion between search engines over a decade ago. Although simplistic as [...]