Robots.txt serves as a tool to guide obedient search engine spiders to the content a webmaster wants indexed. It also blocks those spiders from reaching content as the webmaster desires.

It’s a relatively simple concept to implement, and it began out of a cooperative discussion between search engines over a decade ago. Although simplistic as it is, robots.txt wants it to remain simple for Webmasters.

Robots.txt has been primarily used as an exclusion tool because people were only concerned about getting rid of spam, but now, there is a new element to it. Companies are now concerned with improving their search content. In an interview with Webpronews at the SES conference, Priyank Garg, Project Manager of Yahoo Search Priyank Garg spoke with us about this.

“It’s not just about keeping the bad guys out, it’s about pulling the good content in.”

Robots.txt is expected to bring about improvements within the next year, but they want to be very careful to keep the simple parts. Partial indexing and a continuous crawl are just a couple of the issues that are on the table.

We will continue to bring you more information from the SES conference.

If you enjoyed this post, make sure you subscribe to my RSS feed!