“Make one URL for each piece of content or exclude the other using robots.txt.”

If the content of a search doesn’t show up, many times it can be found in the supplemental results tag. Whalen touches upon a few matters that she posted on her site dealing with common misconceptions such as scripts from external files, altering the templates of each page of your website, and not commenting in code.

Whalen concludes by discussing the High Rankings Seminars that are held two or three times each year. The seminars include five or six speakers that offer basic information that is easy to implement.

It’s a relatively simple concept to implement, and it began out of a cooperative discussion between search engines over a decade ago. Although simplistic as it is, robots.txt wants it to remain simple for Webmasters.

Robots.txt has been primarily used as an exclusion tool because people were only concerned about getting rid of spam, but now, there is a new element to it. Companies are now concerned with improving their search content. In an interview with Webpronews at the SES conference, Priyank Garg, Project Manager of Yahoo Search Priyank Garg spoke with us about this.

“It’s not just about keeping the bad guys out, it’s about pulling the good content in.”

Robots.txt is expected to bring about improvements within the next year, but they want to be very careful to keep the simple parts. Partial indexing and a continuous crawl are just a couple of the issues that are on the table.

We will continue to bring you more information from the SES conference.