Robots.txt serves as a tool to guide obedient search engine spiders to the content a webmaster wants indexed. It also blocks those spiders from reaching content as the webmaster desires.
It’s a relatively simple concept to implement, and it began out of a cooperative discussion between search engines over a decade ago. Although simplistic as it is, robots.txt wants it to remain simple for Webmasters.
Robots.txt has been primarily used as an exclusion tool because people were only concerned about getting rid of spam, but now, there is a new element to it. Companies are now concerned with improving their search content. In an interview with Webpronews at the SES conference, Priyank Garg, Project Manager of Yahoo Search Priyank Garg spoke with us about this.
“It’s not just about keeping the bad guys out, it’s about pulling the good content in.”
Robots.txt is expected to bring about improvements within the next year, but they want to be very careful to keep the simple parts. Partial indexing and a continuous crawl are just a couple of the issues that are on the table.
We will continue to bring you more information from the SES conference.
If you enjoyed this post, make sure you subscribe to my RSS feed!
Tags: Priyank Garg,
robots.txt,
SES,
yahoo Download MP4
Add this video to your site
6 users commented in " SES: Yahoo Search Product Manager Priyank Garg "
Dear Sir,
What is the best of the robot text content?
There is no catchall ‘best’. Each site is different, and your robots.txt should be written expressly for the strategy you wish it to perform. Just as many think there is a catchall .htaccess solution that they can copy into each new site they build, robot.txt should also be built unique to each website according to the layout and structure of the site itself, and it’s goals overall. So in my opinion there is no ‘BEST’, only what works best for that site itself.
C. Crawford
http://www.chuckcrawford.com
Wouldn;t we all figure that with now more than 10 years of the Internet, and the Internet now grown up, can’t Yahoo!, Google, and MSN agree on a website submission search engine standard and give everyone an equal chance on ranking? My meta-tags are down, stire-map is on, robot.txt is uploaded, what else are they gonna make us run and do?
whats next?
dont forget the abuse usage, spams, fraud clicks etc… this is life and we need improvements and new things! otherwise it will be boring
[…] […]
see this ( http://www.indoswall.blogpost.com ).
Leave A Reply