Speaking on sitemaps specifically, Watlington tells WebProNews that people still do not understand how to use them, even though they have been around for 5 years. She has recently encountered multiple sites that have thousands of pages that are not getting picked up by the engines. The problem is that many of their pages have not even been visited.

To correct this, she suggests that site owners rethink their strategy for pushing content. In other words, it is better to have less content that is high quality and link-heavy, than to have lots of content that the spider cannot see.

Watlington goes on to offer advice for situations pertaining to when a company makes changes to its site. She says by pushing a sitemap with content that has new urls or urls that it wants to be found through 301 redirects, the company essentially makes the spider notice the changes faster. She compares this process to sending out change-of-adddress cards in bulk.

“One of the most important things is to use the sitemaps as a strategic tool, not as a blunt club,” says Watlington.

Sitemaps are very powerful and useful, if utilized correctly. Are you creatively using sitemaps to push your content?

On a side note, Watlington also advises marketers to make improvements to their site in order to take advantage of the “early bird” opportunities that will occur when the economy recovers completely.

If you do have a real issue, Vanessa Fox suggests that you benchmark. Look at the top ten queries that give you the most traffic and find out what pages rank for them and what the position is. By doing this, you can better determine where the problem is.

To verify a crawling issue, Vanessa advises that you run a script over the server logs to look at search engine bot activity. This step allows you to see how well your site is crawled, how often it is crawled, and also helps to categorize your pages.

For more information on these issues, Vanessa has several checklists listed on Jane and Robot, which include: Discoverability Checklist, Accessibility Checklist, and User Conversion Checklist. Also, Jane and Robot is hosting a Developer Summit Friday, June 12 and will examine these very issues. You can access the registration information here.

First, know whether or not the search engines are indexing all your pages. If they aren’t, then you need to check a few things out. Make sure you have a sitemap that is working correctly, and check your own spider to make sure you’re being seen. Also, look at your combined PPC and organic data to find out what is converting and what is not.

Although having a big site or numerous sites might seem like a pain, there are some perks involved. You can create a set of templates and use them multiple times and optimize them accordingly.

Another hint for managing a large site is to make sure your links are specific. Inbound links that only link to your homepage aren’t really benefiting you. For instance, if someone is using your products or services, have them link to that page instead of the homepage. This method will help both the searcher and the company.

Large sites are a bit overwhelming, but just need to be broken down. Everyone simply needs to work together and focus on the end user’s wants and needs.

“There’s no difference in indexing based on how we found a page.”

Fox advises going along with the rules, such as making sure webmasters have a crawlable site, that’s well linked, and has excellent exclusive subject matter.

“You know, links to your site, internal links, make sure we can access and crawl the pages, make sure the content is unique and valuable, and know there’s a lot of things that go into it. We’re not just going to take the sitemap and index all these pages, no matter what they are, just because you submitted a sitemap. I mean, clearly. But, why not do both things, because a sitemap will add to, there’s a couple of times when a sitemap is particularly valuable.”

Fox feels a sitemap doesn’t take the place of those features, but works as an extra tool for webmasters.
Fox included that a sitemap may make it easier for webmasters to find problems in their sites.

“If the site has pages with errors that prevent us from crawling, the pages won’t appear in the index and those pages will be listed in the Crawl Errors section of Webmaster tools. We may not have attempted to crawl some of these pages if they weren’t in the Sitemap, so in this case, a Webmaster might be alerted to problems not otherwise known.”

Fox looks forward to future advancements to allow webmasters know about more problems. Fox thinks Sitemaps enable webmasters to get a complete look at their site.

“A sitemap is a way for you to say here is every single page on my site, here’s a comprehensive view. And, that helps us out, and it helps the site out.”

Fishkin doesn’t believe that sitemaps help.

“The data you get from the natural crawl IS valuable, and submitting an XML feed (or any other format) can cause that natural process of inclusion to be lost. If a page isn’t accessible, doesn’t carry enough link juice, or lacks unique, valuable content, I want to know about it, and the Sitemaps process can be a hindrance.”

Fishkin still believes sitemaps cause obstacles that stand in the SEO’s way.

In an exclusive video interview with WebProNews Yahoo and Google announced a collaborative site called sitemaps.org. Tim Mayer states in the video, “This is something we are announcing tonight at around 9 PM tonight (Las Vegas) Google and Yahoo have gotten together to provide webmasters and publishers a unified way to send their content … let our search engines know about new and existing content.”

Vanessa Fox of Google added, “So we are launching a new site, Sitemaps.org and that will be jointly supported by Yahoo and Google and it will have the sitemaps protocol there. Over time as we evolve it as sort of a collaborative effort all of the information will be there. We are really hoping that this will help webmasters to have an open standard and make it much easier so that they don’t have to worry about one type of feed for one engine and another type of feed for another engine.”
.

There are certain partnerships you cannot ignore and if Google and Yahoo are involved, jointly, then most of the free world will take notice. With that in mind, Tim Mayer of Yahoo and Vanessa Fox of Google have announced, exclusively to WebProNews, they have combined forces to launch Sitemaps.org. This service gives webmasters and siteowners a sitemaps.org tool that will be usable by all the search engines willing to participate.

Not only are Google and Yahoo collaborating on this project, they’ve also issued an open invitation to the other search engines willing to join. Microsoft accepted that invitation and will jointly support Sitemaps.org and its efforts.