If you do have a real issue, Vanessa Fox suggests that you benchmark. Look at the top ten queries that give you the most traffic and find out what pages rank for them and what the position is. By doing this, you can better determine where the problem is.

To verify a crawling issue, Vanessa advises that you run a script over the server logs to look at search engine bot activity. This step allows you to see how well your site is crawled, how often it is crawled, and also helps to categorize your pages.

For more information on these issues, Vanessa has several checklists listed on Jane and Robot, which include: Discoverability Checklist, Accessibility Checklist, and User Conversion Checklist. Also, Jane and Robot is hosting a Developer Summit Friday, June 12 and will examine these very issues. You can access the registration information here.