How to fix a mystery delisting from Google
May 13th, 2009 | Author: Dan Harrison
At the beginning of this month (May), I noticed that the EnviroGadget homepage had disappeared from the Google search results. Most of the other pages were still visible in the index, but mainly the home page plus a few other key pages were missing. At first I thought it was a penalty, but then I wasn’t doing anything to deserve one.
After some digging in Google Webmaster Tools, I noticed that there was an error with my sitemaps for most of my sites (which are currently on the same server). This was the error that I was getting:
URL timeout: robots.txt timeout. We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.
After some research, I found an article “Banned” By Google? Find Out How to Entice Googlebot to Recrawl Your Site, which explained that the error was due to the Google bots being blocked in some way by the server, and therefore not being able to check the robots.txt for instructions on what to crawl. Essentially, blocking the Google bots was stopping my site from being crawled.
As a result of not being crawled, my site was removed from the index! Having asked my hosting company to remove the Google bots ip from the firewall rules, my sites started getting crawled again. And thankfully, they now appear in the search results once more.
Phew!
Posted in 
Tags: 
Thanks for posting this Dan, I think I’m having the same problem on a few of my sites, was starting to think Google was dishing our penalties for fun. :o)
I was pretty concerned when I got delisted, so I thought others would benefit from it. Penalties can still happen, but it’s worth looking for the simplest answers first. :)
Dan
Wow great tip! thats the exact error I have on some of my sites thanks for the heads up and fingers crossed for re-inclusion.
Thnaks
No problem Dale. Webmaster tools are damn useful!
Dan