Googlebot, Robots.txt Sitemaps and WordPress Plugins

It’s a bit of a mouthful for a title I know, but I hope this post will be useful for a number of reasons.

Recently, my posting frequency on this blog has been a lot lower than normal because I have been involved in setting up an offline business as well as building a new site for that business. I don’t have the facilities to outsource all these things so it’s been a lot of hours of hard work.

Additionally, if you read my recent post about the DDOS attack on this domain you will know that it caused havoc for far longer than it really should have been allowed to. This is just the background to the events that led to me deciding to write this post because it was a combination of all these things that led me to some important discoveries that will hopefully spare others a lot of frustration should they encounter similar problems.

When dealing with SEO and blogs and WordPress there are all sorts of things to consider, and anyone who knows me will be aware that I’m no great techie, SEO or code expert. Very far from it but I know enough to get by. This serves to warn you however that you really do need to be careful and keep a close eye on matters involving your sites and the search engines.

Following the DDOS attack, I was not surprised to see TheCaymanHost.com disappear from the SERPS. The crawlers will eventually stop visiting if they keep butting up against errors. I was not too concerned as I knew once the site was live again the bots would return and find me and restore my listings. I kept an eye on things and noticed that although some results were appearing in Google, quite a few previously well performing pages were not there at all. I figured I just needed to be patient which, as it turns out was a mistake.

I was then occupied with building a new WordPress site and giving it a kickstart in the traffic stakes so I didn’t pay much mind to TheCaymanHost.com for a couple of weeks. I got the new site ready, added some content, installed all the plugins I needed and begun with some basic SEO – social bookmarking, directory submissions and other link building efforts. Sure enough I soon started appearing in the SERPS through those link building efforts, but the site itself didn’t seem to be getting indexed. After two weeks I realized that something was definitely not right – Google will normally spider you far more quickly if you follow the techniques I had implemented already, often within hours or at least days.

I headed over to Google’s webmaster tools to see if I could find a problem and very quickly I found several.

I will make a very embarassing admission first to get it out of the way. It’s been a while since I set up a new WP blog but I had made a very rookie mistake. WordPress has a privacy settings option and, you guessed it, I had left it at the default setting, thus preventing the search engines from visiting!! Thinking that was my problem, I changed the setting and went to bed rather red faced. If you set up a WordPress blog, make sure one of the first things you do is change that privacy setting if you want your site to be indexed!!

However, although this discovery was definitely one reason for the crawlers’ lack of interest in my site, it was far from the only issue. The next day I realized that things were no better as my sitemaps still had big red X’s next to them in the Webmaster tools area.

Initially I thought the culprit might be the Google XML Sitemaps plugin so I went in and tweaked the settings a little and rebuilt my sitemaps before submitting them to Google again. Nothing I tried had any effect – Google was still saying that it could not access my robots.txt file and could not therefore read my sitemaps. I tried removing my robots.txt file completely and turned off the plugin as it generates a dynamic robots.txt from within WordPress. This also failed to sove the sitemaps issue.

I then decided to check my sitemaps for thecaymanhost.com and found exactly the same issues.

It dawned on me that there were probably only two possible culprits – something in my htaccess file or a server side issue. I had made some htaccess changes on thecaymanhost.com but the new blog did not have the same htaccess file, so I was pretty sure it had to be something I needed to talk to my host about.

Before doing so I ran some tests using the tools at Web-Sniffer.net and SEOConsultants.com and sure enough, with all the user agents for the major search engines my sites were throwing up 500 internal server errors. I checked some of the other sites with whom I share server space and sure enough, they were all delivering up the same results! If you are on a shared hosting plan you can see who your close neighbors are using the service at Find_IP-Address.org

So, I had some indications that after tightening up server security, my hosts had inadvertently shut the door on the good guys as well as the bad ones.

However, checking .htaccess was my final port of call before contacting them and I discovered some very odd lines in both of my .htaccess files. They were puzzling because they were not lines I had added manually and I had no idea where they had come from. I have to assume it was a plugin although I still haven’t figured out which one unfortunately. I removed all the lines (four in total examples in the link at the foot of this post) and tried to submit my sitemaps to Google again.

Had I not been in the process of building a new site and trying to rank well for my keywords, it probably would have been a lot longer before I noticed all this. The drop in traffic on this domain I just put down to a recovery period following all the downtime. Instead of stats showing a few thousand different keyphrase searches for a month I was showing only a few hundred. Some of this was down to the closure of the article directory for sure, but the real problem was more serious than I thought.

I am now able to see the robots all crawling again and will hopefully see a full recovery and start getting traffic back to the way it was and a lot more activity on the new WordPress blog!!

The moral of the story is that you really must keep an eye on your stats and investigate unusual behavior immediately. There are so many possible causes but by following the steps I did you will be able to find a solution by a process of elimination. Google webmaster tools are your friend, so you really should be familiar with them whatever kind of site you are running.

There is no doubt that if you want to improve your WordPress site’s SEO performance you need to do a little work. Using sitemaps and a good robots.txt file are just part of that process, but even if you’ve done everything right in those areas it can all be for nothing if you don’t check periodically to make sure all is working as it should be.

The solution to my problem has been posted on the WordPress.org forums and you can read the relevant thread at WordPress .htaccess and robots.txt problems

TCH