11/08: Does Yahoo Really Suck?

SLURP........BURP!

Well, it would seem that when it Slurps it does indeed suck, and largely your bandwidth is want it craves, more than a Humvee craves the Middle East's finest vintage. If you have ever looked at your server stats, you will undoubtedly have noticed Slurp, Yahoo's crawler, muddling its way through your site, but did you notice how much more of a pig it is compared to the spiders of the other leading engines?

Now, personally I don't have too much of a problem with this, because, sadly, I never  come close to approaching my monthly bandwidth limits in any case. However, having read the very interesting article over at The Friday Traffic Report where the blog's owner has become so fed up with the greedy Yahoo spider, he has disallowed it in his robots.txt file.

It makes my little complaint about ASK.com in my last post look a little tame doesn't it? To be fair, the author at FTR has a point - the activity from Slurp should mean a translation into some reasonable listings and visitors, should it not? Yahoo is supposed to have huge numbers of users and be one of the highest traffic sites on the web.

Continuing on from my ASK.com experiment, I ran the same search on Yahoo. Now, to give Yahoo its due, its results are way more relevant than those found on Ask.com, but where am I? Nowhere. This is strange indeed, because I actually know, that I used to appear on the first page of their results, but now, after clicking through fourteen pages I still haven't turned up. As the "Yahoo slap" is not generally known, I don't think it's due to any bad behavior on my part. Something odd has happened however.

Tagging the word "hosting" on the end of the search string does bring me a result on page two, but only because of a del.icio.us bookmark, so, I tried adding "Cayman" instead and finally find an entry for this domain. Alas it is not a link to a specific article, just the home page of the blog. Now, obviously nobody will be using such a search term, but what is Yahoo doing?

Well, it would appear that for reasons known only to itself, Yahoo Slurp, isn't indexing all of my posts, and even some of the ones it does, it is not appending the full URL to them, but using the default /item/# I have no idea why this is happening, but it does explain why I do not appear  for the chosen term. I haven't changed the structure of my permalinks and I know that Slurp has been blundering around through my domain on a regular basis biting all my individual posts, and yet it is failing to index them properly??

All I can think of is that when I recently redesigned my pages, there was a brief period when not all of my permalinks were in place and this has caused such an overnight disappearance. We shall see if, in time, Slurp will return the pages in question to their results. 

Ultimately I don't think it really matters, because even with good results Yahoo just doesn't seem to drive much in the way of traffic. I can only assume this is because Yahoo users just aren't an audience for this type of site. If you are interested in doing all you can to help with the Slurping Spider this article on Yahoo search gives some facts and figures about just how big the "second largest search engine" really is and how to please it.

It's an interesting discussion and the post and subsequent comments at The FTR might persuade you to look a bit closer at your own Yahoo Slurp give and take ratios :-)  I don't intend to stop Slurp from guzzling anytime soon, but maybe a lesson in table manners wouldn't  go amiss  At least I posted something without mentioning Google.....oh damn!

Tomorrow, I'm going to take look at some of the blogs I've been reading lately and some that I haven't.

TCH

Web Hosting Directory