Many spiders index the web every day and cause no problems. In fact, we don’t even notice them do their job. But there’s an exception to every rule and among crawlers, MSNBot is the ugly duckling.
Judge for yourself:
1. Microsoft claims that MSNbot will only request pages once every 10 seconds. In reality, MSNBot could be making 5-10 requests per second. In fact, MSNBot gets so bad at times that it could trigger a DOS attack – alone! If don’t believe me, check out what CPAN testers have to say about MSNBot.
2. There have been multiple reports about MSNBot ignoring (or not recognizing?) nofollow and noindex rules. Apparently, it is illiterate in its own language because it often does not (not sure if intentionally or not) read and follow the robots.txt. If you don’t believe me on this one, see what GitHub has to say about their experience with MSNBot.
There has even been a story about MSNBot using a different website’s robots.txt!
3. MSNBot could be crawling your website but not including it in the index. I can’t tell you how many times I have been wondering why. And I am not alone: SEO Chat forum has a thread on this.
4. MSNBot could be crawling your website and removing pages from the index, replacing them with outdated content that should be deleted. Whether it does this because it’s stupid, or as part of the global evil plan, this sucks. Search Engine Roundtable talked about MSNBot crawling fake file names some time ago.
5. MSNBot is not shy and is likely to try to access a private server. Knock-knock, damnit.
6. There have been instances when MSNBot was not recognized as a crawler altogether! Apparently, WebTrends does not like stupid.
7. Slow indexing is another well-known fault but I’m sure we’d be forgiving of this if there was solid intention to change for the bet and failedter. Really, I am not the only person who has tried to report MSNBot problems to Microsoft (see Toolserver Journal’s experience when they tried to make MSNBot better). Indeed, you could try to visit the Bing Webmaster Center troubleshoot link but it’s unlikely you’ll get anything out of it. You could also email email@example.com to get a user unknown.
In my experience with MSNBot I have gone from thinking that it’s stupid to believing that it is intentionally evil. And the lack of desire from Microsoft to fix this is really something they should be ashamed of. These issues are causing good, reputable websites to elect to block MSNBot, block it for good. Really, Microsoft, do something before we all give up on it.