Silicon Valley Watcher - Former FT journalist Tom Foremski reporting from the intersection of technology and media

MSN sends out aggressive RSS bot, apologizes

Posted by Richard Koman - August 23, 2005

by Richard Koman for SiliconValleyWatcher

Ooops, Sorry.jpgLike a pledge who hits on every woman at a frat party in the first hour, only to be taken aside by a brother and told to chill, MSN Search is apologizing for its overly aggressive RSS bot, which was hitting feeds every 10 minutes. According to SEW there is a robots.txt command to slow things down. Says Eytan Seidman, MSN Search program manager, (and kudos to him for knowing about it, linking to it, and responding quickly):

At this point we are just experimenting and suffice to say that in our short experiment we have learned a lot. We have been a bit more zealous in our crawling then we would have liked and we are going to fix that. Thanks for the feedback!

Story link | Subscribe free | Categories:


The Holmes Report names Tom Foremski one of the top 25 Innovators of 2013.