MSN sends out aggressive RSS bot, apologizes

Posted by Richard Koman - August 23, 2005

by Richard Koman for SiliconValleyWatcher

Like a pledge who hits on every woman at a frat party in the first hour, only to be taken aside by a brother and told to chill, MSN Search is apologizing for its overly aggressive RSS bot, which was hitting feeds every 10 minutes. According to SEW there is a robots.txt command to slow things down. Says Eytan Seidman, MSN Search program manager, (and kudos to him for knowing about it, linking to it, and responding quickly):

At this point we are just experimenting and suffice to say that in our short experiment we have learned a lot. We have been a bit more zealous in our crawling then we would have liked and we are going to fix that. Thanks for the feedback!

Permalink | Subscribe free | Categories:

Top Stories:
• Preserving The (French) Web -- Revisiting Its Ideals
• IHS: A Korean War Would Be Catastrophic To Global Electronics Business
• Paris Is Putting "Entrepreneur" Back Into The French Language
• Worries About The Web And Google's Place In It
• Psychedelics And Technology: Albert Hoffman Five Year Anniversary
• Robert Scoble's Naked Enthusiasm
• ValleyWag Wrong On Arrington
• Do You Think Opinion Startup State Is Similar To Swipp?
• A New Meaning To "Glassy Eyed Stare?"
• How Many Fake Journalists Does Reuters Employ?
• WeekendWatcher: De Young's Friday Nights Season Is A Huge Dud
• Google Glass Is A Great Idea For The Old And Demented

Most Recent:
• Every Company Is A Media Company - My European Tour
• CultureWatch: The Genius Of Johnny Marr At The Fillmore
• CultureWatch: Haight-Street Fair Upcoming, And The Remarkable Pablo Heising 'The Mayor Of Haight Street'
• WeekendWatcher: A Killer Story At The Marsh in Berkeley
• About SVW: Heading To London, Amsterdam, Berlin, And Warsaw - Presentations On Corporate Media EC=MC
• Analysis: The Hand Of Retired CEO Andy Grove Is Seen In Intel's New Leadership

---

---

Comments powered by Disqus