MSN sends out aggressive RSS bot, apologizes
By Richard Koman - August 23, 2005
Like a pledge who hits on every woman at a frat party in the first hour, only to be taken aside by a brother and told to chill, MSN Search is apologizing for its overly aggressive RSS bot, which was hitting feeds every 10 minutes. According to SEW there is a robots.txt command to slow things down. Says Eytan Seidman, MSN Search program manager, (and kudos to him for knowing about it, linking to it, and responding quickly):
At this point we are just experimenting and suffice to say that in our short experiment we have learned a lot. We have been a bit more zealous in our crawling then we would have liked and we are going to fix that. Thanks for the feedback!
« If Silicon Valley is to become the Las Vegas of Innovation it has to improve the local public schools | Main | [Publisher's Note] Mea Culpa: Lost syndication feeds...the comments problem (fixed!) ...I'm way backed up on my emails (sorry!) ...and I'm moving apartments »
August 23, 2005 | Permalink | Comment | Category: | Subscribe to SVW
- Top Stories:
- Tech Awards For Benefiting Humanity
- The Death Of The Search Algorithm? Techmeme Has Six Editors
- TEDxSF - Little TED Just Like The Big TED
- SNCR Research: Social Media IS Influencing Business Decisions
- What's Next? Beyond Real-Time...
- PearlTrees: A Novel Approach To Human Mapping Of The Internet
- MediaWatch Analysis Part II: Google Has More To Lose Than Murdoch
- MediaWatch Analysis: Murdoch Will Negotiate Payment For Access To Basket Of Content With GOOG et al
- WeekendWatcher: The Sheer Number Of Things Will Devalue Them
- ChipWatch - Where Will The Next Generation Of Engineers Come From?
- Public Healthcare Could Cut Startup Costs And Help Spur Innovation
- Is GOOG's $750m AdMob Buy Strategic Or Dumb? An alternate view...