MSN sends out aggressive RSS bot, apologizes
By Richard Koman - August 23, 2005
Like a pledge who hits on every woman at a frat party in the first hour, only to be taken aside by a brother and told to chill, MSN Search is apologizing for its overly aggressive RSS bot, which was hitting feeds every 10 minutes. According to SEW there is a robots.txt command to slow things down. Says Eytan Seidman, MSN Search program manager, (and kudos to him for knowing about it, linking to it, and responding quickly):
At this point we are just experimenting and suffice to say that in our short experiment we have learned a lot. We have been a bit more zealous in our crawling then we would have liked and we are going to fix that. Thanks for the feedback!
By Richard Koman - August 23, 2005 | Permalink | Comment
| Category: Microsoft [MSFT]
| SVW Toolbar | SVW Newsletter | SVW Mobile
- NEW STORIES:
- 2009: The Year Of The Great Repression
- Letter From London . . .
- FutureWatch: The End Of The News Aggregators And The Future Of News
- Saturday Post: Stemming Deflation When Thrift Is The Answer . . .
- The State Of VC Funding In Southern California
- Shrinking Mass Media Masses At Googleplex
- The Black Swans Have Yet To Come Home To Roost . . .
- No Men Allowed: Girls In Tech Expands To New York, LA, And Beyond
- Let's Take A Lesson From The Chip Industry: Turn The Big 3 Auto Makers Into Car Foundries . . .
- Doug Engelbart In 1968 - The Computer Demo That Changed Lives!