Will search become less important? The future of the online bot armies . . .
By Tom Foremski - October 31, 2005
My recent post on Battling the online bot armies of the search engine giants and startups brought out some interesting discussions.
Over on the VeriSign Infrablog, Michael Graves, 'techno-evangelist' wrote:
.
. .there are problems with crawling. Over time, the number of crawlers grows. There used to be just a handful that got broad coverage, now there’s a dozen or more. In the future, there may be a hundred or more. At some point, Tom Foremski’s argument will become undisputable, and crawlers will have to be managed much more carefully than they are now through the use of Robots.txt or some other means.
Michael goes on to propose full content pings as a solution to the crawler problem:
For example, if this post were submitted in full as part of the ping, Googlebot and the gang wouldn’t need to come fetch this post to analyze it for inclusion in their databases and indices. It would be available from the ping server directly. Search engines could maintain a high-bandwidth, always-on connection to the ping server, and have the full content of newly published articles in hand, without having to do any fetching at all from the origin server.
Full post: I, Robot
Veteran media entrepreneur Mitch Ratcliffe, at Ratcliffeblog notes:
What I see happening is what happens with any medium as it matures—people will stop looking for new content as much as they do at first and start settling into relationships with trusted sources. This conforms to the conclusion Tom Foremski arrived at, but I believe search will play a bigger role in the Web (hell, let's call it "Web 2.0" to separate it from the first decade's worth of Web) because so many more sources are introducing new content.Long term, though, we're going to see the value of relationships, which are largely built on content. If, to reach the people I want to have relationships with I need to allocate a lot of bandwidth, I'm happy to do it.
Full post: Search engines add value, but that value is diminishing
And Niki Scevak over at Bronte Media checks my math: You Say Toe-mato I say Tomar-to
« Weekend Wrapper : More thoughts on GOOG database; the drag on internet performance from the bot armies; and SVW readers get discount to upcoming Churchill Club Mossberg/Swisher gadget fest! | Main | "Life is too short to click on an unknown" - No blind links on Blogs says Web Design Guru Jakob Nielsen »
October 31, 2005 | Permalink | Comment | Category: SearchWatch | Subscribe to SVW
- Top Stories:
- Tech Awards For Benefiting Humanity
- The Death Of The Search Algorithm? Techmeme Has Six Editors
- TEDxSF - Little TED Just Like The Big TED
- SNCR Research: Social Media IS Influencing Business Decisions
- What's Next? Beyond Real-Time...
- PearlTrees: A Novel Approach To Human Mapping Of The Internet
- MediaWatch Analysis Part II: Google Has More To Lose Than Murdoch
- MediaWatch Analysis: Murdoch Will Negotiate Payment For Access To Basket Of Content With GOOG et al
- WeekendWatcher: The Sheer Number Of Things Will Devalue Them
- ChipWatch - Where Will The Next Generation Of Engineers Come From?
- Public Healthcare Could Cut Startup Costs And Help Spur Innovation
- Is GOOG's $750m AdMob Buy Strategic Or Dumb? An alternate view...
