By Tom Foremski, Silicon Valley Watcher.com

My recent post on Battling the online bot armies of the search engine giants and startups brought out some interesting discussions.

Over on the VeriSign Infrablog, Michael Graves, 'techno-evangelist' wrote:

.

. .there are problems with crawling. Over time, the number of crawlers grows. There used to be just a handful that got broad coverage, now there’s a dozen or more. In the future, there may be a hundred or more. At some point, Tom Foremski’s argument will become undisputable, and crawlers will have to be managed much more carefully than they are now through the use of Robots.txt or some other means.

Michael goes on to propose full content pings as a solution to the crawler problem:

For example, if this post were submitted in full as part of the ping, Googlebot and the gang wouldn’t need to come fetch this post to analyze it for inclusion in their databases and indices. It would be available from the ping server directly. Search engines could maintain a high-bandwidth, always-on connection to the ping server, and have the full content of newly published articles in hand, without having to do any fetching at all from the origin server.

Full post: I, Robot

Veteran media entrepreneur Mitch Ratcliffe, at Ratcliffeblog notes:

What I see happening is what happens with any medium as it matures—people will stop looking for new content as much as they do at first and start settling into relationships with trusted sources. This conforms to the conclusion Tom Foremski arrived at, but I believe search will play a bigger role in the Web (hell, let's call it "Web 2.0" to separate it from the first decade's worth of Web) because so many more sources are introducing new content.

Long term, though, we're going to see the value of relationships, which are largely built on content. If, to reach the people I want to have relationships with I need to allocate a lot of bandwidth, I'm happy to do it.

And Niki Scevak over at Bronte Media checks my math: You Say Toe-mato I say Tomar-to