24
November
2009
|
00:23 AM
America/Los_Angeles

A Single Search Index Would Speed Up The Entire Internet - A Zero Carbon Speed Boost

Yesterday I argued that a single search index administered by a non-profit could address issues around de-listing and indexing out-of-print books.

Google's founders supported the idea when they were at Stanford university.

A single index would allow Google and others to apply their analysis and their algorithms to the same data set creating a level playing field.

A single index would also speed up the entire Internet. For example, looking at my server logs for SVW, it shows 16 robots/spiders visiting the site. In total they are responsible for 37% of the hits and 45% of the bandwidth.

These spiderbots are creating their own index of the Internet and they are taking up nearly one-half of my bandwidth and slowing down my server. SVW has fresh content everyday so it gets more attention from spiderbots, but this is happening to tens of millions of other web sites every day.

Spiderbots are a huge drain on Internet resources

If there were a single search index that could be held in the public domain, administered by a non-profit, it would cut down on the spiderbots and speed up the entire Internet -- all without having to install any new routers, servers, or lay new fiber optic lines!

Google would still be Google as would Microsoft's Bing because their value lies in their analysis and ranking of the search index. But it could also spur innovation because startups wouldn't need to spider their own index. We could see all sorts of innovation in algorithms and applications.

Larry Page and Sergey Brin were once strong supporters of the idea that the search index should be run by a non-profit.

On page 39 "Inside Larry and Sergey's Brain" by Richard Brandt (referral link).

Andrei Broder, who led the team that created the AltaVista search engine, the best of its time, talks about meeting Larry and Sergey. "When the discussion turned to the topic of making money from the technology, Broder found that Page had a profound difference of philosophy on the subject. "It was a very funny thing about Larry," Broder recalls. "He was very adamant about search engines not being owned by commercial entities. He said it should all be done by a nonprofit. I guess Larry has changed his mind about that."

Brian Lent, now CEO at Medio Systems:

"The problem with the Google search engine at the time, Lent recalls, is that Larry and Sergey didn't want to commercialize it, and Lent was anxious to become an entrepreneur. Their mantra at the time was more socialistic than entrepreneurial. "Originally, 'Don't be evil' was 'Don't go commercial,'" says Lent.

In one fell swoop Google can:

- Overcome many of its problems with indexing content
- Dramatically speed up the entire Internet at almost no-cost
- Plus the Internet gets a zero carbon speed boost

GOOG's founders have a chance to live up to their ideals, fulfill their mission to index all the world's information, and keep all their money.

It's win-win all around.

- - -
Please see: GOOG Founders Originally Believed Search Should Be Non-Profit - Why Not Do It Now?