09
July
2006
|
19:09 PM
America/Los_Angeles

How much Internet bandwidth is used by spiderbots from GOOG, YHOO, MSFT et al...?


Google, Yahoo, MSN and the many other search sites and aggregators wandering the Internet with their spiderbots could be in trouble, if their version of net neutrality doesn't survive.


That's because the spiderbots eat up a huge amount of bandwidth, and if bandwidth gets more expensive, the spiderbots are going to suffer. I get 5 per cent of my traffic from more than 18 spiderbots, as they scour the Internet copying everything in their path. They use up about one-third of my bandwidth.


That's a key reason why Google, Yahoo and others, are arguing for everyone to have equal access to bandwidth--at least the last mile pipe to the home--the most important pipe.


If companies are going to have to pay extra to the telcos or cable companies for bandwidth to reach their users, they might not be so pleased to be paying for the bandwidth of the swarms of spiderbots.


I'm fortunate that more than 92 per cent of my readers come directly through bookmarks or RSS, so they know where I live. Many sites depend on 30 per cent to 60 plus per cent of their traffic from the search engines.


And they spend a lot of money to optimize their sites to attract more search engine traffic. But often, this is not quality traffic, it is fly-by-night web surfers.


Web sites should optimize for their readers, not the spiderbots. Let the search engines optimize themselves, that's their job.


If the telcos/cable companies get away with raising fees from the many online companies, to guarantee they have the bandwidth to reach their potential customers, then the spiderbots will be in trouble.


When audience numbers stabilize for a web site, and very few new readers come in from the search sites--yet the spiderbots suck up one third of the bandwidth--then things will change. More and more web sites will be posting a Robot.txt file that tells the spiderbots to go away. They will change because the overall visitor experience is slowed down by the bandwidth hungry packs of spiderbots.


We used to have estimates of how much bandwidth is consumed by email, by SPAM, etc, how about spiderbots? Does anybdy have access to such data?


I would love to know: how much Internet bandwidth is used up by the legions of spiderbots, in their constant search to find and copy new content.