05:51 AM

The lie of distribution--search engines return very little value to news/blog sites yet hog bandwidth and increase server loads

. . .or is it just me?

By Tom Foremski, Silicon Valley Watcher.com

Search-and-scrape sites such as Google, Yahoo, MSN, and oodles of others claim they bring traffic to web sites. And they do--but at what cost? It was a question I asked myself following a chat with Jim Buckmaster, ceo of craigslist, and its recent complaint that Oodle was scraping its listings way too aggressively and slowing down the entire system.

I took a look at my server stats from mid-October.

The search-and-scrapers sucked out one-third of my bandwidth and provided just 3.7 percent of the traffic!



Microsoft is by far the most egregious of the lot. Over the past three weeks it sucked out 4.6GB or 18 per cent of my bandwidth and returned...275 page views--0.0007 percent of the total!!


It is news/blog/original content sites that are being targeted by these over-zealous bots because they provide fresh content, and without fresh content the search-and scrapers have nothing. And the more often you post fresh content the more attention and visits you get from the bot army.

And lets not talk about the masses of cached pages out there that are hit and viewed but do not show up in the server stats--yet are counted and monetized by the search-and-scrapers.

It is inevitable that content owners will increasingly choose to glue down their content. You can only get it here and you have to come here to get it--will be the mantra.

Why do you think Yahoo is trying to scramble as quickly as it can into producing original content ;-). It knows the writing is on the wall.

Tell me what your AWstats are showing...