Craigslist: Battling the spider and bot armies of the swarms of VC-funded search-and-scrape startups . . .a chat with ceo Jim Buckmaster
By Tom Foremski - October 26, 2005
The complaint filed against Oodle by Craigslist is fascinating. Oodle, an online classified ads aggregator/scraper was recently asked to quit aggressive scraping of Craigslist and it complied.
(See: SVW Oodles of lawsuits--the battle over content will escalate)
I had an interesting chat with Jim Buckmaster, ceo of craigslist, about this issue. Jim said that Oodle was the most aggressive in checking its listings and this was slowing things up for users.
Jim showed me a chart of craigslist traffic and how much traffic Oodle was bringing, and you could barely see Oodle's red line graph coming up off the x-axis, while the blue line of craigslist was flying high up in the logarithmic realms of the y-axis.
"We try and be fair and reasonable but aggregation sites like Oodle put a big strain on our infrastructure," he said. "We don't want our users suffering because of this."
Oodle and all the other google-like search-and-scrape sites insist that they bring traffic to the original content sites such as craigslist. They provide much needed distribution, is their line.
But the data shows that the cost of the tiny amount of traffic driven to craigslist is massive because of the huge amount of bandwidth and server load caused by repeated hits and scrapes of its data.
"I'm sometimes asked how much benefit we derive from a site such as Oodle, and I estimate it is about a minus 0.5 per cent because it slows up overall performance," Jim said.
With more than 3 billion page views each month, 0.5 percent drag on performance is severe--it means about 15m page views are sucked up by Oodle.
Oodle claims it sent 1m page views to Craigslist in September. That is about a 15 to 1 burden that craigslist has to carry, and its entire community has to bear because of slower system performance.
I told Jim my take on all of this: Companies such as Oodle says classified ads customers want as wide a distribution online as possible, but that is NOT true. If customers wanted wider distribution they would have taken wider distribution because there is nothing stoppong them from advertising on many other web sites, including Oodle!
VCs are targeting craigslist
In my view, craigslist acted fairly and responsibly because it is protecting its community from resource-hungry bots that give back a fraction of what they take.
The VC community continually salivates at ways of creating a craigslist. Or better yet, creating businesses that can syphon-off and commercialize craigslist listings. One VC told me about all the money craigslist leaves on the table by refusing to monetize all its traffic, (collecting revenue just for job listings.)
"We are constantly being approached by other organizations to partner in some way or other. But we feel we don't have any responsibility to help other companies become profitable, the only responsibility we have is to our users," Jim said. "And we don't want to be a target for every new startup that wants to be in the classified ads market."
The problem ahead is that creating a search-and-scrape business is easy, and the oodles of googles coming online funded by massive amounts of VC money are going to be hitting and scraping craigslist at an ever increasing pace and scale.
And it is not just craigslist that is the target *ALL* news/blog sites are being targeted--see next post.
- - -
Here are some blog posts from Craig Donato co-founder and CEO of Oodle, who appears mystified by craigslist's complaint.
. . .for consumers trying to sell items. They want to reach the biggest audience so they can get the best deal. Services like Oodle, that help bring prospects to a consumer's listing, are helping them not hurting them.
Check out the post and its comments section :-)
By Tom Foremski - October 26, 2005 | Permalink | Comment
| Category: Tech Watch
| SVW Toolbar | SVW Newsletter | SVW Mobile
- NEW STORIES:
- Intel Experiment Could Save Millions in Data Center Power Costs
- Will East Coast Flood West Coast in Search of Jobs?
- GOOG Founders Could Buy All US Newspapers and Still Have $12bn
- Microsoft in Bay Area Recruiting Blog Sites for AdCenter Ad Network
- Silicon Valley Rocks! Charity Event for Local Schools
- Fishwrap: Changing Media and PR . . . Plus a Great Pep Talk
- Top of my 2008 Watch: Berlin Based Plista . . . and Online Dopplegangers
- The "Experiential Gap" . . . and the Growing Cosmos of Twitter Applications
- FT Anger on AIG Bailout
- Shift Happens . . . A Visit With One of My Favorite PR Companies
Comments (8)
As far as I can tell, there are quite a few folks who appreciate the one-stop approach of an Oodle. Who in their right mind appreciates searching for stuff on multiple sites when you could just go to one. And, Tom, I believe Oodle allows you to search multiple sites. It does not enable posting, as you suggest above.
Posted: October 26, 2005 2:40 PM
Widest distribution and biggest audience is not what craigslist is about; it's local! I use craigslist all the time and, in some instances, would prefer to search or post to multiple cities, ie, my patent research business that is not dependent upon location, but craig's focus is on the local market. No worries about shipping, etc, typically the seller and buyer meet for the exchange (I'd actually be interested in any data craig has on the type of interactions its classified sections generate)
Posted: October 28, 2005 5:05 AM
This is the inherent problem with a lot of the "We b 2.0" plays right now. They keep AGGREGATING - like an online WalMart of rehashed Internet services and applications. While there are still opportunities in SEO (I mean, how good is search today? Maybe a little better than 5 years ago), the true value is improving what you have today in your day-to-day life.
The above post is truly what CL is all about - Location, Local, what is around me, etc. The ability for me to connect virtually in order to build physical connections and relationships where I am living with people, goods, services - my daily life - that is key. The days of being fascinated about accessing, aggregating and filtering the WHOLE thing so that "nobody knows you are a dog on the Internet" and "Look! I am IMing someone in Italy!" - are over. How many neighbors on your block can you IM? Not that many I bet. :)
The goal for an individual online is about being known, establishing an identity that can be global AND local. The fact that CL (ads & forums) helps do this is not lost on the population. That all being said, there are HUGE opportunities to make it better and more useful - look for more plays like this very soon.
Posted: October 28, 2005 3:37 PM
I see craigslist point but still do not understand how he came up with minus 0.5%.
Posted: January 5, 2006 1:14 PM
The minus number comes from the load on Craigslist servers that the robots and spiders create. For example, on my modest site, SVW, the spiderbots use one third of my bandwidth yet bring less than 5 per cent of my pageviews. The resr of my traffic comes from RSS and direct bookmarks--my readers know where I live, which is what you want. And this will become true for many other sites, which means the value of the search engines will decrease for news sites, imho.
Posted: January 5, 2006 1:33 PM
Craiglist.org is in the Boming phase. every business man wants to get on regionally and craiglist.org. Pls go thru this post
"How Craiglist helps Online Marketers"?
http://blogs.profitimo.com/2007/04/19/how-craiglistorg-can-promote-online-business
Posted: April 24, 2007 9:09 AM
Google bot is probably the number one user of server resources on the Internet. But most companies would happily provide more if they get more traffic although Google is making money using other people's content.
Spidering and organizing the ever-growing has become one of the most important functions and should not be regulated unless it is done for malicious reasons like spamming.
It is also questionable if web sites owners with user generated content claim any rights to the content.
Posted: May 10, 2007 2:48 PM
I realize that CL is designed for "local" markets, but some very "global" (or "national") products still appear. An example is aircraft. What possible reason would there be for a local-only FSBO posting on an aircraft? Yes, CL users shouldn't post such ads in the first place, but they do. When CL starts banning things that appeal to a national audience, people will stop searching CL for products with nation-wide appeal.
Posted: January 13, 2008 5:10 AM