Craigslist: Battling the spider and bot armies of the swarms of VC-funded search-and-scrape startups . . .a chat with ceo Jim Buckmaster
The complaint filed against Oodle by Craigslist is fascinating. Oodle, an online classified ads aggregator/scraper was recently asked to quit aggressive scraping of Craigslist and it complied.
I had an interesting chat with Jim Buckmaster, ceo of craigslist, about this issue. Jim said that Oodle was the most aggressive in checking its listings and this was slowing things up for users.
Jim showed me a chart of craigslist traffic and how much traffic Oodle was bringing, and you could barely see Oodle's red line graph coming up off the x-axis, while the blue line of craigslist was flying high up in the logarithmic realms of the y-axis.
"We try and be fair and reasonable but aggregation sites like Oodle put a big strain on our infrastructure," he said. "We don't want our users suffering because of this."
Oodle and all the other google-like search-and-scrape sites insist that they bring traffic to the original content sites such as craigslist. They provide much needed distribution, is their line.
But the data shows that the cost of the tiny amount of traffic driven to craigslist is massive because of the huge amount of bandwidth and server load caused by repeated hits and scrapes of its data.
"I'm sometimes asked how much benefit we derive from a site such as Oodle, and I estimate it is about a minus 0.5 per cent because it slows up overall performance," Jim said.
With more than 3 billion page views each month, 0.5 percent drag on performance is severe--it means about 15m page views are sucked up by Oodle.
Oodle claims it sent 1m page views to Craigslist in September. That is about a 15 to 1 burden that craigslist has to carry, and its entire community has to bear because of slower system performance.
I told Jim my take on all of this: Companies such as Oodle says classified ads customers want as wide a distribution online as possible, but that is NOT true. If customers wanted wider distribution they would have taken wider distribution because there is nothing stoppong them from advertising on many other web sites, including Oodle!
VCs are targeting craigslist
In my view, craigslist acted fairly and responsibly because it is protecting its community from resource-hungry bots that give back a fraction of what they take.
The VC community continually salivates at ways of creating a craigslist. Or better yet, creating businesses that can syphon-off and commercialize craigslist listings. One VC told me about all the money craigslist leaves on the table by refusing to monetize all its traffic, (collecting revenue just for job listings.)
"We are constantly being approached by other organizations to partner in some way or other. But we feel we don't have any responsibility to help other companies become profitable, the only responsibility we have is to our users," Jim said. "And we don't want to be a target for every new startup that wants to be in the classified ads market."
The problem ahead is that creating a search-and-scrape business is easy, and the oodles of googles coming online funded by massive amounts of VC money are going to be hitting and scraping craigslist at an ever increasing pace and scale.
And it is not just craigslist that is the target *ALL* news/blog sites are being targeted--see next post.
- - -
Here are some blog posts from Craig Donato co-founder and CEO of Oodle, who appears mystified by craigslist's complaint.
. . .for consumers trying to sell items. They want to reach the biggest audience so they can get the best deal. Services like Oodle, that help bring prospects to a consumer's listing, are helping them not hurting them.