Silicon Valley Watcher - conversations and observations at the intersection of technology and media

When the first iPhone came out two years ago I was astounded that there was no way to search its contents: Searching For Search On the iPhone

One year later, in 2008, after many updates to the software and new hardware: still no search function! It's taken another year to add search in the most recent update iPhone 3.0 in June. It's taken two years to add search.

That's not accidental it has to be deliberate. A year ago I wrote: Is Apple About To Launch Apple Search?

And it makes sense. It has its own browser and having its own search site makes perfect sense.

The browser and search service are essentially the operating system for our modern times. Apple knows all about owning the OS and that has been a winning strategy for Apple.

Take a look at how Apple would be able to quickly build a large search engine business:

1: The Apple fanboy market would leave Google in a heartbeat. No question. That's about 5 per cent of the computer market, and that's not counting the iPod, iPhone users.

2) All the users of Safari, the Apple web browser on Macs and Windows systems would be default users of Apple search.

3) Search is all about brand. Tests have shown little difference in the quality of search results between search engines. Microsoft rebranded its search service Bing and it now has traction. Apple knows "brand" very well and it knows how to parlay that expertise into new businesses.

Study: Good Brand Can Make Search Seem More Relevant

The study showed that when a searcher was given an identical result set across Google, Yahoo, Windows Live Search and an in house search engine, Google and Yahoo came out as more relevant. Why? Because of the brand of the search engine.

Despite the results pages being identical in content and presentation, participants indicated that Yahoo! and Google outperformed MSN Live Search and the in-house search engine.

http://www.pcworld.idg.com.au/index.php/id;229165979

One drawback for Apple is that GOOG pays Apple for placing its search engine in the search bar in Safari. Industry whispers report that this payment is enough to fund development of Mac OS X.

However, that revenue is jeopardized by Google's announcement of its Chrome browser, and also there is competition from Google's ChromeOS, and its Android cell phone software.

While the industry focuses on Microsoft and Google competition, Google and Apple are becoming ever more competitive with each other. That's why Eric Schmidt is leaving Apple's board of directors.

- - -

Please see:

Since There Is No Objective Way To Gauge Search . . . Brand Will Win

I went to Google's biennial Searchology event this morning. This is where top execs unveil their latest advances in search.

This year there was a lot of new stuff, and a lot of it revolved around understanding the semantics of search, to understand the meaning of what users want, rather than what they typed. If you misspelled something, or used a seemingly cryptic search term, GOOG execs said it was their responsibility to figure out what you meant by that.

For example, a search for "sfll playoffs" could mean a misspelling of "nfl playoffs" or, more likely, "San Francisco Little League playoffs." Google is using location data and other factors as part of what it calls "rocket science" search technology to try to determine what the user really meant -- not what they typed.

Udi Manber, vice president of core search, spoke about how difficult it was to determine the semantics of user queries, how to understand what was meant.

The demos they showed were impressive. But, how will Google sell advertising against an increasingly semantic set of results? Currently, Google sells keywords to advertisers, and those keywords trigger advertising served up on the results page.

Keywords are very specific. How will this work when the results that people want, aren't related directly to a keyword they typed? Is there such a thing as a "semantic keyword?"

If Google tries to guess what the advertiser wants and what the user wants, it creates more opportunities for a mismatch. Since Google gets paid on clicks, that could be a problem if it misses the intentions of both user and advertiser.

How do you sell "semantic" advertising?

[This is a guest post by Todd Hogan founder of Surchur, which specializes in real-time search. Please see http://blog.surchur.com/. If you'd like to offer a guest post please contact tom(at)siliconvalleywatcher.com.]

- - -

By Todd Hogan

This past week you've seen the mighty Google show its interest in the upstart Twitter. Now, whether those discussions really were as serious as many at first believed, the takeaway is the same: what is happening "right now" is becoming increasingly important to many internet users. One important trend growing from this is the need to search through all of the "right now" content that is being generated every moment and make it accessible. Our team at surchur.com is one group trying to make that happen.

But who cares about what's happening "right now?" Here are 5 profiles of users that are very interested in real time search:

Foremski's Take: What would happen if somebody developed a better search engine than Google? Would it be able to unseat Google? And what would happen to Google?

Nothing would happen to Google. Because if you could build a better search engine GOOG could monetize it way better than you or anyone else.

GOOG would partner with the better search engine because it has the monetization platform, it has the ad delivery network, and ad sales representatives in nearly every country.

The market has moved on--it's not enough to be better than Google in search, you also have to have be better in all aspects of the business.

That's what Microsoft needs to focus on, building a great monetization platform for search - search alone is not good enough even if it's great.

Reuters: Yahoo reaches Google ad deal, Microsoft talks fail

Yahoo said it had agreed to let Google put search ads on its site in what it called an $800 million annual revenue opportunity that would boost cash flow by $250 million to $450 million in the first 12 months.

Yahoo's ads and Google's would be pitted against each other in an auction style process that could make a deal easier to pass regulatory approval.

"Yahoo is being a reseller of Google whenever it makes sense and that is likely to be a lot of the time given how much more effective Google Web search ads have proven to be," Global Crown Capital analyst Martin Pyykkonen said.

Monday morning I was down in Mountain View at the Googleplex, GOOG's HQ for a briefing on new search initiatives and to find out about Google Health - a potentially wonderful and problematic service.

[For the first time, I was using Twitter to "live blog" from the event. Twitter is particularly well suited to such an activity, plus my "Twits" served as a decent note taking device for this post. I'm tomforemski on Twitter if you'd like to follow me.]

A planet sized wiki...

On search there was a lot said about local search and how difficult it is to do because place names are not standardized and some facilities share the same address and phone number. GOOG is very excited by image search because there are about 100 billion photos taken every year and Google loves new content (after all why go back to the Internet if there is nothing new?)

Google is trying to create a 3-D virtual earth. Its satellite and street level imagery are lacking. What it wants is a 3D rendition such as the one for San Francisco, which is very impressive. But I think that Google will have trouble getting users to continue to create that 3-D content because the novelty will subside so it'll have to think up some games and incentives.

Google likes images and the geo-tags that can go with them, but again, this relies on the continuing goodwill of users to upload and tag images, etc.

What this adds up to is a wiki-earth that users create using Google tools, or other tools, and also annotate and tag the locations. It would be a massive project and on top of that it would require fresh content - good luck with that. I just don't see legions of users willing to update Goog's database time and again, and again. Businesses will-- but that's advertising and that doesn't require payments to Google, (there could be a shot in the foot here ...).

Google Health very important...

Google Health is interesting and I would say it is Google's most important business launch since its AdSense/AdWords text-link ads. Not only is this a great way to integrate health data from multiple sources by having the individual do it but it also enables Google to access billions of dollars in pharma and medical services marketing.

Government regulations on health data make it very difficult for medical practitioners to pull an individual's data from many sources. But a Google Health user can consolidate their medical records in a highly secure Google database then authorize sharing that data with their doctor or other third party, and even allow that data to cross-borders. How about MyIndianMD.com...? (Hold on one second while I visit GoDaddy...)

Google Health could reduce health care costs because tests won't have to be repeated, and individuals could choose not to have expensive procedures just because their doctors are trying to cover their legal liabilities.

The problems with this approach is that the individual is being asked to make choices which their doctors would be best at making but because of legal and other issues, they cannot. Similarly, Google has to be careful not be providing medical advice. It gets around that - it provides users with "choices and options."

I spoke with Martin Harris, M.D and Chief Information Officer at Cleveland Clinic, one of the beta test partners for Google Health. He said that one problem was that computer literate users would be the first to benefit. "It's part of what I call the Internet divide," he said. "Ideally we should be able to let people access their Google Health accounts through their TV set top box or through the phone. That will come soon."

I spoke with a couple of people on the Google Health team and they said users would in the future be able to designate "delegates" to administer their Google Health records on their behalf so that the elderly could have their families help them.

Building businesses on top of Google Health . . .

There is also a great opportunity for a range of third party services to be built on top of Google Health and it's open API. Google for example, is offering an application that allows people to monitor how much they walk and will give $100k to charity.

I can think of linking gym machines to my Google Health record, what I ate that day, even what I ate at a restaurant automatically uploaded with calories and vitamin info. How much I slept, etc. This could become many people's home page.

There are some obvious potential problems if insurance companies or employers seek permission to view health records. What about potential spouses opening their hearts . . . and their medical records to each other?

How to monetize search, Google Health . . .

Nothing was said about how Google would make money from the improved search or from Google Health.

I asked about Google's search and if it improves then fewer pages are served. . . and therefore the performance of Google ads has to improve just as fast as search improves.

I was told that the advertising teams are ahead of search, and with better search there will be more search, therefore more pages will be served. I don't see that as being true.

It would have been good to have a presentation about the business side of Google's search and Google Health.

Here is further discussion of the business opportunities: The Google Health Problem...And The Quest For Pharma Gold

- - -

Notes: Google did a nice job pulling together top media and allowing good access to its executives and also to Google Health partners. Its PR people didn't get in the way but facilitated interviews etc. This meant that journalists could each have a chance in coming away with original quotes and material that wasn't part of the general presentation. I always appreciate the opportunity to come away with some unique content.

I recently visited Powerset to get a look at the company's much buzzed-about search technology based on the semantic understanding of language. Powerset is not the next Google but it does have an interesting technology thanks to its dozens of noted academics in lingustics and artificial intelligence.

The first product is a hand-optimized use of its technology to search Wikipedia along with Freebase, an open-source database of information. The results are fascinating in that the search results provide a new way to navigate through subjects. You can see what Powerset calls "factz" similar to facts, which are terms drawn from the subject matter dispersed throughout the two databases, that highlight and summarize complex subjects.

It takes a lot of computing power and time for Powerset to process a web page and it generates a lot of metadata. Thanks to Moore's law and Powerset's computing architecture, this processing time will continue to shrink but right now it cannot be used to process the entire web in a similar way to that of the major search engines because of those constraints.

There are about 2.5m pages in Wikipedia which is a lot of information and a good size to demonstrate Powerset's technology. Wikipedia was chosen because of its open license.

I can think of many uses for Powerset's approach such as improved ad serving based on meaning rather than contextual keywords.

What will be Powerset's next product? I wonder how many types of searches require a semantic understanding? It'll be interesting to find out. In the meantime I encourage you to take Powerset for a spin.

Here is Powerset's news release:

. . . An end to Enron and other corporate scandals?

Friday I met with Michael Lynch, CEO of Autonomy [AUTN], the second largest European software company with a market capitalization of more than $4 billion.

I wrote about its new software that can identify illegal content in corporate communications such as blogs, emails, any document, and phone recording, and even in video files.

The Autonomy Information Governance (AIG) software makes sure that a corporation complies with huge numbers or regulations, it also monitors any document being created by an employee, including a blog entry, an email, even audio and video. It checks in real-time to see if any laws or company policies are being violated.

AIG also makes it quicker and cheaper to find documents related to a lawsuit.

When a corporation is sued, it often has just 90 days to produce huge quantities of documents, emails, etc, related to the lawsuit. It faces huge fines if it doesn't meet the deadline. The software understands the meaning of documents and can reveal which documents should be reviewed by lawyers.

Companies being sued or investigated, must lock-down all their documents so that they can be searched for evidence. This has to be done even if a company has not been sued or investigated but has a reasonable chance of that happening.

"With 14,000 separate records retention regulations out there and the complexities and costs being incurred just trying to comply with legal hold requests, a company doesn't have the capability to manage this without advanced technology," said Browning Marean, a top lawyer, and partner at DLA Piper US LLP.

"I was on a panel recently and one of the panelists said that the Internet is different and people will behave better. I don't think it will change human nature at all. Human nature hasn't changed in hundreds of years, if you read Shakespeare. And if you look at Roman and Greek times, human nature is pretty much the same thousands of years ago."

"It's seems really arrogant for people to say that in this one generation human nature will be different," Mr Lynch said.

The AIG software also carries out document retention policies, which in the corporate world of double-speak refers to: how soon can we shred or delete this document just in case it comes back to bite us.

Autonomy is trying to bolster its position in the multi-billion dollar e-discovery market, which is lead by Recommind, based in San Francisco. You can read more about the software here.

Foremski's Take:There are some good and bad aspects to this software. The bad is a big brother type use for it. I can imagine the Chinese government using it to monitor Internet use and communications in real-time. You'd get a message pop up: "Report to your local police station immediately - you just committed an illegal act." (The Chinese police sent out text messages to Tibetan demonstrator's cell phones to report to the police.)

It could be used to restrict blogging. A lot of people tell me that large corporations are scared of blogs violating a regulation and so every corporate blog entry has to be run through lawyers-- it has to be "lawyered." This can take time, days, even weeks.

Paradoxically, I think AIG could be used to clear a blog post in real-time and could thus increase the amount of good, legal information that company workers can share in public. Either way, it automates some of the tasks of a lawyer, and that means less need for lawyers, and smaller legal bills for corporations.

Less lawyering, means lower operating costs, which maximize share holder value, and that's what corporate officers are required to do. It's the ultimate SEC regulation.

Please also see: Autonomy CEO says tags don't work

News.com's Charles Cooper: What's interesting is what we say it is. Really?

Last week I attended a briefing by Autonomy, a company based in the United Kingdom and San Francisco. On Monday, Autonomy will announce a product designed to assist companies with governance compliance. This likely will be a big deal for IT administrators and law firms that are scrambling to enact internal information management policies in the wake of the subprime mortgage and credit crisis.

Please see: Mike Lynch on the meaning of meaning based computing.

About a year ago, Internet mogul Barry Diller appointed Johnny C. Taylor Jr. president and CEO of a new business group at his IAC corporation called Black Web Enterprises Inc. Mr Taylor was previously head of human resources at IAC, which owns large Internet properties such as Ask.com, Match.com, LendingTree, Ticketmaster and 54 other leading Internet brands.

Black Web Enterprises is the first business to be created by IAC rather than acquired. I met with Mr Taylor recently and he explained the background to tomorrow's launch of a new search site: Rushmore Drive targeted at the Black Internet user.

"We felt that the Black community could be better served by the Internet and so we carried out a lot of focus groups. We found that Black people in the US do a lot of search, they look at news, and they look for jobs. Out of that work we decided to launch a search site called Rushmore Drive that offers mainstream search but also something extra for the Black community." It also has its own editorial department staffed by journalists, and a jobs site where people can also network similar to LinkedIn."

Is "Black" considered a good designation? "Yes, Black designates a large number of people. Barry Diller asked why we didn't use the term "Afro-American. First of all, "afro" is a hair style and secondly there are many people such as those from the Caribbean that don't identify themselves as African-American."

IAC estimates that there are 40 million African-Americans and an additional 20 million people who fall under the designation "Black."

Suspicious of a Black web...

I asked if there was a suspicion by the Black community about a search site focused on them, because there have been many cases where minorities paid more for cars, loans, etc.

"It's interesting. When we ran focus groups I was surprised that there was a lot of suspicion from people about a large corporation targeting the Black community, because I don't really pay much attention to being Black myself. I was watching a lot of hostility through a two-way mirror. But when I stepped into the room and people could see me, they immediately changed their attitude and were much more welcoming of the idea. They also told us that they want great search, they don't want just a bunch of Black targeted search results."

Not a Black Ask...

Mr Taylor said that the site is not a Black version of Ask.com. It provides high quality search results plus more. The term Black is hardly used on the site and it won't be used in marketing or advertising.

Radio ad spots will be targeted at Black audiences only in the sense that a radio station has a Black audience, and not in the advertising message itself. "It's important to be inclusive," Mr Taylor says.

Classy...

The search site's name Rushmore Drive is based on the address of the business group, located on 1115 Rushmore Drive, in Charlotte, North Carolina. "It's like Rodeo Drive, it is a classy name." Classy is an important quality, Mr Taylor says, which is why the right type of advertiser is important.

"I could have tripled my first year numbers from just one advertiser that wanted to be part of the site but I turned down the business because I did not like the association with the product." Mr Taylor declined to name the advertiser (I would guess it could be a malt liquor brand) and said he wanted top advertisers such as Gucci and Ferragamo.

A Black Linkedin...

Part of the job search at Rushmore Drive is a network feature similar to Linkedin. "One of the first things Black people ask each other is which church do you go to, and what professional associations do you belong to," Mr Taylor says. Those two answers are key to the networking feature on the site.

Black writers...

News on the site will come from mainstream sources but there is also an editorial department with Black writers that will engage users in discussions around topics that are of importance to that community.

"For example, the recent death of R&B singer Sean Levert was a big deal in the Black community but it didn't make much news elsewhere. We would promote that news and also discuss why he was in prison for failing to pay child support. We would look at the issue of paying child support in the Black community and not shy away from uncomfortable issues," Mr Taylor says.

IAC is investing many millions of dollars in this first home grown business. "I can't say how much we are investing. Barry Diller told me to make sure that we build a quality product. I can tell you that it's not cheap setting up a search site, I've got a large group of engineers and also a large group of marketing people."

Foremski's Take:

It is certainly new territory for a search site. And maybe it will spur the development of other search sites focused on specific types of users. For example, I recently wrote about the possibility of an Apple branded search site.

Mr Taylor's challenge will be to demonstrate how Rushmore Drive can provide a better search experience that is inclusive while at the same time exclusive in its strong focus on the 60m strong Black community.

- - -

(Thanks to Steve Gillmor for the headline inspiration.)

Additional Info:

Latest results from Comscore show a slight drop of 0.2 percentage points in Google's share of search in December 2007 compared with November 2007 to 58.4 percent.

Second place Yahoo gained 0.5 percentage points to 22.9 per cent over the same period.

Third place Microsoft remained at 9.8 per cent.

Fourth place Times Warner gained 0.1 per cent to 4.6 per cent.

Ask lost 0.3 percentage points to 4.3 per cent.

ComScore said:

Americans conducted 9.6 billion searches at the core search engines, representing a 3.9-percent decline versus November. With many Americans traveling and spending time away from home during the holidays, search activity typically experiences a seasonal decline during December. Google Sites saw 5.6 billion core searches during the month, while Yahoo! Sites recorded 2.2 billion.

Additional info: www.comscore.com

I recently met with Aggregate Knowledge and Proximic, both companies have technologies that serve up relevant content to web site visitors, especially on media sites. Relevant content means visitors are likely to stay on the site and that provides another chance to try and monetize that visit.

- Aggregate Knowledge, based in Silicon Valley, could be called "Aggregate Web Analytics" because it watches the behavior of web visitors and then guides future visitors to the same pages that most other visitors visited. It can also guide visitors to higher margin pages that the web site owner is seeking to monetize.

"Most sites have click through rates of abut 2 percent, with some of our clients we can get 22 per cent click through rates," said Paul Martino, CEO of Aggregate Knowledge.

Kleiner Perkins is the main investor in the company. Randy Komisar, a partner at Kleiner Perkins Caufield & Byers, said that the web is changing. "It is not about search it is about discovery. Search is only good if you know what you are looking for, 'discovery' means serving up content that is relevant and that you didn't know was there," said Mr Komisar. (Kleiner Perkins was an early investor in Google.)

- Proximic, founded in Munich, Germany tells a very similar story of being able to serve up relevant content on media sites. It claims a radical search technology that indexes pages by "patterns" not by links or keywords. Pages with similar "patterns" represent similar and relevant content.

Philip Pieper, CEO of Proximic, says that the beauty of their approach is that their index is easily scalable and very compact. "We have even written our own disk operating system, to speed up searches. We believe that our technology can help media companies better monetize their online operations because we can provide relevant content," said Mr Pieper. The index can be held in just a few gigabytes of memory, it could be loaded onto a flash drive. Proximic says that its search technology could be run on a bank of iPods.

Proximic is working with Nature magazine to boost its online revenues, and also with the British newspaper "The Independent."

Foremski's Take: They may get there in different ways but Aggregate Knowledge and Proximic serve up similar messages to web site owners. They promise to improve online revenues by making the content on those web sites more relevant to visitors, including the advertising.

And neither of them collects personal data on visitors. This is a key advantage because privacy issues are only going to get bigger and messier--you don't want to be in a business that relies on collecting and using personal user data.

Media sites in particular, need all the help they can get to monetize their online operations. However, making sites more "sticky" produces value only if the site publisher can monetize those extra clicks. And there aren't many media sites that are good at that. Even if media sites doubled their online revenues, it is still not enough to support their businesses as they transition from traditional media to digital media.

The success of both companies is tied to the ability of their customers to figure out their own online business models. Yes, Aggregate Knowledge and Proximic can help in that regard, by serving up advertising that pays better, and is more relevant to visitors, but that will likely not be enough to help media sites make money, and survive.

Working with online retailers to help optimize their sites is likely to produce longer lasting customers than in the media sector.

- - -

Please See Information Week: Aggregate Knowledge Aims To Be A Mind Reader On The Web

Proximic empowers the web publisher for the post-2.0 world

Technorati Tags: adsense, proximic, aggregate knowledge

About a year ago I met with Bob Chandra, founder of GrayBoxx. At the time, GrayBoxx was still in very stages of development around the idea of getting recommendations for local businesses from the contact info in people's address books and telephone logs.

Mr Chandra says,"I already know who my favorite businesses are because they are the ones I call and they are the ones whose phone numbers I already have." That's a great start to finding out the recommendations of other people, without any tagging or reviews or community action to create that content.

GrayBoxx also uses other sopurces of data to compile an impressive database of listings. It has 185m recomendations compared with 3m for Yahoo Local.

It is still in stealth mode but is building up its user base with plans for a broad launch in early 2008. The Sierra Ventures backed company says it has developed its own version of GOOG's PageRank, but one that is tuned to the recomendation of local services.

The local search market is extremely competitive but it is an area that is poorly served. Yes, some sites are good for say, restaurant reviews, while others for automotive repairs. None offer a comprehensive service, at least none so far...

GrayBoxx just announced a board of advisors that includes Rajeev Motwani, one of the top experts in search and co-developer of PageRank. It's all about the algorithm...

Terry Semel, the embattled head of Yahoo! has been ousted and replaced by co-founder Jerry Yang. The move comes in the wake of shareholder criticism of Yahoo! in its bid to keep up with Google (GOOG).

Mr Semel will remain as a non-executive chairman and advisor.

Susan Decker has been appointed president.

Here are details from Yahoo! and SVW analysis will soon follow:

Google said searches will now combine all other types of results such as video, images, books as well as web pages.

It is a move away from silo search engines that focused on video, images, etc. Marissa Mayer, VP Search Products & User Experience, said that it was confusing for users to pick from search engines. "You almost needed a search engine for search engines."

Foremski's Take: This is bad news for the huge SEO industry because there is less space to get web pages into the first page, new opps for video placement. The focus now is on the first two results...

Over on New Rules Communications I was writing about Google's ad networks and the bad economics for media companies. 

Link to: This Is Why Online Ad Nets Can't Save Media Companies

I noticed that there is an interesting conflict of interest emerging at the heart of Google's business model.

Please check my reasoning:

Google makes almost all of its revenues from two ad networks:

-AdWords: Customers advertise on Google web sites.

-AdSense: Customers advertise on Google partner sites, which includes many media companies.

Google's revenue in 2004 was about evenly split between AdWords and AdSense.

Since then, Google's revenue from its own sites has grown by 24 percent to 62 percent of total revenues. AdSense has fallen.

And this makes sense because Google makes far more money from its own sites than from partner sites. It is better for Google's shareholders that it channel more of its revenues through its own sites because:

-Google gives back about 80 per cent of AdSense revenues to its partners.

-It keeps all of its AdWords money.

And this is where it has a mammoth conflict of interest:

Which advertising network should Google invest in?

-A dollar invested in its AdWords produces far more profit than invested in AdSense, its partner network. Management has a fiduciary duty to its shareholders to maximize profits.

-Google can boost overall profits by undercutting AdSense at anytime it wants, say  by offering a discount on AdWords compared with AdSense. A 20 percent discount on AdWords would make more money for Google than the corresponding loss of business through AdSense partners.

-Google can undercut AdSense in other ways, and is already doing it, by investing in technology that improves AdWords conversions over AdSense. Google can apply technologies to its own sites that make them more efficient at selling ads. It can't do that with partner sites. And partner sites don't have the resources to improve their advertising conversions at a similar pace.

Which means AdSense revenues for media companies will continue to fall because AdWords is more efficient.

---

Media companies that partner with Google in its AdSense program do it because they don't know what else to do. The economics of partnering with Google are poor and the relationship is unsustainable because of the inherent conflict of interest.

Why Would GOOG Maintain AdSense?

-There are strategic purposes, it forces media companies in its network to compete with its far more profitable business model which weakens them as potential competitors.

-Also, it keeps third-party sites out of rival ad networks.

-AdSense is a great "cookie jar" because if GOOG ever needs to meet its numbers for its quarter, it can push more ads through its own sites rather than through partners.

UPDATE: Independent Advertising Network Advantage 

Independent advertising networks, which don't compete with their publisher partners, such as Blue Lithium, Federated Media, and others, will be able to attract partner sites away from Google because they can invest in technologies to improve revenues for the entire network.

But how much freedom do large publishers have in leaving the Google network? Some have contracts with Google that could tie their hands for years.

FAST Search and Transfer, the leading European search firm, introduced a search based business platform for media companies and said that search traffic from its customers has surpassed Yahoo! and will overtake Google in 2 years.

The Norwegian based company said its FASTMedia software platform is the first to match content to individual users, in addition to supporting all types of online advertising models. The suite of applications is designed specifically for media companies and enables them to run their own ad networks.

FAST said that its top 35 media customers already generate search traffic that exceeds that of Yahoo!.

The market research firm IDC estimates that 70 per cent of search queries do not come from search engines.

Instead, people are choosing to bypass search engines and are going
directly to their preferred information sources, retailers, and other
websites and searching there.

FAST recently completed a survey of its top 35 media customers:

Together this "FAST Media Network" is generating as much search traffic as Yahoo!, with search traffic growing more than twice as quickly.  At the current pace, searches within the "FAST Media Network" collectively will surpass Google in two years...

FAST wants media companies to "turn search from the disrupter of their worlds"  and use the technology to grow their businesses.

For example, newspaper publishers have stood by and watched Google and Yahoo sell massive amounts of advertising based on their content without any direct benefit--unless a reader clicks through to their news site, and then happens to click on a Google or Yahoo text ad.

FAST wants to cut out the middle-men by selling search-based business software to media companies to run within their own data centers and operate their own adverting businesses.

Global Media Warming

The battle for media markets is heating up.  Google recently acquired DoubleClick for $3.1bn and today Yahoo! acquired Right Media for $680m, to beef up their advertising services.

The competition is for a very lucrative business--running massive advertising networks for global media companies. The search giants keep about 20 per cent of total advertising revenues.

But this revenue share is skewed towards a handful of large media companies with the clout to negotiate better deals. Most publishers receive far less, between 60 and 40 per cent of ad revenues.

The FASTMedia software supports pay-per-click, contextual, classifieds and all other types of online advertising services, plus self-service, and unique features. It's  "Featured Content" technology matches content on-the-fly with online users' searches. Publishers can use this for targeted premium content services.

FAST Search and Transfer, a leading European enterprise search company, said it will build an advertising exchange for its media company customers, who between them have an online audience that rivals Yahoo! in size.

The move will help to establish it as serious contender in the battle for large media advertising accounts. Yahoo and Google have recently signed big ad distribution deals with some of the largest newspaper, radio, and TV companies.

The difference with FAST's approach is that media companies use FAST software to run their own advertising networks within their own data centers. And they get to keep all the revenues instead of paying advertising networks such as Google and Yahoo as much as a 40 percent share.

LinkedUp

FAST's advertising exchange would link together media companies that use its advertising network software. The value of the ad exchange is in providing publishers with opportunities to sell excess inventory and gain the liquidity of a much larger network.

There would also be opportunities for advertising alliances and new types of ad services between publishers.

The FAST software can be configured to run many types of advertising models, such as pay per click, impressions, classified, and auction based models. It includes self-service features. Customers install and run the software within their own data centers rather than using third party networks.

From its founding in 1997 in Norway, FAST has been steadily building a strong name in enterprise search within global corporations, It has about 800 staff in the US, and about 200 in Norway, with offices around the world.

Autonomy in the UK, is the market leader in this sector but FAST has managed to win some large customers and also large media companies. IBM and the New York Times are among its search customers.

It's expansion into advertising software builds on its core search technologies, which are used to match advertising with content and users--the same as Google, Yahoo, and Microsoft advertising services.

Don't expose the customer

But using Google AdSense or Yahoo Publisher Network, not only reduces revenues, it exposes customer relationships that the ad network giants can target with additional services. Their publishing partners get none of this extra revenue.

"We think that newspapers and magazine publishers should keep more of their advertising revenues, and their customers, " John Lervik, CEO and co-founder of FAST, told SVW during a recent visit to San Francisco.

Joint Ventures

Mr Lervik said that FAST will also partner with some of its media customers in joint ventures around online advertising. One of these is a joint venture with Softbank in Japan.

The details of the FAST media enterprise software suite are under embargo. It builds on its FAST AdMomentum product.

...

Please See: The Vikings are in GOOG's rear mirror and coming up FAST

Recently I wrote "Is Search Broken" about the amount of help search engines need to find and index web pages. Now Matt Cutts, GOOG's chief evangelist in the blogging community, has asked people to report web sites which are selling paid text links.

Those web sites would then be given a lower pagerank and might even be banished from GOOG's index.

It seems a bit rich for Google, the world's biggest seller of paid text links, to ask for help in identifying sites that sell paid links.

Threadwatch.org, a leading site about search, weighs in:

Google is still indexing those lolita preteen results, ranks all these .edu ringtone pages, and lets not forget that Google continues to deliver AdSense ads on sites they banned for being spam.

If Google doesn't CLEARLY mark their own paid links, encourages publishers to blend them into content, and doesn't police their own network, why do they think they have the right to police other sites?

Google Wants Reports of Paid Links ... What a Joke | Threadwatch.org

Paid links can confuse Google's pagerank system which pays attention to links in web pages to help find relevant search results. Mr Cutts says that Google has some technology in the works that will identify paid links.

In the meantime, he wants webmasters to clearly identify paid links, and also content that has been paid for.

You could put a badge on your site to disclose that some links, posts, or reviews are paid, but including the disclosure on a per-post level would better. Even something as simple as “This is a paid review” fulfills the human-readable aspect of disclosing a paid article.

Matt Cutts Gadgets, Google, and SEO » Hidden links

Does this mean if a magazine publisher pays a freelancer to write a review and to mention specific companies this has to be clearly marked? Does it make the review less trustworthy or less worthy of being indexed? There's a potential Pandora's box of issues here if you start applying this to mainstream media...

Over at Threadwatch.org, the paid links issue continues to rankle:

The more I think about it the more I realize why Google doesn't like the various flavors of paid links. It has nothing to do with organic search relevancy. The problem is that Google wants to broker all ad deals, and many forms of paid links are more efficient than AdWords is. If that news gets out, AdWords and Google crumble.

The Real Reason Google Doesn't Like Paid Links Threadwatch.org

Mike Lynch, the CEO of Autonomy, the UK enterprise search company, was in town this week (video is coming). I used to meet with Mr Lynch regularly when I was at the Financial Times, and when the dotcom boom was in full swing.

The dotbomb affected Autonomy along with tens of thousands of other companies, and Mr Lynch's visibility suffered too. So it was good to catch up with Mr Lynch and we joked that the "bubble" was back (it's not, of course).

We chatted about some of the trends in Autonomy's space, which is how to deal with unstructured data--estimated at about 87 per cent of all data.

Autonomy's technology uses statistics to find correlations between data, documents, video, and audio. It doesn't need artificial intelligence to understand the connection between data--it understands the probability of that data being related.

Autonomy uses Bayesian probability developed by Thomas Bayes, an 18th century British Presbyterian minister. The advantage of this approach is that Autonomy can find stuff without knowing any keywords or tags or taxonomy--it can determine the taxonomy on the fly.

During a presentation, Mr Lynch slammed the popular practice of tagging web content and says that it won't help to organize information. Mr Lynch quoted an essay by Cory Doctorow, the science fiction writer, titled Metacrap. "Tags don't work because people lie, they are lazy, and they use different tags. And there is a huge amount of information that will never be tagged."

I agree, and I resent the work in having to tag everything (See: Is search broken?) But, tagging does work to some extent, and in some applications.

Autonomy's technology could be used to improve "tagging." Often, it is not clear what tags to apply--technology such as Autonomy's could help identify the appropriate tags. For example, are the Technorati tags at the end of this post the right ones to use to associate this post with other, similar posts?

...

Earlier this week, Autonomy introduced a new product called Virage ACID (Automatic Copyright Infringement Detection) which uses its technology to search through video images. It can automatically detect copyrighted videos.

 Being totally independent of media format, ACID can not only detect whether distributed video infringes copyright, but also whether audio content ripped from a copyrighted video or audio track that is overlaid on legitimate video has been uploaded.

Seems like it was designed for YouTube...

----

» Is search broken Tom Foremski IMHO ZDNet.com

Under revisions announced late Wednesday, Google promised to wrap a cloak of anonymity around the vast amounts of information that the Mountain View-based company regularly collects about its millions of users around the world.

Google believes it can provide more assurances of privacy by removing key pieces of identifying information from its system every 18 to 24 months. The timetable is designed to comply with a hodgepodge of laws around the world that dictate how long search engines are supposed to retain user information.

Google to Adopt New Privacy Measures By Michael Liedtke, AP Business Writer

Interesting. Especially since GOOG and the other search engines have often said they are interested in behavioral data rather than individual data. If GOOG didn't collect identifiable data in the first place it wouldn't need a privacy policy. And it wouldn't be vulnerable to this:

Authorities still could demand to review personal information before Google purges it or take legal action seeking to force the company to keep the data beyond the new time limits.

Nevertheless, Google's additional safeguards mark the first time it has spelled out precisely how long it will hold onto data that can reveal intimate details about a person's Web surfing habits.

I hope it is not the same type of unidentifiable data that AOL released last year, which didn't take much work to identify some users.

The unguarded thoughts of the digital haves...

The most compelling content on the Internet, by far, is AOL's release of search terms linked to individual users. This is a glimpse into the human condition that goes way beyond anything else we have seen, beyond Dostoevsky, Dickens,...

Technorati Tags: privacy - search -

TOKYO (Reuters) - The online collaboration responsible for Wikipedia plans to build a search engine to rival those of Google Inc. (Nasdaq:GOOG - news) and Yahoo Inc. (Nasdaq:YHOO - news), the founder of the popular Internet encyclopaedia said on Thursday.

Wikia Inc., the commercial counterpart to the non-profit Wikipedia, is aiming to take as much as 5 percent of the lucrative Internet search market, Jimmy Wales said at a news conference in Tokyo.

"The idea that Google has some edge because they've got super-duper rocket scientists may be a little antiquated now," he said.

Describing the two Internet firms as "black boxes" that won't disclose how they rank search results, Wales said collaborative search technology could transform the power structure of the Internet.

Link to Wikipedia founder says to challenge Google, Yahoo - Yahoo! News

Interesting. I've been writing about people-powered search compared with machines and algorithms:

Is search broken?

Microsoft thinks that it is - at least, it concedes that MSN Search is not doing the trick (Google market share: 53.7%; MSN marketshare: 8.9%). So at TechFest - Redmond's three-day chance to show off to reporters and employees the neat tricks MS Research is working on - the word of the week is "search," says the Times' John Markoff.

Lili Cheng, a user-interface designer for the Windows Vista operating system, showed off a new service called Mix that will allow Web surfers to organize search results and easily share them. She said Mix would be released in six to nine months.

A second tool demonstrated, called Web Assistant, is intended to improve the relevance of search results and help resolve ambiguities in results that, for example, would give a user sites for both Reggie Bush and George Bush.

Microsoft is not the spiffiest when it comes to product names, but these widgets are not products yet.

Personalized Search compares web-search results with an index of content on the user's hard drive (that's generated by Desktop Search, quite similar apparently to Google Desktop). It does a neat trick, though.

Susan Dumais, a veteran Microsoft search expert, demonstrated the effectiveness of the program by searching for Michael Jordan. By culling through local information on her hard drive, the program was able to discern that she was interested in finding the Michael Jordan who is the machine-learning expert at the University of California, Berkeley, not the basketball player.

Search in the future will look nothing like today’s simple search engine interfaces, she said, adding, “If in 10 years we are still using a rectangular box and a list of results, I should be fired.”

About 7,000 employees participate in the event as well as reporters and business partners. Rick Rashid, head of MS Research, originally opposed the festival idea but he's now a fan.

“We realized we were clearly tapping an underserved community,” he said. Now the company uses the event to move technology from its research division into products.
That includes monitoring which employees visit which lectures and booths and looking for patterns, he said.

(Thank you all for your comments on this topic of search...)

The holy grail within search is to be able to serve up the right search result from one user click.

The "I'm Feeling Lucky" button next to the "Google Search" button constantly tests this ability. One day it might very well be the only button found there because the underlying search technologies will be hugely improved.

And researchers within the search engine communities will be widely recognized for their groundbreaking work--I'm certain that there will be future Nobel prize winners among them.

But improvements in search, through the development of better algorithms and IT architectures, will very likely be of lesser value over the next few years compared with what can be achieved developing "people-powered" search technologies.

It can be seen in sites such as Digg, which is a  better news aggregator using people-powered technologies, than machine-powered Google News, which states near the bottom of its page:

"The selection and placement of stories on this page were determined automatically by a computer program."

It is not just at Digg, there are "people-powered" search efforts all across the globe, involving tens of millions of people,  laboring every day to help improve the search experience. This is done by adding tags, site maps, headlines, etc, -- they are creating ever larger amounts of valuable search metadata about content.

And we are just at the beginning of this trend. It is a markedly new phase in the development of search, one best described as "people-powered and technology-enabled."  As opposed to mostly "machine-powered" through algorithms and servers.

What will be interesting to see if the people-powered component outweighs the value the machine-powered component of a search service. Because that would determine where the competitive edge is to be found. Is it in the algorithms or in the people?

And it begs questions such as: Is GOOG's lead in machine-powered search technologies large enough to keep it ahead of competitors harnessing people-powered search?

Search engines say they use complex algorithms to help users find exactly what they want. Google's "I'm feeling lucky" button (btw, does anybody use it?), right below the search box implies that very thing.

The legions of top Ph.Ds working for the search engines publish oodles of scientific papers on complex mathematical concepts related to search.

 Recent Papers Written by Googlers

• A complete, co-inductive syntactic theory of sequential control and state

• Achieving Anonymity via Clustering in a Metric Space

• An Assertional Correctness Proof of a Self-Stabilizing l-Exclusion Algorithm

• On-the-fly sharing for streamed aggregation

• Programmable clustering

• Using Many Machines to Handle an Enormous Error-Correcting Code

It all looks very impressive but it seems to have more to do with contributing to the mythology surrounding search--that it is very complex and scientific--than to the actual reality of how search is done.

From my vantage point as an online publisher, it is clear that search is increasingly "people-powered" rather than machine-powered. There are millions of people helping the searchbots find information.

Here are some examples and gripes:

- There are many publishers that try to make sure their headlines catch the attention of the search engines rather than catch the attention of readers. The same is true for content, editors increasingly optimize it for the search engines rather than the readers.

- Why should I have to tag my content, and tag it according to the specific formats that Technorati, and other search engines recommend?  Aren't they supposed to do that?

- Google relies on a tremendous amount of user-helped search. Websites are encouraged to create site maps and leave the XML file on their server so that the GOOGbot can find its way around.

- The search engines ask web site owners to mask-off parts of their sites that are not relevant, such as the comment sections with no-follow and no-index tags.

- Web sites are encouraged to upload their content into the Googlebase database. Nice--it doesn't even need to send out a robot to index the site.

- Every time I publish something, I send out notification "pings" to dozens of search engines and aggregators. Again, they don't have to send out their robots to check if there is new content.

- Google asks users to create collections of sites within specific topics so that other users can use them to find specific types of information.

- The popularity of blogs is partly based on the fact that they find lots of relevant links around a particular subject. Blogs are clear examples of people-powered search services.

And there are many more examples. If the search engines are so great at doing what they do, then how come we have to do all of the above?

I resent the fact that I have to create all this content describing my content--the search engines should be creating this "metadata."

I just want to write stuff,  and leave it up to the search engines to find it, classify it, index it, and do all the other things their mythology suggests that they do.

In the world of enterprise search, companies such as FAST, Vivisimo, Autonomy, etc, have to find information without the benefit of aids. Corporate documents have no pagerank or tags or much metadata of any kind.

Yet in consumer search it seems as if nothing would be found without a huge amount of help from millions of people every day. Why is it that we have to help the search engines do a job they are supposed to be doing by themselves?

I wonder about the productivity cost to society from all this human labor--work that is supposed to be done by robots.

It's as if these searchbots are blind, and we have to lead them patiently along the street and point things out to them, while they tap away at the world with white canes.

...

Part 2: Search seems to be broken...

FAST Search and Transfer has suddenly popped into my sphere of attention and I mean really popped. I got to spend some time at FAST's user conference at the end of last week, and it was an educational experience that got me interested in search again.

This Norwegian based enterprise software search company has made the subject of search compelling again. For too long GOOG has made it appear as if it had already won the search wars--anything better would be an incremental improvement.

Yet enterprise search--which is where FAST has staked its expertise--is a much more interesting subject than I imagined, and much more interesting than consumer search. Enterprise search is much more difficult problem, and one of the most challenging problems in IT.

Consumer search can be vague and still be successful. It can bring up a list of nearly relevant sites or documents, and usually that is all that is needed. But in the enterprise, search is usually needed to find something very specific, a contract, a purchase order, a memo. 

And there are all sorts of conditions associated with access to data, some security based, others are regulatory. Search quickly becomes quite a complex process and one that can lead to other things.

Enterprises use a lot of structured data, but there is also a massive amount of unstructured data too. Search in the enterprise could potentially bring the two data world's together.

You might even be able to create enterprise applications by using modified search algorithms.

This type of scenario gets very interesting: enterprise applications by algorithm. This is already happening in business intelligence, I wonder how far, in theory, such an approach could be taken. Could you create CRM applications through search algorithms?

I'll be writing more about this subject, and FAST, over the next few days.

Susan Feldman, a senior analyst at IDC, will release on Friday the results of a groundbreaking study that shoots down one of the largest myths in search engine marketing: that the majority of traffic to web sites comes from the top ten search engines.

By comparing publicly available traffic data from companies such as Nielsen Research, with research of its own, IDC found a big discrepancy in terms of the number of search queries tracked. Ms Feldman said, "Our model showed that there were seven to ten times more search queries being made and that the large search engines had only about 30 per cent of the search query traffic."

Ms Feldman said that the missing search queries, the dark matter of the search engine world, were coming from direct queries. People would go to a web site such as Amazon.com and type in a search query.

"This means that there is a massive business opportunity still to be had. The top search engines do not own the web, at least yet," said Ms Feldman.

The IDC results will be released at a Friday session at a conference organized by Fast Search & Transfer, a client of IDC and a vendor of search enterprise software. Earlier this week Fast introduced its AdMomentum product which allows online publishers to set up their own advertising networks instead of sharing revenues with Google, Yahoo or other ad networks.

. . .

Foremski's Take: The IDC findings are not a revelation for any online publisher. Peeking into the server logs reveals where traffic is coming from. For example, SVW gets less than 5 per cent of its traffic from search sites, and that is great because my traffic is not "surfer" it knows where I live and comes in direct. Yet, the bandwidth used by the search engine robots is one third of my total--to serve less than five per cent of my visitors.

The IDC numbers will help to dispel one of the big myths about the Internet, that the search engines drive substantial amounts of traffic therefore sites need to optimize for the search engines.

I've long said: optimize for your customers/readers not for the search engines. SEO, beyond basic principles, is not worth it, yet many companies spend a lot of money making their sites searchbot friendly rather than user friendly.

- - -
Additional Info:

Fast Forward conference blog

Fast Forward Conference Agenda

SVW:

The lie of distribution--search engines return very little value to news/blog sites yet hog bandwidth and increase server loads

GOOG: give us your content for free!

craigslist: Battling the bot armies

The lie of distribution

Posted by Tom Foremski on October 26, 2005 5:51 AM

By Tom Foremski

ComScore Networks' latest analysis of search engine activity was for December 2006, it showed Google gaining share, and at a faster pace than second placed Yahoo!

Google increased its lead by 0.4 share points to 47.4 percent of the total US market, compared with November 2006. Yahoo added 0.3 share points with 28.5 percent of the total. Microsoft sites were third with 10.5 percent, followed by Ask Network with 5.4 percent, and Time Warner with 4.9 percent.

Google and Yahoo gains were at the expense of Microsoft, which lost 0.5 share points, Ask, which fell by 0.1 share points, and Time Warner which lost 0.2 share points.

ComScore assembles its data from monitoring the Internet activities of more than 2m consumers.

Foremski's Take: Google's lead shows no signs of flagging. Yahoo is doing a decent job in gaining share but not enough to catch Google. Google continues to grow much faster.

Third, fourth and fifth places in search rankings are all declining. Is there a place for these and specialized search engines on the Internet? Or is it that only the top two Internet businesses in each category have the best chances to succeed?

From ComScore Networks:

* Americans conducted 6.7 billion searches online in December, up 1 percent versus November.  Annual growth rates in search query volume remained strong with a 30-percent increase since the same month a year ago.

* Google Sites led the pack with 3.2 billion search queries performed, followed by Yahoo Sites (1.9 billion), MSN-Microsoft (713 million), Ask Network (363 million), and Time Warner Network (335 million).

Local search and local online commerce are the next battlegrounds for the giants such as Google and Yahoo, but also for many startup companies. The lure is the billions of dollars spent on Yellow Pages advertising by local businesses.

We have Yahoo Local and Google Local, plus Max Levchin's Yelp, CitySearch, Ingenio, plus local newspapers and other companies--all trying to grab a piece of the local search and local commerce ad spend.

But tapping into the local businesses market through online services is hard. The same factors that make scaling a global online service easy on the Internet become reversed when applied to local businesses.

For example, a local pizza parlor gets nearly all its business from within two miles of where it operates. Reaching China or even a neighborhood five miles away is something the Internet does well for many companies. But it doesn't make much sense for a local pizza parlour, and the same is true for most local businesses--they all nned to reach customers in their neighborhoods.

I recently spoke with two startups, Smalltown and Grayboxx, with two different approaches.

Smalltown targets neighborhoods and small communities

I was very impressed with Smalltown's approach to local search because of the elegant design and simplicity of the site. Simplicity is not easy to do but it is extremely vital in the online space.

Last week I spoke with Smalltown CEO Hal Rucker. "To succeed in local markets we belive that you have to be local," said Mr Rucker. "So we've created 'webcards' which are a type of virtual index cards with their own web address that can be created by anyone local, by a business or an individual."

An example of a webcard:

Webcards can be attached to each other, and also sent to people. They provide an easy to use template that allows busy, small business owners to create a web page about their business without having to have a website.

And the process is simple enough that any customer of a local business can create a webcard that acts as a recommendation, or a warning, by filling in text and adding photos.

"With webcards we are helping to build what we call the 'local web.' And it creates 'social advertising,'" said Mr Rucker.

The roll out of Smalltown will be town by town. The first two are San Mateo and Burlingame, in the heart of Silicon Valley.

If they can get the formula right, then they can reproduce it in other small towns and city neighborhoods.

Each Smalltown would have a local person/agent to help evangelize the service and also aid local businesses and people to produce webcards. Smalltown is =offering cash bounties to people of $5 per five webcards, a smart way to create a large inventory of webcards, which creates a potentially useful library of local content.

SVW's take: Smalltown has a great user interface and I love the concept of webcards, each with their own web address, each discoverable through the search engines, as well as the local Smalltown portal.

What I'd like to see with such approaches is a compelling front end, by which I mean a reason to go to the site even if I'm not actively searching for something local. What would draw me to it if I did not have a need to find something? I've got a couple of ideas...

Coming up next in SVW: Interview with Bob Chandra, founder and CEO of Grayboxx (still in stealth mode). This local search startup taps unseen databases to create ranked lists of top local businesses favored by their communities. . .

Baynote came out of stealth mode today with a software as a service product it calls "content guidance." It helps users find what they want on web sites with thousands of documents by studying the behavior of prior visitors.

Type in a search term and up pops a selection of relevant documents that are "useranked" according to how others found what they wanted.

"Our technology works as a silent observer and we can tell by a user's behavior if they found what they were looking for. When you consider the fact that studies show only 17 per cent of web site visitors find what they were looking for, that means you are wasting 83 per cent of your money spent on online marketing campaigns to drive users to your site," says Jack Jia, CEO of Baynote.

Typically, it can take six clicks to find relevant content and today's visitors are increasingly less patient and more likely to give up. Mr Jia claims Baynote's technology provides a 20 times increase in users finding the content they want, usually in just one click.

Mr Jia is a former CTO of Interwoven, so he has a good understanding of web site design and content plus his team makes use of the latest discoveries in artificial intelligence. The Baynote technology tracks 12 key behavioral characteristics displayed by web site visitors in order to determine whether they found what they wanted or left frustrated.

The service also offers a variety of usage reports and is available for $995 per month for large businesses and $95 per month for small businesses.

SVW's take: The price seems a bit steep for small businesses but for large web sites serving many different types of user groups, Baynote potentially provides a way to avoid leaving lots of frustrated visitors.

The service is also a good way to test web site designs to make sure the most popular content is easily accessible and where to tweak navigational elements.

One problem is that search on large web sites is notoriously bad because you can't apply the traditional page rank methods to determine the importance of a specific web page or document. And so people's bad experiences with site search on large corporate sites means that they will often avoid the search box and thus might skip the little icon that signifies Baynote's search technology is being used.

Baynote does offer other ways of connecting users with relevant content by clever use of AJAX and mouse-over techniques that popup pages in a Browster-like way without requiring users to commit to a click. The background loading of pages into cache however, will skew resident analytical software. But Baynote provides an analytics component as part of the service, which provides detailed usage reports.

Interestingly, Baynote does not use its technology on its own web site - there is no search box!!!!

I recently met with Raul Valdes-Perez, the CEO and co-founder of search engine company Vivisimo. It is not a well known company but it is one that is doing very well, and doing it very profitably--all without any venture capital money.

I first met Mr Valdes-Perez nearly two years ago. At the time, this Pittsburgh, PA based company was ready to take on Google and the others with its unique technology that can index information into subject groups or clusters, by automatically detecting the underlying taxonomy within a set of information, on-the-fly.

Founded in June 20002 by Carnegie Mellon University researchers, Vivisimo offered an new approach to search. I became intrigued by the promises of Vivisimo's technology. I wondered if it could be used in other ways too; if it might reveal emerging social trends before they become visible in traditional ways.

And the clustering approach always seemed much more useful for researching various topics. For example, by seeing the various groupings of a term, i.e.. "bond" which could be James Bond, or a chemical bond, or a family bond--you could find unexpected, serendipitous connections to a term without requiring serendipity to reveal them.

I would sometimes fantasize about using Vivisimo's technology for local search sites. I own the domain names SearchSiliconValley.com and SearchBayArea.com, and the same "searchxyz.com" format for the San Francisco Bay Area counties and towns. Vivisimo's spider is not designed to crawl the entire web, but it is very good at crawling thousands of web sites, perfect for specialized applications, such as indexing a specific region. Clustering the results would be a great way to quickly find all the local Mexican restaurants, or public libraries, and many other uses.

One of these days I might get to try out Vivisimo's technology for such an application, but in the meantime the company has been doing very well in the enterprise search market, especially in government applications. Earlier this year it won a prestigious contract, following stiff competition, to provide the search technology for FirstGov.gov, the official portal for the US government.

Search results are from government web sites and also from related web sites outside of the government, and Vivisimo has been winning considerable praise for a much improved search experience.

I was glad to hear that Vivisimo was doing well. And Mr Valdes-Perez was kind enough to remind me of some advice I gave him a while ago, when Vivisimo was launching Clusty.com, the consumer version of its technology, and dreaming of taking on and winning against Google and the others.

The advice I gave was that the noise level in the consumer search space would soon become very intense and would require massive amounts of money to stay visible. For a small company such as Vivisimo, it would not be a game it could play without having to raise considerable capital. For Vivisimo, the opportunities would be in more specialized approaches where it could offer unique advantages, and fly under the radar while the big companies battled for the prized consumer markets.

And that's what the company has been doing. Apart from its FirstGov and other government related business, it is also doing well in the Life Sciences arena, where finding clusters of information is exceedingly important. And there are other such opportunities in other industries.

Clusty.com did not usurp Google, but it is a very good business for Vivisimo. "Clusty gave us experience in creating simple user interfaces. Too often, our competitors in the enterprise search space pay very little attention to the user interface, which makes their products difficult to use." says Mr Valdes-Perez.

Specialized search applications such as the FirstGov contract require specialized approaches unique to each implementation. FirstGov for example, requires at least 58 boolean terms for each search query. And the algorithm has to distinguish between different parts of the web page--not all dot-gov web pages are created or formatted in the same way.

The Google, Yahoo, Microsoft approach to search is far more general than the tuned, specialized approach taken by Vivisimo. And that means the search giants won't find it that easy to expand into the enterprise search markets.

Mr Valdes-Perez is also a critic of the behavioral technologies that the large search engine companies use to try and improve the search experience by collecting personal data. Clusty.com does not collect any user data which means that there can be no privacy breaches, accidental or subpoenaed.

"Users search based on their whims at the time, and not on past behavior. It is much better to provide a user with several options on what they are searching for and allow them to choose," he says.

That certainly makes sense to me. I hardly ever run the same search twice (except for ego surfing :-) so why collect my personalized search data?

Here is an excellent story on the subject of click fraud from the consistently good CIO Today magazine.

Click Fraud Gets Smarter

http://www.cio-today.com/story.xhtml?story_id=41828

Does Google and the others have the technology to filter out click fraud? Greg Boser is going to put that to the test:

Web consultant Greg Boser has an ingenious method for sending loads of traffic to clients' Internet sites. Last month he began using a software program known as a clickbot to create the impression that users from around the world were visiting sites by way of ads strategically placed alongside Google search results. The trouble is, all the clicks are fake. And because Google charges advertisers on a per-click basis, the extra traffic could mean sky-high bills for Boser's clients.

But Boser's no fraudster. He cleared the procedure with clients beforehand and plans to reimburse any resulting charges. What's he up to? Boser wants to get to the bottom of a blight that's creating growing concern for online advertisers and threatens to wreak havoc across the Internet: click fraud.

The article makes a good case for the need for an independent third party to monitor this problem.