Web search in a post-Twitter world

We all contribute to the news cycle when we post timely content online, even if it's 140 characters or fewer, and this week we learned that our little bits of information have substantive value when search giants Google and Microsoft announced that they will index our tweets and status updates.

But the sheer volume of content that we produce could be a problem, and Twitter users who find day-to-day value in the service may scoff at the idea that the "information firehose" of live content can be tapped and made searchable. It doesn't take extensive use to see that most tweets are in fact worthless, and that thousands of bots simply use Twitter as a tool for free promotion, by tagging links to a particular Web site with trending topics.

But this seemingly uncontrollable flow of information is shaping the way news gets to us, and it presents an important evolutionary hurdle for the search industry: We know there is extremely valuable information there, but how do we make it work for us?

Gerry Campbell has been dealing in the search space since the days of CompuServe, and was one of the first to work on monetizing Web search at AltaVista nearly 11 years ago. Campbell is an angel investor in nearly a dozen social search companies, including Summize, which came to power Twitter's own on-board search. He currently heads Collecta, a real-time search engine which shows how far we've come in terms of live news: It's a continuous query search engine that streams news articles and relevant tweets (which are filtered for content rather than indexed) as they happen until the user decides to hit "pause."

Campbell today shared some perspective on the trends in content creation and consumption. "There are two trends that are fascinating to me, the bright lights of all the trends: The first is the current velocity of content creation. Google has a trillion documents indexed, and about half a billion new indexable pieces of content are being published every single day. But the nature of what's being published has changed dramatically since Google began about ten years ago. Today it's short-lived, quickly published documents. They're high velocity. Old documents were meant to have more weight to them so they could be read longer. Information now is valuable for less time, and an important space is in finding a better way to surface this."

Startup search services like Worio attempt to do just this, and provide an equally valid mix of results from both traditional sources and "real-time" social sites. Worio combines keyword- and context-based search with "Web discovery" algorithms like the ones that suggest content in Pandora and Netflix. Topics which are spiking in interest on the social sites are more likely to be included in search results.

"You want current news, but you want to carefully look at what is interesting rather than just what is popular. iPhone, for example is historically more popular than Palm Pre. However, on the date such as the Pre's launch, search results for 'smartphone' would have been more skewed toward the Pre because that topic had more traction at the time. Some search results are always more popular than others, but social media activity helps us determine how interesting a topic is," Worio CEO Ali Davar said today.

The second important trend that Gerry Campbell pointed out today is the Web's changed role in our daily life.

"The Internet isn't a destination any more, we don't make time in our schedules to get on the Internet," Campbell said. "The emerging model goes beyond even the 'always on' concept. I've got my phone on, my notebook open at every meeting, with information constantly flowing to me at all times. I don't necessarily want a summary explaining what happened...I want to be involved as the story evolves. All of those little bits of information have a daily value to us which gets less valuable over time. This is the evolution of news.

"The topics have a shelf life based upon the format in which they're presented. A study recently showed the diffusion of a news story: it starts in Twitter and on social sites, then as it moves into blogs, it gets colored with interpretation and opinion, and the value goes up. Finally, mainstream media gets it and value goes up even further with trustworthy analysis."

With the increased flow of real-time data and increased daily use of mobile devices, Web search in a post-Twitter world is not going to be a destination model, but rather a model of integration.