Burnham's Beat: Persistent Search: Searchâ€™s Next Big Battleground (via Mowser)

« Real Estate, Cars, Jobs: Watch Out World, Google Base Has Only Begun To Stir | Main | Launching Inductive Capital ... Finally »

04/10/2006

Persistent Search: Searchâ€™s Next Big Battleground

What do you get when you marry ping servers and RSS with stored queries? A whole new type of search that is destined to become the search industryâ€™s next big battleground: Persistent Search. While Persistent Search presents search companies with difficult new technical challenges and potentially higher infrastructure costs, it also gives them powerful mechanisms for building much stronger user relationships which may increase the value of their advertising services.

The Search That Never Stops
Simply put, Persistent Search allows users to enter a search query just once and then receive constant, near real-time, automatic updates whenever new content that meets their search criteria is published on the web. For example, letâ€™s say you are a stock trader and you want to know whenever one of the stocks in your portfolio is mentioned on the web. By using a persistent search query, you can be assured that you will receive a real-time notification whenever one of your stocks is mentioned. Or perhaps you are a teenager who is a rabid fan of a rock group. Wouldnâ€™t it be nice to have a constant stream of updates on band gossip, upcoming concerts, and new albums flowing to your mobile phone? Or maybe you are just looking to rent the perfect apartment or buy a specific antique. Wouldnâ€™t it be nice to get notified as soon as new items which roughly matched your criteria were listed on the web so that you were able to respond before someone else beat you to the punch? Persistent search makes all of this possible for end users with very little incremental effort.

Something Old, Something New
While the technical infrastructure required for Persistent Search services leverages existing search technology, there are several new elements that must be added to existing technology to make Persistent Search a reality. These elements include:

Ping Servers: Most blogs and an increasing number of other sites now send special â€œpingsâ€ to so called â€œPing Serversâ€ every time they publish new content. The ping servers do things such as cue crawlers at a search engine to re-index a site or provide a summarized list of recently published information to other web sites. Because ping servers are the first to know about newly published content, they are critical to enabling the real-time nature of Persistent Search. RSS: RSS feeds can be used both to feed raw information into Persistent Search platforms (in a similar fashion to what GoogleBase does) as well as to take processed queries out. RSS is a polling based mechanism so it does not provide real time notification, but it is good enough in most cases. Stored Queries: Stored queries are simply search queries that are â€œsavedâ€ for future use. Ideally, the stored query is constantly running in the background and it flags any new piece of content that meets the search criteria. While this is a simple concept, it presents some very difficult technical challenges. The easiest way for a search engine to do a stored query would be to execute the stored query into its existing index at some regular interval, say once an hour. However, executing each unique stored query 24 times a day could start to become very expensive if Persistent Search starts to take off. One could easily imagine search companies in the near future executing billions of incremental stored queries an hour. Processing these added queries will take lots of extra resources, but will not generate the same amount of revenue that traditional ad-hoc search queries generate because stored queries will often return no new results. One alternative would be for search companies to use the â€œstream databaseâ€ query techniques pioneered by start-ups such as StreamBase. These techniques would allow them to query the new content as it flows into their index, not just reducing overall query load but also improving latency. However changing their query approach is a huge step for existing search companies and therefore one that it unlikely to be undertaken. One more likely approach might to use special algorithms to combine stored queries into â€œmaster queriesâ€. This reduces the number of queries that need to be executed and uses simple post-query filters to â€œpersonalizeâ€ the results. Given their critical important to overall quality of persistent search, the design of â€œstored query architecturesâ€ is likely to become one of the key technical battle grounds of search companies, much the way query result relevancy has been for the past few years.

Once these three pieces are put together, search companies will be in position to provide rich Persistent Search services. The results of those services will be distributed to end users via e-mail, RSS, IM, SMS, or some pub-sub standard, depending on their preferences and priorities.

The Business of Persistent Search
From a business and competitive perspective, Persistent Search has a number of very attractive aspects to it relative to traditional ad-hoc queries. Traditional ad-hoc search queries tend to result in very tenuous user relationships with each new query theoretically a competitive â€œjump ballâ€. Indeed, the history of search companies, with no less than 5 separate search â€œleadersâ€ in 10 years, suggests that search users are not very loyal.

Persistent Search presents search companies with the opportunity to build rich, persistent relationships with their users. The search engine that captures a userâ€™s persistent searches will not only have regular, automatic exposure to that user, but they will be able to build a much better understanding of the unique needs and interests of that user which should theoretically enable them to sell more relevant ads and services at higher prices. They will also stand a much better chance of capturing all or most of that usersâ€™ ad-hoc queries because they will already be in regular contact with the user.

It is this opportunity to build a long term, rich relationship directly with a uniquely identifiable consumer that will make persistent search such an important battle ground between the major search companies. Persistent search may be especially important to Google and Yahoo as they attempt to fight Microsoftâ€™s efforts to imbed MSN Search into Windows Vista.

It should also be noted that enterprise-based Persistent Search offers corporations the opportunity to improve both internal and external communications. For example, itâ€™s not hard to imagine the major media companies offering persistent search services or â€œchannelsâ€ to consumers for their favorite actor, author, or singer.

The State of Play
As it stands, Persistent Search is in its infancy. The world leader in persistent search is most likely the US government, specifically the NSA, however the extent of their capabilities appears to be a closely guarded secret. Some of the commercial players in the space include:

Pub-Sub is in many ways the commercial pioneer of the space. It operates one of the largest ping servers and has been offering customized key word-based â€œreal timeâ€ persistent searches for some time. Some start-ups are even building vertical persistent search services on top of pub-subâ€™s infrastructure. However Pub-Sub lacks the infrastructure and resources of a complete search engine and lacks the consumer awareness and multi-channel distribution capabilities needed to reach critical consumer mass. Google itself offers Google Alerts, a service that enables users to be e-mailed the results of stored queries, however this service relies on Googleâ€™s general index and so it could take days for new content to appear. It also does not appear to directly incorporate ping servers and does not offer any distribution mechanisms beyond e-mail. In addition, as a beta service, itâ€™s not clear that it is capable of scaling efficiently should demand take off. Real Time Matrix is a start-up focused on aggregating RSS feeds and re-broadcasting them based on user preferences. The net effect of their technology is to deliver real-time Persistent Search-based RSS feeds to consumers. Technorati, a popular blog search engine, allows users to create â€œwatchlistsâ€ of persistent searches. However Technorati limits its index to blogs and so does not offer a comprehensive search service. Yahoo and MSN offer their own alert services and while these services provide very good distribution options (e-mail, IM, SMS), they do not integrate into their search engines but just offer canned updates for things such as news stories and the weather. WebSite-Watcher and Trackengine offer services that track specific sites and/or RSS feeds for changes, but they do not allow fine grained free text queries and are just focused at the site level.

Despite this activity, no one has yet to put together an end-to-end Persistent Search offering that enables consumer-friendly, comprehensive, real-time, automatic updates across multiple distribution channels at a viable cost. That said, the opportunity is clear and the competitive pressures are real, so I expect to see rapid progress towards this goal in the near future. It will be interesting to see how it plays out.

April 10, 2006 in Blogs, Internet, RSS | Permalink

Other Articles In This Blog By Topic: Blogs Collaboration Content Managment CRM Database Development Tools EAI ERP Internet Middleware Network Management Open Source Operating Systems Operations Management PLM RSS Security Software Stocks Supply Chain Venture Capital Wall Street Web Services Wireless

Comments

PubSub has *all* the technology bits -- including the use of the Jabber/XMPP-based pub-sub for doing "real" realtime results, not just RSS/Atom polling.

Of course, this is IM client dependent, but they could easily run a service whereby users could sign up, add a PubSub "bot" as their buddy, and voila! real time results today.

Now if only they would hire a usability/design firm to rebuild their website from scratch...a developer community would probably be a good idea, too.

My guess: Google may get their first, since GoogleTalk is built around XMPP as well.

Posted by: Boris Mann | Apr 10, 2006 4:46:33 PM

See http://immedia.at and http://zaptxt.com

Cool stuff.

Posted by: Marshall Kirkpatrick | Apr 10, 2006 6:20:42 PM

About Google, didn't you forget the RSS/Atom alerts in blogsearch.google.com?

It takes only minutes for content published by RSS/Atom to be in the index of blogsearch. So, you permanent searches in BlogSearch are fed with very recent data

didier

Posted by: d.durand | Apr 10, 2006 8:47:14 PM

Great article but you missed the best one. Windows Live.

Posted by: Jason | Apr 11, 2006 9:01:17 PM

You wrote: "One could easily imagine search companies in the near future executing billions of incremental stored queries an hour." At PubSub, we normally do billions per minute and trillions per day... In the last minute (1:16am EST), we did 2,858,901,318 query evaluations. That works out to over 4 trillion queries per day. The future is now.

You wrote: "â€œstream databaseâ€ query techniques pioneered by start-ups such as StreamBase." Actually, PubSub has been providing services much longer than StreamBase. PubSub is still the only Internet-Scale near-real-time persistent search provider (we call it "prospective search").

You wrote: "[No one provides] real-time, automatic updates across multiple distribution channels at a viable cost." For free, we deliver via XMPP/Jabber, REST, SOAP, RSS, Atom, SMS (via partners) and email (some topics). We'll be adding more delivery mechanisms in the future.

bob wyman
CTO, PubSub.com

Posted by: Bob Wyman | Apr 11, 2006 10:29:06 PM

*cough* BoardTracker ;)

We provide alerts by email and jabber IM and have a page showing all your alerts/matches and have rss feeds for searches (ordered by date or relevancy) and browsable categories.. we even have an rss feed for tag clouds - how web2.0 is that? ;)

Lots more nifty features too.. boardtracker.com

Posted by: BoardTracker | Apr 12, 2006 8:34:06 AM

To respond to a few of the comments saying, in so many words, that so and so firm is already doing this, while a lot of firms have put several of the pieces in place no one has yet to do everything. It's important to point out that persistent search is not just focused on Blogs and/or RSS feeds. Ideally it incorporates all web-data no matter how it is accessible. RSS feeds and ping servers obviously negate some of the crawl latency but everything is not accessible via a feed yet, so a full fledged persistent search offering really has to have a crawler farm and has to cover more than just blogs/feeds. Also, the real time or near real time nature of persistent search is very important as well and very few folks have matered this. My ideal persistent search firm would marry something like pub-sub on the back end with Google in the middle and something like Zaptxt on the front end.

Posted by: Bill Burnham | Apr 12, 2006 1:22:52 PM

I've been listening to the debate about persistent search or 'prospective search' for a while now. Timeliness of information is usually cited as the top benefit, which I totally get... Bob Wyman has an excellent example of this (an eBay auction alert) on his blog. However, I haven't seen much discussion around the *relevance* of search results. I have a few PubSub searches, and I get some good info and a lot of 'noise'. In my mind, relevance is as or more important than timeliness in many cases, but I don't see much discussion around improving relevance (again, other than the time aspect of relevance...).

Posted by: David Andrzejek | Apr 13, 2006 10:05:18 AM

Davdi, You raise a good point. I believe that since folks such as pub-sub are using simple key word queries/filters that relevance has suffered, especially for generic terms. Theoretically, if the persistent searches are run through the existing index of the search engines the results should benefit from the same technologies (page rank, click thru rates, etc.) that have improved relevance in the ad-hoc query world. The big problem is that these ad-hoc indexes are not real time so someone has got to figure out how to marry the two while preserving persistent searche's near real-time nature. It's problems like this which makes me believe that persistent search represents the next big technical battleground in search as ad-hoc relevancy has been somewhat commoditized at this point.

Posted by: Bill Burnham | Apr 13, 2006 10:27:22 AM

David Andrzejek wrote: "relevance is as or more important than timeliness in many cases"
Clearly, "relevance" is just as important in prospective search as it is in retrospective search -- but it is also different. In a retrospective search system, relevance is used to order results; however, in a prospective search system, relevance is used to filter results. In one system, relevance determines what is at the top of a result list, in a prospective system, relevance determines whether or not you see a specific result at all.

PubSub computes "LinkRank" numbers for all feeds in order to assist users in filtering based on relevance. Thus, at PubSub, when you create a weblogs subscription, you can say that you want only posts from the "Top 1%" of bloggers, or the "Top 5%", etc. Our LinkRanks are different from traditional PageRanks in order to address the specific needs of prospective searches. Thus, while PageRank gives the same weight to all pages and links no matter how old they are, the LinkRank calculation gives a higher weight to newer links than to older links. There are also a number of other differences in the calculations that reflect specific prospective needs.

PubSub also allows users to filter results based on "community lists." Thus, you can say that you are only interested in posts that come from known A-List bloggers in the PR space or from a similar list of "Legal Bloggers."

LinkRank and community list filtering are not a complete answer to the problem of relevance in a prospective system -- but we've found them to be very useful starting points. There is still much interesting and exciting discovery work to be done in this area.

bob wyman

Posted by: Bob Wyman | Apr 13, 2006 2:15:38 PM

Yeah, these were called "agents" in the mid-90's. Remember all those companies launched in 1995 that all ended in C-A-S-T? They all were working on what's described in the 2nd paragraph. Didn't make money then, either.

Persistent Search. New term, old idea.

Posted by: Michael | Apr 13, 2006 5:49:05 PM

â€œLinkRank and community list filtering are not a complete answer to the problem of relevance in a prospective system -- but we've found them to be very useful starting points. There is still much interesting and exciting discovery work to be done in this area.â€

One mans treasure is another mans garbage. Billboard top 50 list is completely useless for someone who is a German Goth metallic fan. Relevance has to be based on relevance of the author and relevance of the content to the consumer. Once relevance is established filtering mechanism based on reputation of the author and content can be used to devise ranking systems personalized to a users need. Further more in the fragmented space of multiple devices that enable access to information of multiple channels, information needs to be customized to meet the UI limitations of devices.

I agree this is not an easy problem to solve. At ConnectedMix we are building upon the innovations of the likes of PubSub, Myspace, 4Info and developing reputation enabled community aided vertical search approach to tackle this issue. Interest based communities narrow down the relevance of the content created by like minded people. Open reputation for content and author creates filtering mechanism. Personalized access to data customized for your medium of choice help address the UI challenges.

Posted by: Lalit Sarna | Apr 14, 2006 6:04:25 AM

I tried pubsub.com using "Persistent Searc" and here is the result:

http://atom.pubsub.com/34/07/608d85fef84172697d4c109b8e.xml

I like pubsub.com but clearly there are more work to do.

Posted by: Hu Dou | Apr 15, 2006 9:46:55 AM

There is an interesting (differing view) discussion of this topic here:
http://www.threadwatch.org/node/6186

David

Posted by: David Andrzejek | Apr 15, 2006 3:33:03 PM

It's called Intelligent Agents, and its been 'on the way' for a long time...

Posted by: Lyndon Samson | Apr 16, 2006 3:43:15 AM

Possibly the most useful application of persistent search are the job search engines. At indeed.com you can do a search, refine it, and then subscribe to it via RSS and have new jobs that show up anywhere on the Internet sent to you automatically.

Most job seekers who know about Indeed.com and their competitors are doing this and its a very popular feature.

Posted by: fred | Apr 29, 2006 3:29:56 AM

You can run any job search on Indeed.com and save it as an email alert, RSS feed or Instant Message alert. Here's how to get jobs by IM:
http://www.indeed.com/tools/jobsbyim/

And it's all free!

Paul
http://www.indeed.com - one search. all jobs.

Posted by: Paul | May 4, 2006 9:23:49 PM

Hmmm, isn't this a keyword-driven subset of what first version of Stratify's Discovery Engine did? Persistent search=intelligent agent, although only as intelligent as the search engine and the quality of the content it covers. While there wasn't a business model for such a service back in the day, perhaps the time has come... (I refuse to refer to the company as PY ;)

Posted by: David Patchen | May 10, 2006 4:49:31 PM

We agree with alot of these assumptions. Aggregating the latest material on the web and then keeping users updated with that content will eventually become commonplace. The "Pointcast" model is coming back in a big way - as the volume of inforation grows on the web, it becomes impossible to search and keep track of it manually. Ultimately, it's about spending "less" time on the web so that the information you care about is sent to you and at your fingertips 24x7.

Chase Norlin
CEO, Pixsy

Posted by: Chase Norlin | May 11, 2006 5:21:36 PM

Chase Norlin
CEO, Pixsy

Posted by: Chase Norlin | May 11, 2006 5:21:45 PM

AI and the Future of Search:

I heard a rumor Google bought-out an MIT based search engine that is optimized for WiFi networks -- using AI to improve local search -- for >$20 million pre-emptively? Is this true?
http://www.thefinalmile.net/blog/?cat=11

>www.wi5d.net
homepage says "SOLD" but gives no details...

Posted by: Vijay Shaw | Jul 14, 2006 9:45:37 PM

Funny that you bring up persistent search discussion. I was working on a persistent search technology at buiyee.com (stealth mode and hence not much in mentioned on the website) and I presented to Bessemer and CRV and got different responses. Bessemer just did not get the concept at all. From their point of view if their is anything to be done then Google will do it and rest will just wither away.
For CRV, they liked the concept but probably were a little intimidated as the exits do not look that good for players.

Posted by: Sukanta Ganguly | Jul 23, 2006 1:46:28 PM

I dunno but Google (news) alerts combined with Copernic Agent's "track search" function does just fine for me...

Posted by: Evert | Jul 31, 2006 5:13:58 PM

google gets like 60% of all search ..

http://www.ibtimes.com/articles/20060803/google-search-volume.htm

Posted by: bj | Aug 3, 2006 12:52:41 PM

Hi Bill,

I had a small paper describing an alternative ranking algorithm, that'd be useful for applications like persistent search. Look for "Searching association networks for nurturers" to find my IEEE Computer article.

Posted by: Bharath Kumar | Aug 27, 2006 9:15:29 AM

Great Post.

Posted by: Rhon Daguro | Sep 7, 2006 2:52:31 PM

It is great to get so much feedback in this area.

Posted by: Steve | Sep 26, 2006 7:53:10 PM

I don't think that other companies will able to be better then Google, Yahoo and MSN. Now these search engines have 95% of search traffic.

Posted by: BoardReader | Feb 6, 2007 6:01:40 PM

Legal Disclaimer

Brad Feld's legal disclaimer is a classic but since I am not as funny as he is I will just state that the opinions on this blog are mine and mine alone and not affiliated in any way with Inductive Capital LP, San Andreas Capital LLC, or any other company I am involved with. Nothing written in this blog should be considered investment, tax, legal, or financial advice. These writings are just my personal opinions, no matter how misguided and ill-informed they may seem.

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/t/trackback/14841/4640436

Listed below are links to weblogs that reference Persistent Search: Searchâ€™s Next Big Battleground:

» Persistent Search Today from Russell Beattie Notebook
Just read Bill Burnham's post about Persistent Search - this stuff is already available via Yahoo! APIs. Simply combine our Search RSS feeds with our Feed Alert system and you can get an IM, Email or an SMS (in the US) when the search changes. After sp... [Read More]

Tracked on Apr 10, 2006 5:48:38 PM

» Persistent Search from Another Node
Bill Burnham has a great post:Persistent Search: Searchs Next Big Battleground Certainly Ill visit this topic again later. ... [Read More]

Tracked on Apr 12, 2006 6:14:05 AM

» Lots of links from JD on EP
Lots of links: I've kept lots of webpages open this week, but don't have sufficient original content of my own to prompt a top-level entry... if you're looking for interesting reading this weekend, then try some of the articles in the extended entry he... [Read More]

Tracked on Apr 14, 2006 3:26:48 PM

» Persistent Search: Searchâ€™s Next Big Battleground from chew shop
Bill Burnham has a great post on persistent search:Persistent Search allows users to enter a search query just once and then receive constant, near real-time, automatic updates whenever new content that meets their search criteria is published on the web. [Read More]

Tracked on May 7, 2006 8:43:13 PM