WebProNews spoke with Jack Moffitt, the CTO for Collecta, about the challenges these real-time engines face. The main idea behind real-time search engines is getting data as soon as it is published. Moffitt says this is challenge for three reasons.

First of all, it’s a challenge because the real-time engines have to convince publishers to send them their data. Secondly, they have to make sure that they get the data quickly. In other words, real-time search engines need publishers to push content to them instead of pulling it themselves.

The third reason it is a challenge is due to user privacy issues. Facebook data, for instance, is private, which means Collecta and other real-time engines cannot use it. In addition, there is the issue of users changing their minds about their data. Sometimes users go back and delete their data or decide to make it private. This creates a challenge for real-time search engines since technologies that distribute these types of action do not exist.

Despite these challenges, real-time search engines are growing in value and will only improve with time with the introduction of new tools and developments.

As Collecta CEO Gerry Campbell tells WebProNews, their approach to real-time search is different from the other real-time engines. Collecta provides real-time streaming search results of not only social networks, but also from news publications and sites across the entire Web.

As content is published online, Collecta send it directly through to the users in approximately a split second. The engine is based on XMPP technology, which is the same protocol that Google Wave is based on. Collecta does run filters to detect for any spam and to apply relevance.

To better explain how Collecta is differentiated from other search engines, Campbell shares an analogy using bodies of water. Other search engines gather search results and put them into a pile to sort, which essentially creates an ocean of information. Collecta, on the other hand, is already watching everything that is happening online and as content is published, it builds a stream and flows the information to users.

Unlike the traditional search engines, these real-time search engines return the most current information from the Web in the form of blog content and comments, Twitter messages, images and video, social services, news sources, and more.

Why does our society crave this continuous stream of information? Collecta CEO Gerry Campbell told WebProNews that this need to publish and consume information actually goes back to ancient times.

“If you go back all the way to the progression of cave paintings through monks copying Bibles to Gutenberg’s press, we’ve seen an acceleration in the ability to publish information. In addition to that, we’ve seen an acceleration and a growth in people’s ability to consume information… now, we’re bringing together the ability for anybody to publish and for anybody to consume information.”

Despite all the perks of real-time search, the real-time search engines have their share of challenges to overcome. There are complaints that the results are not relevant and that they are filled with spam, among other things. While challenges are present, real-time search is still in the early stages and Google, Facebook, and users still express strong interest in the area.

Where do you think real-time search will lead us going forward? How will it impact the search industry?