Technology overview:

In the 1980s, scientist Tsien Hsue-shen (Qian Xuesen), a co-founder of the Jet Propulsion Laboratory and father of the Chinese space program, advocated scientific investigation on human thinking – noetic sciences. Hongfeng Yin, then a graduate student at Chinese Academia of Science under professor Dai Ruwei, proposed a human think model with simulation methodologies based on Tsien’s proposal. Tsien regarded Hongfeng and Dai’s paper “On thinking and simulated intelligence” as a great epochal groundbreaking paper. At that time, scientists from Japan and other countries proposed intelligent machine plans. To build a truly intelligent machine, Tsien and Hongfeng believed that the research should focus on human intuitional thinking instead of logical computing. Hongfeng focused his investigation on associative memory models with artificial neural networks to simulate the intuitional thinking. In 1990s, Tsien and Dai proposed giant complex intelligent system theories and pointed out these systems should be constructed with human and machines together.

Since the beginning of this century, Hongfeng has explored the possibility of applying this research and the resulting theories to build a human-like world’s knowledge system for web search. While human-generated knowledge systems, such as Wikipedia, web page directories and so forth, are fairly easy to create for web search applications, these systems are limited in size to only millions of items even with massive human participation. On the other hand, machine-based knowledge systems can process trillions of items. Until now, the main difference has been that the results for machine-based knowledge systems are not as accurate as human generated results. To bridge the gap, our approach is to combine human intelligence with machine algorithms to build a world’s knowledge base. Actually, we believe that search services based on the knowledge base is the intelligent machines which were proposed in 1980s.

The invention of computer has brought about an information technology and information revolution. Information can now be generated at an explosive speed. To effectively use this information, it is necessary to have a means of converting it into knowledge. To fully harness the incredible advances in information computing, there must be a way to make this information easily accessible for people to utilize. In the last few years, new technological developments have finally made this conversion of information to knowledge a realistic possibility. Cloud computing, map/reduce/Hadoop infrastructure and data mining algorithms have created a knowledge revolution and brought us to a new era in knowledge technology.

What is the main difference between information and knowledge?

Information is usually unstructured, static and redundant and it is data centric. Knowledge is structured, connected, categorized, ranked with meaning and it is human centric.

At the beginning of the Internet, there were few web sites and pages. Pioneers like Jerry Yang and David Filo just listed the web sites. When more information was on line, Yahoo! developed a web directory to search the information. When the Internet grew to hundreds of millions web pages in the late 1990s, early search engines took the major role for finding information by listing all relevant web pages for a query. This has not essentially changed in the past decade. Currently, one query may generate a list of millions of results. From a data structure point of view, the linear list is not a good structure for finding information from a large data set. The optimal structure for human use is a tree or graph structure for efficiently finding targeted information. Until now, this optimal structure has not been possible to utilize because, on the one hand, it is not scalable to build web search directories for all queries with only human power while, on the other hand, pure algorithms cannot generate human desired results. Until now…

Our approach and what makes our efforts so groundbreaking is our integration of algorithms with human knowledge to build a web directory for each query and each user. We have developed a group of algorithms of association, clustering and categorization for automatically generating knowledge for search concepts, web sites, web pages and users. We also integrate human labeled information with our algorithms.

We expand the search in several dimensions: higher – summary of top sites and categories for queries; wider – related search terms; longer – results of expansion terms for the queries; deeper – inside links and keywords of search result pages;

Using the Amazon.com clouding computing service, we have built a knowledge base for about 10 million concepts over 1 billion web pages. Our technologies are scalable to build a knowledge base of 100 million concepts over 10 billion web pages in the next phase.

The knowledge based search can fundamentally change the search result presentation, finally expanding beyond the confines of the linear structure. Unlike 10 links in a traditional search page, we can present hundreds of links with categories in a similar space of a page. This can largely reduce users’ time to find desired information from the search results. It also provides rich channels for users to research and explore a topic.

Currently, to complement our knowledge based search, we provide traditional search results for the queries that are not in our knowledge base. In a few years, we will be able to provide non-linear knowledge results for the majority of potential searches.

We are just at the beginning of the knowledge revolution. In a few years, we believe that the majority of searches will return knowledge results. However, the impact of this technological development will expand beyond simply human web search applications. This revolution in knowledge technology will make possible the generation of truly intelligent machines which are going to greatly impact our industries and our life.