Searching from Among Searchers

WebCompass is a metasearch tool, which means it can pass requests to other search resources and then process the results. Thus WebCompass adds value to search engines, but it is not dependent on any particular one.

The first part of metasearching--interacting with search resources such as Yahoo--is straightforward. WebCompass does essentially what a human user of Yahoo would do: Enters a search term and submit s the search to Yahoo. It's only in processing the search results that WebCompass gets complicated.

The WebCompass Agent downloads (as a background process, without human intervention) the documents whose URLs were returned by the search. It then uses a variety of AI techniques to analyze the documents, including natural-language parsing for extracting noun phrases. The Agent next uses a combination of statistical and heuristic rules to rank the noun phrases in the document. For example, it might note the frequency of a phrase (a statistical method) or promote a phrase because it falls in the first sentence of a paragraph (a heuristic method).

The Agent uses the noun phrases to derive a summary, or abstract, of each document. This summary (not the whole document) is stored in the local database for future reference. You can remove the abstract from the database when it is no longer useful.

WebCompass employs the sentence rankings to group similar documents, another AI technique called conceptual clustering. Once the Agent has decided which documents are similar, it analyzes the similarities to produce a title that describes that group of documents. This title appears as a hyperlink that you can use to jump be tween related groups of documents.

An artificial intelligence that passes the Turing test may be some time in the future. But the efforts of AI researchers are clearly bearing fruit in agent-based products like WebCompass.

---

AI Comes to Searching

illustration_link (4 Kbytes)

WebCompass uses AI techniques to massage Internet search results and group related hits.