Backyard frontier

This year marks the 200th anniversary of the Lewis and Clark expedition, sent by Pres. Thomas Jefferson to explore and report back on the newly acquired territory of the Louisiana Purchase.Today, government's challenge is to explore a vast new realm'the data within its own servers. Although agencies have done an admirable job posting hundreds of millions of pages on intranets and public Web sites, finding information in them can still be a harrowing adventure.'Governments, like large corporations, have silos of information attached to different applications,' said Susan Feldman, research vice president for content management and retrieval software at International Data Corp. of Framingham, Mass. 'They can't get the whole picture of what they know, and neither can their constituents.'Another flaw is obtaining only relevant search results. Let's say an inexperienced user is looking for the federal portal FirstGov.gov. The term 'United States' entered into MSN Search gets 8,164 results: the Louisiana State University Library at No. 12, U.S. Grand Prix at Indianapolis at No. 21 and the All About Irish Web site at No. 112. If the searcher continues to 300, the FirstGov portal still wouldn't appear. Google.com and Yahoo.com, on the other hand, give it as the first result to the same query.Different search engines, then, have quite different capabilities and produce wildly varying results for the same query. Even applications such as Microsoft Outlook and Word and content management systems incorporate a search feature.In addition to the inconsistency of results from different products, it can be cumbersome to jump from one screen to another trying to locate information. Enterprise search engines simplify the process by scouring all pockets of potential information to present the most appropriate hits, regardless of format or location.The search market itself is fairly new and undergoing rapid changes. Many small vendors have been bought out, but others keep springing up with specialized technology. According to IDC's Feldman, the search and retrieval market grew to $423.3 million last year and is forecast to reach $1.3 billion by 2006.Consequently, agencies can choose from hundreds of options. They extend from hosted applications that do keyword searches of Web-site content, to tools that aggregate information from numerous sources and give custom views based on criteria such as job title or search context.There are two key barriers: the sheer volume of document types and the dispersed nature of information. But for the most part, modern enterprise-class search tools rise to the challenge. Most can search more than 200 file types and extract information from both formatted and unformatted documents.What about the other barrier'being able to simultaneously extract information from a variety of repositories? Much of an entity's knowledge exists in databases rather than documents. Other valuable content resides on outside agency or commercial sites.A district attorney's office, for example, needs access to the FBI's National Criminal Information Center and an online legal research provider such as Westlaw.com or Lexis/Nexis. National Institutes of Health researchers, say, might need to search drug interaction databases and online medical journals in addition to NIH's extensive collection of reports.Certain search engines, in addition to spidering their own content'locating documents by following hypertext links, have the ability to initiate database queries or search content on other sites.Another important stage in the evolution of enterprise search engine technology is toward accuracy. It takes real search intelligence to interpret exactly what the user is looking for and understand the content of potentially relevant documents. For example, the statements 'I am going to buy into the soybean futures market' and 'In the future I am going to the market to buy soybeans' mean quite different things. Yet they get the same keyword search results.Search engine makers have refined several linguistic areas to increase intelligence, such as:Stemming. This involves determining the root of a word in a document or query, and including all other versions of it. For example, if someone enters 'education,' stemming will bring up documents with the words 'educate' and 'educator.'Synonyms. Many concepts can be expressed with more than one word. This feature catches them. If, for example, someone searches for research papers on cancer at a health department site, documents discussing carcinomas or tumors would also appear. A state motor vehicle department site should recognize that a person querying about cars would also be looking for items covering automobiles or vehicles.Misspellings. Some search engines can make a good guess at the intended meaning of a misspelled word and sometimes return correct results while suggesting the correct spelling.These advances are designed to increase the number of document hits. But just as important is restricting the number of results displayed, so the user gets no more than he needs.Among the techniques are:Disambiguation. Words and phrases can have more than one meaning, and search engines must distinguish between them based on the context. A document with the words 'bomb' and 'theater' might trigger a security alert unless the search engine can understand that the subject is a play that performed poorly.Classification and taxonomy. Some search engines can automatically set up taxonomies for an agency's documents by analyzing their content. Others depend on the administrator to create the taxonomy or use a combination of automatic and manual methods.Individual documents are classified according to taxonomy. This helps searching in several ways: One method is to let a person restrict the search to a certain type of data or category of information. On the IRS site, for example, visitors can limit the search to documents pertaining to individual income taxes, rather than also pulling up information applicable to businesses and nonprofits. Search engines also can limit a search based on the section of the Web site or intranet a visitor is in.Personalization. Some search engines customize their results for the person doing the searching. This requires recognizing the person's security level so the search doesn't return results that person is not allowed to access. They also can customize returns according to department or profession, or by learning preferences set by the search history.Ranking. Most search engines use one or more algorithms to rank the search results so that the most relevant results appear in the top five. Google, for example, weighs more than 100 factors in determining a document's relevancy, including the layout of the Web page, the colors used and where specific words appear.Each search engine has specific strengths and weaknesses. There is no surefire way to decide what your own organization needs, nor any meaningful benchmarks that consistently differentiate one search engine from another. You must have actual users test the engines on actual data stores to see which is best for a particular organization. Even so, different users or applications within your organization might be better served by different search engines.In addition to basic search and retrieval functions, there are many other tools that integrate with a search engine to present more relevant information to employees and constituents.'The more tools you combine, the better off you are in terms of technologies and algorithms,' Feldman said. 'This means providing for browsing, searches of all kinds'including Q&A and document retrieval'[and] analysis and alerting.'Such tools typically come from smaller vendors that have developed a particular piece of technology, and they frequently wind up being bought out by larger search firms for integration with another product line. Several of them appear in the accompanying chart.As a final caution, although the features of these search engines and tools are potentially useful, they do take work to set up and maintain, and organizations tend to underestimate the task.'You should avoid becoming enamored of features and capabilities that look really interesting but not immediately practical,' said Laura Ramos, director of Giga Information Group in Cambridge, Mass. 'If you can't figure out how to make it start paying back in six months, save it for the next implementation.'Drew Robb is a freelance IT writer in Glendale, Calif.