Going where no search engine has gone before

 

Connecting state and local government leaders

Connotate Technologies uses information agents to extract data from Deep Web.

Google, one of the most popular search engines, at best can index and search about 4 billion to 5 billion Web pages, representing only 1 percent of the World Wide Web.But officials from Connotate Technologies, a company based in New Brunswick, N.J., said they have developed technology that can mine and extract data from the Deep Web, which contains an estimated 500 billion Web pages, and deliver it in any format and through any delivery mechanism. The Deep Web refers to content in databases that rarely shows up in Web searches.Through the use of intelligence-based software modules called information agents, corporate and government organizations can quickly and easily target specific unstructured data from intranets and password-protected Web sites on a continual basis."What the agents do is they automate time-consuming Web interaction," said Bruce Molloy, the company's chief executive officer. "So an agent can act on your behalf, type in information, search terms, can click on links, can know your password — but we would keep it protected — can automatically go to sites and bring back information, format and cut and paste results."Such information agents can monitor pages as often as once per second and deliver real-time results, he said. In addition to the financial and energy sectors, some federal agencies, such as the Defense Department, use the technology. Company officials would not comment on how DOD uses the technology. But Connotate officials said they are talking with intelligence organizations and the Homeland Security Department about the technology.Connotate was formed in 1999 by three Rutgers University professors, whose Web-mining technology research was funded by the Defense Advanced Research Projects Agency and the university.Ken Hambright, information technology manager at Quadel Consulting, a firm based in Washington, D.C., said the company began using information agents about three years ago to help monitor several multifamily housing programs as required by its Department of Housing and Urban Development contract. HUD also requires Quadel to enter data into government systems, which company employees initially had been doing manually."What the information agents allow us to do is kind of automate that procedure so that when we enter things in our system they automatically get entered into the HUD system," Hambright said. "It saves a lot of keystrokes. It saves errors because the systems are always in sync."Molloy said information agents can go to complex Web sites and databases, extract information — such as dates, names or contract identification numbers — and automatically deliver that data in any format."What we're able to do is actually connect on a data level and pull information back, or we can take information and actually place it onto Web sites so the agents can provide a kind of data-entry function," he said.Company officials said setting up an agent is easy and takes only five minutes in some cases. The company sells software licenses, and it also hosts an Information Agent Library in which users can manage their subscriptions to various sites, including news, corporate, government and others.For example, a user could open a Web browser to a news Web site and highlight a section that provides financial news. Another way to build an agent is through a keyword filter. The agent would essentially learn what information the user is targeting."It's a lot like showing something to a small child for the first time," said Chris Giarretta, Connotate's customer relationship manager. Essentially, he said, the more you show what a user wants, the better the agent will get at finding it.Users can personalize subscriptions by setting how often they want to receive data and through what medium, such as e-mail, instant messaging or Really Simple Syndication (RSS) feeds to any electronic device. The information can also be placed in spreadsheets or databases or published in a newsletter format. Plus, data can be delivered to an alert monitor — a personalized desktop ticker — which Molloy likens to an RSS feed. Subscribers can set up a distribution list to automatically send data from sites to several people at once.The agent can access intranets and other sites that need authentication. Essentially, it serves as the user's proxy to enter those sites, said Dan Haughton, Connotate's vice president of marketing."Whatever an individual can do in terms of accessing a site, Deep Web navigation, filling out forms, an agent can do that," he said. "If you have a subscription and password to the site, then the agent can have access. If you don't have a password, that site would be closed."John Blossom, president of Shore Communications, a research and analysis firm, said search engines typically either identify a document or provide an index and list relevance rankings. Web-mining technology not only crawls through sites but also takes content from a Web page and normalizes, analyzes and packages it in useful formats, such as Extensible Markup Language or other means, he said."Since the engine can be prepared to look for specific kinds of information, you're not at the mercy of a general crawling algorithm," Blossom said.He said people are becoming aware of this technology. Several companies, such as Inxight Software, Mark Logic and Zoom Information — formerly Eliyon Technologies — provide text-mining capabilities using different approaches, he added.Blossom said government agencies and other organizations want to get as much value from raw data as they do from structured or normalized information. He said in knowledge management, people are generally moving toward this mixture of structured and unstructured content being brought under a common processing umbrella to extract meaningful information and intelligence.Another benefit of Connotate's technology is that users can effectively apply it at an individual or institutional level. Other similar technologies work at one level or the other, but not both, Blossom said.Pricing starts at a little more than $100,000, Molloy said.













Learning the ABCs






















Web-mining tech emerging











NEXT STORY: BlackBerry 7100t does it all

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.