Chiliad takes virtual consolidation route to big data analytics
Connecting state and local government leaders
Discovery/Alert 7.0 installs nodes at each data location, allowing analysts to query structured and unstructured data simultaneously.
Organizations ramping up a big data program often have to gather, copy, normalize and cleanse the data as well as build a data warehouse to store it, adding cost and complexity to projects.
Chiliad says it can eliminate some of those barriers to adoption of big data analytics with Discovery/Alert 7.0, an iterative information retrieval system that lets analysts search any data warehouse or data set across clouds, agencies, departments and organizations.
Discovery/Alert eliminates the need for data consolidation efforts by installing nodes at each original data location. These nodes collaborate to create a virtual consolidated data center, allowing analysts to query structured and unstructured data simultaneously, as if all data resources were in a single repository, according to Chiliad officials.
The software is in use at the Defense and Homeland Security departments and the FBI. For instance, analysts at the FBI and DHS use the software to monitor all of the flights, ships, trucks, automobiles and passengers entering the United States. Tasks that once took 32,000 hours to complete now take 30 minutes, Chiliad executives said.
Discovery/Alert 7.0 was announced last year, but since then Chiliad officials have overhauled the platform, rewriting application programming interfaces and Web services to provide analysts with a new, simpler user interface, said Michael Young, product manager for Discovery/Alert 7.0. The thin-client, standards-based user interface allows users to access data via browsers. No plug-ins or other client software has to be installed on desktops. The user interface also can run on Android-based mobile devices, Apple iPhones and tablet computers.
The goal is to let government or business analysts access data as simply and intuitively as possible, Young said. Analysts can query data using conversational language with a host of built-in visualization tools that help them find relevant information and uncover new relationships between concepts, with little or no training.
This type of flexibility is important as agencies deal with increasingly massive amounts of data and technology that continues to change rapidly, said Phil Mudd, former deputy director of the FBI and the Central Intelligence Agency and now head of Mudd Consulting. If an agency has to extract data over 70 different data sets that might be different next year, it can't realistically alter its configurations every year and constantly re-train people, Mudd said.
“You have to have flexibility,” because you don’t know what type of data and systems will be in place next year, Mudd said. Agencies are looking to acquire systems that are affordable, especially in an era of budget tightening. “You have to figure out how to manipulate data and, at the same time, [senior agency managers] are saying spend less money.” Budget considerations will force agency managers to make tough choices about how they think about big data projects, he noted.
Discovery/Alert 7.0 gives the intelligence community added flexibility by supporting the Ozone Widget Framework, developed by the Defense Department and National Security Agency and now available to the open-source software community, said Rick Brown, Chiliad’s senior director of product management. OWF lets organizations develop and deploy self-contained applets that have a common look and feel. An agency can develop its own widgets, for example, and put them in a store or clearinghouse for other agencies to use. If an intelligence or DOD agency does not want to use the Chiliad APIs, analysts can replace them with OWF, giving them more flexibility in how they use the company’s user interface, Chiliad’s Young said.
Discovery/Alert provides global ranking of results to help analysts find relevant information in massive collections of data, Young noted. Most information retrieval systems return query results of dozens of pages of documents, leaving the user to determine what documents are useful, Young noted. “We actually give you a unified, holistic global ranking of all of your results,” he said. As organizations scale out to billions of records and petabytes of data, relevancy becomes more important, he noted.