Intelligent files
Connecting state and local government leaders
In-Q-Tel adopts indexing software to improve data management.
In many ways, gathering data is the easiest part of the intelligence business. The hard part is making sense of it. So In-Q-Tel has invested in data retrieval software that promises to make flat-file storage systems as efficient as a database, at a lower cost.
In-Q-Tel is a strategic-investment firm created by the CIA in 1999 to invest in companies with commercial products that would be useful in the work of the intelligence community's 17 acknowledged agencies.
The volume of data that the spy agencies are storing is growing rapidly, and they need cost-effective ways to retrieve and manage it, said Troy Pearsall, In-Q-Tel's executive vice president of technology transfer.
The firm has announced an investment and licensing deal with CopperEye for its Greenwich product and software development kit. The data retrieval software sits between the user's business intelligence application and the file system being searched, enabling the use of SQL queries to retrieve data from the files.
'Given the volume of information gathered by organizations within the public and private sectors,' Pearsall said, 'it made perfect sense to invest in an innovative data access technology that will potentially meet the critical needs of the intelligence community.'
The alternatives for data storage in most organizations today are relational databases or flat-file storage.
"Databases scale very nicely but are expensive,' said CopperEye chief executive officer Kate Mitchell. 'And it's not just the software cost, it's the hardware under it.' Database administrators also are scarce and expensive, she said.
File storage is cheaper and scales better, but, without indexing, it is difficult to track and retrieve data.
'There has been this trade-off,' in which organizations keep current data in databases and voluminous archives in less-accessible flat files, Mitchell said.
market has been the telecommunications industry, which generates and stores huge volumes of call transaction data. With proper support, Mitchell said, there is room for great expansion for CopperEye.
'We are just scratching the surface with telecom.' European data retention requirements for companies create another possible use for the software. The next logical markets are financial services and fraud detection, work that can also involve the law enforcement and intelligence communities.
Why invest in a company that already is established and has a product on the market?
'There are always features that can be added to make it valuable to our customers,' Pearsall said.
'Given the volume of information ... it made perfect sense to invest in an innovative data access technology.'
'You would think indexing would not be a big problem' in information technology, Mitchell said, but the last big innovation was 30 years ago.
CopperEye has come up with a new set of indexing algorithms that handle SQL requests for flat files. This reduces the input/output requirements of the retrieval operation by 99 percent with a corresponding improvement in performance, she said.
The Greenwich product has been around since 2002, and its initial Most of the investment will go toward improving production performance and speeding the product's development life cycle.
For CopperEye, In-Q-Tel's investment gives the company credibility and helps open the door to the intelligence and general government markets. 'It's like a big, fat Good Housekeeping Seal of Approval,' Mitchell said.
Since its inception, In-Q-Tel has operated as an independent, private, not-for-profit company whose aim is to get cutting-edge technologies into the field faster than traditional government procurement practices have allowed.
NEXT STORY: Radio coverage