Fast times in database research

At least a few government agencies are taking a close look at a new form of database technology, called in-memory databases, that promise faster transaction speeds than standard relational database management systems.Raytheon Co. of Lexington, Mass., is incorporating two inline databases for the some of the shipboard electronic and combat systems of the Navy DDG 1000 Zumwalt Class Destroyers, which the integrator is helping build for Northrop Grumman Corp.And last month, In-Q-Tel, the private venture capital firm created by the Central Intelligence Agency, made a strategic investment in StreamBase Systems Inc., a Lexington, Mass.-based provider of in-memory database software and associated analysis tools.In-memory databases are databases optimized for working in the working memory of machines. Usually, databases are stored in main memory, typically on hard drives. When new material is generated, it is written to disk first, and when a query is made of the database from a program, that material has to be read off the disk.In contrast, in-memory databases reside entirely in the working memory, or RAM, of a server or cluster of servers (though they can be archived on disk). Material is only written later to disk, if at all.'The working dataset that the application will be using is resident, or persistent in memory,' said Patrick Moor, head of government contracting and manufacturing for Ants Software Inc. of Burlingame, Calif., one of the companies chosen for the Raytheon work. The other company was TimesTen, another inline database company now owned by Oracle Corp. of Redwood Shores, Calif.RAM works faster than hard drives, though it also is far more expensive on a per-byte basis. It also is volatile, meaning the data is lost once the power is shut off. But because these in-memory databases reside in RAM, they are generally able to ingest hundreds of thousands of transactions per second. They also can be queried against more rapidly.'For many applications where you need to capture, react to and analyze that data instantaneously, a database is just too slow,' said Bill Hobbib, vice president of marketing of Streambase. With traditional RDMS 'you are storing the data before you query it. We can query the data at the moment it arrives.'In tests, Streambase has shown that its software can ingest as many as 500,000 messages per second, whereas an RDMS can, at most, take in about 3,000 messages per second, Hobbib noted.For its destroyer work, Raytheon's Integrated Defense Systems was looking for a database that could ingest a lot of information from radar and sonar systems. Raytheon even generated an acronym to describe the environment, CRUD'create, update, replicate and delete, said Paul Rivot, a director of competitive technologies for IBM, which is supporting Raytheon's work.Traditionally, to tackle the problem, government contractors would write a custom program that would run an entire database in working memory in such a way that new material wouldn't be written to disk first. 'It was so expensive and you would have to custom develop it for each application,' he said. 'Using an off-the shelf memory and database product, it is obviously cheaper.'The Ants software combines an in-memory database and regular RDMS, so that material can be stored through regular Structured Query Language commands.For its own software, Streambase also kept close with SQL as well. It keeps the basic syntax, programming primitives, and declarative nature of SQL. But the Streambase software extends the language with additional capabilities, such as handling data that arrives out of sequential order, matching complex sequential patterns and detecting patterns over periods of time.A database programmer could learn Streambase extensions in about a day, Hobbib said. The company's extension are not overseen by a standards body but the company is looking into that possibility.In-memory databases are not the only way to speed transaction and analysis speeds. You could also just make the hard drive much faster.Texas Memory Systems Inc. of Houston offers hard storage systems that appear to an operating system as hard drives yet consist entirely of much-faster random access memory units. When outfitted with a standard RDMS, such solid-state drives could process hundreds of thousands of transactions per second, Woody Hutsell, executive vice president of Texas Memory Systems, said. Each unit can hold up to 128GB and can be tethered together for more capacity.For In-Q-Tel, in-memory databse technology shows much promise for the intelligence community, so investing in the company made sense, said In-Q-Tel's executive vice president for technology, Troy Pearsall. 'We're looking to accelerate [Streambase's complex-event processing] capability in the marketplace,' he said.The CIA and its Directorate of Science and Technology founded In-Q-Tel in February 1999 to locate and support new commercial technologies that could aid the intelligence community's work. The idea behind In-Q-Tel was to build a bridge between the government agency and commercial technology innovators.The two sides agreed not to release any specific financial details, but Pearsall said the firm traditionally invests between $500,000 and $3 million in a technology company.With the money, Streambase plans to build some new capabilities in its software. Hobbib declined to discuss those features, citing national-security concerns.Hobbib said what interested In-Q-Tel about the technology was its ability to 'consolidate data across lots of different sources, integrate real-time and historical data, and basically monitor for complex conditions to find patterns of events that indicate some need for immediate actions.'Such areas of use could be in network monitoring, real-time message or text analysis, combat theater and other forms of surveillance.In-Q-Tel's equity stake in StreamBase should also help the company court other federal customers, Hobbib said. As part of the agreement, In-Q-Tel will cite StreamBase as the preferred vendor in its dealings with the intelligence community and systems integrators that work in the intelligence field.