Stream of consciousness

 

Connecting state and local government leaders

Complex-event processing engines can make sense of data flowing in from many sources.

An emerging class of software that can monitor data bombarding an organization's information technology infrastructure from multiple sources is making inroads into the government sector.Known as complex-event processing, or CEP, the software can detect patterns in intricate situations from multiple sources, giving analysts a deeper understanding of their business processes and events.First used to analyze trading transactions on Wall Street, CEP engines are being applied to other areas, such as intelligence and surveillance, battlefield command and control, and network monitoring, industry experts say.That's why In-Q-Tel, the CIA's venture-capital arm and technology incubator, is interested in the technology. The company made a strategic investment in StreamBase Systems, a developer of CEP technology, in February.'We're seeing the emergence of devices that are flooding businesses with events,' said Troy Pearsall, executive vice president of technology transfer at In-Q-Tel.'These devices range from your standard network monitoring devices to radio frequency identification tags to sensor networks that are really upping the ante in terms of the volume of data being flowed into organizations,' he said.Traditional databases and their emphasis on historical data don't give business users and analysts the level of analysis they need to make decisions on the fly, Pearsall said. 'Users want to take action more quickly, and that leads to the need for complex event processing. You want to make decisions as events are flowing into your organizations, not after the fact.'In-Q-Tel chose StreamBase, a four-year-old company, because of its strong development environment, he said. 'They have a real strong graphical and text-based development environment that allows mere mortals to stand up applications very quickly.'StreamBase also can connect to many sources of information, and the company's CEP engine can integrate historical data and real-time data to answer questions, Pearsall said.CEP is an emerging market, and ' as in the business intelligence and standard database markets ' there will be a lot of complementary technologies coming out in the next few years, he said.'Certainly, we're always looking for opportunities,' Pearsall said about the possibility of investing in more CEP vendors.'There are three attributes that really define complex-event processing: high data volumes, instant response and complex analytics,' said Bill Hobbib, vice president of marketing at StreamBase. 'If you have all three together, it's a complex-event processing problem.'The StreamBase software platform handles high-volume data. The company is working with users in the areas of intelligence and surveillance, intrusion detection, network monitoring and battlefield command and control, he said.StreamBase runs on Microsoft Windows, Sun Microsystems Solaris and Linux platforms. 'So, it runs fast on commodity platforms,' Hobbib said.The company released in June a new version of its software platform, StreamBase 5.0, which speeds the development and deployment of real-time applications.The new release offers built-in support for IBM's DB2 data server, WebSphere Front Office and xSeries hardware.Other new features include out-of-the box application development frameworks, end-to-end application development, expanded support for advanced data types, flexible pattern matching, enterprise security and integration with a broad range of market data, and messaging infrastructure systems.Flexible pattern matching is significant, and StreamBase plans to expand more into this area, Hobbib said.For example, with pattern matching, a network administrator might say, 'If a network event occurred, and its characteristics look like X, we're going to consider this person an intruder and kick him out of our network.' But with flexible pattern matching, network analysts can say, 'If A occurred followed by B and then C, this is a particular sequence of events or patterns, and this is really significant.' Or they can say, 'A occurred, and we think it is suspicious, but because it wasn't followed by B and C, then maybe it's not so suspicious that we need to give it the highest level of alert. It can be given a middle level,' Hobbib said.This concept of looking for patterns in real-time events is applicable to the battlefield, intelligence and network monitoring, he said.Another direction for the company is support for binary large objects, or BLOBS, which would aid battlefield command, control, intelligence and surveillance. This involves working with partners to extract features from video images and audio files, Hobbib said.Other areas of expansion include the ability to process advanced data types and enterprise security tightly integrated with Lightweight Directory Access Protocol and authentication systems.Two other companies, ANTs Software and Coral8, have teamed on a next-generation CEP environment. ANTs' high-performance SQL database management system will work in conjunction with Coral's CEP engine, company officials said. The combination of ANTs Data Server and the Coral8 Engine will boost application performance in the CEP arena, said John Morell, director of product marketing at Coral8.'It's important to recognize that you just don't process and generate events and throw them out into the ether,' he said. 'When these high-level events happen, you need to be able to store them somewhere, and sometimes in multiple places.' Users need to be able to access data to get at all the information relevant to an event.'No matter which way you store it, you need a high-speed storage system like the ANTs database that can keep up with all of the events that are flagged through the system,' Morell said. 'You're talking hundreds of thousands ' sometimes half a million events per second.''That's what we are doing for Coral8 ' the ability to do a high number of inserts per second into our database,' said Cesar Rojas, ANTs' director of marketing.The Coral8 application rapidly presents data, but the ANTs database can easily keep up because of its ability to do as many as one million inserts per second, he said. Also, ANTs lets users quickly process data, he added.ANTs is no stranger to the defense market. The high-performance database has been used in conjunction with the Navy's DDG 1000 Zumwalt-class destroyer project, said Patrick Moore, an ANTs vice president.In that capacity, the product has integrated with real-time extensions to Red Hat Linux developed by IBM and IBM's service-oriented product, Real Time WebSphere, to support mission-critical ship operations, he said.Another CEP engine used in government is Progress Software's Apama. One could call Apama the godfather of complex-event processing.The Apama algorithmic engine was introduced in 1999 by two researchers from Cambridge University with some assistance from developers at Stanford University and the California Institute of Technology, said Mark Palmer, vice president and general manager at Progress' Apama division.Apama was one of three vendors that sold complex processing engines at the time, he said. The company and product were acquired by Progress Software in April 2005.Apama's origins are in the capital trading market, but government is a growing sector.'We've worked in government applications with radar feeds of intelligence information that come from surveillance planes,' he said. That information is transmitted back to a central, event-driven architecture environment that analyzes it and distributes it to appropriate analysts for defense purposes.The Progress Apama Event Processing Platform is a complete environment for creating applications that monitor event streams, detect event patterns and take action. The platform includes Business Activity Monitoring capabilities and a sophisticated CEP language, company officials said.Truviso, formerly Amalgamated Insight, a new player in the CEP arena, uses standard SQL language for querying, multiple heterogeneous data streams, said Mike Trigg, co-founder and executive vice president of marketing and business development at Truviso.'Compared to some of the earlier products in this space, we think taking a standard approach using SQL ' a language with 30-plus years of proven work behind it in the relational database world ' is an important part of what we are doing differently,' he said.Three main things differentiate the company's CEP product from others, he said. The core engine has an adaptive query processor, which lets the product support a wide set of SQL queries, such as user-defined functions, user-defined aggregates and subqueries. Some vendors that have taken an SQL approach have not included that capability, Trigg said.Adaptive query process also allows the product to run thousands of concurrent queries against incoming streams of data. This lets Truviso perform sophisticated analytics on the fly, he said.Another critical function is visualization capabilities. The product has a full Web-based user dashboard, which is critical for helping users understand what's going on with the data they are analyzing, Trigg said.The third piece is a full database embedded within the product, not just hooked on the side. The company took the PostgresSQL open-source database and added Truviso's extreme processing capabilities to the Postgres engine to give users streaming and more traditional relational database capabilities. In the same engine, they can have queries run over streams and tables, and they can do caching and archiving, Trigg said.'In the real-world use cases that we see, you're not just analyzing incoming streams, but you want to compare those streams to aggregates, historical information, trends and averages,' he said.

Current with Events

CEP can be used to track shipments, other event-driven systems

Where could complex-event processing work better than standard database analysis? During the May JavaOne conference in San Francisco, Thomas Bernhardt offered one answer: tracking shipments of valuable items.

Bernhardt is head of a project to develop complex-event processing software called Esper. He said the software could easily keep track of warehouses of items bearing radio frequency identification tags.

Say, you are in the military overseeing warehouses full of ammunition. How do you keep track of all the items, especially as they move from building to building? How could you set up an alert to tell you if a box 'fell off the truck' or was 'lost in transit,' in the parlance of yore? RFID readers would detect a box of ammunition leaving or entering a building. But what back-end software could interpret and coordinate these messages? In a busy environment, their sheer numbers would quickly overwhelm a traditional database trying to log them.

Nor would standard database analytic software necessarily be able to coordinate all these messages into something meaningful. If a box of ammunition were moved from one building to another, how could the manager tell it had made the journey successfully, short of manually following the progress of each RFID number? Online analytical processing may be good for parsing multidimensional datasets, but it doesn't do well with making sense of data elements in terms of when each element was logged in.

'It is hard to deal with temporal streams of data,'Bernhardt said.

Bernhardt characterized systems such as an RFID-equipped warehouse as having 'or needing ' an event-driven architecture. These systems are built around detection of and reaction to events. Timing and aggregation are essential ' and difficult. 'It [is] very hard to construct an event-driven application manually,' he said.

In the industry, the basic processing for this sort of environment is called event stream processing (ESP). The idea is to pull the right events out of a constant stream of data points and aggregate them in a meaningful way. For instance, someone might want to calculate the average price of a stock over the past 30 ticks of trading, which involves constantly recalculating the average of numbers that are being constantly updated.

But even ESP isn't strong enough to handle more-complex jobs, such as watching RFID tags as they move about. Here is where CEP comes in. CEP is the advanced analytical detection of patterns in a series of events. A CEP function specifically looks for complex behaviors, such as two successive gains in stock price of one company followed immediately by a drop in the price of another company's stock.

Bernhardt's open-source Esper software can do both ESP and CEP. Under benchmarks, Esper has been able to process and make sense of about 110,000 events per second. The Esper container works in either a Java Enterprise Edition application server or the Microsoft .Net framework.

What Esper does is run all the messages it receives through a set of queries defined by a user. Instead of committing these messages to a database and then analyzing them, ESP analyzes the data as it comes across the wire. As soon as one of the query engines detects a set of messages that meets some predefined set of conditions, Esper alerts the user or another program.

Users describe the events they are looking for in Esper's Event Query Language, an SQL-like language with a number of unique characteristics for identifying data on the fly. EQL can do many things SQL can't. For instance, EQL can define a range of parameters as a unique set. You can define a warehouse based on the geographic coordinates within that warehouse, so you know any items identified at one of those coordinates are in the warehouse. Although SQL lets you specify a certain point in time, EQL lets you specify a period, or window, of time. So you can easily compile a list of items that have entered a warehouse within the last 30 seconds, 30 minutes or some other window. And, like SQL, it can bring in additional sets of data for comparison through the join function. You could join data from two database tables to watch how materials are flowing from one location to another.

Using this function, Bernhardt said, you could write a query that could watch when a large number of items moves from one warehouse to the next and send you an alert when one or more of the items that left one location fail to reach their destination. And so CEP solves a problem with an ease other tools can hardly match.

To see Bernhardt's JavaOne presentation, go to GCN GCN.com/806. To download Esper, go to esper.codehaus.org.

' Joab Jackson











































Teaming for CEP














Old school, new school














Building on experience











NEXT STORY: Planning for Longhorn's harvest

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.