The CIA is embracing big data to dramatically speed up the time it takes to analyze and act on the sea of data its sensors and agents collect, a top agency official said.
There have been profound changes caused by technology in recent years, much of it driven by social, mobile and cloud applications, said Ira “Gus” Hunt, the CIA’s chief technology officer, speaking at the recent AFCEA Emerging Technologies Symposium in Washington DC. One example of this is the Arab Spring uprising, which would not have been possible without those technologies, he said, adding that cheap, ubiquitous and networked devices change how the public and private sectors do business.
“If you succeed, you can scale fast and win. If you don’t succeed, you can fail cheap,” he said.
Related stories:
As more devices are linked together, this “Internet of things” will represent a network of sensors in everything from smart phones to refrigerators that communicate and share information with each other. Big data techniques will be used to manage and mine all that data.
“Big data is a big deal,” Hunt said, noting that researchers are moving swiftly from petascale to exoscale computing. This work is creating new computing architectures, which will require designers to get data as close to computer processors as possible to reduce latency, he said.
This is the “high noon in the information age,” Hunt said. Computing and networking technology has only just now become mature enough to merge and manage data from multiple sources on a large scale.
Such data collection and aggregation techniques are very useful for intelligence gathering. Agencies such as the CIA can use big data methods to detect terrorist activities in a sea of information, Hunt said. This data gathering also applies to the private sector, he added, noting that Facebook is scheduled to have 1 billion users by this August.
All of these new developments affect how the agency does its business, Hunt said. The CIA’s primary goals are: collecting information, processing this data, and conducting covert action. To support its activities, he said the agency is focusing on several technology areas: big data, operational excellence (how to get the best value for its IT dollar), supporting the intelligence community, and talent management.
The CIA is concentrating on five key technology enablers, Hunt said, including advanced mission analytics, end-user services and widgets; security as a service; big data computing engines or "data harbors;" and cloud computing.
One of the agency's major goals is to take latency out of the agency’s information processing systems. Two years ago, it took the CIA 63 days to process collected data, Hunt said, adding that it can now do this in 27 minutes.
Based on these new technologies, the CIA has launched several pilot programs. The first is I2, which was the original intelligence community cloud proof-of-concept program, he said. Since then, the agency has also launched the Mass Analytics Cloud, a classified big data processing effort.
But challenges remain, such as faster processing to reduce latency — especially in petascale computing. “Latency breeds contempt,” Hunt said. Besides acquiring new technologies, the agency is also trying to emphasize training and continuous learning at the organizational level to keep up with the rapid evolution of systems. All of these efforts are to make the CIA more agile — what he refers to as an “ahead of” organization.
The agency also must revise how it acquires and licenses software by looking at alternate processes, Hunt said. Traditional methods slow down the process and limit the agency’s ability to bring new capabilities in. “The old models inhibit the ability to bring new technology into the environment,” he said.
