How video analytics helps reconstruct Boston Marathon bombings

Video from surveillance cameras and people’s cell phones are increasingly valuable resources in helping investigators collect and analyze data from crime scenes, such as the finish line of the Boston Marathon. Investigators are combing through hours of digital feeds and thousands of photographs to identify suspects responsible for detonating two bombs there on April 15.

On Thursday morning, authorities were reportedly set to release photos of two suspects in the bombings, although the analysis of all that footage will undoubtedly continue, as the police and FBI seeks to piece together the chain of events. (UPDATE: The FBI released photos of the suspects Thursday night, asking for help in identifying them. Later, one of the suspects, identified as brothers, was killed in a confrontation with police that also left a security officer dead, and the second suspect was still being sought.)

But how do investigators weed through terabytes of video in different formats, whether 30-second snippets from cell phones or hours of footage from a surveillance camera at a nearby store? Going through all that footage is still largely a labor-intensive task, but video analytics and digital forensics tools can help investigators compress video, pinpoint areas of interest, look for anomalies and find relevant details, according to government and industry experts.

Many video surveillance systems come packaged with analytics that can detect anomalies, such as a package left behind or a person entering a restricted area, said Maj. David Mulholland, commander of technical services with the United States Park Police.   Because humans can’t watch multiple security feeds without being overwhelmed or losing attention, analytics software signals -- visually or through audio --  if someone enters a stairway where no person should be, he said.

There are also video analytics tools that compress long hours of video. Video Synopsis, a tool for CCTV surveillance systems from Briefcam, an Israel-based company with offices in Connecticut, lets investigators pinpoint an area of interest and show only the moments where something was different in that picture. Admittedly, that would be more challenging in a race, where something is changing every second than, say, at an office overnight where someone might take a folder off a desk, make copies and return the folder, Mulholland said. But instead of watching eight hours of video, investigators can compress the footage down to the three-minute period in which the folder was taken from the desk. That’s a start, a baseline, he said.

“The next thing you can do with the analytic capability is identify an area of interest within the camera frame,” Mulholland said. For example, once investigators have identified the origin of a detonation of a bomb, they can draw an area of interest. There may have been 500 people who walked in that general area, but the analytics piece will ignore that and flag anything that changed in that one specific area—such as a backpack being left behind. So instead of spending 20 minutes looking at video in which nothing happens, the investigator can hit a button and in 30 seconds go to the area of interest and then begin to dissect what actually happened, Mulholland noted.

There are different flavors of this software out on the market. BRS Labs’ AISight, a behavioral analysis system for video surveillance, adaptively “learns” behavior patterns in complex environments. The video surveillance software uses a reason-based approach versus legacy rules-based technology, company officials say. Because humans are not required to define the rules for object or behavior recognition, the system can easily scale to thousands of cameras.

But the ability to define parameters is a critical feature for video analytics software, Mulholland said. For example, there might be a gateway into the viewing area of a venue that people are only supposed to exit through — so someone entering through the exit could be of interest. You would want to define a parameter saying, “show me if someone is going against the normal flow of pedestrian traffic.” Or if investigators know a suspect was wearing a red shirt, they could put that into the parameters and say, “show me someone who is wearing a red shirt.”

Data and time-based review video submitted by ordinary citizens and the use of various image and pattern recognition software might also help identify potential suspects in the bombing, according to Lee Neubecker, president of Forensicon, a developer of digital forensic tools.

Neubecker demonstrated how he analyzed video of one of the bomb blasts that was submitted to Boston.com.  By slowing the video down and colorizing it, he was able to show debris from the explosion hurling through a window of a nearby building, which could point investigators to debris from the pressure cooker bomb. Then investigators could determine where the pressure cooker was manufactured and analyze sales receipts from local retailers.  Using facial recognition software and video forensics, they then could cross reference security video of people buying pressure cookers with images — such as those of the two suspects — taken at the marathon finish line, Neubecker said.

Because they consider no piece of information or detail too small, the FBI and Boston police have urged people to send visual images, video, and/or details regarding the explosions along the Boston Marathon route and elsewhere, to boston@ic.fbi.gov. By mid-day Tuesday, the day after the bombing, over 2,000 tips had been received by law enforcement.  With thousands and thousands of hours of video, law enforcement will not be able to move quickly through them without some form of video analytics, Mulholland said.

Some challenges remain on the surveillance side too, according to Mulholland. Many older cameras use proprietary formats, which cannot be read by analytic tools. “We have to make sure the systems we are putting out are in a standardized format where we can apply any type of forensic tool,” he said.

With so many cameras, whether personal or surveillance, in use, crimes like the Boston Marathon bombing will be documented with terabytes of images and footage. Analytics tools that help investigators sort through the events will be increasingly important in solving those crimes.