VR drives next-gen situational awareness for public safety

A new software solution puts command centers in front of first responders at the scene of an emergency using virtual reality (VR) headsets.

Headwall developed the Headwall XR Command Center with the idea that VR is better than two-dimensional screens for presenting map and spatial information, said Geoffrey Bund, the company’s chief executive officer.

“We’ve taken it for granted that all information has to be transferred to a 2D format to then be transmitted and viewed,” Bund said. “Virtual reality allows [geospatial information systems], mapping systems and 3D information to maintain its native format through to viewing.”

He put the idea to the test during the recent four-phase CommanDING Tech Challenge, sponsored by the National Institute of Standards and Technology. To improve the quality and usability of incident command dashboards, NIST asked companies to create “new dashboards with improved user interface/user experience (UI/UX) designs that incorporate video, sensor and map data, as well as real-time indoor tracking and video analytics, using mobile and web-based solutions,” according to an announcement earlier this month that named Headwall the winner with a prize of $130,125.

Headwall’s solution allows live video feeds from command centers and any other data that would be displayed on a video wall to be viewed in a head-mounted display of an on-scene commander. That means responders in the field can have access to the same level of situational awareness as those in the incident command center.

For the final phase of the competition, NIST provided a scenario in which an explosion goes off in a school and first responders have 20 minutes to find and rescue victims. The simulation took place in a 90,000-square-foot, four-story building at the University of Memphis with 20 actors. Synthetic data from surveillance cameras, mapping apps, sensors and other devices, such as vital-sign health trackers, was streamed in real time during the 20-minute event.

“What the challenge envisioned is that the XR Command Center would be utilized by people at a headquarters location and then also [by commanders] in the field on-site,” Bund said. “We can allow that command center to be deployed in a tiny form factor, and you can feel like you’re in a physical command center.”

Now, Headwall is working to translate those integrations with simulated data into real-world ones. For instance, instead of using the NIST surveillance camera, crowd detection will work with Genetec, which provides physical security solutions. And instead of annotated 3D point cloud models, which NIST provided for the challenge, Headwall will work with Esri, Bund said.

The software integrates via an application programming interface with first responder agencies’ video management systems. 

“We’ll come in at around the same time that the actual headquarters video wall or command center is being renewed, refreshed or built from scratch, and the AV integrator will provide hardware that sends us video,” Bund said. “We build an API that allows them to send commands and send information … and we can send information to the video management system from our software.”

For instance, if a fire chief sees footage from body-worn or surveillance cameras through the headset that would be of interest to folks at headquarters, they can flag it in XR Command Center and send it.

Additionally, Headwall can load the enterprise licensing of agencies’ mapping software so that it can be ingested and visualized. “Most of the time, that software has met us halfway,” Bund said. “They have hooks into a game engine called Unity that we use to build our visualization, so we’re not having to actually build all of the maps from scratch and bring it in. We can basically import their visualization data into our platform.”

The company is porting over to work with the Meta Quest Pro headset. “What that allows us to do is get rid of the computer, get rid of the wire, and we just will have a standalone headset,” he said. “There would be no setup at all…. What we envision is that someone would just have to open a laptop and launch the application, and they would immediately be in the headset.”

Other potential use cases for the XR Command Center include school shootings and enterprise security, Bund added. 

NIST’s Public Safety Communications Research (PSCR) Division has conducted prize challenges since 2017, looking at tech such as VR, haptic feedback, the internet of things, augmented reality, drones and UI/UX. A main reason for the responder-focused challenges is that it can be hard for public safety agencies to use commercially available technologies, said Scott Ledgerwood, UI/UX lead for PSCR.

The PSCR challenges address problems that exist in public safety communications today “and put resources behind that to push for innovation and try to bring that technology closer to a marketplace deliverable for first responders,” Ledgerwood said. We want to get new technology “in their hands and allow them to really have an impact on their missions in their operations.” 

All of the challenge participants were impressive, according to Sarah Hughes, a prize competition and challenge specialist with PSCR Open Innovation team at NIST.

“For all of our research areas to date that we have at PSCR … we’ve had a roadmap of where we want to go and where we want to push communication technology” based on what we’ve heard from first responders, Hughes said. “If we’re going to design communication technology for public safety, they need to be at the forefront of the design.”

Stephanie Kanowitz is a freelance writer based in northern Virginia.