Kentucky plows through big data on winter roadways
Connecting state and local government leaders
By outfitting snow and ice removal trucks with automatic vehicle location technology, officials hope to improve safety and save money.
This winter, through the Kentucky Transportation Cabinet’s (KYTC) Intelligent Transportation Systems program, about more than 400 of the state’s 1,400 snow and ice removal trucks will be outfitted with automatic vehicle location (AVL) technology to improve roadway safety and save money.
AVL uses vehicle-mounted sensors that collect and transmit data on the trucks’ location, road conditions, plow position, temperatures and salt application rates. The data gets sent back to headquarters where it is combined with data from about 12 other sources including social media, traffic conditions, Doppler radar, air and pavement temperatures and precipitation to give officials information they need to make changes to operations in real time -- rerouting trucks, avoiding double salting or simply helping a stranded driver.
Together, the data can total 80,000 records in one minute, KYTC’s Data Architect Vineet Kumar said.
Python scripts read and aggregate the data feeds every 10 seconds and bring them together to be processed using Apache Spark and then warehoused using Hadoop. The data is compared against the state’s 94,000 roadway segments, enabling KYTC officials to break up the data into more manageable chunks. For instance, an engineer can view all the data on a certain segment to understand traffic speeds, air and road temperatures or how much salt was applied.
“It’s really organizing our data so that the engineering staff can … see if there’s any changes that need to be made,” said Chris Lambert, systems consultant intelligent transportation systems at KYTC.
The open source Hadoop and Spark technologies were game changers, Kumar added, cutting the records processing time from 45 minutes using a traditional database to about 10 minutes with Hadoop and then 35 seconds with Spark.
The system also uses Esri’s GIS platform for the spatial component, such as finding and comparing information from Waze, a community-based traffic and navigation app, with other traffic data.
“Once the data comes from the trucks, we gather that data using [Apache] Flume, an open source product,” Kumar said. The raw data is stored and processed. The summary data gets fed “back to SQL server, and then from SQL server it goes to the public using Esri’s ArcGIS,” he said.
During initial tests of the system last winter, the system received 1,400 records every 10 seconds from the AVLs on each of 150 to 200 trucks. “We did do some load testing with additional trucks -- pushing 600 to 800 -- and we were still hitting that timeframe,” said KYTC Systems Consultant Jeremy Gould said.
Before, managing snow and ice operations was largely a manual process, with engineers determining how many trucks and workers were available and prioritizing routes to be cleared, Lambert said. The exact savings from being able to adjust operations in real time are yet unknown, but officials are targeting numbers in the millions of dollars.
“Once we have full implementation, which is all 1,400 trucks, one of our goals will obviously be making the most efficient use of tax dollars,” Lambert said. “I just hate the word ‘save’ because no two winters are the same, and we don’t have the historic data to truly show ‘savings.’ Even after this is implemented, our cost on snow removal may still increase just due to the severity of the weather.”
Kentucky’s snow season, which lasts about half the year, typically costs the state $45 million to $75 million, but the expenses from winter storms between 2013 and 2015 exceeded that range, propelling the need for a new approach.
Lambert said he isn’t sure when the project will come out of the pilot phase. Two things have to happen, he said. First, he wants to have sensors in place on all the trucks, and second, he wants them to be properly calibrated to ensure the data’s accuracy.
This means cost is a factor. Some trucks are already fitted for AVLs, but the equipment costs about $500 to $600 and then there’s the $40 per month fee for cellphone service that each unit requires, Lambert said. Retrofitting trucks with both a harness and controller can cost about $4,000 per truck.
“It’s a long-term project,” Lambert said. “This is not going to happen overnight.”