Piranha: Decoding the genomic tree in millions of documents
Connecting state and local government leaders
ORNL's text analytics software sees documents as mathematical vectors, clustering them according to their similarities.
The Piranha text analytics software developed at Oak Ridge National Laboratory, which can scale from working on a single PC to a supercomputer, speeds up the search for relevant information by grouping and comparing documents.
Piranha essentially represents documents as mathematical vectors, which lets users of the software perform similarity comparisons, explained Thomas Potok, senior scientist and leader of the Computational Data Analytics Group at the Energy Department’s lab. Users can compare the two vectors and determine how closely they resemble each other.
“What we are able to do in a highly parallel fashion is to create and compare these vectors, so in essence we are comparing every word in every document to every word in every other document,” Potok said. From that point, researchers can say how similar the documents are.
The Piranha clustering algorithm is similar to how animals are grouped in a genomic tree, where horses and cows are close to each other but much different from a squid or an amoeba, Potok explained. “It kind of gives you a sense of how you look at similarities in documents, what things are similar or how things are dissimilar.”
Piranha’s software agents work differently than more traditional software agents, Potok said. It is not a piece of software running around a network but can be assigned to a specific computer or group of computers. For example, if an agency used the software with 10 computers, the software agents would move around the 10 computers to process information faster. A law enforcement officer might install Piranha on his computer, but not on any other. For an agency that has a huge volume of data, users might put Piranha on a cluster of computers, a very large cluster of computers or even on a supercomputer if they needed that level of computing power.
ORNL’s Computational Data Analytics Group is working to extend Piranha’s capabilities. “We are looking to analyze a trillion documents,” Potok said, noting that the team is using the lab’s supercomputer to try to solve this task. But challenges still exist with how text analysis tools tag items, he said. An analyst might want to pull out and highlight a name in a document, for instance. But if he sees the name “Washington,” he doesn’t know if that refers to a person, city or street. So information tagging still needs to be addressed, he added.
ORNL’s computational data analytics team is also focused on how to look at text information in a time view. For instance, with Twitter, people tweet things at certain times but also from certain locations. Law enforcement or intelligence analysts might have information before an event that could be significant, and after an event has occurred wished they had seen the information. So the team is working on “how do you deal with time and location,” Potok said.
Finally, “perhaps one of the biggest challenges is: How do I deal with the volume of data and the ugliness of data?” Potok said. “You have all of these documents and raw information. Well, what is the value, what is useful and what is noise of no value at all?”
ORNL is offering a prototype version of Piranha that can be downloaded from the ORNL site.
NEXT STORY: Should agencies dump e-mail for social networks?