Unlocking the black box of AI reasoning
Connecting state and local government leaders
Research with generative adversarial networks may explain how neural networks learn and make decisions.
While artificial intelligence has proved effective at many tasks critical to government -- such as protecting power grids against hacking -- some agencies have been reluctant to employ AI tools because their inner workings are unintelligible to humans. How can a solution be trusted if nobody knows how it works?
David Bau, a Ph.D. student at the Massachusetts Institute of Technology, thinks generative adversarial networks may help show how AI algorithms reach their conclusions. Bau and others are testing GANs not only as tools for performing tasks, such as pattern recognition, but for examining how neural networks made decisions.
GANs -- which gained recent notoriety for being used to create the first AI painting sold at auction and, more disturbingly, for superimposing the faces of celebrities on porn stars in videos -- work differently than other AI algorithms, according to Bau.
“Most neural network training is done is as a one-player game, where we set up … rules and the network learns to beat the game,” Bau said. Training a GAN uses a two-player game model in which the goal is for the GAN to achieve better results than a neural network adversary in generating accurate images, he said.
“Under normal training we would train a neural network to mimic the human behavior,” Bau said. In this case, we “show a neural network a huge number of images, completely unlabeled by a person.” It’s up to the GAN to find any discernable structures in the images -- airplanes, cars, gates, trees -- without cues from humans.
Bau said there are only two ways an algorithm can accomplish the task. One is to memorize the pixel-by-pixel characteristics of images and then compare those images to freshly provided images. The other strategy, he said, is one that humans use --specifically, a compositional strategy, analyzing and image and breaking it into its parts.
To see how the GAN made its choices, the team cracked it open partway through a job, Bau said. “We stopped the computation halfway through, and we looked at the internal variables that the GAN produced.”
The researchers looked for signs that certain patterns of numbers being processed by the GAN correlated with structured data in the image. Did one sequence of numbers represent the concept “tree”?
Conceding that analyzing the processing of GAN would bring even veteran programmers to their knees, Bau said that the team used a different neural network to look for patterns and correlations in the GAN’s process.
And the team found that, indeed, the GAN had isolatable number sequences, which researchers dubbed “neurons,” that corresponded to objects in images. “We found that this neural network has neurons that correspond to trees, other ones that correspond the doors, other ones that correspond to rooftops,” Bau said. “The correlations are high enough that it is really suggestive that the network is breaking down the images in terms of objects that humans would call compositional, things that make up the scene.”
The team then tested that finding by erasing selected neurons and then running the GAN again. Sure enough, when they erased the algorithm’s neurons that correlated with “tree” the image eventually generated by the GAN was missing trees.
Perhaps even more surprising, “when we get rid of the trees, the neural network keeps on painting a reasonable picture,” Bau said. “It doesn't just draw blotches all over the place. Instead of the tree it will draw the building that was behind the tree.”
The researchers' paper is posted online and includes an interactive photo manipulation app that demonstrates this kind of object-level control. Each brush stroke activates a set of neurons in a GAN that has learned to draw scenes.
In short, the algorithm seems to be learning how to capture the real structure of the world without human intervention. “That’s one of the holy grails of machine learning,” Bau said.
NEXT STORY: Marching toward intra-soldier wireless