Don't look now, but everybody (CIA, DHS, etc.) is watching

It started with businesses concerned about their brands. Intelligence agencies have moved into it, raising alarms among privacy advocates. And many other government agencies are just beginning to explore its potential. It is social media analytics.

Companies and government agencies alike are using tools to sweep the Internet — blogs, websites, and social media such as Facebook and Twitter feeds — to find out what people are saying about, well,  just about anything. The companies are generally interested in complaints about products or looking for sales leads. Intelligence agencies are looking for, among other things, warnings about potential terrorist threats.

“As consumers spend more time using social media, it should not be surprising that extremist organizations are as well,” said Rebecca Garcia, director of sales at SAS, a business analytics software firm. “They see social media as a primary tool for bringing together communities of interest, regardless of geographic location and for discussing ideas in a seemingly anonymous manner.”

Related coverage:

5 thieves, 5 cities, 12 hours: can Twitter catch them? 

DARPA's $40K quest tests social media's ability to help in an emergency

Garcia said many of the techniques used to distill the essence of online conversations for businesses can also be applied to intelligence efforts. “The area of social media analytics could feasibly emerge as an indispensable counterterrorism tool by enabling proactive monitoring, analysis, and engagement through extremist social networks and associated digital properties,” she said.

The potential for social media to serve as a critical conduit during emergencies has been recognized by the Defense Advanced Research Projects Agency. From Feb. 23 through March 8, DARPA sponsored the CLIQR Quest Challenge, an online contest in which participants try to locate Quick Response codes over social media. The prize for the winner was $40,000. 

Similarly, the State Department launched the “Tag Challenge” on March 31, in which participants will compete in trying to use social media to locate five criminals in five cities. The winner will be rewarded $5,000.

Not surprisingly, intelligence agencies have already been looking at social media as a source of information. The Homeland Security Department has been analyzing traffic on social networks for at least the past three years. Although the department has not commented on specific uses of such tools, a senior DHS official was recently reported to have said the department does not monitor dissent or track American citizens’ views. 

And the department claims that it does not routinely monitor Facebook or Twitter. Instead, it says it checks such sites for information when it receives a tip.

The privacy line

Ginger McCall, director of the Electronic Privacy Information Center’s Open Government Program, disagrees with DHS’s characterization of its efforts.

“As far as the Department of Homeland Security is concerned, they don’t actually have the authority to engage in this sort of monitoring,” McCall said. “The Department of Homeland Security is charged with preventing and handling natural disasters, man-made disasters and terrorism. But their monitoring here is far broader than that. They are explicitly monitoring for criticism of the government, for reports that reflect adversely on the agency, for public reaction to policy proposals.”

McCall said EPIC has asked Congress to suspend DHS’ programs for monitoring social media. “Most people, when they comment on a news story, when they comment on a blog, they don't assume when they are doing that that a law enforcement agency is going to be coming along and possibly recording comments and noting them down, and storing them,” she said.

DHS declined a request for an interview.

And DHS is not the only security agency getting into social media. The CIA has reportedly invested in Visible Technologies, a company that offers software that crawls across blogs, forums and social networks to monitor traffic. And the FBI in January put out a request for vendors to provide information about available technologies for monitoring and analyzing social media.

In addition to performing routine searches, the agency specifically said the FBI’s analysts need to be able analyze social media to, among other things:

• Detect specific, credible threats or monitor adversarial situations.
• Geospatially locate bad actors or groups and analyze their movements, vulnerabilities, limitations, and possible adverse actions.
• Predict likely developments in the situation or future actions taken by bad actors (by conducting trend, pattern, association, and timeline analysis).
• Detect instances of deception in intent or action by bad actors for the explicit purpose of misleading law enforcement.
• Develop domain assessments for the area of interest (more so for routine scenarios and special events).

GCN had questions about some of these requirements, but the agency declined our request for an interview. Indeed, because of the potential power of social media monitoring and analysis — and the potential for abuse — it’s a topic most agencies decline to talk about. And most vendors we contacted also declined to talk about their work with government agencies and departments.

At the same time, the potential of social media analysis for other areas of government is also great. Government agencies, just like private businesses, need to serve their customers.

The global pulse 

“In order to understand your constituency, you need to have a good random sample, and it can be difficult to get feedback from consumers,” SAS' Garcia said. “So social media could be used as a way for government organizations, just like commercial organizations, to understand whether folks are happy or unhappy with the services they are receiving. During tax time, are they frustrated by the help they are receiving online or from the advisers who are available online?”

SAS demonstrated some of the potential power of social media analytics with the “Global Pulse” project. Using SAS Social Media Analytics and SAS Text Miner software, the United Nations project analyzed more than half a million blogs, forums and news sites to find out whether a country’s online “mood” can predict spikes in unemployment. Tracking two years of data from the United States and Ireland, the software checked those sites for references to unemployment and how people were coping with it (downgrading housing, selling cars, returning to school, etc.).

The analysts found that online conversations gave indications of employment rate changes as much as three months in advance of official figures. The analysts also learned things about coping tactics and other areas of impact, such as loss of housing in the two months after a significant increase in unemployment.

One of the strongest advocates of using social media analysis in government is Craig Fugate, head of the Federal Emergency Management Agency. Fugate said information culled from social media could be used to better direct assistance during emergencies.

“Something we've never really had the ability to do is to see what people are dealing with and see the information that they are putting out about their experiences to help us better shape our response,” Fugate said at a conference in 2010. “The conversations taking place in social media add a whole different dimension to how we can provide assistance and look at making sure our assistance is tailored to needs in the disaster area.”

‘Sentiment analysis’

In fact, the tools being developed to analyze social media can do a lot more than search social networks and websites for keywords. 

For one thing, there is the potential to geolocate those posting messages. Although Garcia says SAS software does not currently take advantage of that capability, she added that “much of the global population’s use of social media sites is driven from mobile devices, which increasingly contain coordinates of a user’s location.”

Although geolocating postings may be in the future, the tools offered by SAS and other vendors already allow users to perform sentiment analysis on social media content. Is a posting on Facebook that refers to burning a church a joke or serious? Sentiment analysis can tell the difference. 

But algorithms for sentiment analysis need to be tuned by specialists for the particular use of the client. “The sentiment rules that you would apply if you are dealing with the United Nations or unemployment versus the sentiment you would use for someone not happy with their brand or the sentiment that you might employ in different regions of the country or in different Spanish dialects, for instance, are different,” Garcia said.

There are a host of other context-based factors that must be considered in designing sentiment analysis rules, including changing language patterns and the age of the writer, for example. 

How accurate is sentiment analysis? “That's one of the great debates,” said Nathan Gilliat, an analyst with Social Target, a North Carolina-based social media analytics consulting firm. “People who sell it will tell you it's really good. Some of the people who buy will tell you they just can't use it.”

But if the reliability of sentiment analysis is somewhat suspect at present, Gilliat says, things are changing quickly. “How well sentiment analysis works varies by industry,” he said. “This is a very active area of research, and somebody is always working on a better way to do it in unison. As they think they've got something, it seems to show up in a product very shortly after, which makes an interesting space because there's so much work going on.”

Social media meets big data

Tools are also available to users to measure and track the influence and connections of contributors to social media. 

Klout, for example, specializes in measuring individuals’ online influence by tracking reactions on social networks to their online postings. Klout tracks activity on Twitter, Faceboook, Google+, LinkedIn and Foursquare. 

SAS and i2 both offer tools to analyze how people are connected to each other over social networks in order to identify, say, those who are “gatekeepers” to linked networks of people.

Gilliat said that when he began specializing in social media analysis he started by making a list of all the companies providing software and services. That list now has more than 300 companies on it.

“What I really find interesting is the variety of approaches in the different ways these companies are coming to market,” he said.  “There are companies that pull data from online communities, and there are companies that aggregate data from lots of blogs. If you want a feed of Twitter, there are companies that partner with Twitter, and they will sell you a feed of data. And there are companies that specialize in analyzing people's influence, such as Klout.”

What has Gilliat really excited is the prospect of combining social media analytics with big datasets already maintained by business and government. “This is where we are really going to see a lot of the value — when companies are able to take their social media data and do something interesting with it in combination with other data they have,” he said. 

“Social media is a source," Gilliat said. "It is very easy to think of it as something you can draw a line around. But I think we’re going to find it's more powerful if we find ways to erase that line and combine it with other things to make it useful.”