How can tweets affect policy decisions?
Global Pulse Director explains how to predict societal changes and create effective policies in real time by analyzing social data
Recently, SAS teamed up with the United Nations Global Pulse on a unique research project titled "Unemployment Through the Lens of Social Media." This project investigates how social media and online user-generated content can be used to enrich the understanding of the changing job conditions in the US and Ireland. It analyzed the moods and topics discussed in unemployment-related conversations from the open social web and then compared them to offi cial unemployment statistics. The analysis revealed that increased chatter about cutting back on groceries, increasing use of public transportation and downgrading one's automobile could, indeed, predict an unemployment spike. After a spike, surges in social media conversations about such topics as canceled vacations, reduced health care spending, and foreclosures or evictions shed light on lagging economic effects. Such information could be invaluable for policymakers trying to mitigate negative effects of increased unemployment.
In this recent interview, SAS Executive VP Mikael Hagstrom discussed the study's implications with Robert Kirkpatrick, Director of UN Global Pulse. The conversation covers everything from real-time data to data privacy, and how this type of information could be used to influence and analyze the results of public policy decisions almost immediately.
Hagstrom: Understanding unemployment is one of the biggest challenges for any government. Are there any ways that you feel governments can address unemployment differently by using this type of information? What is the value of having big data and predicting unemployment spikes?
Kirkpatrick: This was really powerful from a UN perspective because the work that was done is actually comprised of two different types of analysis: the prediction of unemployment spikes based on mood analysis before the event and the impacts of unemployment. In other words, you have early-warning capabilities, but you can also look back on this data for insights that are essential to understand future vulnerabilities.
Hagstrom: What is your definition of big data and why do you care about it and big analytics?
Kirkpatrick: The IT department's definition of big data hasn't really changed because it's simply about the storage and processing power and transport, yet the term is being used to refer to an incredible diversity of data types. So I think for us, big data is data about human behavior.
I heard the term "big behavior" last week when I was talking to a behavioral psychologist who works in analytics. That interests me as a term to describe the real-time capabilities of big data analytics that can provide new insights into human behavior. Data that can give us a real-time picture of changes in human behavior is something that we have never had before. The real-time analytics provides capabilities that allow us to make a decision fast enough to change the outcome.
This is the most important aspect, but real-time is relevant. Real-time detection of malnutrition could be measured in months; real-time detection of an Ebola outbreak can be measured in days; real-time detection of a natural disaster can be measured in a matter of hours. We're trying to fix outcomes, and realtime analytics is what helps us do that.
When we did the work, we found significant differences between the US and Ireland in terms of how their populations were affected. In the US, we worry about losing our houses. In Ireland, you don't have that worry. But in Ireland
there was significant discussion about people canceling travel plans. So if you are a government for a country where you knew unemployment spikes tended to lead to an impact on your tourism industry, you can use the lead time to predict unemployment spikes, as well as the lag time three months after an unemployment spike to predict a drop in tourism. You now know that in six months, you will need to take steps to protect the tourism industry. These are rudimentary models, but there is tremendous potential. That project is an initial proof of concept, so now we really need to start building on this foundation to develop it into a reliable methodology.
Hagstrom: Based on your experience, how can you see this leading to the more preventive type of application that you mentioned?
Kirkpatrick: What's powerful about these data sources is that they are real-time, which means it's not about a linear process of detecting something and then initiating a response. The monitoring doesn't stop there; it is continuous. The real application is to initiate a response and evaluate the effectiveness of the response. And if that response didn't have the desired outcome, you can make changes very quickly. It's really about faster feedback for results.
It's a much more agile and adaptive approach to public policy.
Hagstrom: Let's assume for a moment that there had been the possibility to predict, from early indicators, how the events in Syria were likely to unfold. Do you foresee a future where technology can help quantify measures and build a case around risk that would enable earlier decisions that would lead to less damage and a greater chance of preventing the crisis from even occurring?
Kirkpatrick: The short answer is, yes. Populations that have never had a voice before now have a voice. They're communicating publicly and they're sharing in real time what happens in their lives. Conflict and early warning aren't areas that fall within Global Pulse's scope of work. But it certainly is the case that precursors of conflict are the familiar challenges of inclusion and poverty, hunger and disease, and issues of discrimination. These are very much issues that we are looking at. So there is certainly the potential in these new data sources to have a better and earlier warning of when a situation is escalating. The real question is, how will the information be used?
We've got UN colleagues who are very interested in understanding the issues that are facing the Middle East and North Africa right now and how they can be more attentive and provide the best assistance using these new tools.
Hagstrom: How can big analytics help Global Pulse keep international development on track and serve the world's most vulnerable populations?
Kirkpatrick: I'll give you two answers. At the highest level, it's about this agile and adaptive approach. We're getting feedback on whether what we're doing is working, which is going to let us have the best possible impact from a targeting standpoint. It's going to let us have a positive impact sooner, in addition to being more cost-effective in an age where there's not much development aid out there.
At the tactical level, what does it mean for a development practitioner? What I envision is a world where checking socioeconomic conditions in a community will be as easy as checking the weather. Right now, I can log on to any weather website and see current information such as wind speed, temperature and barometric pressure. I can get analytics about the current trends and where things appear to be headed, and I can even have early warning alerts sent to me if the sensors detect certain patterns like rotating clouds. Checking on socioeconomic conditions should be just as easy.
Hagstrom: Do you think that analytics can help policymakers be better informed and make more up-to-date policies?
Kirkpatrick: Yes, I think the challenge for policymakers over the next decade is learning how to operationalize the integration of these new, real-time data sources with the traditional work flows and data that they already base decisions on. When you're dealing with real time data, however, there's a tradeoff between speed and accuracy. Is this a blip, an anomaly or the beginning of a trend? So the challenge is helping policymakers develop the practices that are going to let them understand what kinds of decisions they can make based on real-time data and where they need to look for more information.
Hagstrom: What are your thoughts on data privacy with regard to these data sources?
Kirkpatrick: At Global Pulse, we recognize that protecting the vulnerable means fully protecting their privacy, without exception. This entire enterprise must be about experimenting with these new types of data and tools to understand what is possible without compromising privacy.
Discussing data privacy is a very polarizing event. It seems to come down to a struggle between those who believe that privacy is dead and those who believe that any reuse of data without explicit consent is dangerous. We see an opportunity to bring to the discussion a recognition that data can be used for public good, but that means there is no simple black or white framework we can use to treat all big data. We're going to have to grapple with different kinds of data to answer different problems and develop methodology to protect privacy on a case-by-case basis. But we need to create the space to experiment and learn, and that requires shifting the debate.
In our case, Global Pulse is looking at big data and analytics for policy responses. So our level of interest is what's happening at the community level. It's not about understanding the specifi cs of an individual who lost his job and had to sell his assets to survive. It's about discovering that a lot of people in a hard-hit community are trying to sell livestock at an odd time of year, recognizing what that means and finding ways to help.
Hagstrom: Was there anything that surprised you about the findings or about what you could do with analytics? Was anything validated from your initial expectations?
Kirkpatrick: We found what we had hoped to find: When peoples' needs change, the way they use digital services change, depending on what's actually happening at the household level.
We also found something that was unexpected: There are insights in these new digital data sources that we can't get any other way. That is what is incredibly exciting. Statistics can tell us when people are losing their jobs, but statistics can't tell us how that's changing people's lives.
Hagstrom: So what's next for Global Pulse? What other societal challenges can you tackle now?
Kirkpatrick: We've started work in Indonesia with a number of partners, including SAS, looking at two areas. One is food security, which is tied to youth unemployment and migration to cities. We're facing challenges all over the world with urbanization, and we've realized that we do not have a good understanding of what happens economically to the people who are in urban environments. But these new data sources provide an ideal window to understanding that.
We are also beginning to look at discrimination and barriers to entry into the workplace for women. We just started discussing this with the International Labour Organization to see if there's online chatter that can give us insights into why women have a diffi cult time entering the workforce and what sorts of challenges they encounter once they get there.