Applying “Big Data” to Development: Revealing Initial Research Results

As Secretary-General Ban Ki-Moon noted in a briefing about Global Pulse to the UN General Assembly last month: “The private sector is analyzing this new data to understand its customers in real-time. Much of this data contains signals that are relevant to development. We must use it to tell us what is happening, while it is happening.”

That is why, over the past 6 months, Global Pulse has teamed up with leading data-research companies and institutions Crimson Hexagon, Jana, PriceStats, SAS, and a consortium of French centers led by the Complex Systems Institute of Paris Ile-de-France & IFRIS on a series of collaborative research projects testing the feasibility of adapting new tools and methods in real-time data collection and analytics for global development.

And today we are very pleased to publicly present the findings and methods from this initial series of experimental research. The five main projects included:

Real-Time E-Pricing of Bread

This project, a collaboration with PriceStats, entailed the construction of a daily bread price index for six Latin American countries. The findings illustrate how the online retail prices reveal offline street price changes weeks before official sales numbers were able to reflect the inflation, potentially allowing policy makers to better prepare for the negative effect on consumers.

Unemployment through the Lens of Social Media

Online conversations on blogs and forums about bleak job prospects and job satisfaction were occurring months in advance of actual job losses, predicting the rise of unemployment in the US and Ireland and simultaneously revealing the strategies that unemployed are pursuing to overcome their new economic situation. In collaboration with SAS, analysis revealed that the increases in the volume of on-line employment-related conversations in Ireland which were characterized by “confusion” show up 3 months before increases in unemployment, while in the US conversations about the loss of housing increased 2 months after unemployment spikes.

Twitter and Perceptions of Crisis-Related Stress

A research project with Crimson Hexagon analyzed Twitter data from the US and Indonesia to measure the degree to which cost pressures related to food, fuel, housing, and the economy are discussed in online conversations. The research analyzes trends in these topics in conjunction with themes such as “afford,” showing how the volume and topics of the conversations change over time reflecting populations concerns.

Tracking the Food Crisis via Online News

This multi-partner project with the Complex Systems Institute of Paris Ile-de-France & IFRIS analyzed thousands of news items related to food security issues in French language press. The analysis shows that as the 2008 global economic crisis unfolded, news coverage shifted from a focus on humanitarian issues to food price volatility discussions, whereas in 2011, the news focus has shifted to social unrest.

A Global Snapshot of Well-being through Mobile Phones

This project surveyed a sample of the 2.1 billion mobile phone subscribers that can be incentivized through Jana’s global research platform. Simultaneous mobile phone surveys were deployed in dozens of countries around the world, demonstrating how household-level information can be gathered rapidly and at unprecedented scale.

We embarked on these “proof of concept” projects with the hope that they may eventually help establish new methodological approaches for analyzing real-time data, identify tools which can provide a faster and clearer understanding of population behavior during periods of stress, and hopefully contribute to the development of new proxy indicators for real-time tracking of development.

These preliminary findings represent just the beginning of what new real-time data sources can tell us about human well-being. A great deal more work and investigation, particularly around validation, sample size and causality, still needs to be conducted, as we enter this emerging field of study. We look forward to working closely with data scientists, development experts, and others to bring these methods to a more robust state.

We are thrilled to have had the opportunity to learn together with all of our exceptional research partners who worked tirelessly with us over the past few months to innovate and adapt the latest tools and techniques in data science for the cause of global development. Stay tuned for more information about the process and findings from our research projects here on our blog.