Jonathan Stray is a journalist and a computer scientist, and the Interactive Technology Editor at the Associated Press in New York.
In a crisis situation, my journalist colleagues and I are charged with witnessing, documenting, and rapidly publicizing the events on the ground. This has been the journalist's role since the invention of the news periodical centuries ago. But the speed and scope of this work are undergoing a massive transformation: journalism must now be real-time, and often depends hugely on public collaborations.
As I write these words, Libya is in a state of tremendous political unrest, with nearly 200 protestors reported killed. Think for a moment how we know this, given that Libya has allowed almost no independent journalists to enter or operate in the country. In such circumstances, professional reporting on Libya must rely on techniques like telephone interviews with anonymous hospital workers. But our knowledge of the situation is also greatly improved by the Libyans themselves, saying what they see, posting reports and photographs on the internet.
A person I know only as @Arasmus is collecting these reports, compiling them into a crowd-sourced map of unrest and violence.
This map is an astonishing resource, the product of new forms of communication and organization that did not exist until only a few years ago. (Google Maps launched in 2005, Twitter in 2006.) It is a resource produced entirely outside of the normal operations of traditional journalism; certainly, no news organization could produce such a map without dozens of reporters spread across the country. In Libya, as in other recent crises, it seems obvious that the residents of a place are a rich source of information — and often able to respond faster than institutions.
But if we look at this map a little closer, it also shows us how little we really know about real-time, collaborative reporting. The first problem is verification. How do we know that any of the aggregated reports are true? Experiments with crowdsourcing have repeatedly demonstrated that the majority of contributors are well-intended, but they aren't necessarily well informed. Do multiple reports of the same events correspond to independent witnesses, or are they repetitions of a popular rumor? Similarly, it would be all too easy for someone to fabricate reports or otherwise inject misinformation into the map, and in any volatile situation there will be many parties with the motive to do so.
For that matter, how can we have any assurance that the map is complete? We cannot interpret these reports without understanding who has access to communications technology — from DSL lines to mobile phones — and what their demographics are. It's far too easy to map connectivity instead of reality.
These are the sorts of questions that must be answered before collaborative, real-time reporting can be incorporated routinely into the practice of journalism. When I see maps like this, I wonder why it was an enterprising individual instead of an enterprising news organization who created them. But professional journalists have responsibilities that individual citizens do not: we must be very clear about the sources and accuracy of the information we endorse. When the sources are thousands or millions of individual reports, traditional verification practices fall apart. We cannot hope to "call and confirm" each tweet. We need new techniques, and we need new tools.
Further, not all important events produce a flood of information. It's easy to see that something big is happening in Libya, but sometimes, a single report contains critical information — or perhaps a link to crucial documents, photos, or video. So far, journalists have only the most rudimentary tools for sifting through the flood of social media and other "data exhaust." We are reduced to watching our Twitter feeds scroll by, hoping that we're following the right people, and that we haven't missed something important over lunch. Report detection is just as important as verification, and may be especially consequential at the community level, where local situations may not produce an international flurry of re-tweets.
All of this is to say that UN Global Pulse is not alone in its need for a new generation of real-time monitoring systems. International organizations, governments, NGOs, journalists, technologists, researchers, educators, corporations and citizens all have a stake in building and maintaining our collective information ecosystem, in times of crisis and at other times as well. And indeed, many different groups are attacking similar problems in their own fields. UN Global Pulse is the most ambitious attempt that I am aware of within an international organization. Professional journalism is just waking up to the changed landscape in which it operates. Meanwhile, the financial and law-enforcement communities are investing heavily in real-time technologies.
It's easy to see that these many, somewhat disconnected efforts have something to gain by co-inventing the necessary technologies. But there is another reason to collaborate: each of these potential developers of real-time monitoring and collaboration systems has somewhat different aims and challenges. More data might mean better crisis response, but it can also be an invasion of privacy. The journalist's instinct is to publish, but sometimes sources must be protected by concealing their identity. And other stakeholders might be concerned about the security of their systems or the proprietary nature of their algorithms. Real-time global monitoring is coming, but we must ask questions about who owns the technology and what ends it serves.
That why I believe that if we are serious about applying these techniques for the global public good, we must develop them in the open, for all, confronting the tensions between competing interests head on, and tapping into talent wherever we may find it. I am excited about various grass-roots communities with relevant expertise, such as Random Hacks of Kindness, the Open Government and Open Data movements, and the Hacks/Hackers network of journalist-programmers.
It's my hope that UN Global Pulse will become one nucleus around which important projects and technologies can coalesce. All those who deal in information have woken up to discover that we have already built a real-time peer-to-peer global communications infrastructure. But we have not yet created the technologies, processes, and polices that will allow us to harness the efforts of billions of networked individuals for the greater good.