US gov't agency looks to online data for early warning system

Iarpa (Intelligence Advanced Research Projects Activity) wants to find ways to automatically track and analyse publicly available data to provide early warnings of political or humanitarian crises, mass violence or migration, economic instability, resource shortages, disease outbreaks and responses to natural disasters.

Its Open Source Indicators (OSI) programme is currently considering proposals from academia and industry to “develop and test novel methods to assist intelligence analysts in anticipating significant societal events”.

Such events, Iarpa says, are often preceded by changes in how people move, communicate and consume, which it believes could be used to predict what will happen, if they can be observed through publicly available data such as web search queries, internet traffic, social media and financial market activity.

“OSI’s methods, if proven successful, could provide early warnings of emerging events around the world,” said programme manager Jason Matheny. Iarpa, which says it invests in “high-risk, high-payoff” programmes, says it wants to “beat the news” by combining data from multiple sources.

Methods will be evaluated based on whether they perform in the real world.

Google has already experimented with using search data to predict real events. Its Flu Trends service, introduced in 2008, tracks searches for flu-related terms in order to estimate where the disease is prevalent and how it is likely to spread. The results closely matched data on actual infection rates – but provided an advance warning. It is now using the same methodology to predict outbreaks of dengue fever.

A separate project by the defence research body Darpa has been looking at how the military can track and respond to the spread of information and ideas in social media.