What is the data revolution?

Recent years have seen an exponential increase in the volume, quality, and types of data available. Combined with new technologies that permit faster processing of larger data sets and new statistical methods, this presents unprecedented possibilities for using these data for informing and transforming society. As the U.N.’s Independent Expert Advisory Group observed in their report, A World That Counts, there has been “…an explosion in the volume of data, the speed with which data are produced, the number of producers of data, the dissemination of data, and the range of things on which there is data, coming from new technologies such as mobile phones and the ‘internet of things,’ and from other sources, such as qualitative data, citizen-generated data and perceptions data.” But the data revolution is not only about supply. As the IEAG also noted, it is characterized by “… a growing demand for data from all parts of society.”


… a growing demand for data from all parts of society.

How will the data revolution help countries reach the sustainable development goals?

The SDGs are a broad, ambitious program of action. 

They will place new demands on governments in all countries to deliver more effective policies and programs, cutting across sectors and responding to economic, social, and environmental challenges. To design and implement new programs, governments and citizens will require more data – on who is affected, how they are affected, and who is being left behind. The data revolution offers many opportunities to meet that need, with more  information now available than ever before. Governments are responding. They are making commitments to open data, providing greater access to government data. And new opportunities exist for collaboration with the private sector and non-governmental organizations. But greater efforts are required to make that information useable and useful for governments and others who will contribute to achieving the goals. New technologies, crowd sourcing, and improved connectivity can empower people with information on the progress towards the targets. Without a data revolution, the SDGs will be little more than hollow promises, lacking one of the essential means to take action and achieve the goals. 

How is the data revolution affecting people's lives?

The data revolution brings new opportunities for using data to improve people’s lives.  

Consider, for example, Namibia, where cell phone records were used to trace travel patterns and satellite images were used to identify places likely to harbor concentrations of mosquito and parasite populations. Bringing these data sets together allow the ministry of health to target its limited resources on the 80,000 people most likely to spread the disease to other parts of the country.  At a more local level, GPS tracking devices have been used in Nairobi, Kenya, to map routes of private buses, providing bus drivers and commuters with information about optimal routes; and in Seoul, South Korea, cellphone data records have been analyzed to improve late night bus routes.  These are among many examples of ways in which innovations borne of the data revolution have affected people’s lives. 

What does the data revolution mean for governments, statistical offices, civil society, and business?

As the data revolution moves ahead, governments, businesses, researchers and citizen groups will need to innovate and adapt to the new world of data, a world in which data can be acquired faster and data sets are larger and more detailed than ever before.   

For governments there are opportunities to improve the functions of government, to better match programs to citizens’ needs, and to respond more quickly and effectively in moments of crisis. Statistical offices are an important center of expertise in the collection, management, and analysis of large data sets.  But their role is changing to focus less on producing data and more on interpreting the data for government use.  They should be at the forefront of the data revolution, taking advantage of new capacities to produce reliable measures of social, economic, and environmental conditions. Some businesses are already turning data into new products; others are reinventing their business processes to take advantage of new methods of managing their organizations, reaching their customers, and identifying ways to increase their social impact.

Is the data revolution only for rich countries? WIll poor countries be left out?

The opportunities and benefits of the data revolution cannot and should not be limited to rich countries.

data revolutions far
Already cell phone subscriptions cover 87 percent of the population in developing countries and 66 percent of the population in Sub-Saharan Africa and nearly 30 percent of the population in developing countries accessed the Internet in 2013.  Rates of cellphone and Internet use continue to rise, but developing countries are falling behind in their investment in new technologies. A World That Counts cites evidence that “… African countries spend about 1.1% of GDP on investment in and use of Internet services, less than a third of what, on average, is spent by richer countries.” To catch up and claim their full share of benefits, developing countries will have to do more than invest in technological infrastructure. They need to invest in human skills and create a legal environment in which data can be used and reused to benefit people. With the help of development partners including the private sector, they also need to do more to improve the quality and coverage of traditional data collection tools such as surveys, censuses, and administrative data.

What are the risks of the data revolution? How can these risks be managed?

As much as the data revolution can speed the progress of development and benefit people’s lives, there is a risk it will widen gaps in the ability to use and benefit from information and communication technologies. 

Poverty, lack of education, prejudice, and social isolation exclude many people from the new world of data and information. Whether the data revolution results in a world of greater equality or inequality will be determined by the decisions of public and private actors to treat data and information as public goods, available to all, and to make the necessary investments in education, infrastructure, and legal frameworks necessary to increase knowledge and lay the basis for sustainable development.

The data revolution also poses risks of political misuse or violations of personal privacy that may undermine the trust of people in their institutions. As the IEAG report, A World That Counts, observed, “As more is known about people and the environment, there is a correspondingly greater risk that the data could be used to harm, rather than to help. People could be harmed in material ways, if the huge amount that can be known about people’s movements, their likes and dislikes, their social interactions and relationships is used with malicious intent, such as hacking into bank accounts or discriminating in access to services. People and societies can be harmed in less material, but nonetheless real ways if individuals are embarrassed or suffer social isolation as a result of information becoming public.”  New regulatory frameworks may be needed to manage these risks, and new norms and principles to guide data use and increase trust between the different actors in the data ecosystem.

What is big data? How is it connected to the data revolution?

“Big data” is a part of, but not all of, the data revolution. Big data commonly refers to very large sets of unstructured data, often gathered as a by product of other activities, such as cell-phone records, Internet postings, and commercial transactions, as well as remote sensing and autonomously generated data. 

The capacity to collect and analyze big datasets is part of the data revolution but not all of it. Censuses, surveys, and the administrative records of governments also generate large data sets, but they are not usually considered to be big data, although they may be important for analyzing big data sets.

Big data are often collected in real time and may contain detailed information about the location and actions of the subjects. Using high-speed computers and new statistical techniques, big data can be mined to monitor rapidly emerging events, intervene in crises, and to identify small- scale occurrences that would escape conventional data collection and analysis methods. For the same reasons, the uncontrolled use of big data techniques may pose threats to privacy, security, and personal freedom.

What new technologies and innovations are driving the data revolution?

The wide distribution of computers and mobile devices, the falling cost of data storage, and the spread of the Internet have propelled the data revolution. 

With them have come new types of data: citizen generated data, including photos, exchanges on  social media, and crowd-sourced data. Another productive source has been so-called “digital exhaust,” records left by phone calls and other network interactions. And enormous datasets recorded by satellites and other remote sensing instruments exist and are yielding new applications. The data revolution has led to new ways of “mashing up” data sets – quantitative and qualitative — and exploring their interlinkages. New visualization techniques that make it easier to present large quantities of data in understandable ways are also important.

Technology has been and will continue to be a driver of the data revolution.  To harness the benefits of new technology, large and continuing investments in innovation are required at all levels, but especially by governments in lower-income countries that are currently lagging behind.  There needs to be an increased access to information technologies by, among other things, increasing access to broadband, increasing literacy, including adult literacy, and increasing the use of ICT in schools worldwide, to ensure that all people, including the poorest, have access to the technologies that can improve their lives.  Other areas of activity could be developing systems for global data sharing, filling research gaps, and engaging in creative incentives for data challenges.

Is the data revolution just about new technologies and statistical methods?

More than technology and methodology, the data revolution is about realizing the value in vast quantities  of data that are created every day and finding new ways  to use data to benefit people. 

It is about counting things that have not been counted well enough such as environmental conditions, the status of women and girls, poverty and inequality around the world, and the performance of governments. It is about better quality and more open data for all. It is about data usage and improvements in data literacy and empowering more people to use and produce data to better their lives and the lives of those around them.

What is the role of the private sector in the data revolution?

The private sector has an important role in the data revolution.

PrivateSecChartsIt has been an important source of innovation in the use of data to drive decision making and improve the delivery of services. Businesses and research institutes should be encouraged to keep innovating and creating new products. But they should also be encouraged to share data with the public sector and civil societies and partner with them in areas where they can add value to meet sustainable development goals.

What are the roles of NGOs and CSOs in general in the data revolution?

Non-governmental organizations and civil society organizations can raise their voices as users of data and articulate their need for more and better data. 

They can look for places where data have had an impact and highlight them. They can participate in local, regional, and global data initiatives, not leaving the work to a few. NGOs and CSOs should demonstrate high standards and transparency in their production, use, and sharing of data, and provide feedback on government services.  They can advocate for and ensure personal privacy protections are adequate. The same is true for citizens who use data: they should ask questions from the producers, articulate demand for better data, and challenge the misuse or misrepresentation of data.

Data revolution: there is a role for everyone

THE PRIVATE SECTOR

 

Role: Innovate and create new products, but can also share its data and partner with the public sector and civil societies to meet the sustainable development goals. Benefit: Data Revolution will result in more people with access to information systems, increasing the value of innovative business ventures. It will also improve the quality of data, helping with better business decisions.
NGOs AND CSOs
Role: Advocate for more and better data; highlight examples of data impact; participate in local data initiatives; and maintain high standards in their own data sharing practices. Benefit: With better data and tools to access and use them, CSOs and NGOs will be able to better serve their mandates and focus on priority areas.
NATIONAL GOVERNMENTS
Role: Create the rules and legal framework necessary to encourage innovation while guaranteeing data privacy and security, as well as training citizens to be productive members in an information society. Benefit: Through the increased sharing of local and global knowledge, governments can benefit from the experiences of others while continuing to foster more transparent data systems and more knowledgeable and informed citizens.
INTERNATIONAL ORGANIZATIONS
Role: Provide aid and technical assistance for statistical capacity building; finance analytical research; develop standards and guidance for data quality; and provide open, quality data from their operations. Benefit: Gain from collaborations with many stakeholders in the public and private sectors and with other international agencies, working collectively to pursue common goals such as the SDGs.
CITIZENS
Role: Participate in actively using data; develop skills needed by a new information society; generate and share data; and hold data collectors and producers accountable for transparency and privacy.​ Benefit: Benefit from improvements in living standards due to data driven innovations, improved applications to access and use data, and more global attention to data privacy and security issues.

What can national governments do to further the data revolution in their countries?

National governments should make a commitment to open data, creating an environment that encourages the innovative use and reuse of data to solve outstanding problems. 

As the IEAG report says, “If the new world of data is to be based on public trust and public consent, there has to be a confidence that governments can and will play this role, at least in part through the creation and enforcement of new rules.  Governments working with private institutions, civil society, and academia can set and enforce legal frameworks to guarantee data privacy and security of data for individuals, and ensure its quality and independence.  It is governments that can balance and create systems that foster incentives without creating unacceptable inequalities, adopt frameworks for safe and responsible use and manage the international system that can transfer finance and technical expertise to bring the least informed people and institutions up to the level of the most informed.”
Governments should also “walk their talk” of evidence-based decision making, weighing all evidence fairly and making decisions transparently. They should increase domestic investment in statistical capacity. Partnerships with the private sector can create opportunities for productive sharing of data and technology. Through the educational system, governments, in partnership with the private sector, should provide training in the skills needed to be productive members of an information society, evaluate the quality of data and use them for their own decisions, as well as to fully participate in initiatives to foster citizenship in the information age. And by becoming involved in international initiatives, governments can benefit from the experience of others while contributing from local to global knowledge.

What can international organizations and financial institutions do to further the data revolution?

International organizations and financial institutions should provide resources to countries where capacity is weak, develop standards and guidance to maintain data quality, and provide open, quality data from their international operations.

International bodies can also promote innovations and collaborations among international groups.

Along with providing aid and technical assistance, international organizations should finance analytical research in forward-looking and experimental subjects.  International and regional organizations can work with other stakeholders to improve methods of data collection, analysis, anonymization, and mechanisms for data exchange to ensure that data flows are safely and ethically transformed into global public goods.

One of the important roles of the UN and other international and regional organizations is setting principles and standards to guide collective actions within a global community and according to common norms.  The IEAG report calls for a “Global Consensus on Data” supported by international organizations. This consensus would establish legal, geospatial, and statistical standards to facilitate the use of data across national borders.  These standards should include openness and exchange of data and metadata, including interoperability of data and information systems; demographic and geospatial information, including “geographic semantic” management and exchange; global exchange of information on illicit financial flows; and open data and digital rights management and licensing.

What new investments are needed to prepare developing for the data revolutions?

An assessment of the scale of investment, capacity development, and technology transfer that is required, especially for low-income countries is needed, along with mechanisms and incentives to leverage the creativity and resources of the private sector. 

Funding will be needed to implement an education program aimed at improving the data literacy of citizens, “infomediaries,” and public servants, breaking down barriers between people and data.  Strong collaboration between public institutions and the private sector can help official agencies rapidly adopt new technologies and ways of doing things.  Substantial additional investment is needed to consolidate gains made during the MDG era and to develop reliable, high-quality data on a range of new subjects with an unprecedented level of detail, on subjects such as climate risk mitigation, inequality, and women’s empowerment.

To be useful, data must be of high quality, at a level of disaggregation that is appropriate to the issue at hand, and accessible to those who want or need to use them. There is a global responsibility to ensure that all countries have an effective national statistical system, capable of producing high-quality statistics in line with global standards and expectations.  This requires investments in human capital, new technology, infrastructure, geospatial data and management systems in both governmental and independent systems, as well as information intermediaries.  At the same time, national capacity for data science must be developed to leverage opportunities in big data to complement high-quality official statistics.  Increased domestic resources and international support for developing countries are needed to make a data revolution that contributes to sustainable development.

Funding is urgently needed in developing countries to strengthen national statistical capacity and engagement in the data revolution. It is crucial to use funding effectively, disbursing them in line with national priorities to incentivize innovation, collaboration and whole systems approaches, while also encouraging creativity and experimentation.  Capacity development and improvements to existing arrangements should include upgrading national strategies for the development of statistics. NSDSs to identify cost-effective investments, engage nonofficial data producers in a cooperative effort to speed up the production, dissemination and use of data, and strengthen civil society’s capacity to produce, use, and disseminate data. ​


There is a global responsibility to ensure that all countries have an effective national statistical system

Will there be legal partnership for sustainable development data and how could it support the data revolution?

The data revolution promises many benefits. As world leaders meet to decide on the post-2015 development agenda, there is a unique opportunity to set up a mechanism whereby stakeholders can come together to find shared solutions and spread the opportunities of the data revolution — leaving no one behind. 

The concept for the Global Partnership for Sustainable Development Data is under discussion. It is envisioned to be an inclusive neutral, global, and comprehensive platform for raising awareness and catalyzing action to advance the aims of sustainable development.  Its key goals would include building political and financial support for improved data systems; expanding use of data; convening events and forums to raise the profile of data challenges and to bring together diverse communities to find new solutions for data use and harmonization; fostering partnerships among stakeholders and among initiatives that should but do not currently interact; and encouraging use of norms and standards such as open data; and promoting innovations and knowledge exchange.

What is the world data forum and when will it take place?

The World Data Forum will be an output of the Global Partnership for Sustainable Development Data.

It is envisaged to comprise a series of thematic events, bringing stakeholders together to galvanize commitments and form partnerships among public and private stakeholders. The thematic World Data Forums will feed into an annual World Data Forum. This is expected to generate political commitment and accountability, inspire action, encourage networking, and solve problems among multiple stakeholders to move forward the data revolution for sustainable development. The World Data Forum, as an international conference, is still under development.

How does the data revolution align with the recent support for open government and the launch of the open government partnership?

The belief that open, reliable and transparent data will drive better development outcomes in developed and developing countries is one of the founding principles of the Open Government Partnership (OGP).

Open data can help ensure that knowledge is shared, creating a world of informed and empowered citizens, capable of holding decision-makers accountable for their actions. This is the foundation of a data revolution. The OGP National Action Plans outline relevant commitments policymakers and politicians in OGP governments are already making to support the data revolution.  The OGP’s Independent Reporting Mechanism produces biannual independent progress reports for each country participating in OGP. The progress reports assess governments on the development and implementation of OGP action plans, progress in fulfilling open government principles, and make technical recommendations for improvements.

How does the data revolution align with the open data charter and efforts to expand it?

The Open Data Charter is a set of principles that define Open Data and Open Data practices. 

The G8 governments at their meeting in 2013 proposed the first version.  Building on the G8 Open Data Charter, the G20 commitment to open data principles, the UN Secretary General’s Independent Expert Advisory Group on the Data Revolution, the Data Revolution Task Force, and a number of other multilateral groups and initiatives, a global consultation is underway. The goal is the development of an International Open Data Charter consisting of a set of foundational principles for open data, as well as associated tools and guidelines to enable delivery.  Adoption of the International ODC will encourage the release of data in open formats for use and reuse, expanding the scope of the data revolution.

Is open data a precondition for the data revolution to succeed?

The data revolution will go farther and do more good for more people if data are open.

OpenDataMany benefits of the data revolution can only be realized if innovators are able to apply new technologies to open data and to combine data in new ways. When access to data is restricted, innovation is inhibited. However not all developments of the data revolution will be based on open data. Many of the new and emerging methods of “big data” have been developed for and applied to proprietary databases. The progress of the data revolution will depend, in part, on whether these new techniques become widely available. Open source software, which encourages sharing and collaborative improvement of software and analytical methods, is a powerful complement to open data.

Why a development data revolution now?

  • Because meeting the SDGs will require more data and more complex data.
  • Because there is widespread support for improving statistical systems in developing countries.
  • Because developing countries are taking steps to open up their statistical systems.
  • Because open data encourages government transparency and public participation in decision making.
  • Because producing better, more reliable data will increase the demand for data.
  • Because new technologies will allow developing countries to skip ahead, realizing the benefits of new sources and methods.
  • Because new applications using better data will create new services and business opportunities.

Endnote 

1. The United Nations Secretary-General’s Independent Expert Advisory Group on a Data Revolution for Sustainable Development (IEAG) (2014), A World That Counts, United Nations, page 6. Available at http://www.undatarevolution.org/report/

2. See http://www.dataimpacts.org/project/malaria/

3. See http://www.dataimpacts.org/project/mapping-commuters-streamlines-transit/

4. See http://www.dataimpacts.org/project/mobile-phonedata-optimize-bus-routes/

5. World Bank (2015). World Development Indicators database. The data are from the International Telecommunications Union’s ICT database. Data are available at http://data.worldbank.org/data-catalog/worlddevelopment-indicators.

6. IEAG (2014), page 6.

7. IEAG (2014), page 8.

8. IEAG (2014), page 21.

9. Jessica Espey et al., 2015, Data for Development: A Needs Assessment for SDG Monitoring and Statistical Capacity Development, Sustainable Development Solutions Network. Available at http://unsdsn.org/resources/ publications/a-needs-assessment-for-sdg-monitoring-andstatistical-capacity-development/.

10. See https://www.gov.uk/government/publications/opendata-charter/g8-open-data-charter-and-technical-annex

11. A consultation draft is available at http:// opendatacharter.net/ ​