Better Measurement and Monitoring
of Data for Development
by
Jamison Henninger
and Eric Swanson
Open Data Watch
20 April 2022
Many tools exist that measure the capacity and outputs of statistical systems, but little work has been done on how they differ from or complement each other. What questions do these tools answer and do they provide consistent assessments? A new report by Open Data Watch and the D4D.net, with support from the International Development Research Centre (IDRC), compares 12 indexes and assessment tools measuring the capacity of national statistical systems. The indexes and tools are mapped to the pillars of the new Global Data Barometer and the stages of the Data Value Chain to reveal similarities and differences. The report concludes with a list of recommendations for current and future statistical capacity indexes and tools.
The indexes and tools included in the report are the Data Quality Assessment Framework, European Open Data Maturity Assessment. Ibrahim Index of African Governance, Joined-Up Data Maturity Assessment, Open Data Demand Assessment, Open Data Inventory, Open Data Maturity Model, Open Data Readiness Assessment, OURdata Index, Statistical Performance Index, Use of Statistics Index, and Worldwide Governance Indicators.
The report has several major findings, three of which are summarized below.
Most indexes and tools measure data availability, but not data use and impact.
The indexes and tools are constructed from indicators that measure attributes of national statistical systems. Their indicators were mapped to the four pillars of the Global Data Barometer (GDB) and, in a separate exercise, to the stages of the Data Value Chain. The GDB pillars are data governance, data capabilities, data availability, and data use and impact. Not all indexes and tools were designed or intended to cover the same topics as the GDB but understanding where overlaps occur reveals how they complement each other. Table 1 shows the distribution of indicators across the four pillars of the GDB.
Table 1. Indicator coverage across GDB pillars (% of indicators in each pillar)
| Data Governance | Data Capabilities | Data Availability | Data Use and Impact | |
| Performance Indexes | ||||
| Open Data Inventory | 10% | 0% | 90% | 0% |
| OURdata Index | 42% | 38% | 17% | 4% |
| Statistical Performance Index | 6% | 6% | 76% | 12% |
| Ibrahim Index of African Governance | 20% | 20% | 60% | 0% |
| Worldwide Governance Indicators | 0% | 0% | 100% | 0% |
| Use of Statistics Index | 0% | 0% | 0% | 100% |
| European Open Data Maturity Assessment | 37% | 20% | 5% | 37% |
| Assessment Tools | ||||
| Data Quality Assessment Framework | 14% | 5% | 81% | 0% |
| Open Data Readiness Assessment | 22% | 72% | 6% | 0% |
| Joined-Up Data Maturity Assessment | 37% | 25% | 25% | 13% |
| Open Data Demand Assessment | 0% | 19% | 12% | 69% |
| Open Data Maturity Model | 25% | 67% | 0% | 8% |
None of the 12 indexes and tools provide equal emphasis on all four pillars, but the pillar with the greatest emphasis across (nearly) all measures is data availability. Data use and impact receives the least attention. Five indexes and tools don’t measure data use or impact at all, and three others have only a low emphasis on this pillar. Measuring data use and impact is a particularly challenging area of research. The World Bank’s Statistical Performance Index (SPI), for example, specifies five dimensions of data use, but all but one are still under development (which is why Table 1 shows just 12 percent of SPI indicators fall under data use and impact).
Only a few tools assess the availability of disaggregated data.
Though nearly all indexes and tools measure data availability across countries, very few focus on the availability of disaggregated data needed to measure differences in gender, age, ethnicity, or other characteristics of vulnerable populations.
In the report, each index and tool is mapped to the four stages of the Data Value Chain (data collection, data publication, data uptake, and data impact). The data publication stage was further broken down into data availability, data quality, data disaggregation, and data openness and accessibility to see which aspects are more commonly measured.
Table 2. Examples of indicators found in the data publication stage
| Common Indicators | Less Common Indicators |
|
Data availability
Ex: Availability of specific indicators Data quality
Ex: Adherence to international guides or use of internationally accepted classification systems |
Data disaggregation
Ex. Availability of indicators by sex, age, disability status, and other characteristics Data openness and accessibility
Ex: Availability of data in machine-readable and non-proprietary formats made available free of charge Availability and quality of metadata
Ex: Comprehensiveness of metadata, Usage of internally accepted standards for metadata dissemination |
As Table 2 shows, there is a lack of attention to disaggregated data, data openness and accessibility, and the availability and quality of metadata. Only two indexes include an explicit requirement for disaggregation – the Open Data Inventory (ODIN), which requires key disaggregations for each dataset, and the SPI, which covers the availability of data to measure the Sustainable Development Goals.
Gaps in country coverage limit the usefulness and comparability of indexes.
The assessment tools can be used for self-evaluation, but performance indexes are most useful when they provide cross-country comparisons. Four of the indexes provide global coverage, while the other three provide scores for a limited set of countries, focusing on high-income countries, or in the case of the IIAG, African countries. The selective, regional coverage of these indexes limits comparisons with other indexes, as well as the ability of countries to use the indexes in complementary ways.
Country coverage of the global indexes has increased in recent years. In 2015, ODIN provided scores for only 125 low- and middle-income countries before including high-income countries in 2016 for a total of 173; in 2022, ODIN will include 194 countries. The Use of Statistics index included 69 countries in 2019 but expanded to 174 in 2020.
Read the full report here or view the printable PDF version here.
.
The Global Data Barometer, released on 11 May, is a welcome addition to the existing suite of indexes and assessment tools. We will continue to track how indexes and assessment tools measure countries’ data systems and statistical capacity in future revisions and whether they address some of the shortcomings — such as better measurement of data use and impact – noted in the report.
