This study was produced by Open Data Watch with financial support from the William and Flora Hewlett Foundation.
Summary
As plans for the Sustainable Development Goals coalesce, it becomes increasingly clear that data are a crucial piece to solving the puzzle of poverty eradication and sustainable well-being. Data are essential for tracking progress towards goals. A strong statistical system allow stakeholders to harness data and achieve goals with ever greater effectiveness and coordination. However, statistical systems in many countries are not yet where they need to be, nor do they have the resources needed to increase their capacity. The UN Secretary General’s Independent Expert Advisory Group for the Data Revolution for Sustainable Development report, A World That Counts, calls for a new funding stream for statistical finance and a new global partnership for statistics (Independent Expert Advisory Group 2014). To understand how a partnership can mobilize and coordinate efforts around strengthening statistics, this report offers lessons learned from 27 evaluations of statistical capacity programs. Based on these findings, we make a broad set of recommendations for future statistical capacity building in developing countries.
In quick summary they are to strengthen capacity and use of statistics, to improve effectiveness and efficiency of statistical systems, and to harness the benefits of robust evaluation. Equal focus must be on the demand for data as on their supply. Collective efforts should identify user demands, support mechanisms to consult with users, and widen the scope of official statistics, such as pulling in “big data.” Avoiding duplication of effort through improved coordination, developing standards and techniques to make use of untapped data, and making statistics a core component of the post-2015 agenda will help a collective effort promote effectiveness and efficiency.
Independent, external evaluations provide lessons of widespread value. Evaluations must be part of the initial program design, not just a formal afterthought. This report echoes Willoughby’s recommendation from 2008 for an evaluation database, including design documentation and reports. Through the consideration and implementation of these recommendations, efforts to meet the Sustainable Development Goals will be more on par with the magnitude of their ambition.
Background
The need for data to support development and poverty reduction has been widely acknowledged. At the time of the Millennium Summit in 2000 and the launch of the Millennium Development Goals (MDGs) it was clear that much more and much better data would be needed if the challenge of monitoring development and delivering real and sustained poverty reduction was to be met (Short 1999). While the capacity of the world to provide data on development has undoubtedly improved, much remains to be done (Independent Expert Advisory Group 2014).
The Millennium Development Goals are scheduled to end in 2015. The United Nations will report on the progress that has been made, both at the global level and, more importantly, in developing countries. At the same time, there will be international agreement on a new set of Sustainable Development Goals. While discussions about the goals are ongoing, it is already clear that data will continue to be both essential and a major challenge for many aspects of development. There is great potential for making use of new kinds of data and new technology, but if this potential is to be realized, a real and effective data revolution will be needed (A New Global Partnership 2015).
Delivering a data revolution commensurate to the goals of sustainable development and poverty eradication will not be easy. New sources of data and new methods of analysis can overcome old obstacles and fill outstanding gaps. However, new resources and new partnerships and significant efforts to strengthen and improve national statistical systems are also required (PARIS21 2015). These systems will remain central to the challenge of producing reliable public data needed to build, monitor, and sustain development programs.
Before committing new resources to data production and use, it will be important to learn the lessons from what we have achieved so far. This study contributes to this effort by reviewing and synthesizing the lessons learned from statistical capacity building programs in developing countries. It reviews and analyses the results of formal evaluations and other published assessments of a number of internationally supported statistical capacity building programs and initiatives from 2000 to 2014.
This period has been selected for two reasons. First, it covers the period of the MDGs, which drew critical attention to development statistics and gave rise to efforts to improve the performance of national statistical systems. During this period, as outlined in Section 3 below, a number of important new programs were put in place, and their focus changed from data availability per se to supporting institutions and strengthening capacity. And, second, it provides a manageable set of evaluations and other reviews that are readily accessible. While there were a number of statistical capacity building programs and projects in place prior to 2000, not all were fully evaluated and documents from that time are more difficult to locate and access.
The purpose of the study, therefore, is to use these evaluations, reviews and other assessments to gain an understanding of what the various programs and initiatives were aiming to achieve, their impact, their successes, and their failures.
Approach
This study comprises 27 evaluations from 2000-2014. The selection include broad syntheses, external evaluations, and internal reviews from key international organizations such as the World Bank (WB), International Monetary Fund (IMF), United Nations (UN), and others. There are three syntheses, thirteen external evaluations, and eight internal reviews. The specific evaluations in this study are based on our knowledge of major statistical capacity building programs. In some cases, organizations have provided information through email correspondence with Open Data Watch staff. Otherwise, information is based on publications available on organization websites.
Summaries were prepared for each evaluation, highlighting its methods, findings, and recommendations. Each evaluation was classified as either a synthesis or an external or internal review. The number of evaluations in each synthesis and the number of programs in each evaluation were noted. Greater credence was given to findings of syntheses representing multiple, external evaluations than to internal reviews. Summaries of each evaluation are available in the Annex.
The sample of evaluations in this study is strong, but not necessarily representative. Due to the limited scope of this study, we have not undertaken a full review of all available evaluations to determine whether those included in this report represent the broader scope of evaluation. It was not possible, for example, to carry out a formal meta-analysis bringing together results from the different studies into a consistent data set. In the future, a more exhaustive study may be carried out which could include a more robust analysis of the full scope of available evaluations. The data set we have identified, however, does cover the most important programs and initiatives and a major part of international expenditure on statistical capacity building.
While the data set we have compiled includes 27 separate evaluations, with syntheses representing even more, an examination of the information in the Annex indicates that a relatively small number of individuals were involved in a number of them. To a considerable extent this demonstrates the difficulty of ensuring a genuinely independent evaluation. Many of the programs were quite large in scope and involved a substantial number of international statisticians, either as providers or recipients. The pool of qualified people available to carry out an evaluation, therefore was limited and people who were involved in the design and implementation of one program may well have participated in the evaluation of another. This can mean that issues that cover more than one program are well identified, but it may also reduce the effective independence of the evaluation itself.
It is also important to take into account the limitations of the various evaluations and reviews. With few exceptions, they were commissioned by the agencies responsible for the design and implementation of the different programs and initiatives. Even where an independent, external evaluation was carried out, because of limitations of both time and cost, not all of them followed the OECD standards for development evaluations (OECD DAC 2010). At the same time, some of the programs were not easily amenable to formal evaluation in that baseline data were not readily available, a formal theory of change had not been developed, and it was not obvious what the counterfactual should be. Few if any of the evaluations, therefore, provide much insight into the impact of the program. They do, however, provide useful information on what was done, whether or not the outputs were delivered and some information on what outcomes were achieved. As such, we believe they provide useful information to support the design of future programs and projects to strengthen statistical capacity in the post-2015 era. They also help to identify what worked well and where problems were encountered. We hope, therefore, that this report provides a useful background to discussions about strengthening statistical capacity from 2016 onwards.
Statistical Capacity Building from 2000 to 2015
The Launch of the Millennium Development Goals
The Millennium Declaration agreed in New York in September 2000, in addition to launching the Millennium Development Goals, represented a new approach to development and poverty reduction. It was clear that there was a need not only for new resources, but also a new way of doing business where countries took the lead and results could be demonstrated. The international statistical community had already recognized the need for a more effective process to strengthen statistical capacity in developing countries with the launch of PARIS21 in 1999. The MDGs, with their requirement for data to monitor progress, coupled with the need for countries to prepare poverty reduction strategies to qualify for debt relief, suddenly increased the demand for a wide range of statistics. It was also clear, however, that in many countries statistical systems did not have the capacity to respond (Eele et al. 2009).
The initial period of the new millennium – from 2000 to 2004 – therefore, focused on identifying needs, mobilizing resources, and understanding why previous efforts at improving statistical capacity had had only a limited impact (Short 1999). At the same time, the international community more broadly addressed the issue of aid and development. A series of meetings on financing for development, starting in Monterrey in 2002, advanced the idea of a compact between donors and developing countries. This aimed to increase aid, provided that recipient governments were committed to promoting good governance, to implementing effective policies, to achieving results, and to monitoring progress. These principles were eventually set out in the Paris Declaration on Aid Effectiveness in 2005 (OECD 2005).
The Marrakech Action Plan for Statistics (MAPS)
At the second International Roundtable on Managing for Development Results in 2004 (one of series of meetings to follow up the Monterrey Conference), the Marrakech Action Plan for Statistics was agreed. This presented six action areas with specific targets and timetables, three focusing on countries and three at the international level. The most important was to support countries to prepare and implement national strategies for the development of statistics (NSDSs). The second phase of the post-2000 period, therefore, which lasted from 2005 to around 2009, was concerned with this process, focusing on helping countries to identify their statistical priorities and to develop coherent strategic plans. Support was provided through PARIS21 and from resources such as the World Bank’s Trust Fund for Statistical Capacity Building (TFSCB), established in 1999.
At the international level MAPS supported specific activities to strengthen statistics in key areas such as participation in the 2010 round of population censuses. It also set up the International Household Survey Network, which focused on the coordination, management, documentation, and archiving of household surveys.
Implementation – Building Capacity
While efforts to help countries to develop and to improve the quality of their statistical strategies continued, the third phase of international action, supporting their implementation really began in earnest from about 2008 onwards. The process of developing strategies was found to be both effective and flexible, but implementation was always seen as the main objective (PARIS21 2009). The Dakar Declaration on Statistics, approved in 2009, and the Busan Action Plan on Statistics (BAPS), agreed in 2011, both focused on the need to move from the preparation of strategies to their implementation. A number of international initiatives, including the World Bank’s multi-country Statistical Capacity Building Program (STATCAP), established in 2004, provided improved access to international aid for investment in capacity strengthening.
Review of the Evaluation Findings
In this section we present our findings based on an analysis of the 27 evaluations and other assessments listed in the Annex. We give greater weight to those evaluations that either cover more than one program or were carried out in line with the OECD DAC evaluation guidelines.
We discuss first those features of the programs, partnerships and initiatives that seem to have been more effective. The second part looks at challenges, concerns, and issues.
Characteristics of Effective Programs, Partnerships, and Initiatives
Adhering to Paris Declaration Principles
The Paris Declaration on Aid Effectiveness provides a roadmap to improve the quality and impact of aid. The principles stress the importance of the ownership of aid programs by developing countries, the need for donor countries to align their support with national objectives and priorities, the importance of donors harmonizing their procedures and the need to focus on results. A thematic study, carried out for DFID in 2009, looked at what approaches to statistical capacity building had been most effective and why. The study, which was based on a number of case studies and extensive research, concluded that the key success factors were: (i) a focus on the users of statistics and their needs; and (ii) the explicit recognition of the Paris Declaration principles in the design of capacity strengthening programs (DFID 2009). The study concluded that aid programs that focused mainly on the short-term production of data could have a short-term impact but that this was often not sustained. In the examples looked at, aid for statistics delivered through more comprehensive programs providing coordinated and predictable support was thought much more likely to be successful.
Country Ownership of Programs
Many of the evaluations reviewed here identify country ownership of programs as a key success factor. In the Annex, evaluations 3, 5, 13, 15, 22, 26, and 27 provide examples of this. It is important, however, to be clear about what country ownership actually means in practice. The 2009 DFID study emphasized that ownership means more than mere consent. A 2010 IMF synthesis report concluded that ensuring flexibility in the design of a program is important to accommodate changing government priorities. A 2014 Statistics for Results Facility Catalytic Fund (SRF-CF) evaluation and a 2003 World Bank Trust Fund for Statistical Capacity Building evaluation, found promoting a continuing dialogue between users and producers of statistics was important in developing and maintaining country ownership of the process.
Alignment or Local Relevance
The importance of ensuring that statistical capacity building programs are closely aligned with national priorities was widely stressed. For example, a 2011 World Bank synthesis evaluation covering MAPS, PARIS21, and TFSCB concluded that the impact of all three partnerships was significantly increased because they had highly relevant objectives and were well aligned with local priorities. The 2010 IMF synthesis report also attributed positive outcomes to the fact that the activities promoted country ownership and were responsive to the needs of the participating countries. The two evaluations of the World Bank’s TFSCB found that its relevance and adaptability to country needs and circumstances were important success factors.
It is more difficult to align international programs such as the International Comparison Program (ICP) and the UNICEF Multiple Indicator Cluster Survey (MICS) with individual national priorities. Inevitably, international programs such as these are concerned with building capacity and meeting data needs mainly at the international and regional levels. Evaluations of these two programs, however, suggest that providing sufficient flexibility to meet country needs to some extent is an important consideration to be taken into account in the design phase. See evaluations 16, 17, 18, and 19 in the Annex for examples of this.
Coordination
The Paris Declaration calls for donor countries to harmonize their procedures and to coordinate activities at both the country and the international levels. Coordination at the international level, as discussed below, seems to be more problematic, but there is some evidence that time and resources devoted to coordination in countries has substantial benefits. For example, the 2009 thematic study by DFID concluded that the presence of a lead donor with statistical expertise improved coordination among donors supporting statistical activities. The 2011 World Bank synthesis evaluation of three international partnerships showed that coordination is important for moving the statistical capacity building agenda forward and the evaluation recommended that different approaches to this issue should be properly documented and widely shared.
Efficiency and Effectiveness
According to the OECD DAC criteria for development evaluations, efficiency measures the outputs produced relative to the inputs used, while effectiveness is a measure of the extent to which a program or initiative achieves its objectives. Formal evaluations that were based on the OECD guidelines all reported on both efficiency and effectiveness. In general most of the factors contributing to both were mainly specific to the program design. A 2013 evaluation of the International Household Survey Program concluded that the outputs were generated efficiently because of the light governance structure. This did, however, have other implications that limited the involvement of recipient countries in the governance structure. There may also be some trade-offs between efficiency and effectiveness. For example, the TFSCB evaluations concluded that the operations of the Trust Fund were cost-efficient, though the relatively small size of the TFSCB grants may have limited cost-effectiveness. Similar concerns may also apply to the IMF’s Regional Technical Assistance Centers (RTACs). They were found to be relatively lean organizations, but the focus on technical assistance precluded other forms of support being provided (IMF 2010).
Impacts and Sustainability
Sustainability is concerned with the extent to which the results of a program are likely to continue once funding ends. In a number of the programs, one of the aims of the evaluation was to support an extension or possible expansion of the activities. In these circumstances it is perhaps not surprising that the extent to which outputs and outcomes will be sustained is not always clear. Another concern is the time period over which the impact of the program was thought likely to occur. The DFID thematic study in 2009 by Strode and MacAuslan concluded that one concern with measuring impact and sustainability was the definition of statistical capacity building itself. Without a clear definition of what capacity building is meant to achieve and without clear and measurable targets, identifying to what extent the impact of a program will be sustained is not really possible. While a number of the evaluations consider that specific outputs have been or will be sustained, assessments of the sustainability of the higher level outcomes and impacts have been very limited.
Challenges to Address
Strengthening the Demand for and the Use of Statistics
Many of the international partnerships, programs and other initiatives focusing on statistical capacity building have been concerned with supporting improvements in the operations of statistical systems with a view to improving the quality and availability of statistical indicators and data series. This is perhaps not surprising as the case for the interventions has been built on an analysis of data gaps, where data for specific purposes are identified as being inadequate or, indeed, missing. Following the launch of the MDGs and the international program to provide debt relief in the first decade of the new Millennium, it was clear that there were many data gaps, at both the national and international levels. The need for data for the MDG indicators as well as better statistics to support poverty reduction strategies was clear. The response of the international community was to put in place initiatives to address this.
A number of the evaluations, however, especially those concerned with synthesizing the lessons from programs or projects, have suggested that less attention has been given to how these data were being used. For example, a 2008 inventory of evaluations of different statistical capacity programs carried out for PARIS21 concludes that there was limited impact on the use of statistics in countries. It cites a large evaluation carried out by the European Commission over the period from 1996 to 2005, covering more than 30 projects, which concluded that “… few explicitly addressed the awareness of the statistical importance for evidence-based decision-making.” The report concludes that this is related to the interface between the producers and the users of statistics and illustrates the contrasting mind-sets of national policy makers and the managers of statistical systems.
This concern is also raised in a number of the other evaluations reviewed. The 2009 DFID thematic study concluded “While support to the production of statistics has increased, the link between production and use in country is still far too weak.” The high-level evaluation of partnership programs at the World Bank concluded that “Statistical capacity building programs need to involve the users of statistics more actively” and a 2013 evaluation of the International Household Survey Network and Accelerated Data Program found “… the creation of survey catalogues has not necessarily led to increased survey data use.” On the other hand an evaluation of PARIS21carried out in 2009 found that “Improved capacity to produce, analyze and use statistics in countries has been observed.”
Some evidence of increased access and use of data has been seen within aid agencies and development banks as a result of different capacity building programs. For example, internal reviews by the Inter-American Development Bank (2009) and the Asian Development Bank (2002 and 2003) indicate the importance of data for their research and policy analysis, while UNICEF reports the importance of the MICS data (John Snow Inc 2009). Because large-scale survey programs are mainly financed by donors, there is a clear concern that the drivers behind these programs may focus more on data availability and use at the international level rather than in countries.
Delivering Resources at the Country Level
A second area of concern identified in the evaluations relates to the design and effectiveness of large-scale capacity building projects and programs at the national level. Since about 2009 the focus of many projects has moved from preparing strategic statistical plans to implementing them. Evaluations of some of these projects reveals that implementation is not always a straightforward process. For example, the internal end-of-project review of an early World Bank STATCAP project in Kenya rated the overall outcome as moderately unsatisfactory mainly because “the project never overcame issues related to financial management, procurement and data dissemination which led to low disbursements and slow pace of implementation.” A different approach tried in the STATCAP project in India, which disbursed funds as budget support, was able to overcome the problems of low disbursement, but the evaluation concluded that this would only work in countries with relatively well-developed and sophisticated budget management processes.
The problems encountered in ensuring the success of large-scale projects supporting the implementation of statistical strategies and plans led to the establishment of the Statistics for Results Facility at the World Bank, where specific arrangements were made to promote internal discussion and to provide technical support on the ground as the project was prepared (Ngo and Flatt 2014). However, one consequence was that the time required to prepare and deliver targeted support was substantially increased. It will be important to ensure that the lessons that have been learned over the past 15 years on how to deliver effective assistance to national statistical systems are incorporated into programs being planned for the post-2015 era.
Global Statistical Public Goods
In addition to the provision of financial resources and technical advice to strengthen the capacity of national statistical systems, the various partnerships and programs included in this analysis have also supported the development of global statistical public goods. These have included the compilation of consistent global data sets with a particular focus on the MDG indicators, the development of standards, recommendations of good practice, the provision of technical advice in different areas of statistics and carrying out international data collection exercises, including the International Comparison Program and the Multiple Indicator Cluster Surveys (MICS). In general, the evaluations suggest that these programs have been both effective and reasonably efficient, although sustainability is potentially more of a problem.
An important conclusion from the evaluations of global programs is that the impact is enhanced when efforts to introduce new methods or new technology are accompanied by the provision of technical support and advice to countries that want to adopt them. The availability of technical advice and support was considered to be an important success factor for both the ICP, according to 2007and 2015 evaluations, and the IHSN; it was also seen as important for MICS. All three of these programs, however, are very dependent on external financing. No viable business model has yet been put forward to ensure that they can be continued in the future without continued donor funding.
Of great importance for the future is whether the international statistical system will support the research and development needed to ensure that national statistical offices are able to innovate and take advantage of new technology. Few statistical systems in developing countries have the capacity or the resources to undertake research and development. PARIS21 has called for a much greater effort in this area through its 2009 Dakar Declaration and 2015 Road Map. Another concern emerging from the evaluations is the extent to which innovation is led by the needs and the agenda of international agencies or by the needs of countries themselves (Thomson et al. 2013).
Coordination at the International Level
A number of international initiatives have aimed to encourage greater coordination at both the international and country levels. This was one of the reasons for the creation of PARIS21; it was also one of the mandates of IHSN. In both cases, however, as the evaluations demonstrate, this has not really occurred. International agencies are responsible to their own management structures, not to any international process. While they may be willing to participate in mechanisms such as the MAPS partnership, they are not accountable to it. To a large extent coordination really has to happen at the country level and this has been the focus of recent initiatives such as the SRF.
Our Recommendations
We offer here some conclusions and recommendations that are intended to contribute to the discussions of the sustainable development goals and the post-2015 development agenda. Understanding what has been achieved in the period since 2000, identifying what has worked well, and understanding where and why problems have occurred will be crucial. We focus on three specific areas. First the design and delivery of programs that are specifically concerned with strengthening the capacity of national statistical systems and the use of statistics in developing countries. Second, actions to support and improve the efficiency and effectiveness of the international statistical system. And, third, how best to ensure that future programs, projects and partnerships are properly evaluated so that lessons can continue to be learned and applied more widely.
Strengthening statistical capacity in developing countries
It is clear that, whatever the final specification of the Sustainable Development Goals that will be approved in September, national statistical systems – financed and managed by the governments of all countries – will be central to the data effort underpinning the post-2015 development agenda. To fulfill their commitments to the sustainable development goals, developing countries will need to make substantial investments in their statistical systems and many will need external assistance to do so.
The lessons from earlier efforts to support capacity building, summarized in the synthesis report for DFID prepared in 2008 remain as pertinent and apposite in 2015 as they were then. Key elements, set out in eight dimensions or pillars, are identified as:
- Strengthening the focus on results and the demand for statistics at the top of government;
- Strengthening the accountability of statistical agencies and the statistical system as a whole to government and to users more generally;
- Ensuring an effective legal and institutional framework for the national statistical system that promotes coordination and ensures accountability;
- Having in place an agreed, nationally owned, feasible, medium-term strategy for the development of the national statistical system and effective mechanisms for work planning and budget management;
- Providing predictable financial resources for the operations of the statistical system over the medium-term;
- Ensuring good management and leadership of statistical agencies and the statistical system as a whole;
- Providing enough staff with the right qualifications and skills and ensuring that these continue to be updated; and
- Ensuring that the statistical system has access to the tools, technology resources and infrastructure it needs to operate effectively and making sure these are maintained and updated.
To these requirements, based on our analysis of the more recent evaluations, we would add the following:
- There is a need to provide support and advice during the process of preparing project and program proposals, especially in countries with relatively weak and inexperienced statistical systems, where the need for external support is the greatest. As the evaluation of the SRF indicates, the role of an in-country statistician provided by a lead donor can be very important and useful.
- It is also important to provide sufficient time for effective internal discussions during the preparation phase. Establishing appropriate mechanisms for consultation and coordination involving all the main stakeholders seems to be an important pre-condition for a successful program.
- New ways of providing assistance, such as system-wide approaches, basket funds, and budget support are needed to help countries manage externally funded programs. This seems to be especially important in areas such as financial management and procurement when countries are required to deal with several donors simultaneously. Establishing mechanisms that can be used by all donors is an important way of reducing the transaction costs of recipient governments.
- Donors and international agencies need to work effectively together to integrate and coordinate their financial aid and their technical assistance.
- Implementation processes need to be flexible and allow for changes in plans to be agreed when required. Statistical systems need to be able to respond to changes in the demand for statistics.
The need for capacity building programs to focus as much on the internal demand for data as on their supply has been identified as an issue for some considerable time. In practice, however, this concern has been more difficult to put into practice. Where countries are dependent on donor financing for investment in capacity there is always a danger that it is the demand from these agencies that takes precedence over often poorly articulated needs from within the country. One lesson from the evaluations is that supporting mechanisms to promote real and sustained consultation with users is likely to be important in this regard. Assistance to identify demand from different users, especially those who may have only limited statistical knowledge is likely to be needed.
There is also considerable potential to bring in new players, including both providers and users of data which, until now, have had only limited interaction with national statistical systems. The benefits of widening the scope of official statistics, including making more and better use of “big data” are potentially huge; the problem is that guidance, support and advice are not readily available to countries and they do not have the space or resources to innovate.
International and regional programs
Over the past 15 years and in the post-2015 development world, the international statistical system has had and will continue to have several important roles to play. These include: the mobilization of financial resources and the provision of technical advice; the development and implementation of standards and good practice in all areas of statistics; compiling and maintaining international databases to provide comparable and consistent data for countries on a wide variety of topics, including the SDG targets and indicators; and organizing and carrying out large scale international data collection exercises such as the ICP. Experience from the evaluations suggests that one further role may well be to develop and promote technical and technological innovations for the operation of official statistical agencies and the management of statistical systems. The returns to research and innovation can be high, but the capacity of small and underfunded statistical systems to carry out their own research and development is very limited.
In general, the evaluations of international programs that are included in our review show fairly positive results. Some concerns were raised however, about the extent to which developing countries were able to contribute to the design and the timing and coordination of activities. For the future, one concern is likely to be how the management and coordination of these programs can be set up to strengthen the voice of developing countries, both in setting priorities and in monitoring progress.
While there is clearly a need for international agencies to be involved in research and development of new techniques and technologies for statistics, the evidence from the evaluations suggests it will be important to avoid duplication of effort. In the past there has been some concern about the duplication of software tools for different statistical purposes, especially for managing databases and survey programs. Presenting countries with competing, or at least, overlapping tools, with little or no capacity to assess their usefulness or otherwise, is unlikely to be either helpful or effective (Thomson et al. 2013).
There is also a need for work on the development and implementation of standards and good practice across the whole of the statistical space to be continued and expanded. An area that is likely to be of high priority is the development of standards and techniques for making use of data from new and so far largely untapped sources. The evaluations show that these efforts seem to be most effective when they are supported by related technical assistance and advisory services and when countries have been included in setting priorities.
Developing a sustainable business model for international statistical activities remains a concern. Investment in global public goods has been and remains, problematic (UNSC 2015). At present many programs and other initiatives are dependent on short-term donor funding and this is likely to limit impact and sustainability. The need for these activities is clear and is likely to increase and new ways of financing are needed urgently. There is a strong case for including discussion about statistics and global programs as a core component of the post-2015 development agenda.
Annex: Evaluation Summaries
Asian Development Bank
1. Asian Development Bank. Technical Assistance Performance Audit Report on Selected Technical Assistance for Agricultural Planning and Statistics in Nepal. (2002).
The internal review draws on Technical Assistance (TA) completion reports (TCRs). The implications for statistical capacity building in the review are limited to a note that initial assumptions about the use of the data were too optimistic. The agriculture statistics project unfolded with the Central Bureau of Statistics, and was partly successful in terms of impact and sustainability. This was due to the lack of use of the improved statistics from the crop and livestock survey, whereas the estimated likelihood of their acceptance by the Government was high at the time of the report. The Bureau developed and conducted a major crop and livestock survey, and all staff received effective training. Thus, the report concludes that the intended capacity building to produce agriculture statistics took place.
2. Asian Development Bank. Technical Assistance Performance Audit Report on Selected Technical Assistance for Development Planning and National Statistics in Cambodia. (2003).
The internal review draws on TCRs. Most of the review focuses on issues other than statistical capacity building; however, the report notes that sustainability is a concern in Cambodia. The key findings are that the project was successful in consolidating Cambodia’s National Institute of Statistics’ (NIS) capacity, while focusing on economic statistics. The Asian Development Bank played a crucial role in making the NIS stronger over the years. Capacity increased, although insufficiently for the NIS to operate on its own without further technical support and finance for the costs of regular and survey-related data collection. The TA did not succeed in regularizing government budgets for operational costs of surveys and some continuous data collection. The Asian Development Bank’s TA changed from providing a wide range of assistance in the early years to a more narrow focus. Given the increasing support from other external agencies in social statistics and household expenditure surveys, the review considered this appropriate. In spite of the institution building, however, the report states that the future of economic statistics is uncertain without further external support.
Department for International Development (DfID)
3. Department for International Development. Evaluation of the Implementation of the Paris Declaration: Thematic Study of Support to Statistical Capacity Building: Synthesis Report. (2009).
The evaluation synthesis draws on a literature review and analyses of support to statistical capacity building in five countries, including Bangladesh, Burkina Faso, Liberia, Rwanda, and Tanzania. It also relies on field-work from three country case studies, including Zambia, Niger and Cambodia, and from the Swedish International Development Cooperation Agency (Sida) and DFID partner case studies. Overall, the study identifies two key drivers of successful support: 1) an explicit recognition of who the users of statistics are; and 2) an explicit recognition of the Paris Declaration principles in the design of capacity strengthening programs. According to the report, these two drivers must be addressed to design and deliver effective programs that not only meet the immediate needs of data users, but also generate sustained improvements in statistical capacity.
The report states that the needs of the many different user groups means that country ownership is difficult to establish, which is vitally important. In most cases where the Paris Declaration principles were followed, the results of support to statistics have improved. They were most successful when delivered within larger programs of predictable, coordinated support. Larger scale country based programs (particularly country-held common funds) seemed more likely to meet them. Results-focused governments were more likely to support statistical systems that largely met the principles.
Other findings were on donor coordination, governance, accountability, and effectiveness, among other things. In terms of coordination, as experience in Tanzania shows, the presence in country of a lead donor with expertise in statistics improved donors’ coordination of support to statistical capacity building. Management at a global level makes these initiatives particularly difficult to administer, because they are neither owned by individual countries nor well aligned to their statistical priorities, institutions or procedures. According to the report, accountability of the statistical system to government and an appropriate institutional context are key to ensuring good performance of statistical systems. A focus on results-based management at the highest level of government is the most important factor that the report identified, and is a necessary condition for statistics to thrive in any country. Qualified and suitably trained staff are the bedrock of the statistical system. Appropriate methods and tools are also a necessary input to statistical capacity.
Food and Agriculture Organization of the United Nations (FAO)
4. Dunmore, John and Jan Karlsson. Independent Evaluation of FAO’s Role and Work in Statistics. (2008).
The external evaluation draws on structured interviews with stakeholders, surveys of FAO data users, and thematic studies on FAOSTAT, Fisheries Statistics, Forestry Statistics, Statistical Information Management and Dissemination, and Information Technology for Statistics. The purpose of the evaluation was to assess work in this area from the point of view of clients and users. The evaluation made three main observations: (i) the continued deterioration in member country statistical collection capacity, (ii) the reduction of resource levels and staff capacities below a critical mass for many technical support activities in FAO’s statistical units, and (iii) the need for FAO to generate “estimates” for the production statistics of nearly 70% of the African countries. These observations led to the main conclusion: FAO’s basic statistics program is crumbling.
Other key findings were that most users valued FAO data for use in their work and for decision making. The statistics user community put a certain amount of confidence in the FAO Statistics Program as a global unbiased body of reliable and relevant statistics. Results from the National Statistics Office (NSO) survey indicated that FAO was generally responsive to the statistical needs of Member States, and that FAO’s assistance led to a strengthening of permanent statistical capacity. However, there is anecdotal evidence that national statistical capacity, particularly for agricultural statistics, deteriorated. This happened due to a lack of donor interest in capacity building and a consequent decline in priority and resources at the national level.
International Household Survey Network (IHSN)
5. Thomson, Anne, Graham Eele, and Felix Schmiedling. Independent Evaluation of the International Household Survey Network (IHSN) and Accelerated Data Program (ADP). (2013).
The external evaluation draws on document reviews, interviews, case studies, and surveys. Overall, both IHSN and ADP achieved considerable success. They developed software tools for data archiving and setting up National Data Archives (NADAs). ADP helped train NSO staff in their use and in effective survey archiving. Despite continued legal, political, and technical barriers in some countries, the report shows they made progress towards improving access to microdata. However, challenges still remain to national and global cooperation.
The evaluation presents findings regarding the relevance, efficiency, effectiveness, impacts, sustainability, and governance of IHSN/ADP. The report states that it was relevant to fulfilling the MAPS recommendations. It filled a niche, and fulfilled a good part of their assigned activities. However, the two programs have not addressed improved coordination of internationally sponsored survey programs or improved collaboration between data producers and users. The programs showed efficiency in achieving most outputs, although some ADP activities started recently and had yet to yield significant outputs. Its effectiveness remains unclear as the creation of survey catalogues had not necessarily led to increased survey data use. Its impacts include improved data availability through the IHSN catalogue and assisting countries to establish NADAs. According to the report, the sustainability of its outputs depends on programmer time and institutional commitment for the first and the capacity and commitment of NSOs for the other two. Its governance limited participation opportunities of partner organizations, with no formal monitoring and evaluation system, and limited oversight outside the two implementing agencies.
International Monetary Fund (IMF)
6. International Monetary Fund (IMF) Statistics Department (STA). Evaluation of the U.K. DFID-Financed Technical Assistance GDDS Project for Selected Anglophone African Countries (2001-06). (2008).
The internal review draws on internal and periodic evaluations of project execution and reports by technical assistance providers. According to the report’s findings, the project provided an opportunity to strengthen delivery and coordination of TA to a group of countries trying to build statistical capacity. New challenges emerged through the requirements imposed by external funding and coordination with other Fund departments, the World Bank, and regional TA providers. The STA and DFID funding greatly increased STA’s leverage and reach. The project also intensified the STA’s support to international initiatives through the Marrakech Action Plan for Statistics and the Paris Declaration on Aid Effectiveness. The STA is making greater efforts to strengthen the links between the Genera Data Dissemination System (GDDS) and the poverty reduction strategy approach to enhance national and donor support for statistical reforms. Flexible budgetary arrangements within the project permitted the executing agencies to use their own administrative and accounting practices. This made it easier for STA to incorporate the project activities into its own work program and procedures.
7. International Monetary Fund Office of Technical Assistance Management. Technical Assistance Evaluation Program Findings of Evaluations and Updated Program. (2010).
The evaluation synthesis draws on all relevant evaluations since the 2008 Board paper on the TA Evaluation Program (EP). The findings include two sections, with the first covering only the RTAC evaluations and the second including all others. The RTAC evaluations assess relevance, effectiveness, efficiency, and sustainability of aid activity. The other evaluations differed substantially in focus. They assess sustainability, flexibility, training, long-term experts, recruitment system, and donor visibility.
In general the RTACs received high scores for relevance. Country officials said that RTACs are country owned and responsive to country needs. According to the report, they were well focused, useful in helping to define country TA priorities, consistent with calls for more donor coordination and, overall, linked appropriately with IMF surveillance and lending operations. In general, the RTAC TA was of high quality and effective in building institutional capacity for macroeconomic management. However, in many areas the agenda remains unfinished and it was difficult to attribute any improvements that were identified to the contribution of the RTACs. At times, slow implementation of advice hampered the effectiveness of TAs. In general, the RTACs are lean organizations, but cost indicator measurement needed improving as reporting did not allow for accurate calculations of total costs. The evaluations pointed to the need for more focus on systematically sustaining benefits.
The other evaluations, showed that local ownership and political commitment determines sustainability. Flexibility is necessary to accommodate changing government priorities. Training will mitigate the effects of rapid staff turnover. Long-term experts are more suitable than other forms of TA in the early stages of a reform process and are most effective in behind the scenes activities. Interpersonal skills and flexibility are crucial to their success. Although the expert recruitment system is efficient, there is a need for more transparency. According to the report, greater use of logos, regional seminars, and informing recipients that a particular donor financed TAs will improve donor visibility.
8. International Monetary Fund Statistics Department. Peru: Report on the Statistics Technical Assistance and Training Evaluation Mission (December 12–16, 2011). (2012).
The internal review draws on document reviews, interviews, and surveys. The review assesses the relevance, efficiency, effectiveness, impacts, and quality of reports. TA was relevant, in most cases responding to demand expressed by the Peruvian authorities. TA interventions were efficient with missions remaining focused on STA’s comparative advantages and core expertise. STA TA was reasonably effective, especially in the areas of government finance statistics (GFS) and monetary and financial statistics (MFS). According to the report, results show that the TA had strong and positive impacts in most cases, which demonstrated the long-term nature of the interventions and their sustainability. The mission also found that the quality of the TA reports, backstopping and report management in STA were generally adequate. Documents related to the TA were found without difficulty, with few exceptions. The report stated that, in general, the quality of the TA reports was high.
Inter-American Development Bank (IDB)
9. Inter-American Development Bank Office of Evaluation and Oversight. Analysis of Research Department’s Activities and the Work of the Bank: SIS—Social Information System. (2001).
The internal review draws on interviews conducted with research staff, Chiefs of the Social Development Divisions, project staff, and professionals involved in other information systems within the Bank. The review only considers the use of the data by Bank staff. The principal issues examined in the short-term study are demand, relevance, cost-effectiveness, and the complementarity of the information system with others supplying similar data. Its key findings are that the Social Information System (SIS) is a suitable database that uses primary statistics, an important tool for applied research and for the design of social policy. There is wide use of SIS in bank activities such as (i) evaluating social conditions in the region; (ii) conducting research within the Bank; (iii) designing social projects; (iv) defining and determining the location of a particular target population for a project; (v) preparing project documents; (vi) writing economic assessment reports (EARS); and (vii) providing input for publications outside RES.
Islamic Development Bank (IsDB)
10. Islamic Development Bank. Statistical Capacity Building Program (STATCAP) in Partnership with Islamic Development Bank (IDB). (2014).
The internal review does not share its methods in detail, but states it obtained beneficiary feedback during DFID country staff visits and through email and telephone exchanges with delivery partners. The report provides a summary of the progress since the previous review. The STATCAP program’s original timeframe slipped due to issues with an Egyptian bid and replacing the Jordanian census. However, the program has met the aims of the subsequent re-launch. It has developed and is currently implementing three projects in Egypt, Yemen, Jordan, and the Occupied Palestinian Territories (OPTs). A key lesson identified in the report was regarding communication between delivery partners, the IsDB, and DFID. In 2014, there were times when delivery partners were unclear of expectations and the funding disbursement processes and timescales.
Oxford Policy Management (OPM)
11. Strode, Mary and Ian MacAuslan. Support to Statistical Capacity Building Development of an Evaluation Framework.(2008).
The thematic study includes field-work in three countries and desk studies of five countries, as well as two donor case studies (DFID and Sida). The purpose of this study was to inform the discussions at the High Level Forum on Aid Effectiveness in Accra, Ghana, September 2008. This study emphasizes that the national capacity to produce, analyze, and use high quality statistics is critical to managing for development results. Although substantial progress was made, the study concludes that more needs to be done. In particular, some programs supporting statistics focus mainly on short-term data production. While this approach does address some immediate data needs, the long-run impact on statistical capacity in countries is limited and may even be negative. A more integrated approach is needed, based on Paris Declaration Principles. The study also developed an evaluation framework for statistical capacity building, establishing 8 pillars of capacity building for country statistical systems together with 7 pillars for donors.
The Partnership in Statistics for Development in the 21st Century (PARIS21)
12. Willoughby, Chris, Anne Thomson, Xavier Charoy, and Juan Munoz. An Evaluation of PARIS21: An Initiative to Improve the Contribution of Statistical Services to Development. (2003).
The external evaluation draws on the documents provided by PARIS21 and interviews with stakeholders. The findings showed that the overall effectiveness of the PARIS21 program is promising, although the evidence was not conclusive at such an early date.
In addition, the key findings focused on the effect of the workshops, overall PARIS21 sponsorship, and partial financing. According to findings from one-third of the countries involved in the first two workshops (in 2000-01) there were lasting effects in strengthening interactions between statisticians and policymakers. These efforts helped the countries to respond to the needs for the data that are required for evidence-based policy making and implementation in connection with PRSPs and MDGs. The effects of statistical production and use were only gradually appearing. PARIS21 sponsorship and partial financing enabled the Task Teams to do important work that would not otherwise have been done. They also helped to build consensus about various aspects of statistical development among aid agencies and between aid agencies and developing countries.
13. Spitz, Pierre. PARIS21 Light Evaluation. (2006).
The internal review draws on documentation, interviews with selected respondents and discussions by the Review Group. It is a light evaluation, only designed to fill a gap between the full evaluations of 2003 and 2009. The 2003 evaluation found the efficiency of PARIS21 to be rather on the high side. The answers of the Review Group indicate that P21 relevance and effectiveness seem, in general, satisfactory, with the some provisos. The overall process of reaching the goals is inevitably slow and only a two-year period was reviewed, however it was found that effectiveness was high. There were some concerns about the clarity of the work program and the extent to which some objectives, such as improved coordination, could be achieved.
14. Roberts, Richard and Claudine Voyadzis. Evaluation of PARIS21: Final Report PARIS21. (2009).
The external evaluation draws on documentation reviews, desk studies, and emailed questionnaires to national statistical offices. It also relies on personal field interviews field with current or former PARIS21 Steering Committee members, frequent attendees at Steering Committee meetings, PARIS21 Task Team Conveners, current or former PARIS21 Secretariat staff and consultants, staff of the Development Co-operation Directorate of the Organization for Economic Co-operation and Development (OECD/DCD), and opinion leaders or major users of developing country statistics. The purpose of the evaluation was to review progress towards PARIS21’s Goal and Purpose and to provide recommendations for the future for PARIS21.
According to the report’s key findings, the Secretariat activities and outputs effectively contributed to achieving the partnership goal, purpose, outcomes and outputs. Respondents observed improved capacity to produce, analyze and use statistics in countries. Progress was most notable in relation to NSDSs. There is some evidence of an increase in the attention given to statistics in poverty reduction strategies. There are also indications that there is an increased availability, use and analysis of statistics in Millennium Development Goals (MDG) Country Reports. While there is no specific evidence relating PARIS21 activities directly to more effective decision-making, the work of PARIS21 is specifically recognized in some of the documents reviewed. The report also considers achievements to date as sustainable. The evaluation indicated that staff levels, including recent increases in staff, were compatible with the growth in the Secretariat’s outputs; while staffing levels had increased, more outputs had been delivered.
Swedish International Development Cooperation (SIDA)
15. Schmidt, Martin, Johan Bring, Silvana Rusi, and Agneta Sverkel-Osterberg. Evaluation of Co-operation with the Statistical Office of Kosovo: Final Report, SIDA. (2012).
The external evaluation draws on document reviews and field interviews. Implementation of the SIDA program occurred through short missions wherein staff from Statistics Sweden visited the Statistical Office of Kosovo (SOK) to give hands-on support with a strong technical development element. The DFID support focused on the development of a strategic plan for the SOK and supporting planning and management. While on a technical level the support and temporary results are impressive, both projects’ sustainability levels are low with the a few exceptions.
The evaluation provides findings for both the SIDA and DFID programs. For SIDA, results relate strongly to production and technical skills development, much of which relies on continued support from SCB. There are clear, though limited, indications of adoption of new methodology and an increased production capacity. However, sustainability levels are weak. The SOK had difficulties putting a new methodology or production processes into an independent operational framework. Two obstacles to sustainability are the program’s broad coverage and relative fragmentation. However, cost efficiency appears high on the implementation and output levels. For DFID, the needs identification in the program was urgent and prudent. There is a strong need in SOK for improving management and planning capacities through a strategic plan. The plan itself represents a vital beginning to capacity building in this area. However, the operational planning underpinning priorities and choices set forth by the strategic plan may be insufficient. The evaluation states that this is still the case in 2011 and that there is a strong need for improved planning.
United Nations (UN)
16. United Nations Statistical Commission Friends of the Chair. The Friends of the Chair Review of the International Comparison Program 2005 Round. (2007).
The internal review draws on a survey directed towards all countries participating in the regular ICP 2005 round, a survey comprising countries participating in the Ring comparison, a request asking for the users’ views on the ICP, and self-assessment reports that organizations involved in the present ICP round prepared. As the report was prepared before the final results of the 2005 Round had been released, it was not able to assess the quality of the published data. According to the report, respondents saw the ICP as a major step forward, and expressed a desire for another round of ICP in 2010. Respondents emphasized that this is feasible only as long as the data are timely, reliable and available across a large number of countries. They viewed the substantial initial costs of building consensus on methodologies, and in training national experts, as strong arguments for continuing the program. A large majority of the ICP group countries were satisfied with their participation in the 2005 round and confirmed that they would participate in the next round. Comments from the countries confirmed that the ICP had had wide positive effects on national statistical programs, both in the field of price statistics and national accounts. Overall the governance structure and partnership arrangements worked well, according to the responses of the parties involved.
17. United Nations Statistical Commission Friends of the Chair. Report of the Friends of the Chair group on the evaluation of the 2011 round of the International Comparison Program. (2015).
The internal review draws on an ICP document review, a survey questionnaire, and interviews. It provides findings on general evaluation, governance structure, capacity-building, quality of data and metadata, timeliness and publication of results, and technical aspects and methodologies. There is a general understanding that ICP 2011, with its considerably expanded coverage (from 150 to 199 countries), brought a much higher acceptance compared to earlier exercises. The report concludes that the governance structure generally worked quite well. A large majority of the ICP stakeholders confirm that ICP 2011had wide positive effects on the regional statistical programs. Simultaneously, the regional coordinators confirm that the various parts of ICP had a significant impact on the workload of the regional offices and national statistical institutions. The ICP website is well-organized, providing broad information on the Program, the entities involved, the activities, the survey programs and important uses of PPPs.
United Nations Children’s Fund (UNICEF)
18. John Snow Inc. Evaluation of UNICEF Multiple Indicators Cluster Surveys Round 3 (MICS3): Final Report. (2009).
The external evaluation drew on information from online surveys; key informant interviews at global, regional, and country levels; structured document review; data quality assessment; and a compilation of key indicators (taxonomy). UNICEF commissioned the evaluation to examine the third MICS round (MICS3) to judge if the results justified the expenses and commitments made, and to learn how best to use and improve the MICS initiatives and similar data in support of global goals and targets.
The evaluation assessed the quality, governance and impact of the MICS. In terms of quality, the report states that in most countries examined, the quality of data was on a par with other “gold standard” global household survey efforts. It found that the type and magnitude of non-sampling errors were similar to those in global household survey programs. In terms of governance, UNICEF was able to make effective use of many aspects of its organizational structure, processes, and culture to make MICS3 a high priority. However, UNICEF’s decentralized structure meant that critical technical decisions were negotiated by those with the least knowledge and experience in the conduct of household surveys. In regards to impact, the positive findings have to be balanced against suggestions for further improvement and strengthening of the survey program. The report noted significant data quality lapses, sampling deviations from accepted norms in sampling, and fieldwork procedures that did not adhere to recommended practice.
19. Plowman, Beth Ann, and Jean Christophe Fotso. UNICEF Evaluation of the Multiple Indicator Cluster Surveys (MICS) – Round 4. (2013).
The external evaluation draws a structured document review, interviews, group discussions, country visits, an online expert practitioner panel, and data quality assessment using standard tabulations from round 4 of MICS. Over four rounds, 240 MICS were conducted in over 100 countries, providing data on 23 of the 48 MDG indicators. At the close of Round 4, this evaluation examines responses to the 2007-08 evaluation recommendations and assesses preparations for Round 5.
The evaluation assesses the technical support and human resources, decision-making and governance, quality assurance and timeliness, and the data quality of the MICS. UNICEF significantly expanded the technical support resources to guide MICS implementation through placement of Regional MICS Coordinators in the Regional Offices, use of HQ/SMS/MICS and regionally based consultants, and UNICEF MICS consultants working with country offices. Although the technical support envelope expanded, the organizational structure, communication channels, and decision-making authorities remain unchanged. Furthermore, recommendations from past evaluation were not addressed. The 2008 evaluation noted areas of concern relating to sample sizes, timeliness of reports, adherence to data collection guidelines, and the content/length of the questionnaire. The evaluation states that there were improvements in some areas, but not others. It finds dramatic improvement between MICS3 and MICS4 across all quality indicators, comparable data quality in many indicators, but that the quality of some MICS data still needs improvement. The main factors related to improvements in data quality were in human resources and technical support, due to a more systematic adherence to the standards on training field workers.
The World Bank (WB)
20. Thomson, Anne, Christopher Willoughby, and Ramesh Chander. The World Bank Trust Fund for Statistical Capacity Building: An Evaluation. (2003).
The external evaluation draws on interviews with TFSCB managers, members of the Internal Management Committee with Task Team Leaders (TTLs) where possible; donors; and members of the Advisory Panel; a review of projects undertaken in the first 18 months; and three field visits to specific projects. The Trust Fund evaluation was its first and took place about two years after it became effective. It covered 44 projects that had been approved for funding by the Trust Fund, of which 27 were to developing countries and 17 to international or regional organizations supporting statistics.
The report finds country projects emphasized improvements in skills in poverty measurement and macroeconomic statistics, strategic planning, and dissemination and use of statistics. In contrast, grants to international institutions emphasized improvements in dissemination and use of statistics, and skills in macroeconomic statistics. Statistical capacity building has three major prongs: the culture surrounding statistics, the management of national statistical systems, and technical capacity improvements. The report says statistical capacity building is promising, but highlights shortcomings in guidelines for statistical capacity building, support to project design, project prioritization, supervision of project implementation, and feedback on project outcomes. Projects developed in conjunction with WB staff were straightforward, but others faced problems when trying to access the TF independently. Project implementation was not very smooth due to disbursement issues.
21. Willoughby, Chris and Philip Crook. Marrakech Action Plan for Statistics: Report of an Independent Evaluation. (2008).
The external evaluation draws on discussions with World Bank staff and partner agencies and four case studies from Ethiopia, Malawi, Mali, and Niger. The principal purpose of the evaluation was to examine the eight programs, of which five were relatively major, implemented as part of the Marrakesh Action Plan for Statistics (MAPS), to see whether they should be continued and how they might be improved. The evaluators were guided to focus not on program details but on their overall contribution to development, and especially to extent to which they had contributed to decision making by governments and others about development policies and programs.
The key findings showed that a common feature of almost all the programs was that they involved important capacity-building actions for developing countries at agency headquarters as well as in the countries themselves. The various DGF-supported programs did make a difference in the various environments where they operated, and were having substantial impact on the broader objectives. The various programs were generally well managed. The evaluation did not identify any cases of wasteful or excessive expenditure, but did highlight the significant savings gained from technical innovations, especially new software and other procedures.
22. Snorrason, Hallgrimur, Andrew Flatt, and Jette Jensen. World Bank Trust Fund for Statistical Capacity Building Evaluation Report. (2010).
The external evaluation relied on a document review, surveys, and interviews. The questionnaires received a 43% response rate. The evaluation assessed the distribution of the grants, their relevance, project management and implementation, TFSCB management and governance, and overall TFSCB usefulness. In terms of distribution, Sub-Saharan Africa was the largest recipient, receiving 33% of the US$30.6 million disbursed. The report states that these projects were highly relevant and fully accepted in developing countries. The initial stages (idea, formulation, and approval) were efficient, but external factors affected the timeliness of project activities. Although the project activation period was lengthy, the report argues that it reflects recipient government ownership. Quality of projects was high, but the ease of implementation and timeliness varied greatly. Overall, projects remained within the budget and the proposed timeframe.
The report also provides findings regarding the outcomes and impacts of NSDS and SCB (non-NSDS) projects, regional projects and global projects. Overall, the NSDS projects succeeded in achieving their outcomes, most often in cooperation with several local partners. However in some few cases, countries felt that NSDS work was less successful than they had expected and participation of main statistical producers and users varied greatly. Implementation of the strategies and action plans created the most concern, hampered by a lack of funds. Evidence shows that the SCB (non-NSDS) projects yielded positive results. These grants may have contributed to the sustainability of NSDS results. There were limited and good outcomes of TFSCB funded regional projects. Limited success occurred in contexts of low development levels and political instability. However, they show success in capacity building. Global projects were aligned with the TFSCB guidelines and may have been worthwhile with substantial effects. However, their impact was difficult to assess and thus uncertain.
23. The World Bank Independent Evaluation Group. Marrakech Action Plan for Statistics, Partnership in Statistics for Development in the 21st Century, and Trust Fund for Statistical Capacity Building. (2011).
The evaluation synthesis draws on evaluations of MAPS, PARIS21, and TFSCB from 2008, 2009, and 2010. In addition, it uses interviews and reviews relevant internal materials, including progress reports, results frameworks, meeting minutes, and other information available online.
The evaluation synthesis assesses the independence and quality of the three evaluations. It provides findings on relevance, efficacy, efficiency, governance and management, and financial sustainability of the programs. It also draws lessons from the World Bank’s engagement. According to the report, all three programs continue to have highly relevant objectives. In terms of efficacy, there was moderate progress on achieving the activities and outcomes of the three programs. However, insufficient attention to NSDS implementation challenges, a lack of strategy to stimulate data demand, and inadequate (though improving) attention to support for statistical capacity among the donor community hampered progress towards the programs’ higher goals. Each evaluation found that the programs were cost-efficient. Regarding governance and management, the three arrangements work moderately well to guide and implement activities. In terms of financial sustainability initially, PARIS21 and TFSCB would solicit donors to jointly support these programs, but now TFSCB grants helped finance several PARIS21 initiated activities.
In addition to the findings mentioned above, the report summarized their lessons learned. It stated that effectiveness requires explicit strategies for achieving outcome objectives. Some selectivity may be needed to make progress. Coordinated financial support across donors for statistical capacity building at the country level is important for moving the agenda forward and could benefit from documenting and sharing different approaches widely. The awareness gap on the need for statistical development between DECDG’s professional statisticians and the Bank’s operational staff may require stronger advocacy efforts inside the Bank.
24. The World Bank. Implementation completion and results report on a loan in the amount of USD 107 million to the Republic of India for a statistical strengthening loan. (2012).
The internal review does not discuss the methods that it used to assess the SSL. The report provides assessment of outcomes, risk to development outcome, and World Bank and borrower performance. In addition, it summarizes some lessons learned. The overall outcome is rated moderately satisfactory based on the fact that, despite the delays, the SSL helped to trigger the long term institutional reform process the Government of India (GOI) is carrying out. The review considers the risk to development outcomes as moderate, based on the robust commitment demonstrated by the GOI towards the strengthening of the decentralized statistical system and improving coordination between the Center and States/UTs. Overall World Bank performance was satisfactory while the implementing agency and borrower performance was moderately satisfactory.
25. The World Bank. Implementation completion and results report on a credit in the amount of SDR8.4 million (US$12.9 million equivalent) to the Republic of Kenya for the development of the national statistical system project. (2013).
The internal review does not discuss the methods that it uses. The report provides assessment of outcomes, risk to development outcome, and World Bank and borrower performance. The evaluation rates the overall outcome as moderately unsatisfactory. The project helped to increase data production and trigger the long-term institutional reform process being carried out by the Government of Kenya (GOK). However, the project never overcame issues related to financial management, procurement, and data dissemination. These issues led to low disbursements and the slow pace of implementation. According to the report, the project had substantial risk to development outcomes. The risks included government commitment to statistics; capacity issues; financial sustainability; a complex institutional setup; and weak financial management, procurement, and audit systems. The report states that sustainability of the achievements depends on a concerted effort by external development partners and GOK to provide additional finance and technical assistance for statistical capacity building. Overall, the report considered World Bank, implementing agency, and borrower performance as moderately satisfactory.
26. Ngo, Brian T. and Andrew J. Flatt. Statistics for results facility – catalytic fund (SRF-CF): Evaluation report of the pilot phase. (2014).
The external evaluation draws on telephone interviews, country visits, and surveys. There was limited implementation in Afghanistan, Nigeria, Rwanda, and latterly in Lao PDR, which posed challenges to drawing conclusions. To remedy the lack of information, the evaluation team made extensive use of the survey questionnaire. According to the report, the five SRF objectives are being met overall, but with somewhat differing rates of success.
For the most part, the report’s key findings of the evaluation are summaries of respondent opinions regarding the SRF-CF. In many cases respondents stated it was too early to judge certain aspects of performance. In all three countries where the statisticians were working for some time, there was a general consensus that they had a very significant positive impact. Over half of all respondents, in both implementing and pipeline countries, felt that the concept of the SRF met statistical capacity building needs to a great extent. In terms of forward-looking comments, respondents in all three implementing countries expressed the wish that SRF funding should be continued.
Broad Review
27. Willoughby, Chris. Overview of Evaluations of Large-Scale Statistical Capacity Building Initiatives. (2008).
The external evaluation draws on a review of various evaluation and assessment reports, supplemented by discussions with relevant officials in international agencies concerned with financing and supporting statistical capacity building initiatives. The report is an inventory of assessments of different statistical development initiatives undertaken between 2000 and 2008. The inventory was drawn quite widely, focusing on projects, programs, and policies that were already underway and excluding appraisals of potential new initiatives.
The report concludes that the programs and subsequent follow-up and monitoring had yielded substantial impacts at various levels. However, it identified three worrying themes related to the relationship between the production and the use of statistics. First, there was little direct evidence of data improvements having a direct impact on decision-making. Second, there was apparent weakness in measuring and accounting for the costs of different statistical activities and methods. Third, there was concern that establishing national priorities, taking into account local constraints and resource availability, had not received enough attention.
Bibliography
“A New Global Partnership: Eradicate Poverty And Transform Economies Through Sustainable Development.” The Report of the High-Level Panel of Eminent Persons on the Post-2015 Development Agenda, UN. (2013). Accessed June 22, 2015. http://www.un.org/sg/management/pdf/HLP_P2015_Report.pdf.
ADE s.a. Evaluation of the Commission Support for Statistics in Third Countries: Revised Final Report. European Commission. (Brussels, 2007). Accessed June 22, 2015. http://ec.europa.eu/europeaid/how/evaluation/evaluation_reports/2007/1059_docs_en.htm.
Asian Development Bank. Technical Assistance Performance Audit Report on Selected Technical Assistance for Agricultural Planning and Statistics in Nepal. (2002). Accessed May 11, 2015. http://www.adb.org/documents/selected-technical-assistance-agricultural-planning-and-statistics-nepal.
Asian Development Bank. Technical Assistance Performance Audit Report on Selected Technical Assistance for Development Planning and National Statistics in Cambodia. (2003). Accessed May 11, 2015. http://www.adb.org/documents/selected-technical-assistance-development-planning-and-national-statistics-cambodia.
Department for International Development. Evaluation of the Implementation of the Paris Declaration: Thematic Study of Support to Statistical Capacity Building: Synthesis Report. (2009). Accessed: May 11, 2015. https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/67730/thematic-study-stat-capacity.pdf.
Dunmore, John and Jan Karlsson. Independent Evaluation of FAO’s Role and Work in Statistics. (2008). Accessed May 11, 2015. ftp://ftp.fao.org/docrep/fao/meeting/014/k2921e.pdf.
Eele, Graham. Mustafa Dinc, and Neil Fantom. Financing Statistics for Development Results. (invited paper presented at the ISI International Conference, Durban, 2009). Accessed June 22, 2015. http://www.researchgate.net/profile/Mustafa_Dinc/publication/228474366_Financing_Statistics_For_Development_Results/links/02e7e529f842e2b4d2000000.pdf.
Inter-American Development Bank Office of Evaluation and Oversight. Analysis of Research Department’s Activities and the Work of the Bank: SIS—Social Information System. (2001). Accessed May 11, 2015. http://publications.iadb.org/bitstream/handle/11319/4507/Analysis%20of%20Research%20Department%25C2%25BFs%20Activities%20and%20the%20Work%20of%20the%20Bank.%20%20SIS%20-%20Social%20Information%20System.pdf?sequence=1.
The International Household Survey Network. Accessed June 22, 2015. http://www.ihsn.org/home/.
International Monetary Fund Statistics Department. Evaluation of the U.K. DFID-Financed Technical Assistance GDDS Project for Selected Anglophone African Countries (2001-06). (2008). Accessed May 11, 2015. http://www.imf.org/external/np/pp/eng/2008/032108a.pdf.
International Monetary Fund Statistics Department. Peru: Report on the Statistics Technical Assistance and Training Evaluation Mission (December 12–16, 2011). (2012). Accessed May 11, 2015. http://www.imf.org/external/pubs/ft/scr/2012/cr12233.pdf.
Islamic Development Bank. Statistical Capacity Building Programme (STATCAP) in Partnership with Islamic Development Bank (IDB).(2014). Email to Open Data Watch, June 17, 2015.
John Snow Inc. Evaluation of UNICEF Multiple Indicators Cluster Surveys Round 3 (MICS3): Final Report. (2009). Accessed May 11, 2015. http://www.unicef.org/evaldatabase/index_52700.html.
Independent Expert Advisory Group. “A World that Counts: Mobilizing the data revolution for sustainable development.” Report for the UN Secretary General. (New York, 2014). Accessed June 22, 2015. http://www.undatarevolution.org/report/.
International Monetary Fund Office of Technical Assistance Management. Technical Assistance Evaluation Program Findings of Evaluations and Updated Program. (2010). Accessed May 11, 2015. http://www.imf.org/external/np/pp/eng/2010/061010.pdf.
Ngo, Brian T. and Andrew J. Flatt. Statistics for results facility – catalytic fund (SRF-CF): Evaluation report of the pilot phase. (2014). Accessed May 11, 2015. http://documents.worldbank.org/curated/en/2014/02/19586528/statistics-results-facility-catalytic-fund-srf-cf-evaluation-report-pilot-phase.
OECD. “DAC Criteria for Evaluating Development Assistance.” OECD: Better Policies for Better Lives. Accessed June 22, 2015. http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm.
OECD Development Assistance Committee. “Quality Standards for Development Evaluation.” DAC Guidelines and Reference Series. (2010). Accessed June 22, 2015. http://www.oecd.org/development/evaluation/qualitystandards.pdf.
OECD. “Paris Declaration and Accra Agenda for Action.” OECD: Better Policies for Better Lives. Accessed June 22, 2015. http://www.oecd.org/dac/effectiveness/parisdeclarationandaccraagendaforaction.htm.
OECD, Paris Declaration on Aid Effectiveness, (Paris, 2005), Accessed June 22, 2015, http://www.oecd.org/document/18/0,3343,en_2649_3236398_35401554_1_1_1_1,00.html.
PARIS21. A Road Map for a Country-led Data Revolution. (Paris, 2015). Accessed June 22, 2015. http://datarevolution.paris21.org/sites/default/files/Road_map_for_a_Country_led_Data_Revolution_web.pdf.
PARIS21. Dakar Declaration on the Development of Statistics. (Dakar 2009). Accessed June 22, 2015. http://www.paris21.org/sites/default/files/DDDS-en.pdf.
PARIS21. Paris21 at Ten: Improvements in statistical capacity since 1999. (2009). Accessed June 22, 2015. http://paris21.org/sites/default/files/P21-at-10.pdf.
PARIS21. Statistics for Transparency, Accountability, and Results: A Busan Action Plan for Statistics. (2011). Accessed June 22, 2015. http://www.paris21.org/sites/default/files/Busanactionplan_nov2011.pdf.
Plowman, Beth Ann, and Jean Christophe Fotso. UNICEF Evaluation of the Multiple Indicator Cluster Surveys (MICS) – Round 4. (2013). Accessed May 11, 2015. http://www.unicef.org/evaldatabase/files/MICS_evaluation_Cluster_1__final.pdf.
Roberts, Richard and Claudine Voyadzis. Evaluation of PARIS21: Final Report, PARIS21. (2009). Accessed May 11, 2015. http://www.paris21.org/sites/default/files/p21evaluation-2009.pdf.
Schmidt, Martin, Johan Bring, Silvana Rusi, and Agneta Sverkel-Osterberg. Evaluation of Co-operation with the Statistical Office of Kosovo: Final Report, SIDA. (2012). Accessed May 11, 2015. http://www.sida.se/contentassets/ae7d51cc13744bdfafe2fe4216aa741b/evaluation-of-co-operation-with-the-statistical-office-of-kosovo—final-report_3266.pdf.
Short, Clare. Statistics for the elimination of world poverty. (Speech at a meeting organized by OECD, the United Nations, the IMF and the World Bank, and Paris21, 18-19 November 1999). Accessed June 22, 2015. http://paris21.org/sites/default/files/short-p21-1999.pdf.
Snorrason, Hallgrimur, Andrew Flatt, and Jette Jensen. World Bank Trust Fund for Statistical Capacity Building Evaluation Report. (2010). Accessed May 11, 2015. http://siteresources.worldbank.org/SCBEXTERNAL/Resources/TFSCB_Evaluation_Report-Jan.2010.pdf.
Spitz, Pierre. PARIS21 Light Evaluation. (2006). Accessed May 11, 2015. http://www.paris21.org/sites/default/files/p21evaluation-2006.pdf.
Strode, Mary and Ian MacAuslan. Support to Statistical Capacity Building Development of an Evaluation Framework. (2008). Email to Open Data Watch, May 19, 2015.
Thomson, Anne, Christopher Willoughby, and Ramesh Chander. The World Bank Trust Fund for Statistical Capacity Building: An Evaluation. (2003). Accessed May 11, 2015. http://siteresources.worldbank.org/SCBEXTERNAL/Resources/OPM-Evaluation-of-the-TFSCB-Nov.2003.pdf.
Thomson, Anne, Graham Eele, and Felix Schmiedling. Independent Evaluation of the International Household Survey Network (IHSN) and Accelerated Data Program (ADP). (2013). Accessed May 11, 2015. http://adp.ihsn.org/sites/default/files/IHSN%20ADP%20Evaluation%20-%20Final%20report.pdf.
United Nations Statistical Commission Friends of the Chair. The Friends of the Chair Review of the International Comparison Program 2005 Round. (2007). Accessed May 11, 2015. http://www.bfs.admin.ch/bfs/portal/en/index/themen/05/22/lexi.Document.106239.pdf.
United Nations Statistical Commission Friends of the Chair. Report of the Friends of the Chair group on the evaluation of the 2011 round of the International Comparison Program. (2015). Accessed May 11, 2015. http://unstats.un.org/unsd/statcom/doc15/2015-14-ICP-FOC-E.pdf.
Willoughby, Chris. Overview of Evaluations of Large-Scale Statistical Capacity Building Initiatives. (2008). Accessed May 11, 2015. http://www.paris21.org/sites/default/files/3640.pdf.
Willoughby, Chris and Philip Crook. Marrakech Action Plan for Statistics: Report of an Independent Evaluation. (2008). Accessed May 11, 2015. http://siteresources.worldbank.org/SCBEXTERNAL/Resources/MAPS_EvaluationFinalReport.pdf?resourceurlname=MAPS_EvaluationFinalReport.pdf.
Willoughby, Chris, Anne Thomson, Xavier Charoy, and Juan Munoz. An Evaluation of PARIS21: An Initiative to Improve the Contribution of Statistical Services to Development. (2003). Accessed May 11, 2015. http://www.paris21.org/sites/default/files/p21evaluation-2003.pdf.
The World Bank. Implementation completion and results report on a credit in the amount of SDR8.4 million (US$12.9 million equivalent) to the Republic of Kenya for the development of the national statistical system project. (2013). Accessed May 11, 2015. http://documents.worldbank.org/curated/en/2013/05/17816433/kenya-development-national-statistical-system-project.
The World Bank. Implementation completion and results report on a loan in the amount of USD 107 million to the Republic of India for a statistical strengthening loan. (2012). Accessed May 11, 2015. http://documents.worldbank.org/curated/en/2012/01/15781898/india-statistical-strengthening-loan-project.
The World Bank. “Marrakech Action Plan for Statistics.” Statistical Capacity Building. Accessed June 22, 2015. http://www.worldbank.org/en/data/statistical-capacity-building/marrakech-action-plan-for-statistics.
The World Bank. “Trust Fund for Statistical Capacity Building.” Statistical Capacity Building. Accessed June 22, 2015. http://www.worldbank.org/en/data/statistical-capacity-building/trust-fund-for-statistical-capacity-building.
The World Bank. “STATCAP.” Statistical Capacity Building. Accessed June 22, 2015. http://www.worldbank.org/en/data/statistical-capacity-building/statcap.
The World Bank Independent Evaluation Group. Marrakech Action Plan for Statistics, Partnership in Statistics for Development in the 21st Century, and Trust Fund for Statistical Capacity Building. (2011). May 11, 2015. http://www-wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2011/11/08/000386194_20111108012956/Rendered/PDF/653360NWP0GPR00UBLIC00STATS0GPR2011.pdf.
