Preface

The Multilateral Organisation Performance Assessment Network (MOPAN) is a network of donor countries with a common interest in assessing the organisational effectiveness of multilateral organisations. MOPAN was established in 2002 in response to international fora on aid effectiveness and calls for greater donor harmonisation and coordination.

Today, MOPAN is made up of 16 donor countries: Australia, Austria, Belgium, Canada, Denmark, Finland, France, Germany, Ireland, The Netherlands, Norway, Republic of Korea, Spain, Sweden, Switzerland and the United Kingdom. For more information on MOPAN and to access previous MOPAN reports, please visit the MOPAN website (www.mopanonline.org).

Each year MOPAN carries out assessments of several multilateral organisations based on criteria agreed by MOPAN members. Its approach has evolved over the years, and since 2010 has been based on a survey of key stakeholders and a review of documents of multilateral organisations. MOPAN assessments provide a snapshot of four dimensions of organisational effectiveness (strategic management, operational management, relationship management, and knowledge management). In 2012, MOPAN is piloting a new component to examine an organisation’s development results in addition to its organisational effectiveness.

MOPAN 2012

In 2012, MOPAN assessed six multilateral organisations: the African Development Bank (AfDB), GAVI Alliance (formerly the Global Alliance for Vaccines and Immunisation), the Joint United Nations Programme on HIV/AIDS (UNAIDS), the United Nations Development Programme (UNDP), the United Nations Children’s Fund (UNICEF), and the World Bank.

MOPAN Institutional Leads liaised with the multilateral organisations throughout the assessment and reporting process. MOPAN Country Leads monitored the process in each country and ensured the success of the survey.

Acknowledgements

We thank all participants in the MOPAN 2012 assessment of the World Bank. The World Bank’s senior management and staff made valuable contributions throughout the assessment and document review processes and provided lists of their clients to be surveyed. Survey respondents contributed useful insights and time to respond to the survey. The MOPAN Institutional Leads, Australia and the Netherlands, liaised with the World Bank throughout the assessment and reporting process. The MOPAN Country Leads oversaw the process in the field and ensured the success of the survey. Consultants in each country provided vital in- country support by following up with clients to enhance survey response rates.

Roles of Authors and the MOPAN Secretariat

The MOPAN Secretariat, led by Ireland in 2012 and co-chaired by Germany, worked in close cooperation with the MOPAN Technical Working Group to launch and manage the survey. MOPAN developed the Key Performance and Micro-indicators, designed the survey methodology, coordinated the development of lists of survey respondents, and approved the final survey questionnaire. MOPAN also directed the design of the approach to document review. MOPAN oversaw the design, structure, tone, and content of the reports.

Universalia and Epinion developed the survey instrument and carried out the survey and analysis. Universalia carried out the document review and wrote the reports.

Epinion is a leading consulting firm in Denmark that analyses and evaluates data to support decision making. It conducts specially designed studies for public and private organisations based on data collected among an organisation’s employees, members, customers, partners, and other sources. Epinion has 75 employees and 200 interviewers. Website: www.epinion.dk

Universalia Management Group is a Canadian consulting firm established in 1980 that specialises in evaluation and monitoring for international development. Universalia has made significant contributions to identifying best practices and developing tools in the fields of organisational assessment; planning, monitoring, and evaluation; results-based management; and capacity building. Website: www.universalia.com.

Acronyms

Figures

Figure 1.1 Comparison of MOPAN Results for the World Bank in 2009 and 2012
Figure 2.1 Respondent Rating Scale
Figure 2.2 The World Bank - Distribution of Responses (n=219) on all Questions Related to Micro-Indicators
Figure 2.3 Number of Survey Respondents for the World Bank by Country and Respondent Group
Figure 2.4 MOPAN Ranges and Descriptions
Figure 3.1 Overall Ratings of World Bank Organisational Effectiveness by Respondent Group
Figure 3.2 Overall Ratings on Key Performance Indicators (mean scores, all respondents and document review ratings)
Figure 3.3 Quadrant I: Strategic Management, Survey and Document Review Ratings
Figure 3.4 Quadrant I: Strategic Management, Mean Scores by Respondent Group
Figure 3.5 KPI 1: Providing Direction for Results, Ratings of Micro-Indicators
Figure 3.6 KPI 2: Corporate Focus on Results, Ratings of Micro-Indicators
Figure 3.7 KPI 3: Focus on Thematic Priorities, Ratings of Micro-Indicators
Figure 3.8 KPI 4: Country Focus on Results, Ratings of Micro-Indicators
Figure 3.9 Quadrant II: Operational Management, Survey and Document Review Ratings
Figure 3.10 Quadrant II: Operational Management, Mean Scores by Respondent Group
Figure 3.11 KPI 5: Resource Allocation Decisions, Ratings of Micro-Indicators
Figure 3.12 KPI 6: Linking Aid Management to Performance, Ratings of Micro-Indicators
Figure 3.13 KPI 7: Financial Accountability, Ratings of Micro-Indicators
Figure 3.14 KPI 8: Using Performance Information, Ratings of Micro-Indicators
Figure 3.15 KPI 9: Managing Human Resources, Ratings of Micro-Indicators
Figure 3.16 KPI 10: Performance-oriented Programming, Ratings of Micro-Indicators
Figure 3.17 KPI 11: Delegating Authority, Ratings of Micro-Indicators
Figure 3.18 Quadrant III: Relationship Management, Survey and Document Review Ratings
Figure 3.19 Quadrant III: Relationship Management, Mean Scores by Respondent Group
Figure 3.20 KPI 12: Supporting National Plans, Ratings of Micro-Indicators
Figure 3.21 KPI 13: Adjusting Procedures, Ratings of Micro-Indicators
Figure 3.22 KPI 14: Using Country Systems, Ratings of Micro-Indicators
Figure 3.23 KPI 15: Contributing to Policy Dialogue, Ratings of Micro-Indicators
Figure 3.24 KPI 16: Harmonising Procedures, Ratings of Micro-Indicators
Figure 3.25 Quadrant IV: Knowledge Management, Survey and Document Review Ratings
Figure 3.26 Quadrant IV: Knowledge Management, Mean Scores by Respondent Group
Figure 3.27 KPI 17: Evaluating External Results, Ratings of Micro-Indicators
Figure 3.28 KPI 18: Presenting Performance Information, Ratings of Micro-Indicators
Figure 3.29 KPI 19: Disseminating Lessons Learned, Ratings of Micro-Indicators
Figure 4.1 Development Results Component – Overall Ratings
Figure 4.2 KPI A: Evidence of Extent of Progress toward Organisation-Wide Outcomes, Ratings
Figure 4.3 KPI A: Evidence of Progress towards Organisation-Wide Results/Objectives
Figure 4.4 KPI B: Evidence of Extent of Contribution to Country-Level Goals and Priorities, Rating
Figure 4.5 World Bank Stakeholder Survey - Ghana, Mean Scores by CAS Objectives
Figure 4.6 World Bank Stakeholder Survey - Honduras, Mean Scores by CAS Objectives
Figure 4.7 World Bank Stakeholder Survey - Morocco, Mean Scores by CPS Objectives
Figure 4.8 World Bank Stakeholder Survey – Nigeria, Mean Scores by CPS Objectives
Figure 4.9 World Bank Stakeholder Survey - Philippines, Mean Scores by CAS Objectives
Figure 4.10 KPI C: Contribution to Relevant MDGs, Overall Rating and Criteria
Figure 4.11 Survey Data on World Bank Contributions to MDGs, Mean Scores, by Country
Figure 4.12 KPI D: Relevance of Objectives and Programme of Work to Donors In-Country and Clients, Overall Rating and Survey Mean Scores by Country

Executive Summary

This report presents the results of an assessment of the International Bank for Reconstruction and Development/International Development Association (the World Bank) conducted by the Multilateral Organisation Performance Assessment Network (MOPAN).[1] MOPAN assesses the organisational effectiveness of multilateral organisations based on a survey of stakeholders, a review of documents, and interviews with headquarter-based donor staff. In past years, MOPAN has not assessed an organisation’s development results, but is testing a component on this with four organisations in this year’s assessment.[2]

To assist developing countries worldwide in reducing poverty and achieving sustainable growth, the World Bank provides loans to middle income countries, and credits and grants to low- income countries. It has recently committed to substantially increase its support to fragile and conflict-affected states. The allocated resources are invested by governments in a wide array of development sectors, including infrastructure, agriculture, inclusive growth, health, education, water and sanitation, climate change, and governance in public administration. The World Bank also provides analytical and advisory activities and technical assistance to client countries.

Globally, the World Bank plays a role in the wider multilateral system as a provider of knowledge, a commitment that has recently been reinforced with the launch of the Open Knowledge Repository, and uses its convening power to develop and maintain global development partnerships.

The World Bank has implemented a number of reforms to improve its effectiveness, efficiency and governance. As part of the Bank’s modernisation agenda, it has committed to enhance transparency, increase its focus on results, and become financially stronger and more responsive to clients.

MOPAN Assessment

In 2012, MOPAN assessed the World Bank based on information collected through interviews with World Bank staff, a survey of key stakeholders, and the review of documents. The survey targeted the World Bank’s clients, as well as MOPAN donors based in-country and at headquarters. Eight countries were included in the MOPAN survey: Democratic Republic of Congo, Ghana, Honduras, Morocco, Niger, Nigeria, the Philippines, and Zimbabwe. A total of 219 respondents participated in the survey. MOPAN’s document review assessed the World Bank through an examination of publicly available corporate documents and country programming documents from five of the eight countries selected.

MOPAN assessments provide a snapshot of four dimensions of organisational effectiveness (strategic management, operational management, relationship management, and knowledge management). The main findings of the assessment of the World Bank in these performance areas and in a pilot component on development results are summarised below.

Strategic Management

In the area of strategic management, MOPAN established criteria to determine if a multilateral organisation has strategies and systems in place that reflect good practice in managing for results. Overall, the 2012 assessment found that:

• Strategic management is a clear area of strength for the World Bank and it is seen to actively promote the organisation-wide achievement of development and management results. It received high ratings for its leadership on results and transparency in providing access to information, which the document review recognised as best practice.

• The Bank has strengthened the results focus of its Country Assistance Strategies and country results frameworks have improved in quality and consistency. However, the Bank’s corporate results framework, as presented in the innovative Corporate Scorecard, could better explain the results chain (i.e., the links between outputs, corporate and project outcomes, and development context results).

• The World Bank is considered adequate by stakeholders in incorporating cross-cutting priorities of gender equality, climate change, governance, and fragile and conflict-affected states. The Bank has made efforts to mainstream and/or incorporate each of these themes across its operations.

Operational Management

In operational management, MOPAN established criteria to determine if a multilateral organisation manages its operations in a way that supports accountability for results and the use of information on performance. Overall, the 2012 assessment found that:

• The World Bank was noted for the transparency of resource allocation and allocation of resources based on performance. Its criteria for allocating resources are publicly available and the majority of survey respondents considered that allocations follow these criteria.

• Both donors at headquarters and the document review rated the World Bank adequate in linking its administrative budget to expected results, but the document review found the Bank inadequate in linking expenditures or disbursements to reported results.

• The World Bank has strong policies and processes for ensuring financial accountability and has developed policies and procedures to ensure financial audits, manage risk, and combat fraud and corruption.

• The Bank uses performance information to improve policies, plan new interventions, and allocate resources.

• The Bank was rated strong for its staff performance assessment system. It was rated adequate, however, for its management of staff performance because there is a need to improve the staff compensation system. The Bank is currently reviewing this system to enhance incentives for staff performance.

• The World Bank is considered strong in setting targets to monitor project implementation but adequate in conducting benefits/impact analyses of new operations.

• The World Bank’s business modernisation agenda includes provisions for increased delegation of authority to country level in the interest of reducing bureaucracy and improving its management of client services. This is a work in progress

Relationship Management

In relationship management, MOPAN established criteria to determine if a multilateral organisation is engaging with its clients at the country level in ways that contribute to aid effectiveness. Overall, the 2012 assessment found that:

• Surveyed stakeholders rated the Bank adequate in coordinating with clients in the design of projects, for the clarity of its procedures, flexibility, and responsiveness to changes in the project implementation context. In their written comments, they noted that the Bank’s procedures are rigid and that bureaucratic processes often delay implementation.

• The document review rated the Bank very strong in using national indicators or benchmarks for conditionality, its use of country systems, and avoiding parallel implementation structures. Surveyed stakeholders considered the Bank adequate in using country systems and participating in assessments of its aid effectiveness commitments.

• The World Bank’s input to policy dialogue was considered one of its key strengths.

• The Bank was rated adequate overall in harmonising procedures. In written comments, survey respondents suggested that the Bank needs to improve harmonisation with other donors.

Knowledge Management

In knowledge management, MOPAN established criteria to determine if a multilateral organisation has reporting mechanisms and learning strategies that facilitate the sharing of information inside the organisation and with the development community. Overall, the 2012 assessment found that:

• The Bank was noted for the quality and independence of its evaluations, managed by the Independent Evaluation Group. Surveyed stakeholders consider the Bank adequate in using evaluation findings in decision making and involving clients and beneficiaries in evaluations.

• The Bank has developed innovative mechanisms to provide integrated reports on performance, but it will take time for these to be fully developed. The Bank is considered adequate overall in its reporting at corporate and country level.

• The World Bank is considered strong in the dissemination of lessons learned from performance information. In written comments, survey respondents noted the Bank’s knowledge dissemination as one of the key strengths of the Bank.

Development Results

In the 2012 pilot assessment of development results, the Bank was rated adequate on the four key performance indicators assessed.

• Evidence of extent of progress towards organisation-wide outcomes: Surveyed stakeholders consider that the Bank is making progress towards the strategic priorities defined in the Post-Crisis Directions paper. The Corporate Scorecard and World Bank for Results report show some evidence of progress towards organisation-wide outcomes, however, the values for the baseline and the current year are still the same for a number of indicators, making it impossible to fully measure progress.

• Evidence of extent of contribution to country-level goals and priorities: Surveyed stakeholders rated the World Bank adequate or above for its contribution to results articulated in the CAS/CPS. The CAS/CPS are aligned with national goals and priorities and the World Bank has contributed to these goals across the sampled countries. The documents reviewed provide more evidence of Bank contributions to development results in the areas of economic growth and human development than in governance.

• Evidence of extent of contribution to relevant MDGs: Survey respondents rated the World Bank as adequate for supporting the achievement of national MDGs. Most of the CAS/CPS reviewed discuss the MDGs but the link between the Bank’s programme and contributions to MDGs is explicit in only three out of the five CAS/CPS reviewed.

• Relevance of World Bank objectives and programme of work to stakeholders: Surveyed donors in-country and clients considered the Bank strong in responding to country development priorities, and adequate in providing innovative solutions and adapting its work to changing needs of partner countries.

Trends since 2009

Perception data suggests that the Bank’s performance has remained steady or improved over the past three years. In strategic management, there is increased recognition of the Bank’s results focus at corporate and country levels, as well as its transparency. The views provided on the Bank’s operational management remain largely the same: resource allocation systems and financial accountability are the two areas with consistently strong ratings. The World Bank’s ratings remain steady in areas of relationship management, with some improvements in ratings of its capacity to adjust procedures and make valuable contributions to policy dialogue. In its knowledge management practices, the ratings consistently recognise the World Bank’s strong evaluation practice and range from adequate to strong on its performance reporting and dissemination of lessons learned.

Conclusions

The World Bank has demonstrated strategic leadership in managing for and reporting on development results. While the Bank is viewed as a leader in this area, some shortcomings were noted in the frameworks and systems that it uses to measure and report on organisation-wide results.

The Bank is developing good practices in managing for results at the corporate and country level and has developed mechanisms to systematically track progress and report on both development results and organisational performance. The document review noted the Bank’s establishment of a Results Secretariat, the development of the Corporate Scorecard, and staff capacity-building modules on results management.

At the country level, Country Assistance/Partnership Strategies (CAS/CPS) include results frameworks that link the Bank’s support to country results and national development priorities. The CAS/CPS have improved in quality and consistency over time.

At the corporate level, the management results framework contains a set of implicit assumptions about the results chain between Tiers II and IV. However, the development results framework does not provide any links between country-level outputs and outcomes and the Bank’s organisation-wide development outcomes.[3] In addition, the indicators under Tier II focus on results achieved at the country level, but there is no theory of change that provides a complete picture of how the organisation conceives its contributions to development change.

The World Bank is committed to transparency and has invested in the creation of an Open Bank.

The Bank has demonstrated leadership in transparency with the launch of its Open Data Initiative in 2010 and Open Access Policy in 2012. It has made efforts to facilitate public access to its research data, knowledge repository, and ODA disbursement practices, as well as to ensure that most of its key documents are available on the World Bank website. Stakeholders rated the Bank strong in this area and the document review considered it very strong.

The Bank allocates resources in a transparent manner and on the basis of country performance and its administrative budget is allocated to expected results. However, disbursements are not yet related to corporate results.

The Bank’s criteria for resource allocations are clear and publicly available and IDA resources are allocated based on national performance in the areas of poverty reduction and economic growth. However, while the Bank does link its administrative budget to expected results, it does not report on expenditures or disbursements related to its organisation-wide (corporate) results.

The World Bank is recognised for its strong financial and fiduciary practices.

The document review and surveyed stakeholders rated the Bank strong or very strong for its external and internal financial audits, policy on anti-corruption, systems to handle irregularities, and strategies for risk management. Financial statements, annual audits, and external audits of Bank-supported projects and programmes comply with international auditing standards.

The World Bank’s management of human resources is considered strong overall but it needs to improve its staff compensation system.

The Bank has a performance assessment system for staff on regular, open-ended, term, and local regular appointments, which also applies to senior staff, including Vice-Presidents and Managing Directors. The IEG rated the Bank negatively in terms of performance-based incentives for staff, and the Bank has indicated that it is reforming its human resources policy with a focus on staff compensation.

Surveyed stakeholders rated the Bank adequate for its use of country systems and adjustment of procedures but in their written comments expressed concerns about the Bank’s bureaucratic procedures.

In both the 2009 and 2012 MOPAN surveys, respondents generally considered the Bank’s use of country systems in its operations to be adequate. The 2012 document review, however, considered the Bank strong. It noted that the Bank has committed to the on-going development of Country Assistance/ Partnership Strategies in support of country ownership and continues to use country public financial management and procurement systems as reported to the OECD- DAC in respect of its Paris Declaration commitments.

Survey respondents noted some improvement in the Bank’s adjustment of procedures, which they rated inadequate in 2009 and adequate in 2012. In their written comments in the 2012 survey, however, they raised concerns about the Bank’s bureaucratic procedures and suggested that the Bank should be more flexible.

The Bank is considered strong in its evaluation of results and dissemination of knowledge. It has also introduced integrated mechanisms to report on its performance, but it will take time for these to be fully developed.

The Independent Evaluation Group (IEG), which reports directly to the Board of Executive Directors, has established good practices for the evaluation of results, strong quality control mechanisms to ensure the quality of evaluation activities and performance reporting, and mechanisms for the dissemination of key lessons learned and good practices. The use of evaluation findings in decision making is an area for improvement.

The Bank has developed mechanisms to report on performance, some of which are still being refined; in the Corporate Scorecard and its companion report, World Bank for Results, the links between outputs and organisation-wide outcomes need to be strengthened. The Bank has developed several avenues for knowledge dissemination, including reports, capacity-building sessions, analytical advisory activities, and the World Bank website. Surveyed stakeholders feel the Bank provides sufficient opportunities for sharing lessons learned.

Development Results Component

The Bank is making progress towards organisation-wide outcomes and in reporting on these. Its corporate reports do not yet provide stakeholders with a clear picture of how the Bank’s country level efforts contribute to organisation-wide outcomes.

Surveyed donors at headquarters feel the Bank is making substantial progress towards its strategic priorities. The Corporate Scorecard and World Bank for Results report provide an integrated view of results and performance. However, given the Bank’s focus on results achieved at the country level, the current frameworks do not provide a complete picture of how the organisation as a whole is achieving its organisation-wide outcomes.

The Bank is seen to be contributing to country-level objectives and national Millennium Development Goals.

At the country level, stakeholders gave the Bank generally strong ratings for responding to their countries’ key development priorities and adequate ratings on its relevance overall and its contributions to MDGs.

The majority of stakeholders surveyed consider the Bank adequate in contributing to CAS/CPS country objectives, which are aligned with national goals and priorities. While World Bank reports tend to confirm the Bank’s contribution in most results areas, the Bank’s role or contribution in support of its clients’ efforts to achieve MDGs is more explicit in some CAS/CPS than in others.

The Bank’s country reporting tools are improving and this may lead to better evidence of its contributions to country development outcomes in the future.

Overall MOPAN Ratings of the World Bank

The chart below shows the ratings on the 19 key performance indicators that MOPAN used to assess the World Bank in 2012. These indicators were designed to measure organisational effectiveness (practices and systems), and country-based strategies. The indicators were adapted to the work of the World Bank to encompass its country-driven model and its business modernisation agenda.

The World Bank received ratings of strong on 8 of the 19 key performance indicators assessed in the survey and strong or very strong on 13 of the 17 key performance indicators assessed by document review.

1. Introduction

1.1 MOPAN

This report presents the results of an assessment of the World Bank[4] that was conducted in 2012 by the Multilateral Organisation Performance Assessment Network (MOPAN). In 2012 MOPAN assessed six multilateral organisations: the African Development Bank (AfDB), GAVI Alliance (formerly the Global Alliance for Vaccines and Immunisation), the Joint United Nations Programme on HIV/AIDS (UNAIDS), the United Nations Development Programme (UNDP), the United Nations Children’s Fund (UNICEF), and the World Bank.

Background

MOPAN was established in 2002 in response to international fora on aid effectiveness and calls for greater donor harmonisation and coordination. The purpose of the network is to share information and experience in assessing the performance of multilateral organisations. MOPAN supports the commitments adopted by the international community to improve the impact and effectiveness of aid as reflected in the Paris Declaration on Aid Effectiveness, the Accra Agenda for Action, and the Busan High Level Forum. MOPAN’s processes and instruments embody the principles of local ownership, alignment and harmonisation of practices, and results-based management (RBM).

MOPAN provides a joint approach (known as the Common Approach) to assess the organisational effectiveness of multilateral organisations. The approach was derived from existing bilateral assessment tools and complements and draws on other assessment processes for development organisations – such as the bi-annual Survey on Monitoring the Paris Declaration on Aid Effectiveness and annual reports of the Common Performance Assessment System (COMPAS) published by the multilateral development banks. In the long term, MOPAN hopes that this approach will replace or reduce the need for other assessment approaches by bilateral donors.

MOPAN assesses four dimensions of organisational effectiveness

MOPAN has defined organisational effectiveness as the extent to which a multilateral organisation is organised to contribute to development and/or humanitarian results in the countries or territories where it operates.

Based on a survey of stakeholders and a review of documents, MOPAN assessments provide a snapshot of a multilateral organisation’s effectiveness in four dimensions:

• Developing strategies and plans that reflect good practices in managing for development results (strategic management)

• Managing operations by results to support accountability for results and the use of information on performance (operational management)

• Engaging in relationships with direct partners and donors at the country level in ways that contribute to aid effectiveness and that are aligned with the principles of the Paris Declaration (relationship management)

• Developing reporting mechanisms and learning strategies that facilitate the sharing of knowledge and information inside the organisation and with the development community (knowledge management).

In 2012, MOPAN also piloted a new component to assess a multilateral organisation’s contributions to development results. This component was tested with four of the six organisations assessed this year (AfDB, UNDP, UNICEF, and the World Bank).

Purpose of MOPAN assessments

MOPAN assessments are intended to:

• Generate relevant, credible and robust information MOPAN members can use to meet their domestic accountability requirements and fulfil their responsibilities and obligations as bilateral donors

• Provide an evidence base for MOPAN members, multilateral organisations and direct partners to discuss organisational effectiveness and in doing so, build better understanding and improve organisational effectiveness and learning over time

• Support dialogue between MOPAN members, multilateral organisations and their partners, with a specific focus on improving organisational effectiveness over time, both at country and headquarters level.

The MOPAN methodology is evolving in response to what is being learned from year to year, and to accommodate multilateral organisations with different mandates. For example, the indicators and approach for the 2012 MOPAN review of a global fund and organisations with significant humanitarian programming were adapted to reflect the reality of these organisations.

1.2 Profile of the World Bank

The World Bank is a United Nations specialised agency comprising the International Bank for Reconstruction and Development (IBRD) and the International Development Association (IDA). It has its origins in the United Nations Monetary and Financial Conference (the Bretton Woods Conference), a gathering of representatives of the Allied Nations in July 1944, for discussions on post-World War II recovery in Europe.

The Articles of Agreement for the IBRD became effective on 27 December 1945 and were amended in 1965, 1989 and 2012. The term “World Bank” was first used to refer to the IBRD in a report on the Bretton Woods conference.

In 1960, the IDA was founded on the premise that newly independent developing countries were unable to secure development capital on the terms specified by the World Bank. IDA, a part of the World Bank, was established to make development loans with easier terms to the world’s poorest countries.

The World Bank collaborates with three partner agencies – the International Finance Corporation, the Multilateral Investment Guarantee Agency, and the International Centre for Settlement of Investment Disputes – which, in addition to IBRD and IDA, collectively form the World Bank Group.

Governance and structure

The IBRD and IDA each have separate Articles of Agreement that define their purpose, organisation, and operations, including ownership and governance mechanisms. Each agency is owned by member countries that sign Articles of Agreement and purchase shares. As of 2012, the IBRD has 188 members. To gain membership in the IBRD, a country must be a member of the International Monetary Fund (IMF). The IDA has 172 members and 81 countries are eligible to receive support from the agency. Membership in the IDA is contingent on membership in the IBRD.

As a member of the World Bank Group, the World Bank falls under the purview of the Bank Group governing structure, which is headed by the Board of Governors. The general operations of the IBRD are coordinated by a Board of Executive Directors of the World Bank, which together with the other World Bank Group agencies are responsible for selecting the World Bank Group President for a renewable term of five years. Within the World Bank, the main organisational units are Vice Presidential Units (Vice-Presidencies) that correspond to a world region, thematic network, or central function. With the exception of a few that report directly to the President, Vice Presidencies report to a managing director or chief financial officer of the World Bank Group.

Strategy and services

The World Bank’s work is driven by six strategic themes – the world’s poorest countries; fragile and conflict-affected states; the Arab World; middle-income countries; global public goods issues; and the delivery of knowledge and learning services. Thematic and sector strategies are developed on a rolling basis to address cross-cutting poverty reduction themes, such as the environment, and gender and development.

In 2010, the World Bank refined its strategic orientation to address new challenges for the global development architecture, created in part by the global financial crisis.[5] The new strategic orientation involved the identification of the Bank’s comparative advantage in five areas – presence, knowledge, finance, global public goods, and convening power – and a re- commitment to its core mandate to overcome poverty through a country-based model.

Through its refined strategy, the World Bank has committed to five strategic priorities that involve: targeting the poor and vulnerable; creating opportunities for growth; providing cooperative models, strengthening governance; and managing risk and preparing for crises. It has also incorporated an acknowledged need for: i) collaboration among international and private financial institutions to leverage institutional strengths and resources for maximum impact; and ii) selectivity that uses key lessons learned to adjust the scope of the World Bank’s engagement in development activities.

The World Bank is a provider of financial services to developing countries in the form of investment loans, development policy loans, grants, and co-finance, as well as policy advice, research and analysis, capacity building and technical assistance. The financial services provided are used to fund project activities across diverse areas, including education; health; infrastructure, financial and private sector development; and natural resource management. Trust fund partnerships with bilateral and multilateral donors are also facilitated by the World Bank to support projects in several sectors and regions.

Finances

The finances of the World Bank derive from three main sources – capital markets, member contributions, and donor funding. Over 50 per cent of the World Bank’s lending is facilitated by the IBRD through the sale of bonds on the international financial market. The IBRD is able to earn most of its income through investments in its own capital, comprising reserves and money from member countries. Approximately 40 per cent of the World Bank’s lending is provided by the IDA as low interest or no interest loans, and grant assistance to the world’s poorest countries. The IDA’s finances are replenished every three years by donor countries. Borrower repayment of loan principal is used to regenerate additional funding for the agency.

Modernisation Agenda

In 2010, the Board of Governors of the World Bank Group endorsed a comprehensive modernisation agenda to build on the reforms proposed in the Post-Crisis Directions paper.[6] The modernisation agenda was developed around the Bank’s commitment to support clients to effectively address their development challenges. It encompasses four mutually reinforcing pillars to guide each member organisation of the World Bank Group in modernising services, tracking results, and sustaining high fiduciary standards. The higher objective of the modernisation agenda has envisaged that the World Bank Group will be a multilateral organisation that is strategically focused, financially stronger, and more responsive, transparent and accountable.

The World Bank has achieved several milestones under all four pillars of the modernisation agenda. It has established critical modernisation priorities for fiscal year 2013, and management has endorsed further investment in the strengthening of processes, instruments, policies and cultural change.

For further information on the World Bank, please visit the organisation’s website: www.worldbank.org.

1.3 Previous Assessments

Since its establishment in 2003, MOPAN has conducted four assessments of the World Bank (2005, 2008, 2009, and 2012). Although the MOPAN methodology has been expanded since 2004, findings from previous MOPAN surveys can provide insight into the evolution of perceptions of the Bank’s surveyed stakeholders.

2005 and 2008 MOPAN Surveys

In 2005 and 2008, the assessment was based on the Annual MOPAN Survey, an opinion survey that drew on the perceptions of MOPAN member staff. The survey focused on the partnership behaviour of the World Bank, with emphasis on its in-country performance relative to its mandate.

In 2005, MOPAN surveyed respondents in nine countries – Albania, Bangladesh, Bolivia, Bosnia and Herzegovina, Ethiopia, Nicaragua, Tanzania, Vietnam and Zambia. Stakeholders perceived the Bank to be a leading actor in development policy and cooperation, with a strong country presence. At the national partnership level, government respondents perceived the Bank as a key player in policy dialogue, but respondents from non-government organisations and the private sector did not share this view. Survey respondents thought that the World Bank had become more collaborative in its partnerships with other development agencies but that there was room for improvement.

In 2008, MOPAN surveyed respondents in 10 countries – Albania, Bangladesh, Bolivia, Bosnia and Herzegovina, Burkina Faso, Cambodia, Nepal, Sudan, Tanzania, and Vietnam.[7]

Stakeholders indicated that the World Bank was strong in three areas: policy dialogue with host governments, alignment of its work with national development strategies, and contributing to local donor coordination. They identified areas for improvement in enabling government ownership and capacity development at the local level (as opposed to central government).

In comparing the results of the 2005 and 2008 surveys, surveyed stakeholders perceived a substantial improvement in the Bank’s contribution to inter-agency-coordination and some improvement in alignment and harmonisation.

Trends since 2009

In 2009, MOPAN introduced the Common Approach, which incorporated the views of national partners/clients of the Bank, and thus had a broader scope than previous surveys. In 2010, it added a document review. Figure 1.1 compares the 2009 and 2012 survey ratings of the World Bank on Key Performance Indicators. While it illustrates some interesting trends, some caution is required in making comparisons given that there were changes between 2009 and 2012 in: the survey scale (from a 5-point to 6-point scale), the wording of some questions, and the countries and respondent groups surveyed. In addition, some 2009 survey questions were removed and assessed only by document review in 2012. With these caveats in mind, the 2012 survey illustrates an improving trend in 6 of the 19 indicators assessed.

Figure 1.1 Comparison of MOPAN Results for the World Bank in 2009 and 2012

Strategic management

Operational management

Relationship management

Knowledge management

*The 2009 MOPAN assessment was carried out in nine countries. Four were IBRD countries (Guatemala, Peru, Serbia and Thailand) and the other five were IDA or Blend countries (Ethiopia, Mozambique, Pakistan, Senegal and Uganda).

**The 2012 MOPAN assessment was carried out in the Democratic Republic of Congo, Ghana, Morocco, Niger, Nigeria, Philippines, and Zimbabwe.

2. MOPAN Methodology – 2012

2.1 Overview

Background

MOPAN continues to refine its assessment framework. In 2009, the MOPAN Common Approach replaced the Annual MOPAN Survey, which had been conducted since 2003. The Common Approach is broader and deeper than the previous surveys and includes the following components:

• Expanded survey – The MOPAN survey now brings in the views of direct partners or clients of multilateral organisations, peer organisations (or other relevant stakeholder group), and those of donors, that is, MOPAN members at both headquarters and country level.

• Document review – Since 2010, survey data are complemented by a review of documents prepared by the multilateral organisations being assessed and other sources.

• Interviews – In 2012, MOPAN complemented survey data and document review with consultations and interviews at the headquarters of multilateral organisations assessed.

In 2012 MOPAN also tested a new component to assess the results of multilateral organisations.[8]

As MOPAN’s methodology has changed significantly in the last three years, comparisons of this year’s assessments and previous assessments should take this into consideration.

The following is a summary of the MOPAN methodology in 2012.[9]

MOPAN 2012

In 2012, MOPAN assessed the effectiveness of six multilateral organisations: the African Development Bank (AfDB), GAVI Alliance (formerly the Global Alliance for Vaccines and Immunisation), the Joint United Nations Programme on HIV/AIDS (UNAIDS), the United Nations Development Programme (UNDP), the United Nations Children’s Fund (UNICEF), and the World Bank. The assessment of the World Bank was conducted in Democratic Republic of Congo, Ghana, Honduras, Philippines, Morocco, Niger, Nigeria, and Zimbabwe.[10]

The MOPAN Common Approach examines organisational systems, practices, and behaviours that MOPAN believes are important for aid effectiveness and that are likely to contribute to results at the country level. It groups these organisational capacities in four areas of performance: strategic management, operational management, relationship management, and knowledge management.

Key Performance Indicators and Micro-indicators – Within each performance area, organisational effectiveness is described using key performance indicators (KPIs) that are measured with a series of micro-indicators (MIs).

The micro-indicators are assessed using data from a survey and document review. The survey collects perception data from a variety of stakeholders (see Section 2.2) and the review of documents relies on a set of criteria that provide a basis for the assessment of each micro- indicator (see Section 2.3). However, not all micro-indicators are assessed by both the survey and the document review; consequently, some charts do not show survey scores and document review scores for each KPI or MI. The World Bank was assessed using 19 KPIs and 69 MIs. The full list of MIs assessed is provided in Volume II, Appendix V (KPI and MI Data by Quadrant).

2.2 Survey

To gather diverse perspectives on the multilateral organisations being assessed, MOPAN generally seeks the perceptions of the following primary respondent groups:

• Donor Headquarters Oversight (HQ): Professional staff, working for a MOPAN donor government, who share responsibility for overseeing / observing a multilateral organisation at the institutional level. These respondents may be based at the permanent mission of the multilateral organisation or in the donor capital.

• Donor Country Office Oversight (CO): Individuals who work for a MOPAN donor government and are in a position that shares responsibility for overseeing/observing a multilateral organisation at the country level.

• Clients: Typically, individuals who work for a national partner organisation (government or civil society) in a developing country. Respondents are usually professional staff from organisations that receive some sort of direct transfer from the multilateral organisation or that have direct interaction with it at country level (this could take the form of financial assistance, technical assistance, policy advice, equipment, supplies, etc.).

MOPAN donor respondents are chosen by MOPAN member countries. The other respondents are identified by the multilateral organisation being assessed.

The survey is customised for each organisation assessed and can be completed online in English, French, or Spanish or offline (paper, email, or interview) in these same languages. See Volume II (Appendix II) for the survey. Individual responses to the survey are confidential to the independent consultants managing the online survey or collecting data offline in the field.

Respondent Ratings – Survey respondents are presented with statements describing an organisational practice, system, or behaviour and asked to rate the organisation’s performance on a scale of 1 to 6 as shown below.

Figure 2.1 Respondent Rating Scale

In some cases, not all survey questions are answered, either because: 1) the individual chose not to answer, or 2) the question was not asked of that individual. In these cases, mean scores are calculated using the actual number of people responding to the question. As noted in the methodology (Volume II, Appendix I), ‘don’t know’ survey responses are not factored into the calculation of mean scores. However, when the proportion of respondents answering ‘don’t know’ is considered notable for a micro-indicator, this is indicated in the report. The responses of various categories of respondents on the six choices, plus ‘don’t know’ are summarised across all survey questions in Figure 2.2.

Figure 2.2 The World Bank - Distribution of Responses (n=219) on all Questions Related to Micro-Indicators

While there were responses in all six possible choices, relatively few responses overall were at the ‘weak’ end of the scale. Clients and donors at HQ were generally more positive in their ratings than donors in-country. Close to one-fifth (18 per cent) of the responses of MOPAN donors in-country were ‘don’t know,’ which is not unexpected given the potentially more limited familiarity of these groups with the operational specifics of the World Bank covered in certain questions.

Survey Response Rate

MOPAN aims to achieve a 70 percent response rate from donors at headquarters and a 50 per cent response rate among the population of respondents in each of the survey countries (i.e., donors in-country, direct partners, recipient governments and clients). The number of respondents targeted in each category (i.e., the total population) and the actual response rates are presented in Figure 2.3 below. Response rates of all categories of respondents exceeded the 50 per cent target rate. While there are variations in the response rates by category and location of respondents, the World Bank survey results reflect the views of 219 respondents.

Figure 2.3 Number of Survey Respondents for the World Bank by Country and Respondent

Converting Individual Scores to Group Ratings

As noted above, individuals respond to survey questions on a six-point scale where a rating of “1” is considered a judgment of “very weak” up to a rating of “6” intended to represent a judgment of “very strong.” A mean score is calculated for each respondent group (e.g., donors at HQ). Since mean scores are not necessarily whole numbers (from 1 to 6) MOPAN assigns numerical ranges and descriptive ratings for each range (from very weak to very strong) as shown below.

Figure 2.4 MOPAN Ranges and Descriptions

The ranges are represented to two decimal places, which is simply the result of a mathematical calculation and should not be interpreted as representing a high degree of precision. The ratings applied to the various KPIs should be viewed as indicative judgments rather than precise measurements.

Data Analysis

First level survey data analysis includes calculations of mean scores, medians, standard deviations, frequencies, (including analysis of ‘don’t know’ and missing responses), as well as content analysis of open-ended questions. The ‘don’t know’ responses are removed from the calculation of mean scores, but the proportion of respondents choosing ‘don’t know’ is retained as potentially useful data.

A weighting scheme is applied to ensure that no single respondent group or country is under- represented in the analysis. The weighting is intended to correct for discrepancies/variation in: the number of individuals in each respondent group, the number of countries where the survey took place, the numbers of donors in-country, direct partners, and other respondent groups within each country where the survey took place. Weighted figures are carefully reviewed and analysed before inclusion in the multilateral organisation reports.

Second level analysis examines differences in the responses among categories of respondents and other variables. When significant differences are found, these are noted in the report.[11] The survey included respondents in both IBRD countries and IDA/Blend countries, but there were no statistically significant differences in responses between the two groups.

For a full description of survey data analysis see Volume II, Appendix I.

2.3 Document Review

The document review considers three types of documents: multilateral organisation documents, identified with the help of the organisation; internal and external reviews of the organisation’s performance, found on the organisation’s web site or provided by the organisation; external assessments such as the Survey on Monitoring the Paris Declaration, the Common Performance Assessment (COMPAS) report, and previous MOPAN surveys

Ratings for key performance indicators (KPIs) are based on the ratings for the component micro-indicators in each KPI. For each micro-indicator, a set of criteria are established which, taken together, are thought to represent good practice in that area. The criteria are based on existing standards and guidelines (for example, UNEG or OECD-DAC guidelines), on MOPAN identification of key aspects to consider, and on the input of subject-matter specialists. The rating on any micro-indicator depends on the number of criteria met by the organisation. In cases where the micro-indicator ratings for one KPI are highly divergent, this is noted in the report.

While the document review assesses most micro-indicators, it does not assign a rating to all of them (when criteria have not been established). Consequently, some charts do not show document review scores for each KPI or MI. Documents are also used to aid in the understanding of the context in which the multilateral organisations work.

The document review and survey use the same list of micro-indicators, but some questions in the document review are worded differently from those in the survey. The document review and survey also use the same rating scale, but scores are presented separately on each chart in the report to show their degree of convergence or divergence.

2.4 Interviews

As of 2012, interviews are conducted at the headquarters of multilateral organisations with individuals who are knowledgeable in areas that relate to the MOPAN assessment.

Interviewees are asked to provide knowledge, insight, and contextual information that could assist the MOPAN Assessment Team in analysing document review data, and to identify other relevant documents for the Assessment Team to consider. This helps ensure that the Assessment Team has all the appropriate and necessary documents, enhances the Team’s ability to triangulate data from various sources, and assists the Assessment Team in the analysis of the key performance indicators by providing contextual information.

Interviews are conducted with a small number of staff who work in the primary units that relate to areas of the MOPAN assessment. Interviewees are identified by the multilateral organisation in conjunction with the Assessment Team and MOPAN. An interview guide is prepared and interviewees are advised of the content areas beforehand.

Data gathered during interviews is used to understand the context in which the agency is working, as well as how decisions are made. In the event that survey data present a picture that is very different from the document review, information from interviews can help clarify how the multilateral organisation approached a certain issue.

2.5 Basis for Judgment

From 2003 to 2009, the basis for judgment in MOPAN assessments was the perceptions of survey respondents. With the introduction of the document review in 2010 and interviews in 2012, judgments now draw on a variety of sources that can be compared and triangulated.

To the extent possible, the assessment standards and criteria are tailored to reflect the nature and operating environment of the multilateral organisations under review.

The MOPAN approach uses multiple data sources and data collection methods to validate findings. This helps eliminate bias and detect errors or anomalies.

The MOPAN reports gain trustworthiness through the multiple reviews and validation processes that are carried out by members of the network and by the multilateral organisations themselves.

2.6 Reporting

Institutional Reports

Individual institutional reports are produced for each multilateral organisation assessed. The results of the document review are presented alongside the survey results and discussed in light of the perception-based scores and interviews in order to further substantiate and contextualise the overall findings. For those agencies that were evaluated in 2009, a brief analysis of trends is included.

Country Data Summaries

A summary of survey results is produced for each multilateral organisation in each of the countries surveyed where sufficient survey data exists. Country Data Summaries provide feedback to those who participated in the MOPAN assessment and provide input for a dialogue process. They are not published and are shared only with individuals who attend the country workshop on the MOPAN assessment findings, which takes place in the first quarter of the year following the assessment.

2.7 Strengths and Limitations of Methodology

MOPAN continues to improve methodology based on the experience of each year of implementation. The following strengths and limitations should be considered when reading MOPAN’s report on the World Bank.

Strengths

• The MOPAN Common Approach is based on the core elements of existing bilateral assessment tools. In the long term, the intent is to replace or reduce the need for other assessment approaches by bilateral donors.

• It seeks perceptual information from different perspectives: MOPAN donors (at headquarters and in-country), direct partners/clients of multilateral organisations, peer organisations, and other relevant stakeholders. This is in line with donor commitments to the Paris Declaration on Aid Effectiveness and the Accra Agenda for Action regarding harmonisation, partner voice, and mutual accountability.

• It complements perceptual data with document review and interviews, thus using multiple sources of data. This should enhance the analysis, provide a basis for discussion of agency effectiveness, and increase the validity of the assessment through triangulation of data.

• The reports undergo a validation process, including multiple reviews by MOPAN members and review by the multilateral organisation being assessed.

• MOPAN strives for consistency across its survey questions and document review for each of the multilateral organisations, while allowing for customisation to account for differences between types of multilateral organisations.

Limitations

MOPAN Framework

• The countries are selected based on established MOPAN criteria and comprise only a small proportion of each institution’s operations, thus limiting broader generalisations.

• The Common Approach indicators were designed for multilateral organisations that have operations in the field. For organisations that have limited field presence or that have regional structures in addition to headquarters and country operations, there have been some modifications made in the data collection method and there will be a need for greater nuance in the analysis of the data.

Data sources

• The MOPAN Common Approach asks MOPAN members and the organisations assessed to select the most appropriate individuals to complete the survey. While MOPAN sometimes discusses the selection with the organisation being assessed, it has no means of determining whether the most knowledgeable and qualified individuals are those that complete the survey.

• The document review component works within the confines of an organisation’s disclosure policy. In some cases, low document review ratings may be due to unavailability of organisational documents that meet the MOPAN criteria (some of which require a sample of a type of document, such as country plans, or require certain aspects to be documented explicitly). When information is insufficient to make a rating, this is noted in the charts.

Data Collection Instruments

• Three issues potentially affect survey responses. First, the survey instrument is long and a fatigue factor may affect responses and rates of response. Second, respondents may not have the knowledge to respond to all the questions (e.g., survey questions referring to internal operations of the organisation, such as financial accountability and delegation of decision-making, seem difficult for many respondents, who frequently answer ‘don’t know.’) Third, a large number of ‘don’t know’ responses may imply that respondents did not understand certain questions.

• The rating choices provided in the MOPAN survey may not be used consistently by all respondents, especially across the many cultures involved in the MOPAN assessment. One potential limitation is ‘central tendency bias’ (i.e., a tendency in respondents to avoid extremes on a scale). Cultural differences may also contribute to this bias as respondents in some cultures may be unwilling to criticise or too eager to praise.

• Because one of MOPAN’s intentions is to merge previously existing assessment tools into one, and to forestall the development of others, the survey instrument remains quite long.

Data Analysis

• While the document review can serve to evaluate the contents of a document, it cannot assess the extent to which the spirit of that document has been implemented within the organisation (unless implementation is documented elsewhere).

• Mean scores are used in the MOPAN reports to provide central tendency values of the survey results. The mean has the advantage of being the most commonly understood measure of central tendency, however, there is a disadvantage in using the mean because of its sensitivity to extreme scores (outliers), particularly when population samples are small. The assessment team reviewed the median and standard deviations in analysing the survey results. Volume II, Appendix V provides the standard deviations for each survey question.

Basis for Judgment

• Although MOPAN uses recognised standards and criteria for what constitutes good practice for a multilateral organisation, such criteria do not exist for all MOPAN indicators. As a result, many of the criteria used in reviewing document content were developed by MOPAN in the course of the assessment process. The criteria are a work in progress and should not be considered definitive standards.

• The Common Approach assessment produces numerical scores or ratings that appear to have a high degree of precision, yet can only provide general indications of how an organisation is doing and a basis for discussion among MOPAN members, the multilateral organisation, and other stakeholders, including direct partners.

Despite some limitations, the Assessment Team believes that the MOPAN reports generally provide a reasonable picture of systems associated with the organisational effectiveness of multilateral organisations.

2.8 Testing a New Component in 2012: Assessing Development Results

Overview

Until 2012, MOPAN assessments focused on the organisational effectiveness of multilateral organisations by examining organisational practices, systems, and behaviours that MOPAN believes are important for managing to achieve development results. In 2012, MOPAN tested a component to assess how multilateral organisations measure and report on development results with four of the six organisations assessed: AfDB, UNDP, UNICEF, and the World Bank.[12]

Sub-Components

• KPI A – Evidence of extent of progress towards organisation-wide outcomes[13]

• KPI B – Evidence of extent of contributions to country-level goals and priorities

• KPI C – Evidence of extent of contributions to relevant MDGs

• KPI D – Relevance of objectives and programme of work to stakeholders.

The assessments at the institutional/organisational level (KPI A) and at the country level (KPI B) are separated due to differences in focus, scope and reporting on results at these two levels.

KPIs B, C, and D all focus on the country level and are applied in five countries. Each multilateral organisation is asked to identify the countries where they are likely to have the best data on results.

KPI D assesses relevance as a measure of the extent to which surveyed stakeholders consider the multilateral organisation supports country priorities and meets the changing needs of direct partners and the target population.

Methodology

Various types of qualitative and quantitative data are sought to answer a set of questions about the multilateral organisation’s performance in the indicators noted above. Data are collected using three different methods: document review, stakeholder survey, and interviews with staff at headquarters.

• The document review draws largely on organisational performance reports and country level performance reports and evaluations.

• The stakeholder survey asks donor and direct partner respondents to rate the organisation’s achievement of planned results and the relevance of its activities at the country level. The questions are tailored, as required, to each organisation.

• Interviews are conducted to identify reliable data, and to help contextualise the analysis of results.

Analysis of institutional level data focuses on the extent to which planned results from the strategic period were achieved, based largely on performance reports and thematic evaluations. Analysis of country level data focuses on the organisation’s contribution to results in the sample of countries selected for the MOPAN assessment.

The judgment of an organisation’s performance on each KPI draws on a set of criteria. The Assessment Team uses a “best fit approach,” a criteria-referenced basis for judgment that is suitable when criteria are multi-dimensional, there is a mix of both qualitative and quantitative data, and it is not possible to calculate a simple sum of the data points.[14]

The ratings reflect the Assessment Team’s judgment and analysis of data from all sources. The ratings are qualitative in nature and defined according to a 4-point scale – strong, adequate, inadequate, weak. As in the 6-point scale used in the survey, a rating of “strong” signals that the organisation is approaching best practice, while a rating of “weak” signals that the organisation still has important limitations in demonstrating progress towards its stated results, and particularly its contributions to development outcomes.

Limitations to the Methodology

The methodology was designed to draw on the evidence of results achieved, as presented in the reports of a multilateral organisation. However, there is a critical difference between assessing the actual results achieved on the ground and assessing the evidence of results in the organisation’s reports to its key stakeholders. This is a limitation that is inherent in the current approach.

MOPAN will review the experience with this pilot component and make adjustments in the methodology, as required.

3. Main Findings: Practices and Systems that Enable the Achievement of Results

3.1 Introduction

This chapter presents the findings of the 2012 MOPAN assessment of the World Bank’s organisational effectiveness, that is its practices and systems that support the achievement of results. Findings are based on respondent survey data and document review.

• Section 3.2 presents overall ratings on the performance of the World Bank and summarises respondent views on its primary strengths and areas for improvement;

• Section 3.3 provides findings on each of the four areas of performance (strategic, operational, relationship, and knowledge management).

3.2 Overall Ratings

This section provides a summary of overall ratings. It includes: survey respondent ratings of the World Bank’s overall organisational effectiveness, survey respondent views on the World Bank’s strengths and areas for improvement, and survey and document review ratings for all key performance indicators.

Survey ratings of the World Bank’s organisational effectiveness

MOPAN has defined “organisational effectiveness” as the extent to which a multilateral organisation is organised to support direct partners in producing and delivering expected results. Respondents were asked to rate the organisational effectiveness of the World Bank on a scale from 1 to 6, where 1 means “not effective at all” and 6 means “very effective”. Out of all respondent groups, clients responded most positively, with 51 per cent rating the Bank as 5 or 6, followed by donors at headquarters (46 per cent). Donors in-country responded most negatively overall, with 29 per cent rating the Bank as 3 or lower.

Figure 3.1 Overall Ratings of World Bank Organisational Effectiveness by Respondent Group

Respondents’ Views on the World Bank’s Strengths and Areas for Improvement

The survey included two open-ended questions that asked respondents to identify the World Bank’s greatest strengths and areas of improvement. All 219 respondents provided responses to both questions.[15] The comments summarised below illustrate the range of perceptions; they have been qualified by the approximate proportion of respondents holding such a view.

Overall, survey respondents considered the World Bank’s greatest strengths to be its capacity to generate and disseminate knowledge and the professionalism and expertise of its staff.

Overall, 29 per cent of stakeholders commented on the Bank’s capacity to generate and disseminate knowledge, followed by comments on the professionalism and high level of expertise of its staff (25 per cent), its financial capacity (16 per cent), and its understanding of development issues at the country level (15 per cent).

There were some differences among respondent groups.

• Donors at headquarters: 46 per cent commended the Bank as a global repository of development knowledge, 34 per cent on its convening power and global networking, and 28 per cent on its exceptional pool of talent

• Donors in-country: 33 per cent commended the Bank for its ability to share knowledge, 33 per cent for its robust financial capacity, which allows for the development of large-scale projects with high impacts; and 25 per cent for its exceptional pool of talent.

• Clients: 23 per cent commended the Bank for its exceptional pool of talent, 18 per cent for its ability to share knowledge, and 18 per cent for its long-standing experience in a large number of developing countries.

Survey respondents suggested that the Bank could improve its coordination with other donors and be more flexible in its procedures.

When asked to comment on areas for improvement, 24 per cent of respondents commented on the Bank’s bureaucracy and procedures and 17 per cent on its harmonisation with other donors.

Respondents from all groups commented that the Bank’s rigid procedures create a burden that limits the effectiveness of the organisation’s operations. They also noted that procedures that do not take into account country specificity can significantly delay disbursement of funds and project implementation. Clients suggested that the Bank should simplify its procurement procedures, especially with regard to the no-objection system.

Both donors in-country (40 per cent) and donors at headquarters (30 per cent) felt the Bank needs to improve harmonisation and coordinate better with other donors in the field.

Overall Ratings of Key Performance Indicators

Figure 3.2 below shows scores from the document review and the survey on key performance indicators (KPIs) in the MOPAN 2012 assessment of the World Bank. The white bar presents the survey score, while the black square presents the document review score. For example, on the first indicator, “providing direction for results”, the World Bank received a score of 4.70 (strong) in the survey and a score of 5 (strong) in the document review.

The World Bank received scores of adequate or better on all 19 KPIs assessed in the survey. The World Bank received scores of adequate or better on all 17 KPIs assessed in the document review and 13 of these were strong or very strong.[16]

The survey and document review ratings differed on eight KPIs, all of which were rated higher by the document review than by survey respondents.

Figure 3.2 Overall Ratings on Key Performance Indicators (mean scores, all respondents and document review ratings)

3.3 The World Bank’s Performance in Strategic, Operational, Relationship, and Knowledge Management

3.3.1 Overview

This section presents the results of the 2012 Common Approach assessment of the World Bank in four performance areas (quadrants): Strategic, Operational, Relationship, and Knowledge Management.

The following sections (3.3.2 to 3.3.5) provide the overall survey and document review ratings for the KPIs in each quadrant, the mean scores by respondent group, and findings based on an analysis of survey and document review ratings in each quadrant.

When there were notably divergent ratings between survey respondent groups or between the survey results and document review ratings, these are noted and the information gleaned from interviews with staff is integrated when it has a bearing on the analysis. Differences between respondent groups are reported only if they are statistically significant.[17]

The survey data for each KPI and MI by quadrant are presented in Volume II, Appendix V. The document review ratings are presented in Volume II, Appendix VI.

3.3.2 Strategic Management

Strategic management is a clear area of strength for the World Bank and it is seen to actively promote the organisation-wide achievement of development and management results. The Bank has strengthened its country level results frameworks but the links between outputs, outcomes and impacts could still be improved in its corporate results frameworks.

Figure 3.3 shows the overall survey and document review ratings for the four KPIs in the strategic management quadrant.

In the area of strategic management, the Bank was considered strong in providing direction for results and focusing on results at the country level. While it has demonstrated an increased focus on and commitment to results, there is room for improvement in its corporate results frameworks. The Bank has invested significantly in the incorporation of cross-cutting themes in its work and was rated strong in the document review. Survey respondents rated the Bank’s efforts in this area as adequate.

Figure 3.3 Quadrant I: Strategic Management, Survey and Document Review Ratings

Figure 3.4 shows the mean scores for the four KPIs for all survey respondents, and by category of respondent.

Figure 3.4 Quadrant I: Strategic Management, Mean Scores by Respondent Group

KPI 1: Providing Direction for Results

Finding 1: The World Bank received high ratings from surveyed stakeholders for its leadership on results and transparency in providing access to information. The document review recognised the Bank’s efforts to become an “Open Bank” as moving towards best practice.[18]

Survey respondents noted the leadership provided by senior management on the results agenda and the accessibility of key documents available to the public. The document review, which assessed only one MI on making documents available to the public, recognised the Banks 2012 Open Access policy as an example of the organisation’s commitment to transparency.

Figure 3.5 KPI 1: Providing Direction for Results, Ratings of Micro-Indicators

MI 1.1 – Value system supports results-orientation and client focus

This MI was assessed by survey only. All survey respondents were asked whether the World Bank’s institutional culture reinforced a focus on results, as well as whether it was client focused. Respondents rated the Bank strong for its focus on results and adequate for its client orientation.

MI 1.2 – Leadership on results management

MOPAN donors at headquarters were asked whether senior management showed leadership on results management. More than half of respondents (61 per cent) rated the Bank strong or above.

MI 1.3 – Key documents available to the public

This MI was assessed by both the survey and document review. The majority of survey respondents (71 per cent) rated the Bank’s performance as strong or higher in making key documents readily available to the public. Out of all respondent groups, donors at headquarters rated the organisation the highest, with 85 per cent of respondents giving a rating of strong or higher, and 57 per cent giving a rating of very strong.

The document review found the World Bank very strong in this area. The Bank’s landmark openness agenda has led to the creation of an Open Bank, which commenced with the launch of the Bank’s Open Data Initiative in April 2010 and the Access to Information Policy in July 2010. The Open Data Initiative encompassed a range of reforms leading to free public access to development indicators and information on Bank finances and projects. The Access to Information Policy, which built on the 2001 Disclosure Policy, increased public access to information on Bank operations and the proceedings of the Board of Executive Directors. The adoption of this policy reflects the Bank’s strong commitment to global transparency initiatives such as the International Aid Transparency Initiative (IATI). In 2012, the Bank also implemented a new Open Access Policy for research and knowledge, and launched an Open Knowledge Repository which stores all the Bank’s research outputs.

The Bank has made efforts to ensure that most of its key documents are publicly available on the World Bank website. All documents are available in English and many are also translated into other languages.[19]

KPI 2: Corporate Focus on Results

Finding 2: Stakeholders surveyed rated the World Bank adequate in terms of the strategies and frameworks that determine its corporate focus on results. The review of documents rated the Bank’s practice as strong.

All MIs under this KPI were assessed by document review and two of these were also rated in the survey by donors at headquarters.

In defining itself as a results-focused multilateral organisation,[20] the World Bank has espoused a country-driven model for the achievement of development results at country level, and a business modernisation agenda to guide the attainment of corporate objectives.

The review of documents for this KPI considered both IDA and World Bank frameworks for tracking and reporting on results.

The IDA Results Measurement Framework, launched in 2002 during the 13th IDA replenishment and expanded in IDA 16, now comprises four tiers of performance indicators that are used to track development and agency effectiveness.

In 2011, the World Bank introduced the Corporate Scorecard, an innovative mechanism to track organisation-wide progress towards results achievement. The Corporate Scorecard integrates the IDA Results Measurement Framework and was expanded for use in tracking Bank- supported country results (outputs and outcomes) leading to more long-term development outcomes achieved by the Bank’s member countries. It was first published in print in 2011, followed by an electronic version in 2012. The Corporate Scorecard is described as a “living document” that will evolve and improve over time.

During an interview with World Bank staff, it was highlighted that the organisation is continuously striving to improve its focus on results. As such, three initiatives will be implemented in FY2013, including: 1) expanding client and beneficiary feedback mechanisms to enhance the focus on project design and results; 2) increasing the number of project-level impact evaluations; and 3) testing new long-term institutional development impact indicators.

Figure 3.6 KPI 2: Corporate Focus on Results, Ratings of Micro-Indicators

MI 2.1 – Organisational strategy based on clear mandate

MOPAN donors at headquarters were asked two questions on this MI: one about the clarity of the Bank’s mandate, and the other on the extent to which its organisational strategy is aligned with its mandate. While the Bank received a rating of strong for the clarity of its mandate, stakeholders rated the Bank adequate on the extent to which it links its strategy to the mandate. Overall, the majority of respondents (55 per cent) rated the Bank’s performance as adequate or higher.

The World Bank received a rating of very strong from the document review, based on the articulation of its mandate in the Post-Crisis Directions Paper (PCD) “Synthesis Paper: New World, New World Bank Group”, which provides the long-term strategic framework for the World Bank (IBRD and IDA). The document review also found that the Bank’s strategic priorities are aligned with its mandate for poverty reduction and sustainable growth.

MI 2.2 – Organisational policy on results management

MOPAN donors at headquarters were asked whether the World Bank has ensured the application of results management across the organisation. The majority of respondents (84 per cent) rated the Bank’s performance as adequate or above.

The document review rated the Bank as very strong for its organisation-wide policy on results management which is reflected in a suite of documents that includes: the Corporate Scorecard (2011) and companion report, World Bank for Results 2011, IDA 16 Delivering Development Results (2011), IDA 16 Implementation Framework Report (2010), and Country Assistance Strategies: Retrospective and Future Directions (2009).

In 2003, the Bank established a Results Secretariat within the Operations Policy and Country Services Vice-Presidency to coordinate the Bank’s activities for managing for development results at the corporate and country levels. The Results Secretariat works with client countries to strengthen country capacity in managing for development results. In order to build internal capacity in results management, the Bank administers a mandatory e-learning module on results-based management for new staff, and also offers an advanced e-learning module for all staff. Upon completion of these modules, staff can also take other courses in RBM at headquarters or a regional hub.

The World Bank holds its clients accountable for results-based management and loan agreements stipulate that recipients must regularly monitor, evaluate and report on the results of a project for which a loan was disbursed. In addition, the Bank has been actively working towards strengthening national statistical systems of recipient countries and is part of the CAP- Scan Working Group whose efforts aim at assessing the strengths and weaknesses of countries in managing for results.

In 2009, the Bank launched the Investment Lending Reform with the objective of moving from a process culture to a results culture. Through the IL reform, the Bank has committed to modernising and consolidating its investment lending instrument by (a) adopting a risk-based approach, (b) enhancing implementation support, (c) rationalising a menu of financing options, (d) establishing a better enabling environment, and (e) reforming policy. The first four elements have been implemented and the fifth is currently being addressed by management. In addition, the Bank has introduced a new lending instrument, Program-for-Results, which aims at better linking disbursements to results.

MI 2.3 – Plans and strategies contain results frameworks

This MI was assessed by document review only which rated the Bank strong. The Bank has four-tier results frameworks in its organisation-wide plans and strategies for IDA and IBRD.

The IDA 16 Results Measurement System (RMS) includes expected outcomes and outputs for development results, but these are not labelled as such. The “sectoral outputs” in Tier II, for example, include both output and outcome indicators. As noted below (MI 2.5), the framework provides “projected outputs” as targets for several Tier II indicators, but it is unclear to what extent these are or will be sufficient to increase the likelihood that IDA countries will achieve MDGs or other Tier 1 indicators. The IDA RMS includes performance standards that are adequate to measure the Bank’s management results.

The Corporate Scorecard, introduced in 2011, provides an integrated results and performance framework that applies to the World Bank. It is based on the IDA system but expanded to include additional quantitative indicators. It is also presented in four tiers that group performance indicators along an organisation-wide results chain that goes from Tier IV, which focuses on the Bank’s organisational effectiveness, to Tier I, which tracks development progress in the Bank’s client countries.

MI 2.4 – Results frameworks link outputs to final outcomes/impacts

This MI was assessed by the document review only, which rated it inadequate. The World Bank’s development and management results frameworks are articulated in the Corporate Scorecard, which presents an overview of results achieved based on performance indicators in four tiers:

• Tier I (global development context) and Tier II (country results supported by the Bank) represent the Bank’s development results framework.

• Tier III (development outcomes and operational effectiveness) and Tier IV (organisational effectiveness and modernisation) represent the Bank’s management results framework. They are used to monitor Bank effectiveness and efficiency, and therefore categorise performance indicators under areas of corporate focus.

• Within each tier, the Scorecard notes which performance indicators are associated with the Bank’s strategic priorities (in its Post-Crisis Directions paper) and with the higher development outcomes of the Millennium Development Goals (MDGs).

The management results framework contains a set of implicit assumptions about the results chain between Tiers II and IV. The fundamental premise is that a well-managed organisation (Tier IV) will enhance the effectiveness of projects and other products (Tier III) which will improve the Bank’s contributions to results at the country level (Tier II). However, the development results framework does not provide any implicit or explicit links between outputs and organisation-wide development outcomes. While the indicators under Tier II focus on results achieved at the country level, the Bank’s strategic objectives (reflected in the Post-Crisis Directions) are not only achieved through country level results. Thus, the current frameworks do not provide a complete picture of how the organisation as a whole is conceiving its contributions to development change.

MI 2.5 – Plans and strategies contain performance indicators

This MI was assessed by document review only, which rated the Bank strong for its inclusion of standard performance indicators in its organisation-wide plans and strategies.

In the IDA 16 RMS, Tier II tracks core sector indicators that provide a foundation for the standardised indicators now being used in the Corporate Scorecard. Furthermore, the IDA 16 RMS indicators in Tier II (Section B) include “projected outputs” by 2015, which can serve as targets, assuming there is demand for these services among IDA countries.

In the Corporate Scorecard, both the development results framework (Tiers I and II) and management results framework (Tiers III and IV) contain performance indicators that are adequate, relevant and clear. The Bank monitors indicators in Tiers I and II by comparing current values to baselines, using observed changes to denote progress towards results achievement as well as shifts in demand. The Scorecard notes that output indicators in Tier II will be replaced by outcome indicators as more country data becomes available. As with the IDA 16 RMS, “performance standards” are used to monitor the indicators in Tiers III and IV. All indicators are clearly defined in the Scorecard.

KPI 3: Focus on Thematic Priorities

Finding 3: The World Bank was rated adequate or better for incorporating cross-cutting priorities in its strategic framework.

In its Post-Crisis Directions Paper, which articulated its organisation-wide strategic priorities the World Bank identified four cross-cutting themes: gender equality, climate change, governance, and fragile and conflict-affected states. The Bank has made efforts to mainstream and/or incorporate each of these themes across its operations. Survey respondents rated the Bank adequate overall in mainstreaming cross-cutting themes and the document review provided an overall rating of strong.

Figure 3.7 KPI 3: Focus on Thematic Priorities, Ratings of Micro-Indicators

MI 3.1 – Gender equality

All respondent groups were asked whether the World Bank sufficiently mainstreams gender in its operations. Overall, they rated the Bank adequate. Clients were more positive than other respondent groups and the difference is statistically significant.

The document review rated the World Bank strong in gender mainstreaming. The difference between this rating and the perception data may reflect the Bank’s extensive work at the policy and guidance level which has not yet been translated into visible effects for donors.

The Bank has developed a suite of strategic documents to guide its work in mainstreaming gender, including Operational Policy 4.20 and Bank Procedure 4.20 of the World Bank’s Operations Manual; the World Bank Group Gender Action Plan (2007 – 2010); the Three-Year Road Map for Gender Mainstreaming for the period 2011-2013; and Implications of World Development Report 2012: Gender Equality and Development for the World Bank Group. The strategic framework for the IDA 16th replenishment also included gender as a cross-cutting theme.

The World Bank increased its funding for the implementation of the Gender Action Plan to US $70 million from the original planned allocation of US $25 million. It also incorporated the recommendations of a 2010 independent evaluation of its work in gender and development,[21] by taking steps to strengthen its results framework and management accountability for gender mainstreaming and to disaggregate results by gender. Following the 2010 IEG evaluation of gender and development, the Bank launched the Three-Year Road Map for Gender Mainstreaming in 2011. The Road Map includes specific time-bound performance targets and indicators for Bank results across sectors and regions and is being used to strengthen management accountability.

The World Bank has continued to make progress in mainstreaming gender through capacity building for staff and the publication and dissemination of guidance notes on gender. According to interviews with Bank staff, as of 2012, 100 per cent of Country Assistance Strategies have been gender informed (i.e., gender has been systematically integrated into the development of the CAS document). Despite the notable progress made on gender mainstreaming, an IEG review has mentioned the need to strengthen accountability structures on gender. The Bank is currently taking steps to implement this recommendation.

MI 3.2 – Fragile and conflict-affected countries

All three respondent groups were asked whether the World Bank has sufficient cross-cutting focus on fragile and conflict-affected countries. The majority of respondents (70 per cent) rated the Bank adequate or above. Close to one-third (31 per cent) of donors based in country offices responded ‘don’t know’.

The document review rated this MI as strong.

The World Bank incorporated work in fragile and conflict-affected countries as a strategic priority in its organisation-wide strategic plan and as a special theme in the 16th replenishment of the IDA. It also endorsed the New Deal for Engagement in Fragile States as part of its business modernisation agenda. The New Deal has challenged the development community to support goals of peace and state-building for sustainable poverty reduction and growth. The Bank also created several Trust Funds to support its work in fragile and conflict-affected countries. During the IDA 15th replenishment, 10 per cent of IDA’s resources were allocated to fragile and conflict-affected states. IDA has forecasted a reduction to 6.7 per cent during the 16th replenishment, as countries complete their four-year post-conflict allocation and begin a six-year phase-out period.

Since the publication of the 2009 Mid-Term Review of the IDA 15th replenishment, which highlighted several areas for improvement in fragile and conflict-affected situations (FCS), the Bank has implemented a number of recommendations from the review, namely by focusing on FCS in the 2011 World Development Report, strengthening its partnership on FCS with the UN, and increasing global partnerships by engaging in the OECD’s International Network on Conflict and Fragility (INCAF) and in the International Dialogue process and g7+. The Bank has also strengthened its commitment to FCS by increasing field presence and has recently established a global FCS hub in Nairobi which provides operational and policy advice to country staff. Results statements for this thematic area are included in the Corporate Scorecard and in the Matrix of Action for the report Operationalizing the 2011 World Development Report. Through the adoption of a framework for the provision of rapid emergency assistance in 2007, the Bank has also committed its assistance in cases of crisis or disaster in high risk states.

MI 3.3 – Good governance

Survey respondents from all three groups were asked whether the World Bank sufficiently promotes the principles of good governance in its operations. The Bank received an overall rating of strong for this MI.

The document review rated the World Bank’s performance in this area as adequate.

In 2009, the Bank introduced a compulsory discussion on good governance and corruption during the development of all Country Assistance Strategies. However, the 2011 independent evaluation of the 2007 Strategy and Implementation Plan[22] found that the Bank’s strategy in country-led good governance and anti-corruption had focused on the Bank’s capacities, resources and reputation, at the expense of strategic issues in partner countries. The Bank has since adopted an outward-looking approach to good governance in the 2012 strategy.

In addition to its Updated Strategy and Implementation Plan (2012) for good governance and anti-corruption, the Bank has taken steps to integrate the principles of good governance in its programming. It has recruited dedicated good governance and anti-corruption staff; supported strengthening of sanction regimes; and provided support to global efforts such as the Stolen Assets Recovery initiative. The Bank’s allocation to good governance work increased from US$140 million in 2004 to US$ 169 million in 2010 (a 21 per cent increase). In addition, the World Bank allocated a total of US$1.3 billion (13.4 per cent of its lending) during the first two quarters of 2012 to improvements in performance and accountability of core public sector institutions, as well as the rule of law in client countries.

MI 3.4 – Climate change

When asked whether the World Bank has sufficient cross-cutting focus on climate change, survey respondents gave the Bank an overall rating of adequate. One-quarter of donors in- country (26 per cent) responded ‘don’t know’.

In the document review, the World Bank received a rating of strong for its work in climate change. The Bank has adopted the promotion of global collective action, including the issue of global climate change, as a strategic priority in its organisation-wide strategic plan. Climate change was also identified as a special theme for the IDA 16th replenishment, resulting in IDA finance to climate-affected sectors in the amount of US $ 3.3 billion in 2009 (an increase of 17 per cent over the IDA 15th replenishment).[23] The World Bank Group recently re-committed to climate change in an updated environmental strategy for 2012-2022, Towards a Green, Clean, Resilient World for All. The IBRD is the trustee of the Climate Investment Fund, which comprises the Clean Technology and Strategic Climate Trust Funds. In addition, Tier III of the IDA 16 Results Measurement System indicates that 100 per cent of CASs should discuss climate change and monitors performance against this standard. While no data were available on this indicator in 2011, interviewed staff at headquarters noted that there has been significant improvement in integrating climate change in the development of the CAS in recent years.

KPI 4: Country Focus on Results

Finding 4: Both survey respondents and the document review rated the World Bank as strong for the results focus of its Country Assistance Strategies.

At the core of the World Bank’s country-driven model is the Country Assistance/Partnership Strategy (CAS/CPS) which is used to develop a country-focused approach to results achievement. The CAS/CPS is an indicative management tool that guides negotiations between the World Bank and country authorities for the selection and implementation of activities that are aligned with national development priorities. Each CAS/CPS contains a results framework that is designed to support project monitoring through to completion, including the tracking of progress towards results achievement.

Since the mainstreaming of the results-based CAS/CPS in 2005, the Bank has worked with country authorities to improve subsequent CAS/CPS documents. It has also published a number of documents which give guidance to staff on how to develop a good results matrix for CAS/CPS documents. For the MOPAN assessment of KPI 4, the document review was based on the most recent CAS/CSP in sample countries, and rated the Bank strong on all MIs in this performance area.

Figure 3.8 KPI 4: Country Focus on Results, Ratings of Micro-Indicators

MI 4.1 – Frameworks link results at project, programme, sector and country levels

Clients and donors in-country rated the World Bank strong for the way in which CAS/CPS link results from project, sector and country levels. Clients rated this MI higher than MOPAN donors in-country and the difference is statistically significant.

The document review rated the Bank strong on this MI given the practices evident in the current CAS/CSP. A central component of the CAS/CPS is the results framework, which outlines expected country-specific development results of Bank-supported activities. The CAS/CPS frameworks reviewed included results statements at both output and outcome levels and most statements were appropriate to their level (i.e., outputs were actually outputs; outcomes were actually outcomes). For the sample of countries reviewed, there is a notable improvement in the results framework of the current CAS, when compared to the results framework of the previous CAS period.

MI 4.2 – Frameworks include indicators at project, programme, sector and country levels

Clients and donors in-country were asked whether the results frameworks included indicators at all levels. Overall, respondents rated the organisation strong. Clients responded more positively than donors in-country and the difference is statistically significant.

The document review rated the Bank strong on this MI. All the results frameworks reviewed contained performance indicators at the level of projects, programmes and sectors, as well as at the country-level. All current frameworks reviewed contained indicators that met the CREAM criteria (i.e., clear, relevant, economic, adequate, and monitorable). While the World Bank indicates the sources of data and data collection methods to be used for monitoring projects in project appraisal documents, the results framework of CAS/CPS do not provide similar references for monitoring of expected outcomes in the Bank’s overall country programming. The document review noted that there has been an improvement in quality and consistency of results frameworks since the launch of results-based CAS/CPS documents in 2005.

MI 4.3 – Expected results consistent with PRSP or national plan

Clients and donors in-country were asked whether Country Assistance Strategies contained statements of expected results consistent with those in national development strategies. Overall, they rated the Bank strong.

This MI was rated very strong by the document review. All strategy documents reviewed demonstrated their relevance to the country national development strategy. There was an explicit link between World Bank-supported expected development results and national development strategies across all the countries sampled. As an example, the CPS for Morocco referenced the National Initiative for Human Development.

MI 4.4 – Expected results developed in consultation with clients/beneficiaries

This MI was assessed by survey only. Clients and donors in-country were asked whether the World Bank develops expected results in consultation with clients. The Bank was rated adequate or above by 87 per cent of clients and 70 per cent of donors at country level. Clients were more positive than donors in-country and the difference is statistically significant.

MI 4.5 – Results for cross-cutting priorities included in country level frameworks

Clients and donors in-country were asked whether the World Bank included results for cross- cutting priorities in Country Assistance Strategies (i.e., gender equality, fragile and conflict- affect countries, climate change, and good governance). Overall, stakeholders rated the Bank adequate (87 per cent of clients and 65 per cent of donors in-country). Close to one-quarter of donors in-country (24 per cent) responded ‘don’t know’.

The document review rated the World Bank strong on this MI. All of the CAS/CPS reviewed identified two or more of the organisation’s cross-cutting priority themes and most included strategies to apply the cross-cutting themes.

As noted above in MI 3.3, a discussion on governance and corruption is mandatory for the Country Assistance/Partnership Strategy. In addition, Operational Policy 4.20: Gender and Development stipulates that 100 per cent of all Country Assistance/ Partnership Strategies must be gender informed, and this was the case in the sample of CAS/CPS documents reviewed. Finally, the IDA 16 Performance Standards of the IDA Report Card indicate that 100 per cent of CASs should discuss climate change vulnerabilities.

3.3.3 Operational Management

The World Bank was noted for its strong performance in financial accountability and the allocation of resources. There is room for improvement in the linking of its disbursements to results, ensuring that more client services are managed at country level, and in some aspects of the management of human resources.

Figure 3.9 below shows the overall survey and document review ratings for the KPIs in the operational management quadrant. The World Bank received ratings of adequate or strong on both the survey and document review for all KPIs assessed under operational management.

Figure 3.9 Quadrant II: Operational Management, Survey and Document Review Ratings

Figure 3.10 shows the mean scores for the KPIs for all survey respondents, and by respondent groups.

Figure 3.10 Quadrant II: Operational Management, Mean Scores by Respondent Group

KPI 5: Resource Allocation Decisions

Finding 5: The World Bank was noted for its transparency in resource allocation decisions. Its criteria for allocating IDA resources are publicly available and the majority of survey respondents considered that allocations follow these criteria. The Bank has also committed to improving the predictability of funding.

The criteria for allocation of IDA resources are publicly available on the World Bank’s website. The Bank uses one set of criteria for performance-based allocations and special allocation criteria for post-conflict and re-engaging states. The Bank has indicated that both criteria are revised periodically. In the 2011 edition of Publish What You Fund, the World Bank was identified as a global leader in aid transparency due to the amount of information it publishes on its aid to programmes.

Figure 3.11 KPI 5: Resource Allocation Decisions, Ratings of Micro-Indicators

MI 5.1 – Criteria for allocating funding publicly available

This MI, which was assessed by document review only, received a rating of very strong. The IDA has developed two systems for resource allocation to country programmes. The performance-based allocation system allocates resources to eligible countries based on their policy performance and institutional capacity. Post-conflict countries, as well as those re- engaging with the IDA after prolonged disengagement, receive funding under IDA’s exceptional allocation system; country eligibility for this type of funding is based on performance against the Post-conflict Performance Indicators Framework and/or portfolio rating. Each allocation system is reviewed periodically and is publicly available on the IDA website at http://www.worldbank.org/ida/.

MI 5.2 – Resource allocations conform to criteria

This MI was assessed by survey only. All three respondent groups were asked whether the World Bank allocates funding according to its established criteria. Overall, more than half of stakeholders rated the Bank as strong or above. Statistically significant differences were noted between respondent groups as clients provided higher ratings than other groups. In addition, one-quarter of donors in-country responded ‘don’t know.’ This may be due in part to limited interaction of in-country donors with the World Bank on resource allocations and/or to the complexity of the allocation systems used by the Bank.

MI 5.3 – Resources released according to agreed schedules

This MI was assessed by document review, which rated it adequate. The review was based on data from the 2011 survey on monitoring the Paris Declaration, with emphasis on Indicator 7 (which measures the predictability of aid disbursements).

In 2005, the Bank disbursed 65 per cent of its aid according to schedule; this increased to 71 per cent in 2007, and decreased to 61 per cent in 2010 – below its target of 83 per cent. The World Bank explained that it has become increasingly difficult to maintain the schedule of disbursements as IDA’s financial envelope is replenished according to the commitments of donors who have been facing fiscal constraints as a result of the global financial crisis. To continue to meet the demand of its clients, the Bank injected capital into the financial envelope after the start of the financial crisis in 2008.

The Bank also suggested that there are methodological problems related to the OECD indicator. First, the indicator focuses on annual disbursements as opposed to aid predictability over the short- to medium-term. Second, the indicator does not consider the issue of accountability and combines data from donors and recipient countries. In order to strengthen its reporting in this area, the World Bank included an indicator on aid predictability in Tier III of the results measurement system for the 16th IDA replenishment.

KPI 6: Linking Aid Management to Performance

Finding 6: Donors at headquarters rated the World Bank adequate in linking its administrative budget and its disbursements to results. The document review also found the Bank adequate in linking allocations to results, but inadequate in linking disbursements to reported results.

This KPI focuses on the link between organisational administrative budget resources and results. The MOPAN assessment looks at two different dimensions related to this KPI: whether the World Bank links its administrative budget for its loans and grants programmes to expected results; and whether the Bank's corporate reports on results include the amounts disbursed linked to achievement of outputs.

Donors at headquarters, the only respondent group asked, rated the Bank adequate on both MIs. The document review rated the Bank adequate on the first MI, but inadequate on the second.

It is important to note that the World Bank has been revising its approach to business planning and several changes are being introduced to align the work of units with the Corporate Scorecard, through the establishment of Memoranda of Understanding (MOU) agreed with Senior Management. As these MOU are integrated into practice, it will facilitate the link between the corporate-level budget and the results articulated in the results frameworks (analysed in KPI 2) and performance reports (analysed in KPI 18), both of which refer to the Corporate Scorecard.

Figure 3.12 KPI 6: Linking Aid Management to Performance, Ratings of Micro-Indicators

MI 6.1 – Allocations linked to expected results

Donors at headquarters were asked whether the World Bank links its administrative budget for its loans and grants programmes to expected results. More than half of respondents (57 per cent) rated the Bank adequate or above on this MI.

The document review rated the Bank as adequate in linking administrative budget allocations to expected results at an organisation-wide level. Although the World Bank’s corporate budget does not explicitly link budget allocations to expected development outcomes, it is based on an in-depth business planning process informed by various analytical papers and reporting under Tier IV of the Corporate Scorecard. As an example, the Quarterly Business and Risk Report (QBR) produced by the Bank`s Management provides links between costs of input (staff, travel etc.) to various output targets and measures on operational effectiveness. Given the Bank`s country-based business model, the administrative budget allocation is largely country-driven and aligned with CAS priorities. According to the Bank, the most direct link between results and resource allocations are therefore found at the country level.

This being said, a full implementation of results-based budgeting at the corporate level would mean explicitly linking administrative budget allocations to the Bank’s products and services (including the preparation and supervision of loans and production of economic and sector work and other knowledge products and services).[24]

MI 6.2 – Disbursements linked to reported results

Donors at headquarters were asked whether the World Bank’s reports link the amounts disbursed to the achievement of outputs and outcomes. Close to two-thirds of respondents (64 per cent) gave a rating of adequate or above.

In the document review, this MI thus focused on corporate level reporting on budget and finance in relation to results. The document review rated this MI inadequate as corporate documents on budgeting and financial reporting reviewed do not yet align organisation-wide disbursements or expenditures with results.

At the level of projects, the Bank has increased efforts to link finance to performance. The World Bank recently introduced the Program for Results (P4R),[25] a results-based lending instrument that links loan disbursements to achievement of development results. However, the P4R was not considered in rating this MI as it is a lending instrument and the indicator focuses on organisation-wide or corporate reports. In addition, Implementation Completion Reports provide information on the resources used in relation to the completed outputs and overall project implementation.

KPI 7: Financial Accountability

Finding 7: The World Bank has strong policies and processes for ensuring financial accountability.

All seven MIs in this KPI were assessed by the document review and six of these by the survey.

On average, surveyed stakeholders rated the Bank strong in this area and the document review found it strong or very strong in all areas. The Bank has developed policies and procedures to ensure financial audits, manage risk, and combat fraud and corruption.

IDA and IBRD financial statements are prepared in conformity with accounting principles generally accepted in the United States of America; the annual audits are conducted in accordance with auditing standards generally accepted in the US and in compliance with International Standards on Auditing. External audits are also conducted at country level on Bank-supported projects and programmes, and comply with international auditing standards.

Figure 3.13 KPI 7: Financial Accountability, Ratings of Micro-Indicators

MI 7.1 – External financial audits performed across the organisation

Donors at headquarters were asked whether the World Bank’s external audits are meeting the needs of donors. Overall, 52 per cent of respondents gave a rating of strong or higher.

The document review rated the World Bank very strong for conducting organisation-wide financial audits. The World Bank Annual Report, which covers both IBRD and IDA, provides separate audits of each institution and includes letters from external auditors confirming that these audits have been undertaken according to international standards.

MI 7.2 – External financial audits performed at the regional, country, or project level

Donors in-country and clients were asked whether the World Bank performs appropriate external audits on programmes and projects at country level. Overall, 59 per cent of respondents rated the Bank strong or above. Clients provided higher ratings than donors and the difference is statistically significant.

The World Bank received a rating of strong in the document review on the appropriateness of its country-level external audits of programmes and projects. The document review found evidence of external audits conducted on World Bank-financed projects, by region, over fiscal years 2010 and 2011. These were conducted in accordance with auditing standards that had been approved by the World Bank (e.g., standards issued by the International Organization of Supreme Audit Institutions and the International Standards on Auditing issued by the International Federation of Accountants). The Bank has also been participating jointly with the IMF in the Reports on the Observance of Standards and Codes (ROSC) Initiative which aims, on the one hand, at analysing the compatibility of national accounting systems with international standards and, on the other hand, at proposing a plan of action for countries to improve their accounting practices.

MI 7.3 – Policy on anti-corruption

This MI was assessed only by document review which rated the Bank very strong for its policy on anti-corruption. The corporately approved policy is operationalized by the Integrity Vice Presidency, which is mandated to address allegations of fraud, corruption and staff misconduct across the organisation. In 2009, the World Bank adopted the Integrity Vice Presidency Strategic Framework, followed by a set of integrity compliance guidelines on sanctions and debarment in 2010. The framework outlines operational measures for the elimination of fraud and corruption in operations funded by the Bank. These include: monthly staff meetings to discuss pending investigations, fraud and corruption trends, and lessons learned; sensitisation of operational staff to fraud and corruption; sanctions against firms and individuals accused of fraud and corruption; and capacity building for staff of recipient governments to reinforce national anti-corruption agencies. In 2011, the Bank published revised guidelines for borrowers for the prevention and combating of fraud and corruption in Bank-financed projects.

MI 7.4 – Systems for immediate measures against irregularities

Survey respondents from all three groups were asked whether the World Bank immediately follows up on financial irregularities, including fraud and corruption. Overall, more than half of respondents (52 per cent) rated the Bank’s performance as strong or above. The proportion of ‘don’t know’ responses was high for this MI, especially for donors in-country, 28 per cent of whom responded ‘don’t know’.

The document review rated the World Bank strong for its systems in place for immediate follow- up on financial irregularities. The Bank has developed several policies and procedures that document actions to be taken in the event of financial irregularities, including fraud and corruption, and these are supported by management guidelines. These include Operational Policy 12.00: Disbursement; Operational Policy 11.00: Procurement; Operational Policy and Bank Procedure 10.02: Financial Management; Guidelines – Procurement of Goods, Works and non-Consulting Services; and Guidelines on Preventing Fraud and Corruption in Projects Financed by IBRD Loans and IDA Credits and Grants. Although the guidelines do not specify timelines for the World Bank’s response to irregularities, the document review found evidence of management follow up on audit recommendations. Major and/ or systemic irregularities are reported to the Audit Committee of the Board of Executive Directors by the Integrity Vice Presidency.

MI 7.5 – Internal audit processes provide credible information

Donors at headquarters were asked whether the World Bank conducts internal audits to provide credible information to its governing body. More than half of survey respondents (53 per cent) rated the organisation as strong or above for this MI.

This MI was rated very strong by document review, which found evidence of systems that help ensure that independent and credible information is produced. The Internal Audit Vice Presidency (IADVP) is an independent entity charged with responsibility for the review of the World Bank’s internal controls and processes for risk management and governance. It reports to the President and is under the oversight of the Audit Committee of the World Bank’s Board of Executive Directors. The Terms of Reference for the Internal Auditing Department (IAD)[26] outline the policy of the organisation on internal auditing. IADVP’s quarterly and annual results are outlined in the quarterly and yearly reports, which are made publicly available. IADVP’s Annual Report summarises audit results for the fiscal year and the control, risk management and governance issues at the institutional level arising from IADVP’s body of work. Reports produced by the Audit Committee, such as the 2011 Annual Report of the Audit Committee to the Board, confirmed the Committee’s receipt of the reports of the Internal Audit Vice Presidency.

MI 7.6 – Effective procurement and contract management processes

Clients and donors in-country were asked to rate the effectiveness of the World Bank’s procurement and contract management processes for the provision of services or goods. Overall ratings indicated that the Bank’s performance in this area is adequate. Clients provided higher ratings than donors, and the difference is statistically significant.

The document review rated this MI very strong. In 2011, the World Bank published guidelines for the procurement of goods, works and non-consulting services. In addition, it has developed several procurement and contract management systems, including operational policies and bank procedures, as well as pre- and post- procurement contract reviews. The basic principles of the World Bank’s procurement system are articulated in Operational Policy and Bank Procedure 11.00: Procurement. These principles are grounded in procurement services that are efficient, effective, timely, and of satisfactory quality. The 2011 report Financial Management and Procurement in World Bank Operations indicated that procurement post-reviews were conducted for 8,425 contracts in 803 projects in 2011.

MI 7.7 – Strategies for risk management

Donors at headquarters were asked whether the World Bank has appropriate strategies for risk management. Half of the respondents rated it strong or higher.

The document review rated this MI very strong. The World Bank has developed a suite of corporately approved documents and procedures for the management of risk, including social, financial and climate risk management. In 2010, the Bank also developed the Operational Risk Assessment Framework as part of its investment lending reform. This framework is mandatory for all investment lending and is used to weigh risks at the project level. The Bank indicated that both the framework and documents follow standard international practices for risk management, such as the Enterprise Risk Management Integrated Framework developed by the Committee of Sponsoring Organizations of the Treadway Commission (COSO), and that they have been developed in consultation with key international stakeholders. The document review found that the Board of Executive Directors uses risk management procedures that follow international standards in its review and analysis of financial statements, inclusive of discussions around income allocation and distribution.

KPI 8: Using Performance Information

Finding 8: The document review and survey respondents considered the Bank strong or very strong in most aspects relating to the use of performance information but adequate in managing unsatisfactory initiatives.

The World Bank documents and disseminates performance information on progress towards outcomes that emerges from activities conducted in-country and at the corporate level (e.g., evaluations, reviews, performance monitoring data, and audits). Performance information is used in the development of new Country Assistance Strategies, as well as in the refinement of existing strategies.

Figure 3.14 KPI 8: Using Performance Information, Ratings of Micro-Indicators

MI 8.1 – Using information to revise and adjust policies

When asked whether the World Bank uses project, sector and country information on performance to revise corporate policies, 72 per cent of donors at headquarters rated the Bank adequate.

The World Bank was rated strong on this MI by the document review, which looked for evidence that different types of performance information (e.g., evaluations, reviews, performance monitoring data, and audit) are used to revise Bank policies or strategies. It found a number of examples of the Bank’s use of evaluation information to revise policies:[27]

• In its 2010 evaluation of the World Bank’s Country Policy and Institutional Assessment, the IEG recommended a revision of the criteria used to assess trade, the financial sector, and equity of public resources. In response, management commissioned a thorough review of the policy with the aim of policy revision by the end of 2011. In the 2011 Management Action Record, the IEG rated the Bank’s level of adoption of recommendations as substantial.

• The 2010 IEG evaluation of the World Bank’s safeguards and sustainability policies recommended a revision of the Bank’s policy frameworks in this area. It suggested increased harmonisation of thematic coverage across the World Bank Group, and the enhanced relevance of the frameworks to client needs. In its 2011 update on the Modernisation Agenda, the Bank indicated that it is in the process of updating its safeguard policies in response to the IEG recommendation.

• In 2009, the IEG evaluation of the Bank’s Win-Win energy policy reforms recommended emphasis on policies that induce improvement in energy efficiency to reduce the burden of transition to market-based energy prices. The 2011 Management Action Record indicated that the Bank had made substantial progress in adopting this recommendation.

• In 2007 an independent panel review of the Department of Institutional Integrity made several recommendations to the Bank for improving the operations of the department. These included: restructuring the department and elevating it to a Vice Presidency; creating an independent advisory Board; and increasing the transparency of investigative products. In developing the 2009 Integrity Vice Presidency Strategic Framework, the Bank incorporated the recommendations of the review.

MI 8.2 – Using information for planning new interventions

Donors in-country and clients were asked whether the World Bank uses information on country or sector performance to plan new interventions at country level; 65 per cent of respondents gave the Bank a rating of strong or above.

This MI was rated strong by the document review, which looked at the extent to which performance information was used to inform planning processes at the country level. The World Bank conducts systematic reviews of country level strategies, which are used to inform new projects or an additional phase of existing projects. As an example, the Country Assistance/Partnership Strategy (CAS/CPS) is used to plan World Bank-supported projects and programmes in-country. At the end of the project cycle, a CAS/CPS completion report is produced based on a project’s self-assessment. These findings are used to prepare the new CAS/CPS. The review of CAS/CPS documents for the five countries sampled provided evidence of new interventions that were informed by the findings and main conclusions of the IEG review of the CAS/CPS completion report.

MI 8.3 – Proactive management of unsatisfactory initiatives

Donors in-country and clients were asked whether the World Bank actively manages unsatisfactory projects from the previous fiscal year. The overall rating for this MI was adequate, and the proportion of ‘don’t know’ responses was quite high (31 per cent of respondents). Clients provided higher ratings than donors and the difference is statistically significant.

The document review rated this MI adequate. According to the 2010 COMPAS report, at the end of 2009, the World Bank was executing 1,656 projects, of which 15 per cent (245 projects) had been rated unsatisfactory or marginally unsatisfactory in the achievement of their development objectives. In 2010, the Bank undertook measures to address these projects. As a result, 61 per cent of the projects rated unsatisfactory in 2009 (149 projects) had improved their rating to satisfactory by 2010. Tier III of the 2011 World Bank Corporate Scorecard indicated that 76 per cent of projects completed in 2009 demonstrated satisfactory operations outcomes at completion. In order to achieve satisfactory results, the Bank emphasises implementation support and risk management throughout the project implementation process.

MI 8.4 – Evaluation recommendations are tracked

Donors at headquarters were asked whether the World Bank regularly tracks implementation of evaluation recommendations reported to the Board; 75 per cent gave the Bank a rating of adequate or above.

The document review rated this MI very strong. In its 2011 report on the results and performance of the World Bank Group, the IEG outlined the main steps involved in the follow- up process for the implementation of IEG recommendations. The mandate of the Director General, Evaluation, also specifies that periodic reports are to be made to the Board on the World Bank’s response to evaluation recommendations. At the corporate level, the Annual Reports of the Internal Audit Vice Presidency provide evidence of systematic performance assessment, as well as review and incorporation of recommendations made by organisation- wide audits, performance reports, and evaluations.

The document review also found strong evidence that management responses to evaluations are presented to the Board on an annual basis. The Management Action Record (MAR) is used by the IEG to track the status of implementation of evaluation recommendations. The MAR is an embedded mechanism that has existed within the Bank for several years. In 2008, a thorough self-assessment of IDA internal controls was completed by management, reviewed by the Internal Auditing Department (IAD), and evaluated by IEG. Based on the findings, management adopted a Five-Point Action Plan in which it committed to 22 corrective actions. The 2010 MAR concluded that the plan had been adopted to a substantial extent.

MI 8.5 – Resources allocated based on performance

This MI was assessed by document review only, to determine whether the World Bank’s allocation of resources to countries and projects is based on performance. It was rated very strong.

IDA’s Performance-Based Allocation System (which is available on the World Bank’s website in five languages) outlines country eligibility for IDA resources, as well as the allocation criteria and process. Country performance in implementing policies that promote economic growth and poverty reduction are identified as the main criterion for resource allocation. The Bank conducts a Country Policy and Institutional Assessment (CPIA) that rates the performance of countries in poverty reduction, sustainable growth, and the effective use of development assistance. This performance-based allocation system is reviewed during each IDA replenishment. In addition, there is evidence that the system has allocated more resources to countries demonstrating better performance. For example, during the IDA 15th replenishment, countries in the top performance quintile received higher allocations (2.7 times per capita) than countries in the lowest quintile.

KPI 9: Managing Human Resources

Finding 9: Surveyed stakeholders rated the Bank adequate in managing human resources. The document review rated it strong overall.

The World Bank has a performance assessment system for regular and term staff which is also applicable to senior World Bank staff, including Vice-Presidents and Managing Directors. Recent reviews of the Bank’s human resources systems and practices have recommended that the Bank revise its staff reward system. Survey respondents rated the Bank adequate for the number of staff and quality of staff at the country level.

Figure 3.15 KPI 9: Managing Human Resources, Ratings of Micro-Indicators

MI 9.1 – Results-focused performance assessment systems for senior staff

This MI was assessed only by the document review which rated it strong. In the World Bank Staff Manual, Principle 5 on Organisational Management indicates that procedures are established for the periodic review of the work performance of staff members (including senior staff) on regular, open-ended, term and local regular appointments. The Performance Management Procedures in the staff manual provide an overview of the steps involved in the annual staff review.

During interviews, World Bank management also clarified that managers are subject to a 360 leadership assessment which provides them with feedback on the way they are meeting their responsibilities and delivering on results. Management also added that the Bank has included key performance indicators of the Corporate Scorecard in Memoranda of Understanding between the President and Managing Directors, and between Managing Directors and Vice Presidents.

MI 9.2 – Transparent system to manage staff performance

Survey respondents were asked whether the Bank uses a transparent system to manage staff performance. More than half of respondents (57 per cent) rated the Bank adequate in this area, and 23 per cent responded ‘don’t know’.

The document review, which focused on the link between performance and incentives and rewards, rated this MI adequate.

The World Bank has an operational system for managing staff performance and the Staff Manual indicates that the Bank has aimed to create an environment that is conducive to high standards of performance, and to motivate staff to perform exceptionally well. The Human Resources Committee has made efforts to strengthen the alignment between performance and salary and rewards, and Rule 5.06 of the Staff Manual articulates an explicit link between staff performance and promotion.

Nevertheless, the Internal Audit Vice Presidency’s 2011 Annual Report indicated the need for improved design and implementation of policies, and enhanced monitoring and performance standards. This was reiterated by the IEG in its 2012 evaluation of the World Bank’s organisational effectiveness. The IEG considered the Bank’s performance-based incentives for staff as insufficient, stating that opportunities for staff promotion are infrequent and rewards are inadequate. The Bank confirmed that it is taking steps to reform its human resources policy with emphasis on staff compensation. In its annual review of staff compensation for the World Bank Group, the Bank’s Human Resources Committee confirmed that changes to the compensation were approved by the Board of Executive Directors in 2011. One of the main aims of these changes involves improving the Bank’s pay-for-performance model.

MI 9.3 – Country staff deployment supports partnership development

This MI was assessed by survey only. Country level respondents (donors and clients) were asked whether the number and quality of World Bank staff in country allows for effective country partnerships and rated the Bank adequate on both questions. On the number of staff, 66 per cent of respondents rated the Bank adequate or above. On the quality of staff, 86 per cent considered the Bank adequate or above.

KPI 10: Performance-oriented Programming

Finding 10: Both survey respondents and the document review provided strong ratings for the World Bank’s performance oriented country programming processes.

Figure 3.16 KPI 10: Performance-oriented Programming, Ratings of Micro-Indicators

MI 10.1 – New initiatives subject to benefits/impact analysis

When asked whether the World Bank subjects new loans and credits to impact analysis prior to approval, more than half of survey respondents at headquarters gave the Bank a rating of strong or above.

The document review rated this MI adequate. The Bank’s policy for cost-benefit analysis (OP 10.04 on Economic Evaluation of Investment Operations) stipulates that the Bank is to conduct a cost-benefit analysis (CBA) for all its investment projects. However, the 2010 IEG evaluation of the Bank’s use of CBA expressed concerns over the Bank’s lack of compliance with this policy. It also highlighted that decisions to proceed with projects were often taken before the CBA, thus undermining the purpose. In its Management Response to the evaluation, the Bank explained that traditional CBAs have become less relevant and that other types of analysis are conducted prior to project investment; it therefore considered the policy outdated and, in light of IEG’s recommendations, is currently updating its policy on Economic Evaluation of Investment Operations.

The World Bank also conducts environmental and social analyses that guide the design and preparation of investment projects and has policies and procedures in place for these assessments. Operational Policy 4.00 describes how borrower systems are used to address environmental and social safeguard issues in Bank-supported projects. Through this approach the Bank assesses the environmental and social safeguard systems of borrower (client) countries to determine whether they meet Bank standards. Operational Policy 4.01 mandates the Bank to conduct an environmental assessment of all projects proposed for Bank funding to ensure that they are environmentally sound and sustainable. For the sample of countries reviewed, the document review found examples of environmental assessment reports and social impact assessment reports for several projects.

MI 10.2 – Milestones/targets set to monitor implementation

Survey respondents at the country level (donors and clients) were asked whether the World Bank sets targets to enable monitoring of project implementation at the country level. A majority of respondents rated the Bank strong or above. There were differences between respondent groups: while 74 per cent of clients rated the Bank adequate or better, only 43 per cent of donors in-country expressed the same view. The difference is statistically significant.

The document review rated this MI strong. Project Appraisal Reports were reviewed for a selection of on-going projects in each of the five countries sampled. The document review found evidence of milestones (intermediate targets) in the reports reviewed.

KPI 11: Delegating Authority

Finding 11: Whereas surveyed stakeholders rated the World Bank adequate in its delegation of authority for decentralised decision making, the document review rated it inadequate.

Although the World Bank’s business modernisation agenda includes provisions for increased delegation of authority to country level in the interest of reducing bureaucracy, this is a work in progress.

In this key performance area, country level respondents were surveyed. Clients provided higher ratings than donors in-country on both MIs and the differences are statistically significant.

Figure 3.17 KPI 11: Delegating Authority, Ratings of Micro-Indicators

MI 11.1 – Project tasks are managed at country level

Country level respondents (donors and clients) were asked whether the World Bank’s project tasks are managed at the country level. Overall, 74 per cent of respondents rated the Bank’s performance in this area as adequate or above.

The document review, which looked at whether aid reallocation decisions can be made locally, rated the World Bank adequate. In 1997, the Bank underwent internal reforms by putting in place a decentralization process and now operates on a country-based model.[28] As part of the modernisation agenda, the Bank has committed to increasing its efforts in decentralisation and is now tracking the extent to which it is shifting responsibility for operations and management in Tier IV of the Corporate Scorecard. The 2012 Scorecard indicates that the percentage of services for clients that are managed by Bank staff based in client countries increased from 40.6 per cent in 2008 to 44.4 per cent in 2012. However, there is no performance standard (desired level of decentralisation for management of client services) for this indicator and it is one that has been identified for monitoring in FY2011 and FY2012.[29]

In 2012 the Bank introduced a new Decision-Making and Accountability Framework to guide its delegation of authority, assignment of accountability, and decision making. Through this new approach, it aims to simplify business processes, thereby reducing bureaucracy and strengthening accountability. As noted below, the Terms of Reference for country directors and country managers demonstrate that some key client relationships/services and Bank management decisions are taken at country level (e.g., development and implementation of CASs and budget approval). Furthermore, BP 13.05 on Project Supervision stipulates that minor project changes can be approved by country directors.[30] Significant changes in projects, or project restructuring, can either be approved by the Board of Executive Directors or by the Regional Vice Presidents (RVPs), depending on the nature of the change required.[31]

MI 11.2 – Decision-making authority delegated to country level

Country level respondents (donors and clients) were asked whether the World Bank has delegated appropriate decision-making authority to the country level. The majority of clients (74 per cent) rated the Bank adequate or above. Among donors in-country, 31 per cent rated it adequate or above, 27 per cent rated it inadequate or below, and 31 per cent responded ‘don’t know’.

The review of documents, which looked at whether new programmes / projects can be approved locally within a budget cap, rated the Bank inadequate. As part of the 2006 Budget reform, the Bank adopted a decentralised budget process with the objectives of better aligning resources with strategic priorities, establishing greater accountability over the use of resources, and improving efficiency and effectiveness. The overall envelope allocated to countries is approved by the Board yearly within a three-year cycle. This envelope is then transferred to Vice Presidential Units whose management teams are responsible for allocating and managing the budget for cost-effective delivery of country programmes. However, the extent of financial decision-making authority delegated to country directors is unclear (e.g., it was not possible to identify a framework for delegation of authority to the field including an articulation of risk-based thresholds for procurement, project approval, and budget decisions). During interviews with World Bank headquarters, management clarified that the Bank’s approach for the exact level of authority being delegated largely depends on country context.

3.3.4 Relationship Management

The World Bank has built relationships with its development partners to harmonise technical cooperation and procedures, and with country partners to align priorities and increase country ownership. The Bank continues to be recognised for its contributions to policy dialogue.

Figure 3.18 presents the overall survey and document review ratings for the five KPIs in the relationship management quadrant.

Survey respondents considered the Bank adequate overall in managing relationships and strong in contributing to policy dialogue – an area in which the Bank was rated strong in the previous three MOPAN assessments. The Bank received the highest mean score in the survey in the area of policy dialogue. The Bank ‘s overall rating on adjusting procedures suggests some improvement since 2009, although the Bank’s procedures continue to be a source of concern – as reflected in survey ratings of in-country donors and as expressed by respondents in written comments on areas for improvement in the way the Bank operates.

The document review rated the Bank very strong on supporting national plans and strong on the use of country systems, and adequate in harmonising procedures. In assessing the Bank’s performance in using country systems and harmonising procedures, the document review drew on the Bank’s own review of its performance in meeting its Paris Declaration commitments as well as other sources.

Figure 3.18 Quadrant III: Relationship Management, Survey and Document Review Ratings

Figure 3.19 shows the mean scores for the five KPIs for all survey respondents, and by respondent groups.

Figure 3.19 Quadrant III: Relationship Management, Mean Scores by Respondent Group

KPI 12: Supporting National Plans

Finding 12: Surveyed stakeholders rated the Bank adequate in coordinating with national partners in the design of projects and programmes. The document review rated the Bank very strong in using national indicators or benchmarks for conditionality.

This KPI was developed to measure the extent of the Bank’s collaboration with its shareholders to plan and implement activities that support the objectives of national development strategies. The World Bank’s engagement at country level is encapsulated in the Country Assistance/ Partnership Strategy, a nexus for articulating project/programme activities in-country.

Figure 3.20 KPI 12: Supporting National Plans, Ratings of Micro-Indicators

MI 12.1 – Funding proposals developed with national government or clients

This MI was assessed by survey only. Donors in-country and clients were asked whether World Bank funding proposals are designed and developed with national governments or clients. Overall, the Bank received a rating of adequate for this MI: 85 per cent of clients and 71 per cent of donors in-country rated this MI as adequate or above.

MI 12.2 – Conditionality draws on national benchmarks or indicators

Donors in-country and clients were asked whether the World Bank applies conditionality that corresponds with national government goals and benchmarks. Overall, 73 per cent of respondents rated the Bank as adequate or above.

This MI received a rating of very strong in the document review, which examined the Bank’s development policy lending.

The World Bank’s conditions for lending are specifically applicable to its policy-based loans (development policy lending).[32] Development policy lending includes among its principles the notion of country ownership and the alignment of World Bank operations with national development strategies. The decision to extend development policy lending is conditional upon a country’s capacity and ability, through its institutional framework, to effectively implement World Bank-supported projects and programmes to achieve specific development results.

The World Bank has periodically reviewed and updated its staff guidance and practice notes on development policy lending. Following its 2005 Review of World Bank Conditionality,[33] the Bank published its good practice principles for the application of conditionality in 2006.[34] Subsequent reports in 2007[35] and 2009[36] reviewed the Bank’s experience with conditionality in development policy lending. In 2009, the IEG noted that the World Bank had made significant progress in streamlining conditionality in its development policy operations and had reduced the number of legally binding conditions. There is documented evidence of reports to the Board on the implementation of the Bank’s Development Lending Policy, which was revised in March 2012.

KPI 13: Adjusting Procedures

Finding 13: Survey respondents rated the World Bank adequate for the clarity of its procedures, flexibility, and responsiveness to changes in project implementation. In their written comments, respondents indicated that the speed of procedures remains an area of concern.

This KPI was assessed by survey only, by clients and donors in-country. The World Bank received an overall rating of adequate for the extent to which its procedures take into account local conditions and capacities.

In their written comments, 24 per cent of survey respondents indicated that Bank procedures are rigid and that they do not always consider the particular context in a given country. They also commented that lengthy bureaucratic processes often delay implementation of projects.

Figure 3.21 KPI 13: Adjusting Procedures, Ratings of Micro-Indicators

MI 13.1 – Procedures easily understood and completed by partners

Donors in-country and clients were asked whether the World Bank uses procedures that can be easily understood and followed by clients. Overall, the Bank received a rating of adequate. Among client respondents, 88 per cent rated the Bank adequate or above. Among donors, 33 per cent rated it adequate or above and 32 per cent rated it inadequate.

MI 13.2 – Length of time for procedures does not delay implementation

Donors in-country and clients were asked whether the length of time it takes to complete World Bank procedures delays implementation. Overall, the Bank was rated inadequate but there were statistically significant differences between the respondent groups. The majority of clients (59 per cent) rated this MI adequate or above. The majority of donors in-country (51 per cent) rated it inadequate or lower, 29 per cent rated it adequate, and 21 per cent responded ‘don’t know’. While respondents’ perceptions show that the speed of procedures is an area for improvement, data published in the 2012 Corporate Scorecard indicate that the average time from approval to first disbursement has improved from 12 months in FY2008 to 7.5 months in FY2012.

MI 13.3 – Ability to respond quickly to changing circumstances

Donors in-country and clients were asked whether the World Bank adjusts its overall portfolio quickly to respond to changing circumstances. The majority of respondents (59 per cent) gave a rating of adequate or above.

MI 13.4 – Flexibility in implementation of projects/programs

Donors in-country and clients were asked whether the World Bank adjusts its implementation of individual projects/ programmes as learning occurs. Overall, 59 per cent of respondents rated the Bank’s performance as adequate. Clients provided higher ratings than donors on this MI and the difference is statistically significant. The proportion of ‘don’t know’ responses was quite high among donors in-country (22 per cent).

KPI 14: Using Country Systems

Finding 14: The World Bank was rated strong by the document review for its use of country systems and for avoiding parallel implementation structures. Surveyed stakeholders rated the Bank adequate in its use of country systems and for its participation in assessments of its aid effectiveness commitments.

The MOPAN indicators in this KPI are closely aligned with the indicators of the Paris Declaration. Wherever possible, the assessment has drawn on multiple sources of data including the Bank’s own reporting on aid effectiveness, COMPAS data, Independent Health Partnership and Related Initiatives, and OECD-DAC data.

The World Bank has expressed its support for the Paris Declaration and the Accra Agenda for Action through its work in mainstreaming the principles of aid effectiveness across the organisation and at country level. It has also acknowledged the importance of the Paris Declaration survey in monitoring progress on the global international aid effectiveness agenda. Nevertheless, the Bank has also noted some methodological concerns about the survey measures and questions and the relevance of some indicators, given that the survey only tracks selected Paris Declaration commitments and does not cover many of the Accra Agenda for Action commitments.[37]

Figure 3.22 KPI 14: Using Country Systems, Ratings of Micro-Indicators

MI 14.1 – ODA disbursements/ support recorded in annual budget

This MI, assessed by document review only, was rated adequate. The assessment was based on Indicator 3 of the Paris Declaration and its target of 85 per cent of ODA disbursements recorded in national budgets. In 2010, 74 per cent of the World Bank’s aid flow was aligned with national priorities in partner countries and recorded in their budgets. This result was calculated for the 30 countries that participated in the 2006 baseline survey and the 2011 survey.

MI 14.2 – Use of country systems for operations

Survey respondents at the country level (donors and clients) were asked whether the World Bank uses country systems as a first option for its operations where appropriate. Overall, 54 per cent rated the Bank adequate or above. Among donors in-country, 26 per cent responded ‘don’t know’.

The document review rated the Bank strong on this MI, which was based on data reported by the OECD-DAC and the World Bank for Indicators 5a and 5b of the Paris Declaration. In relation to Indicator 5a (Use of country public financial management systems), the World Bank recorded a score of 69 per cent in 2010, exceeding the target of 51 per cent set by the Paris Declaration. In respect of Indicator 5b (Use of country procurement systems), the World Bank scored 54 per cent, exceeding its own target of 50 per cent. (This target was drawn from the 2008 Accra Agenda for Action, as an overall target was not established under the Paris Declaration.)

The World Bank has supported the use of country-based public financial management systems by conducting periodic reviews of existing country systems to ensure their adequacy, and by providing capacity building for system strengthening. The Bank has been a leading partner in the Public Expenditure and Financial Accountability (PEFA) programme in support of an integrated and harmonised approach to the assessment of public financial management systems. The PEFA programme is a multi-donor partnership between seven donor agencies and international financial institutions.[38] It was founded in 2001 to initiate the reform and capacity building of country public expenditure, procurement and financial accountability systems.

MI 14.3 – Parallel implementation structures avoided

The document review rated the Bank strong on avoiding parallel implementation structures.

This MI is based on Indicator 6 of the Paris Declaration (Strengthen capacity by avoiding parallel project implementation units). In 2010, the World Bank achieved a reduction of 80 per cent of the total stock of parallel project implementation units, exceeding the target of 67 per cent. The result was calculated for the 30 countries that participated in the 2006 baseline survey and the 2011 survey. In order to achieve this target the World Bank coordinated support with country line ministries to leverage existing in-country resources and strengthen national institutional capacity.

MI 14.4 – Promotion of mutual accountability in its partnerships

This MI was assessed by survey only. Clients and donors in-country were asked whether the World Bank participates in mutual accountability assessments of commitments to the Paris Declaration and other aid effectiveness agreements. The majority of respondents (59 per cent) rated the Bank adequate in this area, and 28 per cent responded ‘don’t know’.

KPI 15: Contributing to Policy Dialogue

Donors and clients were positive in their survey ratings of the World Bank’s input to policy dialogue and in written comments identified the Bank’s engagement in policy dialogue as one of its strengths.

Figure 3.23 KPI 15: Contributing to Policy Dialogue, Ratings of Micro-Indicators

MI 15.1 – Reputation for valuable input to policy dialogue

This MI was assessed by survey only. All three categories of respondents were asked whether the World Bank provides valuable input to policy dialogue; 66 per cent rated the Bank strong or above on this MI.

MI 15.2 – Policy dialogue respects partner views

This MI was assessed by survey only. Survey respondents from all three categories were asked whether the World Bank respects the views of clients when it undertakes policy dialogue. The Bank was rated adequate overall on the MI. Clients were more positive than other respondent groups and the differences are statistically significant.

KPI 16: Harmonising Procedures

Finding 15: Survey respondents and the document review rated the World Bank adequate in the harmonisation of its procedures.

In the survey, clients and donors in-country rated the Bank adequate overall on this KPI. In written comments, however, 40 per cent of donors in-country and 30 per cent of donors at headquarters commented that the Bank needed to improve harmonisation with other donors.

The document review found the Bank adequate overall and strong in providing technical cooperation through coordinated programmes.

While the Bank agrees that harmonisation remains an area for improvement, it also highlights in a 2011 report on aid effectiveness[39] that differing policies and institutional mandates between the ever-growing number of actors in the field can make harmonisation rather challenging. Negotiation processes become lengthier, which consequently increases transaction costs. Therefore, while the Bank agrees that coordination with other actors is necessary to increase synergy and avoid duplication, it feels that a more practical approach to harmonisation is necessary.

Figure 3.24 KPI 16: Harmonising Procedures, Ratings of Micro-Indicators

MI 16.1 – Participation in joint missions

Donors in-country and clients were asked whether the World Bank often participates in joint missions. Both groups rated the Bank’s performance adequate. There was a statistically significant difference between groups: 87 per cent of clients rated the Bank as adequate or above, against 67 per cent of donors in-country.

The document review rated this MI adequate. This MI is linked to Indicator 10a of the Paris Declaration and its target of 40 per cent joint missions. In 2010 the World Bank conducted 29 per cent of its donor missions (2,017 missions) to the field as joint missions, representing over 50 per cent of the target.

In its 2011 report on aid effectiveness,[40] the Bank questioned the relevance of Indicator 10a given its recent decentralisation efforts. It also mentioned that this indicator fails to capture coordination at country-level, including with other development agencies.

MI 16.2 – Technical cooperation disbursed through coordinated programmes

When asked whether the World Bank’s technical assistance is provided through coordinated programmes in support of capacity development, 71 per cent of country respondents (donors and clients) gave a rating of adequate or higher. Clients provided more high ratings than donors and the difference is statistically significant.

The document review rated the Bank strong on this MI, which is linked to Indicator 4 of the Paris Declaration (Strengthen capacity by coordinated support). In 2010, 73 per cent of the Bank’s technical cooperation for client countries was implemented in coordination with other development partners. This exceeded the 2010 target of 50 per cent established by the Paris Declaration. In 2011, the IEG noted that the World Bank was effective in coordinating technical assistance with its development partners, and that this contributed to the efficient use of available resources, building national capacity, and the reduction of transaction costs for country beneficiaries.[41]

MI 16.3 – ODA disbursements/support for government-led PBAs

Survey respondents at the country level (donors and clients) rated the World Bank strong overall for its participation in programme-based approaches. The Bank was rated strong or above by 66 per cent of clients and 44 per cent of donors in-country and the difference is statistically significant. The proportion of ‘don’t know’ responses for this latter group was 21 per cent.

The Bank was rated adequate on this MI in the document review. This indicator is aligned with Indicator 9 of the Pars Declaration (Use of common procedures or arrangements) and its target of 66 per cent. The Bank provided 59 per cent of its aid in the context of programme-based approaches and is close to reaching the target.

3.3.5 Knowledge Management

In the area of knowledge management, the Bank was noted for its evaluation of results, managed by the Independent Evaluation Group, and for good practices in the identification, documentation, and dissemination of lessons learned. The Bank has established reporting mechanisms to present performance information at the country and corporate level, but there remains room for improvement in these areas.

Figure 3.25 below shows the overall survey and document review ratings for the three KPIs in the knowledge management quadrant. Survey respondents and the document review considered the Bank is strong in evaluating external results and disseminating lessons learned, and adequate in the presentation of performance information.

Figure 3.25 Quadrant IV: Knowledge Management, Survey and Document Review Ratings

Figure 3.26 shows the mean scores for the three KPIs for all survey respondents, and by respondent groups.

Figure 3.26 Quadrant IV: Knowledge Management, Mean Scores by Respondent Groups

KPI 17: Evaluating External Results

Finding 16: MOPAN donors at headquarters and the document review rated the World Bank as strong in evaluating external results. In their written comments, survey respondents noted the Bank’s knowledge dissemination as an area of strength.

Overall, survey respondents and the document review rated the World Bank strong in its evaluation of results.

The document review, which assessed three aspects of evaluation, found the Bank strong in the independence of its evaluation unit (the IEG) and in the quality of its evaluations, and adequate in evaluation coverage.

Figure 3.27 KPI 17: Evaluating External Results, Ratings of Micro-Indicators

MI 17.1 – Independent evaluation unit

The Bank was rated strong by donors at headquarters and by the document review for the independence of its evaluation unit. The IEG is a structurally independent evaluation unit that evaluates activities across the World Bank.

The IEG is responsible for conducting objective assessments of the work of each member institution of the World Bank Group, and for disseminating lessons learned from experience. It reports directly to the Board of Executive Directors through the Committee of Development Effectiveness, an eight-member standing committee of the Board of Executive Directors. The Director General, Evaluation, is guided by a mandate that specifies the principles of evaluation independence required for the operations of the IEG. Similar principles are found in the Terms of Reference for the directors of the IEG. Together, these documents provide a framework within which the IEG operates independently.

MI 17.2 – Sufficient evaluation coverage of programming activities

Donors at headquarters were asked whether the evaluation function provides sufficient coverage of the Bank’s activities. They rated the World Bank strong on this MI.

The document review rated this MI adequate. Although the IEG is not guided by an organisation-wide evaluation policy, it has a clearly articulated evaluation plan in the mandate of the Director General, Evaluation. This mandate defines the types of evaluation activities that are conducted, for example, reviews of Country Assistance Strategy Completion Reports; project evaluations; reviews of literature, analytical work and project documentation; portfolio reviews, impact evaluations, etc.

The IEG reviews all Country Assistance Strategy Completion Reports, preparing on average, 20-25 reviews annually for consideration by the Board. The document review found evaluations of projects and programmes in the sample countries that were conducted by the IEG in the last three years (2009-2011). Between 2008 and 2011, the IEG also conducted three impact evaluations and six reviews of global and regional partnership programmes. In interviews at World Bank headquarters, staff noted that while all World Bank activities are subject to IEG evaluations, there are fewer impact evaluations as these are quite costly to conduct.

The Bank does not have global standards for evaluation coverage, but does have some targets. For example, the IEG aims for 100 per cent coverage in the review of CAS completion reports.

MI 17.3 – Quality of evaluations

When asked whether the World Bank ensures that its evaluations are of high quality, donors at headquarters rated the Bank strong for the quality standards employed during evaluations.

The document review, which looked for evidence of quality control mechanisms, rated the Bank strong on this MI. The IEG has several practices in place to ensure the quality of its evaluation activities, including: at least two external peer reviewers who provide input on all major evaluations; external review panels that review the quality, recommendations and findings of complex or sensitive evaluations; and the sharing of comments made by external panels with the Committee on Development Effectiveness and with management. The IEG has taken other steps to strengthen the evaluation function, such as: additional training for IEG staff, convening experts for advisory purposes, and an initiative to improve evaluation methodology launched in 2010, which includes a review of consistency, quality and clarity of project evaluations.

MI 17.4 – Use of evaluation findings to inform decisions

This MI was assessed by survey only. Donors at headquarters were asked whether the World Bank uses evaluation findings in its decisions on programming, policy and strategy. They rated the Bank adequate for the extent to which it incorporates evaluation findings in its decision- making processes.

MI 17.5 – Beneficiaries and clients involved in evaluations

This MI was assessed by survey only. All three respondent groups were asked whether the World Bank involves clients and beneficiaries in evaluations of its projects and programmes. While the Bank received an overall rating of adequate on this MI, there were statistically significant differences between respondent groups. Clients rated the Bank strong for its efforts to involve them in evaluation activities. Donors at headquarters and in country offices rated the Bank adequate in this area.

KPI 18: Presenting Performance Information

Finding 17: The Bank has developed innovative mechanisms to report on performance but it will take time for these to be fully developed. Survey respondents rated the World Bank adequate overall in all aspects of the presentation of performance information.

Donors at headquarters were the only respondent group asked about the extent to which the World Bank presents performance information on its effectiveness. The majority perceived the Bank to perform adequately in this area.

The Bank is committed to open and transparent reporting and the document review rated it as adequate or strong in all aspects of presenting performance information.

Figure 3.28 KPI 18: Presenting Performance Information, Ratings of Micro-Indicators

MI 18.1 – Reports on achievement of outcomes

When asked whether reports to the Board provide clear measures of achievement of outcomes, donors at headquarters rated the World Bank adequate.

The document review rated this MI as adequate. The World Bank produces several reports that provide details on the Bank’s achievement of outcomes. At the organisation-wide level, the World Bank for Results report (launched in 2011) provides an integrated overview of results achieved by the Bank. This is the Bank’s main report on organisation-wide achievement of outcomes, and is available to shareholders, partners and external stakeholders. World Bank for Results 2011 provides examples of outputs achieved by projects at the country level (such as kilometres of road constructed), which are the results of the Bank’s loans.

The Bank also reports on outcomes in the Corporate Scorecard, the Annual Report of the Board of Executive Directors, and the IEG Annual Report on the results and performance of the World Bank Group. However, the Bank does not establish clear links between organisation- wide outputs and outcomes in these reports.

MI 18.2 – Reports on performance using data obtained from measuring indicators

MOPAN donors at headquarters were asked whether the World Bank’s performance reports are based on quality data obtained from measuring indicators. They provided a rating of strong for the Bank’s work in this area.

The document review rated this MI adequate. The World Bank for Results report was developed as a companion report to the Corporate Scorecard, which constitutes the World Bank’s integrated Results and Performance Framework. The indicators presented in the report meet the SMART criteria (i.e., specific, measureable, achievable, relevant and time-bound).

In Tier II, which measures country results supported by the Bank, measurement is based on recently-established core sector indicators that are amenable to aggregation. However, these have not yet been measured over time and the baseline and current year are still the same for most indicators. In Tiers I, III and IV there are reported differences between the baseline year and the current year.

MI 18.3 – Reports against corporate strategy, including expected results

When asked whether the World Bank reports against its corporate strategy, donors at headquarters rated the Bank adequate.

The Bank was also rated adequate on this MI by the document review. The World Bank has published its first annual Corporate Scorecard and World Bank for Results, which report against objectives and expected results articulated in the organisation’s development and management results frameworks. The frameworks note linkages to the Bank’s strategy (as articulated in the Post-Crisis Directions paper and other strategic documentation). The first integrated reports represent a positive step towards good practice, and are expected to evolve and improve over time.

MI 18.4 – Reports on Paris Declaration commitments using indicators and country targets

Donors at headquarters were asked whether the World Bank reports to the governing body on performance in relation to its Paris Declaration commitments. They provided a rating of adequate.

The document review rated this MI strong. In 2011 the World Bank reported on its performance against the Paris Declaration commitments in a review paper entitled, The World Bank and Aid Effectiveness: Performance to Date and Agenda Ahead. This paper presented an overview of the extent to which the organisation had achieved each of the Paris Declaration targets. It included a tabular summary of World Bank’s achievements against each indicator to which it had a commitment. While it did not report on the Bank’s achievements by country, World Bank country level data is available on the OECD/DAC website as well as in country reports produced by the OECD.[42]

MI 18.5 – Reports on adjustments to policies/strategies based on performance information

The majority of donors at headquarters (62 per cent) rated the Bank as adequate or above for reporting on adjustments to policies and strategies based on performance information, and 36 per cent gave a rating of strong or very strong.

This MI was rated adequate by the document review. There is some evidence that the Bank is reporting on changes in strategies and policies based on performance information. As an example, in 2011 the IEG made a series of recommendations for strengthening the World Bank’s strategy on governance and anti-corruption, including the strengthening of its results framework. The Bank responded by prioritising the improvement of the results framework in its updated strategy in 2012, with a focus on outcomes and impacts.

MI 18.6 – Reports on programming adjustments based on performance information

Donors at headquarters were asked whether the World Bank reports on adjustments to country- level programming based on performance information. They provided a rating of adequate.

The document review rated this MI strong. The World Bank has robust systems in place for reporting on country level programming adjustments that are based on performance information. First, it monitors and evaluates achievement of development results in accordance with Operational Policy 13.60: Monitoring and Evaluation (revised March 2012). Second, through the Country Assistance/ Partnership Strategies (CAS/CPS), the Bank has developed systematic procedures for supporting country achievement of development results. These include a progress report at mid-point in implementation, which provides an update on milestones attained and proposes mid-course corrections as required. Each of the progress reports reviewed had evidence of such corrections (e.g., resource shifts and re-alignment of strategic priorities). The CAS/ CPS progress reports are publicly available on the World Bank website.

KPI 19: Disseminating Lessons Learned

Finding 18: Survey respondents at headquarters and the document review considered the World Bank strong in the dissemination of lessons learned from performance information.

In the survey for this KPI, which included donors at headquarters only, the World Bank was perceived to perform adequately or above in the identification, documentation and dissemination of lessons learned and/or best practices. The document review rated it as strong in reporting on lessons learned based on performance information. Indeed, one of the central components of the evaluation function is the dissemination of evaluation findings within the Bank and across the wider development community.

Figure 3.29 KPI 19: Disseminating Lessons Learned, Ratings of Micro-Indicators

MI 19.1 – Reports on lessons learned based on performance information

When donors at headquarters were asked whether the World Bank identifies and disseminates lessons learned from performance information, 77 per cent rated the Bank adequate or above and 51 per cent gave a rating of strong or very strong.

The document review rated this MI strong. The World Bank collects and disseminates internal and external lessons learned, including best practices, based on performance information. The IEG regularly conducts evaluative studies on the World Bank’s operations that are made publicly available on the World Bank website as part of the dissemination process. The World Bank also disseminates lessons learned through conferences, analytical and advisory activities, and capacity development sessions.

MI 19.2 – Lessons shared at all levels of the organisation

This MI was assessed by survey only. Donors at headquarters rated the Bank adequate in providing opportunities at all levels of the organisation to share lessons from practical experience.

4. Main Findings: Development Results Component

4.1 Overview

This section presents the results of the 2012 Common Approach assessment of the World Bank in the pilot component to assess the development results of multilateral organisations. It includes four key performance areas:

• Evidence of extent of progress towards organisation-wide outcomes (KPI A)

• Assessment of country-level results and relevance:

- Evidence of extent of contribution to country-level goals and priorities (KPI B)

- Evidence of extent of contribution to relevant Millennium Development Goals (KPI C)

- Relevance of objectives and programme of work to stakeholders (KPI D)

Figure 4.1 provides a snapshot of the findings of this assessment. Please note that the pilot applied a simplified 4-point scale that uses the same “traffic light” colours used elsewhere in the report. As noted in Section 2.8 of this report, the scale was simplified to reflect the methodological approach used in the pilot of the development results component – in which various sources of data are considered together when rating the organisation’s performance on each KPI. The methodology is explained in Volume II, Appendix I.

Figure 4.1 Development Results Component – Overall Ratings

4.2 Progress towards Organisation-wide Results

4.2.1 Overview

This section presents the results of the assessment of the World Bank’s progress towards organisation-wide results. KPI A suggests that an effective organisation should demonstrate progress towards organisation-wide, institutional outcomes. These are usually related to the organisation’s strategic objectives. The assessment draws on the evidence that the organisation has available on its results, particularly on its contributions to development outcomes.

The World Bank’s results and reporting at organisation-wide level

In 2010, the World Bank Group introduced a new integrated annual report on its organisation- wide achievement of results produced by the Independent Evaluation Group: Results and Performance of the World Bank Group.[43] The report, which replaced separate annual reports from each member of the Bank Group, focuses on development effectiveness and was created to provide a more comprehensive overview of organisation-wide results and performance. Information in the 2011 report derived from evaluation reports produced over the period 2008 – 2011.

In 2011, the World Bank launched the Corporate Scorecard and its companion annual report, World Bank for Results that builds on the IDA 16 Results Measurement System. The Corporate Scorecard presents performance indicators in a four-tier structure. Tiers I and II track development results achieved, with Tier I focusing on the global development context and Tier II on the support provided by the World Bank for country results. The MOPAN assessment of KPI A is focused primarily on Tier II and the relationships among the tiers that would support progress towards the Bank’s strategic priorities.

The World Bank for Results expands on the Corporate Scorecard and provides the development context for organisation-wide performance. It is the primary mechanism used by the World Bank to report on results achievement. The Corporate Scorecard is used by the Bank to facilitate dialogue between the Board and Management, while World Bank for Results provides the organisation’s shareholders, partners and external stakeholders with an integrated overview of its results and performance.

The Board of Executive Directors also prepares the World Bank Annual Report: Year in Review to provide an organisation-wide synopsis of fiscal year results, operations, and World Bank policies.

Data used for this assessment

The assessment of KPI A is based on a survey of donors at headquarters and a review of documentation made available by the Bank (reports on the Bank’s performance, including the data in Tier II of the Corporate Scorecard). In the document review, attention was paid to results statements, baselines and targets, and other evidence presented to substantiate the results achieved, including any analysis of contribution. As MOPAN is particularly interested in understanding contributions to development outcomes, the assessment analysed the extent to which the data substantiated the outputs achieved and plausible contributions to higher level results.

4.2.2 KPI A: Evidence of Extent of Progress towards Organisation-wide Outcomes

Overall assessment

Figure 4.2 shows the overall rating for this KPI based on the review of the World Bank’s contributions to development outcomes associated with its strategic priorities – as cited in Bank reports and as indicated by surveyed donors at headquarters. It also presents the criteria MOPAN used to assess each strategic priority to determine the overall rating (criteria met are indicated in blue). Column 8 shows the mean scores of the surveyed donors at headquarters, based on the same 6-point scale that was used in the assessment of the Bank’s practices and systems.

Figure 4.2 KPI A: Evidence of Extent of Progress toward Organisation-Wide Outcomes, Ratings

A total of 61 MOPAN donors at headquarters responded to questions on the World Bank’s strategic priorities. When asked whether the Bank was making contributions to the areas identified as its strategic priorities, respondents rated the Bank strong for strategic priorities 1, 2 and 3 and adequate for priorities 4 and 5 (as shown in Column 8 above). Columns 1-7 are based on the results and evidence found in the World Bank’s integrated reporting on organisation-wide results.

Evidence of World Bank contributions

The following findings are drawn from the Bank’s reports on its contributions to development results and from the perceptions of surveyed donors at headquarters. They also provide examples of results where there is evidence that a result was achieved.

Finding 19: The World Bank has introduced reports that provide an integrated view of results and performance. This is an important step towards assessing the Bank’s organisational performance effectively.

As part of its Business Modernisation Agenda, the World Bank has strengthened its capacity for measuring, reporting and learning from results. It developed integrated reporting mechanisms to capture and document progress towards results, including its strategic priorities. As noted in section 4.2.1 above, these include the World Bank for Results report (companion to the Corporate Scorecard).

The Bank’s Corporate Scorecard reports on indicators which are linked to strategic priorities identified in the Post-Crisis Directions Paper and draws on indicators from the IDA16 Results Measurement System (see Tier II of the Scorecard). Tier II of the Scorecard provides an overview of progress towards development results by monitoring current values against established baselines.

The Scorecard data builds on years of measuring and reporting on results for IDA countries, which has informed the structure of the Scorecard, the development of core sector indicators, and the data that is used as a baseline. The broad application of core sector indicators should increasingly facilitate aggregating data from country to corporate level.

The 2012 Scorecard shows evidence of progress in several result areas. As shown in Figure 4.3, there has been some advancement in the indicators related to immunisation, maternal care, and energy infrastructure whereas progress is lagging behind for social safety nets and climate change. However, at this stage in the evolution of the Scorecard, progress cannot be assessed for a number of indicators given that the value for the baseline is the same as indicator value for the current year. For some of the indicators (such as people provided with access to improved sanitation or countries with strengthened Public Management Systems in Procurement, among others), baseline values have shifted since the 2011 Scorecard, thus creating some uncertainty as to the extent of progress over time. It is unclear why the Bank resets the baseline for some indicators and not others as it produces an updated Scorecard.

Nonetheless, the link established by the Bank between strategic priorities and indicator data available suggests that some progress has been made towards targeting the poor and vulnerable, creating opportunities for growth and preparing for crisis. However, progress cannot be measured for strengthening governance because data is still lacking for most indicators. In addition, only three indicators are linked to promoting global collective action (strategic priority no. 3) and do not fully reflect the Bank’s work in this area, including its global role in providing knowledge services.

Figure 4.3 KPI A: Evidence of Progress towards Organisation-Wide Results/Objectives

Source: Tier II – The World Bank Corporate Scorecard, September 2012 (Printed version).

*Strategic priorities as outlined in the Post-Crisis Directions: 1. Target the Poor and Vulnerable; 2: Create Opportunities for Growth; 3. Promote Global Collective Action; 4. Strengthen Governance; 5. Manage Risk & Prepare for Crisis.

Finding 20: The absence of a theory(ies) of change, and data to support such a theory, makes it challenging to explain the contributions that the Bank is making to the broader development changes outlined in its results framework.

Both the IDA Results Measurement System and the World Bank’s Corporate Scorecard (and its companion report World Bank for Results) rest on the four tiers of the results framework.

These reports, and the Scorecard in particular, are designed to provide a “snapshot” of the organisation’s results in a given period. However, it is difficult to discern the Bank’s contributions to its organisation-wide outcomes because the documents do not articulate how the outputs are connected to the outcomes.

At a basic level, the Bank does not present a theory of change that explains the expected changes and how they are expected to come about. A theory of change provides a rationale for a certain way of working or for making investments in certain areas and not in others, the assumptions or beliefs about change can be articulated at different levels: sector, organisational, or project/programme level.[44] A theory of change also brings to bear available evidence to check and challenge assumptions and strengthen the quality of the hypotheses for change. The World Bank may well engage in this process, but there is little evidence of this in the documents reviewed.

Organisations are increasingly acknowledging the importance of theories of change that define the assumptions underlying their operations and business models. For MDBs, theories of change are more easily applied at the country level, where the organisation’s work is anchored in a defined setting. At regional and global levels, the organisational mandates are broad and the theories of change are less clear. As a result, however, a lot of the work for which the Bank is most appreciated (related to knowledge products and services and the global role that it plays) is not as evident in reporting on its contributions to results.[45]

Finding 21: Donors at headquarters rated the Bank’s contributions toward its strategic priorities as strong or higher.

Donors at headquarters were generally positive about the World Bank’s contributions towards its organisation-wide strategic priorities. The highest mean score was for the Bank’s contribution to opportunities for growth, where 77 per cent rated the Bank strong or above. The perception data on the World Bank’s contribution to the achievement of its strategic priorities is presented in Figure 4.2 above, and in Volume II, Appendix IV.

4.3 Evidence of Extent of Contribution and Relevance at Country Level

4.3.1 Overview

This section presents the results of the assessment of evidence of the World Bank’s contributions to country-level results and its relevance to stakeholders. By separating the KPI at the organisation-wide level from KPIs at the country level, MOPAN recognises the demand- driven nature of many of the activities of a multilateral organisation and the key role that is played by its country programming or strategy document, where expected results at the highest level (outcomes and impact) reflect a shared responsibility between the multilateral organisation and the partner country.

The World Bank’s results and reporting at country level

The World Bank’s approach to the achievement of results is country-driven and tailored to each country’s needs and requests for support. The Bank provides support for specific country-level programming through development finance, policy dialogue and knowledge services.

As noted in Chapter 3 (KPI 4), in order to articulate its strategic priorities at the country level, the World Bank develops Country Assistance/ Partnership Strategies (CAS/CPS) to support the achievement of development results by country authorities. The CAS/CPS is results-based and includes a results matrix to link the Bank’s support to country development outcomes, and to monitor progress towards the achievement of results.

The Bank reports on country level achievements through progress reports produced at the mid- point of implementation of the CAS/CPS, and through completion reports at the end of each strategy period. Progress reports provide an update on the achievement of outcomes, including milestones attained, and facilitate mid-course corrections as required, taking into account country developments; they are reviewed at the regional and corporate levels prior to finalisation. Completion reports provide an overview of development outcomes achieved at the end of the CAS/CPS period, and inform the development of the next CAS/CPS. Each completion report is subject to an independent review by the Independent Evaluation Group.

Data used for this assessment

For this pilot test in 2012, the assessment of the World Bank’s contribution and relevance at country level is based on data from a sample of five MOPAN countries suggested by the World Bank given the availability of results data: Ghana, Honduras, Morocco, Nigeria and the Philippines. In the survey, respondents were asked questions that were tailored to each of the five countries reviewed. In the document review, given the emphasis on results, the key sources of data were completion reports and IEG review of those reports.

The following sections present the overall results of the assessment at country level, using country-specific examples to illustrate key findings.

4.3.2 KPI B: Evidence of Extent of Contribution to Country-level Goals and Priorities

KPI B is based on the assumption that an effective organisation should demonstrate contributions to country-level goals and priorities.

The assessment reviewed survey data from in-country stakeholders (donors and clients) and drew on the CAS/CPS progress and completion reports. Attention was paid to results statements, baselines and targets, and other evidence presented to substantiate the results achieved, including any analysis of contribution. The emphasis was on identifying results areas with illustrative evidence of progress towards expected development outcomes, including evidence of a logical theory of change for each results area.

Overall assessment

Figure 4.4 presents the overall rating for this KPI based on the review of the World Bank’s documentation and the responses of surveyed stakeholders. It also shows the criteria that MOPAN used to determine the overall rating (criteria met are indicated in blue).

Figure 4.4 KPI B: Evidence of Extent of Contribution to Country-Level Goals and Priorities, Rating

*This column is based on the total mean score (all respondents) in each country. A checkmark indicates the organisation received ratings of strong or higher on the majority of focus areas and limited, if any, ratings of inadequate.

Evidence of Extent of World Bank contributions

Finding 22: In most countries reviewed, the documented evidence suggests that the World Bank is adequately meeting objectives articulated in its Country Assistance Strategy. In these countries, there is stronger evidence of contribution in economic growth and human development sectors, where the Bank also has the highest levels of investment.

Between FY2008-2011, the IEG reviewed 64 Country Assistance Strategy Completion Reports CASCRs and found that programme objectives were substantially achieved in 58 per cent of the countries. IEG findings suggest that the Bank performs better in economic growth and human development, where it met 69 per cent and 67 per cent of its country objectives, respectively. The Bank’s development results in the area of governance were less positive, as only 47 per cent of objectives were met.[46]

These results are generally consistent with the document review findings for the small sample of countries reviewed for KPI B.

In countries where funding data was available in the CAS or CAS Completion Report,[47] it was also possible to associate evidence of progress towards CAS objectives (as confirmed by IEG reviews) with levels of investment. Not surprisingly, the Bank’s contributions were more notable in sectors where it has invested the most. The Bank made important financial contributions to growth-related activities in Ghana, Honduras, and Nigeria (48-63 per cent of the overall portfolio), and it also had stronger performance in this pillar than in other pillars of the CAS. The same was true for human development initiatives, for which the Bank committed 19-35 per cent of its overall portfolio in the same three countries. The Bank had not performed as well in the area of good governance, where projected investments reached only 5 per cent of the overall portfolio in Ghana and 10 per cent in Honduras. In Nigeria, however, the Bank made substantial contributions to strengthening anti-corruption agencies, which could in part be explained by the Bank’s higher financial commitment for governance (23 per cent of the portfolio).

The review of the Bank’s results for this KPI focused on completed CAS/CSP. Because these were the “first generation” of results-oriented CAS/CSP, the quality of results frameworks was frequently noted by the IEG as a factor that limited reporting on results. However, as noted in KPI 4, current CAS/CSPs are considered to be more representative of good practice.

Finding 23: Surveyed stakeholders considered the Bank adequate or above in contributing to country development results.

At country level, World Bank support is guided by the Bank’s strategic priorities and areas of comparative advantage, and involves consultation with shareholders and other development partners. The Bank’s Country Assistance/ Partnership Strategies (CAS/CPS) are closely aligned with national development plans. Development objectives are incorporated into each country results framework, which is then used to monitor progress towards results achievement. The assessment reviewed the main national development plans that were used to develop the CAS/CPS.[48]

The majority of survey respondents rated the Bank’s contribution to CAS objectives/expected outcomes as adequate or above, ranging from 71 per cent of respondents in Honduras to 57 per cent in Morocco. Survey responses suggest that the Bank has made the greatest contributions to the objectives in the following areas: energy and transport infrastructure in Ghana; transparency and efficiency in public finances in the Philippines; and increased access to basic services for the poor in Morocco. In four of the five countries sampled, however, there was a high level of ‘don’t know’ responses on the Bank’s contributions to specific objectives.

Evidence of Results Achieved, by Country

There is evidence of the World Bank’s contribution to country-level goals and priorities in several results areas across the five countries reviewed. The country review below presents some examples of development results achieved at country level.

Ghana

When asked about the World Bank’s contributions to its CAS objectives in Ghana, 67 per cent of clients and donors in-country rated the World Bank adequate or above. The majority of respondents (more than 80 per cent) rated the Bank adequate or above for its contributions in energy and transport infrastructure. Donors had high levels of ‘don’t know’ responses (ranging from 38 to 63 per cent) when asked about the Bank’s contributions to increasing access to education, health care, water and sanitation, and information and communications technology.

Figure 4.5 World Bank Stakeholder Survey - Ghana, Mean Scores by CAS Objectives

During the 2008 -2011 CAS period, the World Bank committed[49] the majority of its financial support (60 per cent) to private sector competitiveness which includes, among others, agriculture, natural resource management, land management, energy, and transport; 35 per cent of the portfolio was invested in human development and basic services, followed by 5 per cent for good governance.[50]

The document review found the most documented evidence of World Bank contributions to development results in the areas of agriculture, natural resource management, and land management in Ghana (see sidebar).

During the 2008 -2011 CAS period, the Bank contributed to improvements in the agricultural sector, in terms of increased crop yields per hectare and improved production. In the area of sustainable natural resource management, the World Bank- supported Natural Resources and Environmental Governance credit led to the introduction of key policies in the mining, forestry, and environment sectors that have promoted exports to the European Community. The Bank’s investment in land management has contributed to the development and approval of a spatial development planning framework, which will be used by Government to prepare a comprehensive land-use Bill.

In spite of the Bank’s significant investment in the transport sector, there has been limited progress in increasing transport infrastructure. Good governance is another area where only limited achievements are reported in enhancing decentralisation, managing public sector reform, improving economic governance, and realising evidence-based policy making.

Honduras

When asked about the World Bank’s contributions to its CAS objectives in Honduras, 71 per cent of respondents rated the Bank adequate or above. Close to half of the respondents rated the Bank strong or very strong for its contribution to reducing the impact of natural disasters, improving the performance of water and sanitation services, and increasing the impact of social assistance. They rated the Bank lower for contributions to improving the performance of the energy sector and for enhancing the quality of education. There was a high level of ‘don’t know’ responses (24 to 28 per cent) on the Bank’s contributions to competitiveness in the financial sector, water and sanitation, and the coverage of basic education.

Figure 4.6 World Bank Stakeholder Survey - Honduras, Mean Scores by CAS Objectives

As of 2010, the World Bank had committed a substantial amount (63 per cent) of its total lending in Honduras to economic growth for employment generation, followed by 19 per cent for the development of human capital, 10 per cent for good governance, and 8 per cent for environmental protection and risk management.[51]

When compared to the other four countries assessed, the World Bank’s contributions to development results in Honduras were less evident, which is reflected in the moderately unsatisfactory rating provided by the IEG review of the CAS 2007–2011 Completion Report.[52]

The IEG review provided its lowest rating (unsatisfactory) for the Bank’s contributions in governance, including in the sub-components of fostering transparency and in professionalising the public sector.

However, the review noted that the unstable socio-political environment was a major obstacle to progress in Honduras. In fact, it should be noted that Honduran President Manuel Zelaya was ousted by a coup in 2009, which was followed by a period of significant institutional instability. Due to this political unrest, the Bank’s programme in Honduras was suspended for close to one year. This suspension negatively affected the Bank’s overall performance for this entire CAS cycle.

Despite mitigated development results overall, the World Bank’s development policy lending to Honduras has nonetheless contributed to improving the business environment in particular, and Honduras has made substantial progress towards outcomes in this area.

During the CAS 2003-2006 cycle, the Bank’s lending contributed to the adoption of the Competition Policy Law, which led to the creation of the Comisión para la Defensa y Promoción de la Competencia (CDPC). The Bank continued to support development outcomes in business environment through its Trade Facilitation and Productivity Improvement Project. During the CAS 2007-2010 cycle, the Bank disbursed US$ 15.8 million in support of Honduras’ national strategy to improve the business environment. A review of Honduras’ Competition Policy Law in 2011 indicated that, since 2007, the CDPC “has investigated and sanctioned some of the country‘s largest cartels, in the cement, pharmacies, and sugar sectors (…).”[53] The time required for business registration was reduced from 110 days in 2003 to 14 days in 2010, and the time required for the acquisition of licenses (environmental and other building permits) decreased from 199 days in 2004 to 106 days in 2011.

Morocco

When asked about the World Bank’s contributions to its CPS objectives in Morocco, 57 per cent of country-level stakeholders rated the Bank adequate or above. More than 60 per cent rated the Bank strong or very strong for its contributions to increasing access to basic services for the poor, increasing access to education, and improving water management. Ratings were significantly lower for its contributions to increasing access to financing, and to increasing the competitiveness of small and medium-size agro-businesses. These two sectors also had the highest level of ‘don’t know’ responses (37-40 per cent) from both donors and clients.

Figure 4.7 World Bank Stakeholder Survey - Morocco, Mean Scores by CPS Objectives

The document review found that the World Bank contributed to growth and development in several sectors in Morocco through the provision of development policy lending and technical assistance. In the area of water and sanitation, the Bank supported the implementation of the Output-based Aid pilot project, to extend the supply of water and sanitation services to low- income peri-urban areas. A total of 52,500 persons in these areas benefitted from subsidized access to water and sanitation services through cost-efficient and sustainable means.

The World Bank partnered with the government of Morocco in an impact evaluation of its Education Conditional Cash Transfer pilot project. Findings from the evaluation are being used by the Moroccan government to scale-up the project to the national level.

Nigeria

When asked about the World Bank’s contributions to its CPS objectives in Nigeria, 62 per cent of respondents rated the World Bank adequate or above. Respondents were most positive about the Bank’s contributions to improving access to health care, strengthening public financial management, and increasing environmental sustainability in agriculture. They provided lower ratings on its contributions to strengthening the effectiveness of anti-corruption agencies and improving the performance of public management, where 47 and 29 per cent of respondents respectively have provided a rating of inadequate or lower.

Figure 4.8 World Bank Stakeholder Survey – Nigeria, Mean Scores by CPS Objectives

Almost half (48 per cent) of the World Bank’s original commitments for the period of FY2010-2013 went to initiatives related to non-oil growth such as agriculture, finance, business environment and infrastructures. Governance-related initiatives and human development programmes each accounted for 23 per cent of the planned portfolio.[54]

The Bank has made significant contributions to governance, health and agriculture in Nigeria. Examples of development results achieved are outlined in the sidebar.

The Bank’s contributions to the energy sector in Nigeria were identified as an area for improvement. The 2009 IEG review of the Country Partnership Strategy (CPS) 2005-2009 completion report noted that the targeted increase in power generation and reduction in power outages were not achieved. However, the mid- term progress report on the implementation of the CPS 2010-2013 identified a 21 per cent increase in power generation to 3800 megawatts (from the 2009 baseline of 3000 megawatts).

The Philippines

Overall, 67 per cent of stakeholders surveyed rated the Bank as adequate or above for its contribution to the achievement of its country objectives in the Philippines.

Moreover, 80 per cent or more of respondents rated the Bank adequate or above for its contributions to improving transparency and efficiency in public finances, and to increasing access to education for the poor. They provided lower ratings for the Bank’s contributions to improving capacities of anti-corruption agencies and increasing the productivity of firms. The level of ‘don’t know’ responses was high (43 to 57 per cent) on questions relating to increasing the productivity of firms, implementing development programmes in conflict-affected communities, increasing access to healthcare services, and increasing access to water and sanitation.

Figure 4.9 World Bank Stakeholder Survey - Philippines, Mean Scores by CAS Objectives

In health care, the Bank has contributed to an increase in the percentage of women giving birth in health facilities and to maintaining vaccination coverage above 80 per cent. Several long- term projects to increase access to water and sanitation are underway in the Metro Manila area. While they have shown limited progress thus far, it is expected that outcomes will be met in the upcoming years.

Following the change of government during the period of the CAS 2010-2013, the new administration identified stability and peace as a strategic country priority. The Bank has provided support to capacity building and community-driven programming in the conflict- affected area of Mindanao through a Bank administered multi-donor trust fund and it is anticipated that support will further increase following the signing of the peace agreement between the Government and the Moro Islamic Liberation Front which took place in October 2012. The Bank is on track to achieving expected outcomes set for the 2010-2013 cycle and at least 1000 families affected by conflict are expected to benefit from infrastructure, housing and improved livelihood.

In its efforts to create a stable macro-economy, the Government of the Philippines has also targeted fiscal and financial stability to yield economic growth, and increase country capacity to effectively address and reduce the likelihood of external shocks. It has proposed to improve tax administration and compliance prior to amending tax policy. The World Bank has expressed support for this course of action.

With regard to strengthening capacities of anti-corruption agencies, the Bank has adopted an opportunistic approach depending on the enabling environment. During the past administration, the climate was not very conducive to strengthening anti-corruption agencies but this has greatly improved under the current one. As such, the Bank has now been concentrating more efforts in this area and is on track to achieving expected outcomes for the FY2010-2013 CAS cycle.

4.3.3 KPI C: Evidence of Extent of Contributions to Relevant MDGs

KPI C is based on the assumption that multilateral organisations have made commitments to the Millennium Declaration and have assumed facilitative roles to ensure that finance, aid, and knowledge help countries to achieve the Millennium Development Goals (MDGs). The MDGs are collective, global targets that have, in many cases, been adapted by partner countries in defining their priorities. While partner countries are responsible for making progress toward the MDGs, multilateral organisations ensure that their aid, knowledge, and other types of support facilitate achievement of these goals.

Given the extent of collaboration on MDGs at country level, the World Bank cannot take responsibility for the achievement of specific MDGs, but supports projects and programmes that reflect client country priorities and contribute to the achievement of the prioritised MDGs. In the CAS/CPS of the five countries sampled, all included a discussion on the country’s progress to meeting the MDGs and the likelihood that these objectives would be met. However, only three CAS/CPSs (Ghana, Nigeria, and the Philippines) explicitly aligned the countries’ MDG objectives with support to be provided by the World Bank.

Links between the World Bank’s objectives and the MDGs

At the global level, the World Bank plays a key role in the global partnership for achieving the MDGs. At the 2010 United Nations MDG Summit, country progress towards the achievement of the MDGs was discussed. The World Bank committed to increasing its efforts to support countries that are not on track to meet the MDGs by 2015.

“We are mobilizing substantial new investments to help countries close the gap and achieve the MDGs. Our efforts are focusing on the “Access Agenda”: helping to ensure access to basic health, quality schooling, clean water, energy, food, and jobs. In particular, we are working to assist countries to build social protection systems, enhance the gender focus of our work, and address the special needs of fragile and conflict-affected situations (FCS).[55]

In order to support the monitoring of country achievement of the MDGs, the World Bank publishes the annual Global Monitoring Report in partnership with the International Monetary Fund. The Bank has also launched an online data visualisation tool, the eAtlas of the Millennium Development Goals, to map the indicators that measure progress towards achieving the MDGs.

In the organisation’s Corporate Scorecard, Tiers I and II align global and country-level development results with the higher level outcomes of the MDGs. Some of the Core Sector Indicators included in the Scorecard also measure progress towards achieving the MDGs.

At the country level, the Bank’s Country Assistance/Partnership strategies reflect the national development goals established by the client country, which include the country priorities with regard to MDGs. The CAS must take into account the link between national goals and MDGs and provide an analysis of progress and major gaps in the achievement of national MDGs.[56]

Overall Assessment

Figure 4.10 shows the overall rating for this KPI based on the review of the World Bank’s contribution to the achievement of the MDGs in the five countries reviewed. The main sources of data were reports produced by the World Bank and the perception of stakeholders surveyed.

Figure 4.10 KPI C: Contribution to Relevant MDGs, Overall Rating and Criteria

*A checkmark indicates that the organisation received survey ratings of strong or higher on the majority of MDGs and limited, if any, ratings of inadequate.

Evidence of World Bank contributions

Finding 24: All Country Assistance/Partnership Strategies reviewed showed evidence of the Bank’s support to governments in achieving national MDGs but only three demonstrated an explicit link between the Bank’s expected outcomes and the MDGs.

The document review drew its findings from a review of country-specific results matrices, progress reports on the implementation of the CAS/CPS, project implementation status and results reports, and IEG reviews of CAS Completion Reports for the five countries reviewed.

As demonstrated in KPI B, the Bank’s portfolios in the sampled countries are diversified and include several development projects which are related to the MDGs, ranging from growth and poverty reduction to education, health, and climate change. It is therefore implicit that the Bank has contributed to the MDGs in these countries. However, only three of the CAS/CPS reviewed (Ghana, Nigeria, and the Philippines) included a discussion of the facilitative role that the Bank played in supporting countries to make progress towards the MDGs.

Overview of Survey Data, by Country

Finding 25: Survey respondents were generally positive about the Bank’s contribution to the achievement of national MDGs.

Overall, survey respondents rated the World Bank adequate for supporting countries in achieving national MDGs. Respondents from Ghana, Morocco, and Nigeria provided the highest ratings; respondents from Honduras provided the lowest.

The Bank was seen by stakeholders as having contributed most to MDG 2 (achieving universal primary education). Respondents provided lower ratings for the Bank’s contributions to MDG 5 (improving maternal health) and MDG 6 (Combat HIV/AIDS), especially in Honduras and Morocco.

Figure 4.11 Survey Data on World Bank Contributions to MDGs, Mean Scores, by Country

Ghana

When asked about the extent to which the World Bank contributed to Ghana’s MDGs, 63 per cent of in-country stakeholders rated the Bank adequate or above. They provided higher ratings for Goal 1 (Eradicate extreme poverty and hunger), where 22 per cent of respondents rated the Bank very strong, and for Goal 5 (Improve maternal health), where 45 per cent of stakeholders gave a rating of strong or above. Donors in-country had a high level of ‘don’t know’ responses (38 to 50 per cent) on Goals 2, 4, 5, and 6.

Honduras

Among respondents from Honduras, 64 per cent rated the Bank’s contribution to Honduras MDGs adequate or above. They were most positive about its contributions to Goal 2 (achieve universal primary education), where more than half of the respondents (54 per cent) gave a rating of strong or above, and Goal 7 (Ensure environmental sustainability), where77 per cent of respondents rated the Bank adequate or above

Morocco

When asked about the Bank’s contributions to the MDGs in Morocco, 60 per cent of respondents provided a rating of adequate or above. Among donors in-country, there was a high level of ‘don’t know’ responses (25 to 50 per cent) for all MDGs except Goal1 (Eradicate extreme poverty and hunger).

Nigeria

Overall, 65 per cent of respondents rated the Bank adequate or above for its support to the achievement of national MDGs in Nigeria. They were less confident about the Bank’s contributions to MGD 1 (Eradicate extreme poverty and hunger) and MDG 2 (Achieve universal primary education). There were statistically significant differences between respondents groups on Goal 4 and 5: clients responded more positively than donors.

The Philippines

When asked about the extent to which the World Bank has contributed to the government’s efforts for achieving the MDGs in the Philippines, 59 per cent of in-country stakeholders gave a rating of adequate or above. There was a high level of ‘don’t know’ responses (29 to 57 per cent) among donors in-country on all MDGs.

4.3.4 KPI D: Relevance of objectives and programme of work to stakeholders

For this KPI, MOPAN assessed relevance primarily as a measure of the extent to which a multilateral organisation is seen to support country priorities and meet the changing needs of clients and the target population. The assessment is based exclusively on survey data from the five countries selected for the test of this component.

Overall assessment

Across four of the five countries, donors in-country and clients considered the World Bank strong in responding to partner countries’ key development priorities. In most countries, they rated the Bank as adequate for providing innovative solutions to help address development challenges and for adapting its work to the changing needs of partner countries.

Figure 4.12 shows the overall assessment rating and the mean scores for donors in-country and clients on the three survey questions on which the assessment is based.[57]

Figure 4.12 KPI D: Relevance of Objectives and Programme of Work to Donors In-Country and Clients, Overall Rating and Survey Mean Scores by Country

Overall Assessment: Adequate

*Detailed scores are shown in Volume II, Appendix VI.

5. Conclusion

This conclusion steps away from the specific ratings of the MOPAN assessment and looks at the major messages that can contribute to dialogue between MOPAN and the World Bank and its clients. It draws on the survey findings and principal observations of the assessment of the World Bank’s practices and systems (Key Performance Indicators 1- 19) and the assessment of the World Bank’s development results (Key Performance Indicators A - D).

The World Bank has demonstrated strategic leadership in managing for and reporting on development results. While the Bank is viewed as a leader in this area, some shortcomings were noted in the frameworks and systems that it uses to measure and report on organisation-wide results.

The Bank is developing good practices in managing for results at the corporate and country level and has developed mechanisms to systematically track progress and report on both development results and organisational performance. The document review noted the Bank’s establishment of a Results Secretariat, the development of the Corporate Scorecard, and staff capacity-building modules on results management.

At the country level, Country Assistance/Partnership Strategies (CAS/CPS) include results frameworks that link the Bank’s support to country results and national development priorities. The CAS/CPS have improved in quality and consistency over time.

At the corporate level, the management results framework contains a set of implicit assumptions about the results chain between Tiers II and IV. However, the development results framework does not provide any links between country-level outputs and outcomes and the Bank’s organisation-wide development outcomes.[58] In addition, the indicators under Tier II focus on results achieved at the country level, but there is no theory of change that provides a complete picture of how the organisation conceives its contributions to development change.

The World Bank is committed to transparency and has invested in the creation of an Open Bank.

The Bank has demonstrated leadership in transparency with the launch of its Open Data Initiative in 2010 and Open Access Policy in 2012. It has made efforts to facilitate public access to its research data, knowledge repository, and ODA disbursement practices, as well as to ensure that most of its key documents are available on the World Bank website. Stakeholders rated the Bank strong in this area and the document review considered it very strong.

The Bank allocates resources in a transparent manner and on the basis of country performance and its administrative budget is allocated to expected results. However, disbursements are not yet related to corporate results.

The Bank’s criteria for resource allocations are clear and publicly available and IDA resources are allocated based on national performance in the areas of poverty reduction and economic growth. However, while the Bank does link its administrative budget to expected results, it does not report on expenditures or disbursements related to its organisation-wide (corporate) results.

The World Bank is recognised for its strong financial and fiduciary practices.

The document review and surveyed stakeholders rated the Bank strong or very strong for its external and internal financial audits, policy on anti-corruption, systems to handle irregularities, and strategies for risk management. Financial statements, annual audits, and external audits of Bank-supported projects and programmes comply with international auditing standards.

The World Bank’s management of human resources is considered strong overall but it needs to improve its staff compensation system.

The Bank has a performance assessment system for staff on regular, open-ended, term, and local regular appointments, which also applies to senior staff, including Vice-Presidents and Managing Directors. The IEG rated the Bank negatively in terms of performance-based incentives for staff, and the Bank has indicated that it is reforming its human resources policy with a focus on staff compensation.

Surveyed stakeholders rated the Bank adequate for its use of country systems and adjustment of procedures but in their written comments expressed concerns about the Bank’s bureaucratic procedures.

In both the 2009 and 2012 MOPAN surveys, respondents generally considered the Bank’s use of country systems in its operations to be adequate. The 2012 document review, however, considered the Bank strong. It noted that the Bank has committed to the on-going development of Country Assistance/ Partnership Strategies in support of country ownership and continues to use country public financial management and procurement systems as reported to the OECD- DAC in respect of its Paris Declaration commitments.

Survey respondents noted some improvement in the Bank’s adjustment of procedures, which they rated inadequate in 2009 and adequate in 2012. In their written comments in the 2012 survey, however, they raised concerns about the Bank’s bureaucratic procedures and suggested that the Bank should be more flexible.

The Bank is considered strong in its evaluation of results and dissemination of knowledge. It has also introduced integrated mechanisms to report on its performance, but it will take time for these to be fully developed.

The Independent Evaluation Group (IEG), which reports directly to the Board of Executive Directors, has established good practices for the evaluation of results, strong quality control mechanisms to ensure the quality of evaluation activities and performance reporting, and mechanisms for the dissemination of key lessons learned and good practices. The use of evaluation findings in decision making is an area for improvement.

The Bank has developed mechanisms to report on performance, some of which are still being refined; in the Corporate Scorecard and its companion report, World Bank for Results, the links between outputs and organisation-wide outcomes needs to be strengthened. The Bank has developed several avenues for knowledge dissemination, including reports, capacity-building sessions, analytical advisory activities, and the World Bank website. Surveyed stakeholders feel the Bank provides sufficient opportunities for sharing lessons learned.

Development Results Component

The Bank is making progress towards organisation-wide outcomes and in reporting on these. Its corporate reports do not yet provide stakeholders with a clear picture of how the Bank’s country level efforts contribute to organisation-wide outcomes.

Surveyed donors at headquarters feel the Bank is making substantial progress towards its strategic priorities. The Corporate Scorecard and World Bank for Results report provide an integrated view of results and performance. However, given the Bank’s focus on results achieved at the country level, the current frameworks do not provide a complete picture of how the organisation as a whole is achieving its organisation-wide outcomes.

The Bank is seen to be contributing to country-level objectives and national Millennium Development Goals.

At the country level, stakeholders gave the Bank generally strong ratings for responding to their countries’ key development priorities and adequate ratings on its relevance overall and its contributions to MDGs.

The majority of stakeholders surveyed consider the Bank adequate in contributing to CAS/CPS country objectives, which are aligned with national goals and priorities. While World Bank reports tend to confirm the Bank’s contribution in most results areas, the Bank’s role or contribution in support of its clients’ efforts to achieve MDGs is more explicit in some CAS/CPS than in others.

The Bank’s country reporting tools are improving and this may lead to better evidence of its contributions to country development outcomes in the future.

---

[1] The assessment excludes other entities of the World Bank Group, although in some cases World Bank Group documents were reviewed.

[2] The AfDB, UNICEF, UNDP, and the World Bank

[3] The assessment refers to organisation-wide outcomes for the World Bank, as opposed to the World Bank Group.

[4] This report assesses the World Bank (IDA and IBRD) only, and not the World Bank Group, which also includes the International Finance Corporation (IFC), the Multilateral Investment Guarantee Agency (MIGA), and the International Centre for Settlement of Investment Disputes (ICSID). However, the assessment examined relevant World Bank Group documents that provide strategic direction, policy frameworks, or reporting for IDA/IBRD.

[5] The World Bank. (2010, April). New World, New World Bank Group: (I) Post-Crisis Directions. DC2010-0003. Washington D.C.: Development Committee, The World Bank.

[6] Ibid.

[7] As a result of the volatile socio-political situation in Sudan during the administration of the 2008 Annual MOPAN Survey, the MOPAN process was not completed to the required standard. Country reporting for Sudan was therefore excluded from the final MOPAN report.

[8] This component was tested in 2012 with the African Development Bank, UNICEF, UNDP, and the

World Bank.

[9] The full methodology is presented in Volume II, Appendix I.

[10] MOPAN criteria for country selection include: multilateral organisation presence in-country, presence and availability of MOPAN members, no recent inclusion in the survey, the need for geographical spread, and a mix of low income and middle income countries (middle income countries being subdivided into lower middle and upper middle).

[11] The normal convention for statistical significance was adopted (p≤.05).

[12] These organisations were selected because they were assessed by MOPAN in 2009. The 2009 assessment focused on organisational effectiveness and was based only on survey data.

[13] Different organisations use different terms to refer to their planned results – they may be called goals, objectives, outcomes, etc.

[14] The “best fit” approach is used in public sector institutions (see Ofsted, 2011: Criteria for making judgements)

[15] Respondents who wrote “no comment” or something similar were filtered out of the analysis.

[16] While most KPIs and micro-indicators were considered in the document review, not all were rated. See section 2.3.

[17] The normal convention for statistical significance was adopted p≤.05.

[18] In the MOPAN Common Approach, a rating of ‘very strong’ is given when all five criteria defined in the Document Review Guide for a particular micro-indicator have been met. Further, this is interpreted by MOPAN to mean that the multilateral organisation’s system is moving towards “best practice” in this area.

[19] Arabic, Bengali, Chinese, French, Hindi, Portuguese, Russian, Spanish, Vietnamese

[20] The World Bank. (2011). World Bank for Results 2011. Washington D.C.: The World Bank. (p. v).

[21] The World Bank (2010). Gender and Development: An Evaluation of World Bank Support 2002-08. Washington D.C.: Independent Evaluation Group, The World Bank

[22] Independent Evaluation Group. (2011). World Bank Country-Level Engagement in Governance and Anticorruption. Washington D.C.: The World Bank.

[23] International Development Association. (2009, October). IDA and Climate Change: Progress Report. Washington D.C.: International Development Association.

[24] While results-based budgeting points to outcomes, it is focused on the link between inputs, activities, and outputs. According to OECD/DAC’s definition, results-based budgeting is: “A budgeting method that links appropriations to outcome level performance targets. Expected results justify resource requirements and actual performance in achieving results is measured by predefined performance targets”

[25] The World Bank. (2011, December). A New Instrument to Advance Development Effectiveness: Program-for-Results Financing. OPCS/66193. Washington D.C.: The World Bank

[26] The Internal Auditing Department (IAD) was upgraded to a Vice Presidency in 2009.

[27] A policy is a framework sanctioned by the institution that is used to guide action.

[28] Independent Evaluation Group. (2012) The Matrix System at Work: An Evaluation of the World Bank’s Organizational Effectiveness, Washington D.C.: Independent Evaluation Group

[29] In its 2012 Scorecard, the World Bank marked the status of this indicator as yellow, or “Watch”, meaning that there has been no meaningful decrease or increase in the value.

[30] According to BP 13.05, examples of minor projects are: “renaming of implementing entities, agreed revision of reporting schedules or formats, or changes in the specification of project outputs that have no significant impact on the project”.

[31] According to BP 13.05, first-order restructurings involve a change in project objectives and/or outcome targets and can only be approved by the Board of Executive Directors. Second-order restructurings do not involve a change in project objectives and/or outcome targets and can be approved by the Regional Vice Presidents

[32] In order to assist borrowers with actual or anticipated financing requirements, the World Bank provides support in the form of rapidly disbursed policy-based financing, known as development policy lending. It is provided in the form of loans or grants.

[33] The World Bank. (2005). Review of World Bank Conditionality. Washington D.C.: The World Bank.

[34] The World Bank. (2006). Good Practices Principles for the Application of Conditionality: A Progress Report. Washington D.C.: The World Bank.

[35] The World Bank. (2007, December). Conditionality in Development Policy Lending. Washington D.C.: The World Bank.

[36] The World Bank. (2009). Development Policy Retrospective. Washington D.C.: The World Bank.

[37] The World Bank. (2011, November). The World Bank and Aid Effectiveness: Performance to Date and Agenda Ahead. Washington D.C.: The World Bank.

[38] The members of the PEFA network are: the African Development Bank, Asian Development Bank, European Commission, Inter-American Development Bank, International Consortium on Governmental Financial Management, Organisation for Economic Co-operation and Development, and the World Bank.

[39] The World Bank. (2011). The World Bank and Aid Effectiveness: Performance to Date and Agenda Ahead. Washington D.C.: The World Bank.

[40] Ibid.

[41] Independent Evaluation Group. (2011). World Bank Progress in Harmonization and Alignment in Low- Income Countries. Washington D.C.: Independent Evaluation Group.

[42] For further information regarding country level data, please refer to OECD’s website: http://stats.oecd.org/Index.aspx?lang=en&DataSetCode=SURVEYDATA or http://www.oecd.org/dac/aideffectiveness/2011surveyonmonitoringtheparisdeclaration- countrychapters.htm

[43] The Independent Evaluation Group (2010). IEG Annual Report 2010: Results and Performance of the World Bank Group. Washington D.C.: The Independent Evaluation Group.

[44] Recent literature builds and expands on definitions that closely link theory of change with programme theory approaches in evaluation. For example: Vogel, Isabel, Review of the use of “Theory of Change” in International Development, April 2012; Stein, Danielle and Valters, Craig, Understanding “Theory of Change” in International Development: A Review of Existing Knowledge, July 2012; and James, Cathy, Theory of Change Review, a report commissioned by Comic Relief, September 2011

[45] The Bank’s report The State of World Bank Knowledge Services (2011) notes this dilemma and describes steps it is taking to enhance results, connectivity, and openness associated with work in this area.

[46] IEG. (2011). IEG Annual Report 2011: Results and Performance of the World Bank Group p. xix.

[47] Given different approaches to categorizing individual projects by sectors, we relied on CAS-related documents to provide an indication of how financial commitments during the CAS period are distributed across pillars or key result areas. The CAS documents for Morocco and Philippines did not provide this kind of data.

[48] These included: the National Economic Empowerment and Development Strategy (Nigeria); the Updated Medium-term Philippines Development Plan (2004-2010); the Country Vision (2010-2038) and National Plan (2010 -2022) of Honduras; Ghana’s Growth and Poverty Reduction Strategy II; and the National Initiative for Human Development (Morocco).

[49] In these sections, we provide data on the Bank’s financial commitments as opposed to actual lending. While actual lending is available on a project basis, it cannot be easily categorized by results area or pillar in the CAS. The World Bank country summary lending, which is available online, provides figures on lending by country but classifies these figures by sectors that are not always easy to link to the results statements in CAS documents.

[50] IDA. (2010). Country Assistance Strategy Progress Report for the Republic of Ghana: Report No.52988-GH. p. 4.

[51] IDA and IFC. (2010). Interim Strategy Note for the Republic of Honduras: Report N. 56405-HN. p. 9.

[52] The CAS for Honduras for 2007-2010 was followed by an Interim Strategy Note in 2011 because of political instability. These were evaluated together for the 2007-2011 cycle.

[53] OECD and IDB. (2011). Competition Law and Policy in Honduras: A Peer Review. p.8.

[54] IDA. (2011). Country Partnership Strategy Progress Report for the Federal Republic of Nigeria for the Period FY10-FY13: Report No. 63505-NG. p. 15.

[55] The World Bank (2011, April). Modernizing the World Bank Group: An Update. DC2011-0005. Washington D.C.: Development Committee, The World Bank, pp. iii-iv.

[56] The World Bank (2012, April). Guidelines to Staff for CAS Products. Washington D.C.: Operations Policy and Country Services, The World Bank, pp. 4-5.

[57] At the request of the World Bank, similar questions were asked to MOPAN donors at Headquarters who were familiar with the Bank’s work in each of these countries. However, given the small number of respondents that indicated familiarity with these countries, we have not included their responses in this analysis.

[58] The assessment refers to organisation-wide outcomes for the World Bank, as opposed to the World Bank Group.