Preface

The Multilateral Organisation Performance Assessment Network (MOPAN) is a network of donor countries with a common interest in assessing the organisational effectiveness of multilateral organisations and their measurement and reporting on development and/or humanitarian results. MOPAN was established in 2002 in response to international forums on aid effectiveness and calls for greater donor harmonisation and co- ordination.

Today, MOPAN is made up of 17 donor countries: Australia, Austria, Belgium, Canada, Denmark, Finland, France, Germany, Ireland, The Netherlands, Norway, the Republic of Korea, Spain, Sweden, Switzerland, the United Kingdom, and the United States. For more information on MOPAN and to access previous MOPAN reports, please visit the MOPAN website (www.mopanonline.org).

Each year MOPAN carries out assessments of several multilateral organisations based on criteria agreed by MOPAN members. Its approach has evolved over the years, and since 2010 has been based on a survey of key stakeholders and a review of documents of multilateral organisations. MOPAN assessments provide a snapshot of four dimensions of organisational effectiveness (strategic management, operational manage- ment, relationship management, and knowledge management). In 2013, MOPAN has integrated a new component to examine the evidence of achievement of development and/or humanitarian results to complement the assessment of organisational effectiveness.

MOPAN 2013

In 2013, MOPAN assessed four multilateral organisations: the Asian Development Bank (ADB), the International Fund for Agricultural Development (IFAD), the World Food Programme (WFP) and the World Health Organization (WHO).

MOPAN Institutional Leads liaised with the multilateral organisations throughout the assessment and reporting process. MOPAN Country Leads monitored the process in each country and ensured the success of the survey.

 

Acknowledgements

We thank all participants in the MOPAN 2013 assessment of ADB. ADB’s senior management and staff made valuable contributions throughout the assessment and document review processes and provided lists of their clients to be surveyed. Survey respondents contributed useful insights and time to respond to the survey. The MOPAN Institutional Leads, France and the Republic of Korea, liaised with ADB throughout the assessment and reporting process. The MOPAN Country Leads oversaw the survey planning process in the field and ensured the success of the survey. Consultants in each country provided vital in-country support by following up with country-level survey respondents to enhance survey response rates.

Roles of authors and the MOPAN Secretariat

The MOPAN Chair was held by Australia in 2013 and worked in close co-operation with the MOPAN Technical Working Group to launch and manage the survey. Once the MOPAN Secretariat was established at the OECD in April 2013, the Head of the Secretariat oversaw all related tasks.

MOPAN developed the Key Performance and Micro-Indicators, designed the survey methodology, co- ordinated the preparation of the lists of survey respondents, and approved the final survey questionnaire for each agency. MOPAN also directed the approach to the document review and oversaw the design, structure, tone, and content of the institutional reports.

Universalia and Epinion developed the survey instrument and carried out the survey and analysis. Universalia carried out the document review, conducted the interviews with multilateral organisation staff at headquarters and country levels, and wrote the institutional reports.

Epinion is a consulting firm in Denmark that analyses and evaluates data to support decision making. It conducts specially designed studies for public and private organisations based on data collected among an organisation’s employees, members, customers, partners, and other sources. Epinion has 75 employees and 200 interviewers. Website: www.epinion.dk

Universalia Management Group is a Canadian consulting firm established in 1980 that specialises in evaluation and monitoring for international development. Universalia has made significant contributions to identifying best practices and developing tools in the fields of organisational assessment; planning, monitoring, and evaluation; results-based management; and capacity building. Website: www.universalia.com.

Acronyms

ADB - Asian Development Bank
ADF - Asian Development Fund
ARAR - Annual Report on Acting on Recommendations
CAPE - Country Assistance Program Evaluation
COBP - Country Operations Business Plan
COMPAS - Common Performance Assessment System
CPS - Country Partnership Strategy
DECB - Development Effectiveness Country Brief
DEfR - Development Effectiveness Review
DMC - Developing Member Country
IED - Independent Evaluation Department
IFAD - International Fund for Agricultural Development
KPI - Key Performance Indicator
LTSF - Long-term Strategic Framework
MfDR - Managing for Development Results
MI - Micro Indicator
MOPAN - Multilateral Organisation Performance Assessment Network
OECD-DAC - Organisation for Economic Co-operation and Development - Development Co-operation Directorate
PBA - Performance-based Allocation
PCR - Project Completion Report
PIU - Project Implementation Unit
SAPE - Sector Assistance Program Evaluation
WFP - World Food Programme
WHO - World Health Organization
WPBF - Work Program and Budget Framework
XARR - Expanded Annual Review Reports

Executive summary

This report presents the results of an assessment of the Asian Development Bank (ADB) conducted by the Multilateral Organisation Performance Assessment Network (MOPAN). The MOPAN Common Approach examines organisational systems, practices, and behaviours that MOPAN believes are important for aid effectiveness. It also examines the extent to which there is evidence of an organisation’s contributions to development and/or humanitarian results and the relevance of its objectives and programme of work to stakeholders at the country level.

ADB’s mission is to alleviate poverty in Asia and the Pacific and help create a world in which everyone can share in the benefits of sustained and inclusive growth. In its long-term strategic framework (Strategy 2020), ADB pledges to support three strategic agendas: inclusive economic growth, environmentally sustainable growth, and regional integration. It commits to being selective and placing focus on five core areas that “best support its agenda, reflect developing member countries’ needs and ADB’s comparative strengths, and complement efforts by development partners”, namely: infrastructure, environment, regional co-operation and integration, financial sector development, and education. ADB provides loans, technical assistance, and grants to developing member countries (DMCs) to help tackle poverty challenges and achieve sustainable growth.

ADB has undertaken several processes that reflect its strong commitment to strengthening practices and systems for Managing for Development Results (MfDR) at both the organisational and country levels. The Bank has committed to improve its measurement of and reporting on results achieved at both levels, reinforce selectivity in its operations, review the implementation of Strategy 2020, and strengthen its quality at entry processes.

MOPAN assessment

In 2013, the MOPAN assessment of ADB was based on information collected through a survey of key stakeholders, a review of documents, and interviews with ADB staff. The survey targeted MOPAN donors at headquarters as well as MOPAN donors and ADB clients based in three countries (i.e. Indonesia, Pakistan and Viet Nam). A total of 159 respondents participated in the ADB survey (37 MOPAN donors based at headquarters, 35 MOPAN donors based in-country, and 87 clients). The document review examined close to 400 publicly available corporate documents as well as programming documents for the three surveyed countries. MOPAN also interviewed ADB staff members (12 from ADB’s headquarters and 7 from Resident Missions). This information was not coded or used formally as part of the assessment process, but rather to gain a broader contextual understanding of the organisation’s systems, practices and results-related reporting.

MOPAN assessments provide a snapshot of four dimensions of organisational effectiveness (strategic management, operational management, relationship management, and knowledge management), and of contributions to development results. The main findings of the 2013 assessment of ADB in these performance areas are summarised below.

Key Findings

Strategic management

In the area of strategic management, MOPAN established criteria to determine if a multilateral organisation has strategies and systems in place that reflect good practice in managing for results. Overall, the 2013 assessment found that:

• Since the 2010 MOPAN assessment, ADB has improved its results-orientation at both the organisational and country levels. Its commitment to Managing for Development Results at the corporate level has included: the refinement of its corporate results framework in 2010; the adoption of a revised/outcome- oriented results framework in 2013; and systematic use of the Development Effectiveness Review report (DEfR) as a corporate planning and decision-making tool. Like other multilateral development banks, ADB faces challenges in ensuring a clear and logical progression from lower-level results to results at the impact level.

• MOPAN donors at headquarters considered the Bank strong for having a clear strategy that is aligned with its mandate. The document review concurred and noted the Bank’s review of Strategy 2020 so as to ensure its continuing relevance.

• ADB has strengthened the results focus of its Country Partnership Strategies and both its country and sector results frameworks have improved in quality and consistency.

• ADB was considered adequate or better in integrating/mainstreaming the cross-cutting themes of gender equality, environment, climate change, private sector development, regional cooperation and integration, and good governance. It was rated strong by survey respondents for its support of regional integration and adequate in all other areas.

• While ADB has no clear mandate for adopting human rights-based approaches and does not identify human rights as a priority or cross-cutting priority, the document review noted that the ADB promotes the principles of human rights in many of its policies, including its safeguards policy and accountability mechanism, and has established practices, systems and procedures to ensure that its investments “do no harm”. Survey respondents consider ADB adequate in this regard.

Operational management

In operational management, MOPAN established criteria to determine if a multilateral organisation manages its operations in a way that supports accountability for results and the use of information on performance. Overall, the 2013 assessment found that:

• ADB’s performance-based allocation system (PBA) for concessional funding is transparent and survey respondents rated ADB as adequate in ensuring that resource allocations follow established criteria.

• Donors at headquarters and document review considered ADB as adequate in linking its budgets and expenditures to results. The Bank’s results-based budgeting is evident at the project level and through its CPS and sector strategies and results frameworks. In the development of its administrative budget, there is room for improving the documentation and data that informs the budgeting process. There are already initiatives underway to address this (for example, with regard to the time management system).

• Financial accountability was considered an area of strength of the Bank. ADB was found to have strong internal and external audits practices, anti-corruption measures, and risk management processes in place. The Bank uses performance information to improve policies, plan new interventions, and allocate resources.

• The document review considered ADB’s use of performance information as very strong overall but rated the management of unsatisfactory operations adequate. Survey respondents rated ADB as adequate overall but considered its use of country and sector performance information to plan new interventions as strong.

• While donors at headquarters perceived ADB’s staff promotion processes to be inadequate and to lack transparency, the Bank’s management of human resources was rated strong overall by the document review. The system used to assess the performance of Vice Presidents of the Bank, however, was found to be unclear.

• Survey respondents perceived ADB as adequate in subjecting new loans and grants to benefits/ impact analysis prior to approval. The document review rated it strong in this area. ADB was considered to be very strong in setting milestones and targets to track the progress of project implementation.

• ADB’s delegation of decision-making authority was perceived as adequate overall. The assessment team noted the increased levels of responsibility in Resident Missions that have been put in place as part of ADB’s efforts to decentralize decision-making.

Relationship management

In relationship management, MOPAN established criteria to determine if a multilateral organisation is engaging with its clients at the country level in ways that contribute to aid effectiveness. Overall, the 2013 assessment found that:

• Surveyed stakeholders generally perceived the Bank as strong in ensuring that funding proposals are designed and developed in collaboration with national stakeholders. The document review rated ADB as strong for its use of national indicators or benchmarks for conditionality.

• ADB was seen to be adequate for having procedures that are easily understood and completed by clients, responding quickly to changing circumstances in partner countries, and adjusting these procedures to local conditions and capacities. However, many survey respondents lacked familiarity with these processes.

• Surveyed stakeholders perceived the Bank as adequate in using country systems as a first option for its operations. ADB has made considerable progress in its use of country systems for disbursement and operations but the document review noted room for improvement in the use of national procurement systems.

• ADB was seen as strong for contributing valuable input to policy dialogue and respecting the views of clients.

• ADB was overall considered to be adequately harmonising procedures at the country level. It was rated strong by document review for its participation in joint missions.

Knowledge management

In knowledge management, MOPAN established criteria to determine if a multilateral organisation has reporting mechanisms and learning strategies that facilitate the sharing of information inside the organisation and with the development community. Overall, the 2013 assessment found that:

• ADB was noted for the quality and independence of its evaluations, managed by the Independent Evaluation Department. Surveyed stakeholders considered the Bank adequate in using evaluation findings to inform decision making and involving clients and beneficiaries in evaluations.

• ADB’s use of performance information to report on results was considered adequate overall by surveyed stakeholders. The document review rated the Bank strong overall but noted room for improvement in reporting on programming adjustments based on performance information.

• Survey respondents perceived ADB to be adequate in the dissemination of lessons learned. The document review confirmed that the Bank is a knowledge-based organisation that places strong focus on lessons learned.

• The document review rated the Bank as very strong for making key documents available to the public.

Development results and relevance to stakeholders

In the 2013 assessment of development results, ADB was rated adequate in providing evidence of progress towards organisation-wide outcomes (KPI A) and adequate in providing evidence of contribution to country-level goals and priorities (KPI B). Survey respondents rated ADB adequate for the relevance of its objectives and programme of work to country level stakeholders (KPI C). These findings should be considered in conjunction with the findings above on ADB’s systems and practices for organisational effectiveness.

• Evidence of extent of progress towards organisation-wide outcomes: ADB’s reporting (via the Development Effectiveness Reviews and its current online platform) provides a solid and transparent evidence base of completed outputs, but limited evidence of contribution to higher-level change (i.e. outcome and impact). Recent evaluation studies conducted by the Bank’s Independent Evaluation Department (IED) highlight select areas where ADB’s operations have contributed to higher level change (for example, in the transport sector). There is an indication that recent changes in the Bank’s results framework should improve ADB’s reporting on outcomes in the future. The Bank has also made continuous efforts to improve the quality of data used to support its claims about contributions to results. This is observed in the 2012 DEfR which presents adjusted ratings based on IED-validated data on project effectiveness at outcome level, and therefore enhances the credibility of data presented. Surveyed donors at headquarters considered ADB to be adequate overall but held positive views about the Bank’s contributions to infrastructure development and intraregional co-operation, both of which received strong ratings.

• Evidence of extent of contribution to country-level goals and priorities, including relevant MDGs: ADB’s country-level reporting documents provided mixed evidence on its contributions to country-level goals and priorities in the three countries reviewed. Some reports provided solid evidence of the Bank’s contributions (e.g. Viet Nam), while others did not (e.g. Pakistan[1]). The Bank has committed to strengthen the ‘quality at entry’ of country partnership strategies (CPS) and to improve quality of project completion reports. Surveyed stakeholders in the countries sampled generally gave ADB ratings of adequate for its contributions to country outcomes. Stakeholders in Viet Nam gave ADB positive marks for its contributions to the transport and water sectors.

• Relevance of objectives and programme of work to country-level stakeholders: Surveyed stakeholders in-country considered ADB to be strong overall for responding to the key development priorities of DMCs. They perceived the Bank to be adequate in providing innovative solutions to help address key development challenges and in adapting its work to the changing needs of client countries.

Changes since the 2010 MOPAN assessment

Survey data suggest that the Bank’s performance has improved overall in the past three years. In strategic management, there is increased recognition of the Bank’s focus on results at both corporate and country levels, both of which were considered adequate in 2010 but strong in 2013. Stakeholder perceptions of gender mainstreaming also improved from inadequate in 2010 to adequate in 2013.In operational management, there was a slight decline in stakeholder perceptions of ADB’s allocation of resources; survey respondents rated the Bank as strong in 2010 for allocating resources that follow established criteria, but adequate in 2013. The ratings of the Bank’s use of performance information to plan new interventions improved from adequate in 2010 to strong in 2013. Survey ratings remained steady in most aspects of ADB relationship management, with some improvements in ratings for funding proposals that are developed with national governments or clients and ADB contributions to policy dialogue. Stakeholder ratings also improved with regard to the time it takes to complete Bank procedures, which was seen as inadequate in 2010 but adequate in 2013. In knowledge management, stakeholder perceptions of ADB’s reporting practices declined in two areas – reporting on outcomes achieved and reporting on Paris Declaration commitments – areas rated strong in 2010 but adequate in 2013.

Conclusions on ADB’s organisational effectiveness

As in the 2010 MOPAN assessment, ADB’s strategic management is seen as an area of strength. The Bank’s implementation of Managing for Development Results is widely recognised, and its efforts in this area are seen to be backed by strong leadership from senior management.

ADB has continued to strengthen its implementation of Strategy 2020, with an articulation of core areas of specialisation and key drivers of change that are aligned with its mandate. MOPAN donors at headquarters recognised the Bank’s continued progress in implementing its results agenda and the strong leadership provided by senior management. The positive perceptions of ADB’s results frameworks (both corporate and country level) were confirmed by the review of documents. The most recent revision to the corporate results framework (including the addition of outcome-oriented indicators in level 2) demon- strates an honest intellectual effort to include outcomes in the framework and, in turn, improve corporate reporting. The format of the country partnership strategy (CPS) has improved over the years, and survey respondents accorded strong ratings for the Bank’s country and sector level frameworks. In addition, CPSs are aligned with national development strategies and surveyed stakeholders in-country commended the Bank for its ability to respond to key development priorities of DMCs.

ADB’s approach to promoting regional co-operation and integration comes out strongly in the assessment – it was the cross-cutting theme most highly rated by survey respondents and in which ADB was perceived to make important contributions towards results on the ground.

ADB continues to be valued for being a regionally-owned Bank and for playing a critical role in regional co-operation and integration (RCI). This was evident in survey respondents’ perceptions of ADB’s greatest strengths, their views on the Bank’s promotion of this theme through its strategy and operations, and its contributions to results on co-operation and integration in the region. The Bank’s policies and strategies also provide solid direction for achieving results in this area. Interestingly, however, the reporting on the Bank’s contribution to RCI is limited. This should change as the Bank begins to report on its revised results framework, which has new indicators on RCI. Greater evidence of contribution to regional dimensions may be highlighted in IED’s evaluation expected in 2014.

ADB’s progress on other cross-cutting themes was considered generally adequate, with a few areas noted for discussion with MOPAN members and other shareholders (such as its approach to human rights).

Survey respondents and the document review noted that ADB has made improvements in mainstreaming gender equality in its operations. ADB clients were more positive than other respondent groups about ADB’s work in environment and good governance, which they rated as strong. In the area of climate change, which is key in Strategy 2020, gaps were noted in the frameworks for monitoring results, based on the documents reviewed. The Bank reports that it initiated work on this issue in a concerted manner with other MDBs in 2012.

The issue of human rights, which MOPAN included as a cross-cutting issue for all agencies assessed in 2013, emerged as an important point for discussion between MOPAN donors, other shareholders, and the Bank. While human rights is not an identified priority of the Bank, the review of select ADB policy documents (on gender, education and water) indicates that the Bank either implicitly or explicitly recognises the principles of human rights, has adopted adequate operational policies to ensure compliance with safeguards, and has institutionalised a mechanism for accountability to persons affected by ADB projects.

In its operations, ADB has continued to promote decentralisation to Resident Missions and has made progress in this area. Survey respondents would like to see even more decentralised and streamlined administrative procedures.

ADB has delegated an increasing amount of responsibility to the country level since 2000, including the ability to approve minor changes to projects, emergency projects, and technical assistance projects under USD 225,000. Based on data provided for this assessment, there appears to have been further progress in the last few years in the Bank’s decentralisation agenda. An IED evaluation planned for 2014 should provide a more in-depth assessment of the improvements and any shortcomings. This will be important to follow given that country-level respondents highlighted the Bank’s centralised procedures as one of its weaknesses.

In general, ADB has strong operational policies, especially with regard to financial accountability. It still faces challenges in results-based budgeting and in the area of human resources. ADB has human resource policies and systems in place, but donor perceptions point to the need to monitor and enhance communications about how those policies and practices are being implemented.

ADB is considered an organisation with sound financial accountability mechanisms in place (audit, anti- corruption and risk management). This was confirmed by the views of donor respondents who perceived its internal audit processes to be strong. The ratings on ADB’s human resource policies and systems were adequate or higher, based on the review of documents. The organisation has a results-focused performance assessment system to assess staff performance (with the exception of Vice Presidents who are appointed by the Board of Directors and for whom separate rules apply) and inform promotions and other key human resource decisions. However, MOPAN donors either do not know how promotions take place or express concern about the extent to which staff promotion is transparent or based on merit. This suggests that the implementation of the system may either be inconsistent or not well-known among MOPAN donors.

Conclusions on evidence of ADB’s development results and relevance to stakeholders

At the corporate level, the Bank’s contributions to results on the ground are most evident in infrastructure. The Development Effectiveness Review report (DEfR) provided solid data on the achievement of outputs in select operational areas (infrastructure, education, finance) and indicated improvement from year to year. To date, however, ADB’s reports and other documents provide little evidence of higher-level results. This should change when the Bank begins reporting on its revised Results Framework, which includes more outcome-oriented indicators.

ADB’s contributions to infrastructure development were noted by surveyed donors at headquarters and the document review. Its contribution to intraregional co-operation was also rated positively by survey respondents but reported evidence of this could not be found in the review of the Bank’s reports and evaluations.

ADB provides ample evidence on the achievement of outputs in level 2 in the DEfR. It is important to note, however, that the type of outputs that the Bank supports (e.g. in infrastructure) can often take many years to complete. The Bank is also making improvements in data quality and developing approaches to better capture how projects are contributing to sector outcomes. The recent changes in the Results Framework will give the Bank more outcome-oriented indicators to track and should improve future corporate reporting on results. However, current (2012) reporting on results is still lacking evidence of the Bank’s contributions to outcomes and there are only a few evaluation studies on core sector areas (often with a sub-regional scope) to confirm that outputs are leading to outcomes in the past years (since 2008).

In exploring contributions to development results at the country level, the MOPAN sample of three countries proved to be too limited to make a judgement on the extent of progress and the nature of the evidence, especially given that one of the countries - Pakistan - has undergone a portfolio restructuring in recent years.

ADB’s reporting on results achievement in countries is spread across different types of reports, but the key documents are the CPS final review (CPSFR) validation report, the country assistance performance evaluation (CAPE), and the sector assistance performance evaluation (SAPE). The SAPE is conducted at the final evaluation stage of a country partnership strategy (CPS).

Survey respondents in the three countries sampled provided adequate to strong ratings on ADB’s contributions to key outcomes that had been specified in the CPS. A review of country-level reporting, however, provided a mixed picture; the evidence of contributions to outcomes was more robust in Viet Nam than in Pakistan, for example. It is important to note that the review focused on completed CPS and thus considered strategy documents and results frameworks that do not incorporate the most recent improvements in the CPS.

Across all three countries reviewed, there is a lack of reliable baseline data, which makes it difficult for the Bank to set realistic and achievable targets and also to gauge the extent of ADB’s contribution to country- level results. The Bank recognises this as an area that needs improvement and has committed to strengthening national/DMCs monitoring and reporting capacities as part of its MfDR agenda. The Bank is also committed to enhance its quality control of project completion reports (PCRs) and of country partnership strategy (CPS) results frameworks. Improvement in these areas may help to improve the credibility and reliability of data reported at the country/project levels.

Surveyed stakeholders considered ADB’s ability to respond to key development priorities as one of the strengths of the organisation. They considered ADB adequate in the other areas of relevance assessed: providing innovative solutions to development challenges and adapting its work to changing conditions at country levels.

Overall MOPAN ratings of ADB

The two charts below show the ratings on the key performance indicators that MOPAN used to assess ADB in 2013. The first chart shows the ratings on 21 indicators designed to measure organisational effectiveness (practices and systems), and the second shows ratings on the three indicators designed to assess ADB measurement and reporting on development results and relevance to stakeholders.

Organisational effectiveness– overall ratings

Evidence of ADB’s contribution to development results and relevance to stakeholders – overall ratings

1. Introduction

This report presents the results of an assessment of the Asian Development Bank (ADB) that was conducted in 2013 by the Multilateral Organisation Performance Assessment Network (MOPAN).

Background

MOPAN was established in 2002 in response to international forums on aid effectiveness and calls for greater donor harmonisation and co-ordination. The purpose of the network is to share information and experience in assessing the performance of multilateral organisations. MOPAN supports the commitments adopted by the international community to improve the impact and effectiveness of aid as reflected in the Paris Declaration on Aid Effectiveness, the Accra Agenda for Action, and the Busan High Level Forum. MOPAN’s processes and instruments embody the principles of local ownership, alignment and harmonisation of practices, and results-based management (RBM).

MOPAN provides a joint approach (known as the Common Approach) to assess the organisational effectiveness of multilateral organisations and their measurement of and reporting on development results. The approach was derived from existing bilateral assessment tools and complements and draws on other assessment processes for development organisations – such as the bi-annual Survey on Monitoring the Paris Declaration on Aid Effectiveness and annual reports of the Common Performance Assessment System (COMPAS) published by the multilateral development banks. In the long term, MOPAN hopes that this approach will replace or reduce the need for other assessment approaches by bilateral donors.

MOPAN assesses four dimensions of organisational effectiveness, and evidence of contribution to development and/or humanitarian results

MOPAN has defined organisational effectiveness as the extent to which a multilateral organisation is organised to contribute to development and/or humanitarian results in the countries or territories where it operates. Based on a survey of stakeholders, a review of documents and interviews with multilateral organisation staff, MOPAN assessments provide a snapshot of a multilateral organisation’s effectiveness (see Chapter 3) in four dimensions:

• Developing strategies and plans that reflect good practices in managing for development results (strategic management)

• Managing operations by results to support accountability for results and the use of information on performance (operational management)

• Engaging in relationships with direct partners/clients and donors at the country level in ways that contribute to aid effectiveness and that are aligned with the principles of the Paris Declaration and subsequent related agreements on aid effectiveness (relationship management)

• Developing reporting mechanisms and learning strategies that facilitate the sharing of knowledge and information inside the organisation and with the development community (knowledge management).

In 2012, MOPAN piloted a new component to assess a multilateral organisation’s reported contributions to development results. This component which has become an integral component of all assessments in 2013, examines three areas of performance: evidence of the extent of the progress towards its institutional/organisation-wide results, evidence of contributions to country-level goals and priorities (including relevant millennium development goals (MDGs)), and stakeholder perceptions of the relevance of its objectives and programme of work. See Chapter 4.

Purpose of MOPAN assessments

MOPAN assessments are intended to:

• Generate relevant, credible and robust information MOPAN members can use to meet their domestic accountability requirements and fulfil their responsibilities and obligations as bilateral donors

• Provide an evidence base for MOPAN members, multilateral organisations and their partners/clients to discuss organisational effectiveness and reporting on development and/or humanitarian results

• Support dialogue between individual MOPAN members, multilateral organisations and their partners/clients to build understanding and improve organisational performance and results over time at both country and headquarters level.

The MOPAN methodology is evolving in response to what is being learned from year to year, and to accommodate multilateral organisations with different mandates. For example, in 2011, MOPAN began adapting the indicators and approach for the review of a global fund and organisations with significant humanitarian programming. In 2013, MOPAN is using a modified framework to assess the World Food Programme (WFP), an organisation with a predominantly humanitarian mandate.

1.1 PROFILE OF ADB

Mission and Mandate

The Asian Development Bank (ADB) is an international development finance institution that was founded in 1966. Since its establishment, the Bank has grown from 31 to 67 member countries, with 48 members from the Asia Pacific region and 19 non-regional members.

ADB, headquartered in Manila, the Philippines has 28 resident missions and three representative offices in Tokyo, Frankfurt and Washington, D.C.

The Bank’s mission is to “alleviate poverty and help create a world in which everyone can share in the benefits of sustained and inclusive growth.” ADB provides loans, technical assistance and grants for development projects that “create an economic and development impact” and supports developing member countries (DMCs) in their efforts to tackle poverty.

Structure and Governance

At the top of ADB’s organisational structure is the Board of Governors, comprising representatives from each of its 67 member countries. The Board of Governors meets annually and elects and delegates day- to-day responsibilities to a 12 member Board of Directors (eight from member countries in the Asia Pacific region, and four from countries outside the region).

The Board of Directors conducts its full-time duties from ADB’s headquarters in Manila and is primarily responsible for supervising ADB’s financial statements, approving its administrative budget, as well as reviewing and approving all policy documents, all loan, equity, and technical assistance operations. Two ADB offices report directly to the Board of Directors: The Office of the Compliance Review Panel (OCRP) and the Independent Evaluation Department (IED).

ADB’s management consists of the President, six Vice Presidents and the Managing Director General, who oversee the work of ADB’s operational, administrative, and knowledge departments.

As of December 2012, ADB had 3,045 employees from 61 of its 67 member countries – a workforce that includes specialists such as economists, sociologists, engineers, gender experts and environmental scientists.

Strategy

In 2008, ADB adopted a Long-term Strategic Framework – Strategy 2020 – to guide its operations and support its mission and overall vision of an Asia and Pacific region free of poverty. In order to achieve this, ADB commits to supporting three complementary development agendas of region: inclusive growth, environmentally sustainable growth, and regional integration.

In accordance with Strategy 2020, the Bank currently focuses on five core areas of specialisation that best support its agenda and that reflect the needs of DMCs and ADB’s comparative strengths, i.e. infrastructure development, environment (including climate change), regional co-operation and integration, finance sector development, and education. The Bank has further identified five key drivers of change to be integrated and mainstreamed in all of its operations: private sector development and private sector operations, good governance and capacity development, gender equity, knowledge solutions, and partnerships.

Finances and Services

ADB funding derives from multiple sources, including bond issues, member contributions, earnings from lending, and repayment of loans. ADB offers several types of financial products, which include loans, grants, technical assistance, guarantees and debt management products. The latter are financed from ordinary capital resources (OCR) and various Special Funds, such as the Asian Development Fund, technical Assistance Special Fund, and Japan Special Fund.

In 2012, ADB reported that its operations totalled $21.57 billion, of which $13.3 billion was financed by ADB (Ordinary Capital Resources and Special Funds) and $8.27 billion by co-financing partners.

Change and reforms

Mid-Term Review of Strategy 2020: In May 2013, ADB’s newly appointed president, Takehiko Nakao, commissioned a mid-term review of Strategy 2020 to assess the progress of implementation and ensure continuing relevance of the Bank’s work to the needs of developing member countries.

ADB results framework: By defining itself as a results-driven organisation, ADB adopted a corporate results framework in 2008 and introduced the Development Effectiveness Review as a corporate reporting tool to monitor its progress towards the goals set in Strategy 2020. Following refinements in 2010 and a comprehensive review in 2012, ADB approved a revised corporate framework in January 2013. The new framework introduces changes in the structure of the framework, proposes refinements to the indicators used to assess the Bank’s performance, and sets new intermediate targets for the years 2013-2016.

Country and sector results frameworks: Following a comprehensive review of the ADB Country Partnership Strategy process in 2009, ADB revamped the format of its country and sector results frameworks and developed guidance tools to improve quality.

Sector/thematic operational plans: In 2008, the Bank introduced a number of operational plans to support Strategy 2020 priorities and guide the Bank’s operations in select sector and thematic areas. In response to an independent external evaluation conducted by IED in 2011, the Bank committed to increase the results orientation of those plans in 2013.

Aid effectiveness: ADB is a signatory of the Paris Declaration on Aid Effectiveness and the Accra Agenda for Action. It has committed to the five core principles of the Paris Declaration (ownership, alignment, harmonisation, results, and mutual accountability) and has sought to mainstream these in its operations. Until 2011, ADB’s commitment to aid effectiveness was demonstrated through its annual monitoring of and reporting on 9 of the 12 indicators. ADB’s renewed commitment to aid effectiveness is highlighted in the revised 2013 corporate results framework which allows room for the Busan indicators.

Review of the 2003 Accountability Mechanism policy: In 2010, ADB undertook an extensive review of this policy to improve complaints procedures. The revised policy, adopted in 2012, is expected to provide direct access and allow persons adversely affected by ADB projects to file complaints.

For additional information, please consult the ADB website: www.adb.org.

1.2 PREVIOUS ASSESSMENTS[2]

Since its establishment in 2003, MOPAN has conducted four assessments of the Asian Development Bank (2003, 2006, 2010, and 2013). Although the MOPAN methodology has changed over time, findings from previous MOPAN surveys can provide some insight into the evolution of perceptions of the Bank’s surveyed stakeholders.

Changes since 2010

The 2010 MOPAN survey collected the perspectives of MOPAN members and views of national partners in four countries (Afghanistan, Indonesia, Sri Lanka and Viet Nam). Overall, surveyed stakeholders perceived the Bank’s performance as adequate in all areas of strategic, operational, and relationship. In particular, they commended ADB’s efforts to promote regional co-operation, its allocation of concessional funding, and audit processes. In knowledge management, respondents held positive views about ADB’s monitoring and reporting of results (including the Paris Declarations). Surveyed stakeholders also highlighted several areas that they perceived as needing improvement. These included the Bank’s mainstreaming of gender equality, its management of human resources, as well as the length of time it takes to complete procedures which they perceived as too lengthy and thus negatively affecting implementation.

The 2013 survey, carried out in three countries (Indonesia, Pakistan and Viet Nam), illustrates an improving trend in seven key performance indicators (and relevant micro-indicators common to both the 2010 and 2013 MOPAN assessment) and a decreasing trend in two.

• Providing direction for results – Stakeholder perceptions of the Bank’s efforts in ensuring the application of results management across the organisation and instilling a culture that focuses on clients’ needs improved from adequate in 2010 to strong in 2013.

• Focus on cross-cutting thematic areas – Stakeholder perceptions of gender mainstreaming improved from inadequate in 2010 to adequate in 2013.

• Country focus on results – In 2010, survey respondents rated the Bank as adequate overall for its focus on results at the country level. In 2013, they gave the Bank a rating of strong due to marked improvement in four of the five areas examined.

• Resource allocation decisions – There was a slight decline in stakeholder perceptions of whether the Bank’s allocation of resources follows its established criteria. Survey respondents rated the Bank as strong in 2010 but adequate in 2013.

• Using performance information – Survey responses suggest that the Bank’s performance has improved in using performance information to plan new interventions. This area which was rated adequate in 2010 was perceived as strong in 2013.

• Supporting national plans – There was improvement in survey respondent perceptions of the Bank’s performance in funding proposals that are fully designed and developed with national government or clients. They rated it adequate in 2010 and strong in 2013.

• Adjusting procedures – Surveyed stakeholders considered the time it takes to complete Bank procedures as inadequate in 2010 and adequate in 2013.

• Contributing to policy dialogue – Ratings of the Bank’s contributions to policy dialogue improved from adequate in 2010 to strong in 2013. This improvement was noted in both areas assessed – valuable inputs to policy dialogue and respect for clients’ views.

• Presenting performance information – Two areas were assessed by survey respondents in both 2010 and 2013: reporting on outcomes achieved and reporting on Paris Declaration commitments. Stakeholder ratings declined in both areas, which were rated strong in 2010 but adequate in 2013.

1.3 ADB SURVEY RESPONDENTS

ADB survey results reflect the views of 159 respondents who provided their views on ADB’s performance in the areas of organisational effectiveness and contribution to development results.[3]

Figure 1.1 | Number of survey respondents and total population for ADB by country and respondent group (n=159)

The distribution of responses on the six survey rating options plus ‘don’t know’ answered are summarised by respondent group across all survey questions in Figure 1.2.

Figure 1.2 | ADB – Distribution of responses (n=159) on all questions related to micro-indicators

While there were responses in all six possible choices, relatively few responses overall were at the ‘weak’ end of the scale. Close to a quarter of responses from donors in-country were ‘don’t know,’ which is slightly higher than the level of ‘don’t know’ responses provided by other groups.

2. MOPAN Methodology 2013

2.1 OVERVIEW

The detailed MOPAN 2013 methodology, which includes rating scales, data analysis criteria, and strengths and limitations of the methodology, can be found in Volume II, Appendix I. The following is a brief summary.

MOPAN assessments examine:

• Organisational effectiveness: organisational systems, practices, and behaviours that MOPAN believes are important for aid effectiveness and that are likely to contribute to development and/or humanitarian results at the country level; and

• Humanitarian and development results: the extent to which there is evidence of an organisation’s contributions to development and/or humanitarian results.

As the MOPAN methodology has been refined each year since 2003, comparisons of this year’s assessments and any previous MOPAN assessments of an organisation should take this into consideration.

Data collection methods and sources

Over the years, MOPAN developed a mixed methods approach to generate relevant and credible inform- ation that MOPAN members can use to meet their domestic accountability requirements and support dialogue with multilateral organisations that they are funding.

The MOPAN approach uses multiple data sources and data collection methods to triangulate and validate findings. This helps eliminate bias and detect errors or anomalies.

In 2013, the sources of data included surveys of the multilateral organisation’s stakeholders and of MOPAN donors at headquarters and country level and a review of documents prepared by the organisations assessed and from other sources.

Assessment of organisational effectiveness

In the organisational effectiveness component, MOPAN examines performance in four areas: strategic management, operational management, relationship management, and knowledge management. Within each performance area, effectiveness is described using key performance indicators (KPIs) that are measured through a series of micro-indicators (MIs) using data from the survey and document review.

In this component, survey respondent ratings are shown as mean scores and are presented alongside document review ratings based on criteria defined for each micro-indicator. Not all micro-indicators are assessed by both the survey and the document review. The charts show survey scores and document review scores for the relevant KPIs or MIs. The full list of micro-indicators comprising this component is provided in Volume II, Appendix V.

Assessment of development and/or humanitarian results

In the development results component, MOPAN does not assess an organisation’s results on the ground, but examines how it measures and reports on its contributions to development and/or humanitarian results through three key performance indicators:

• Evidence of the extent of the organisation’s progress towards its stated institutional/organisation-wide results (i.e. goals, objectives, outcomes)

• Evidence of the extent of the organisation’s contributions to country-level goals and priorities, including relevant millennium development goals (MDGs)

• Relevance of the organisation’s objectives and programme of work to stakeholders.

In this component, a “best fit approach” is used in determining the ratings for the first two KPIs above. One of four qualitative ratings (strong, adequate, inadequate, or weak) is assigned following an analysis of data from all sources and confirmed in a consensus-based consultation (with of institutional advisors, a panel of experts, and MOPAN members). The rating of the KPI on relevance is based on stakeholder surveys only.

2.2 DATA SOURCES AND RATINGS

Survey

MOPAN gathers stakeholder perceptions through a survey of MOPAN members (at headquarters and in- country) and other key stakeholders of the multilateral organisation. Donor respondents are chosen by MOPAN member countries; other respondents are identified by the multilateral organisation being assessed.

The survey questions relate to both organisational effectiveness and to the achievement of development and/or humanitarian results. Survey respondents are presented with statements and are asked to rate the organisation’s performance on a six-point scale where a rating of 1 is considered “very weak” up to a rating of 6 which is considered “very strong.” A mean score is calculated for each respondent group (e.g. donors at headquarters).

MOPAN aims to achieve a 70 per cent response rate from donors at headquarters and a 50 per cent response rate among respondents in each of the survey countries (i.e. donors in-country and other respondent groups such as direct partners/clients).

Document review

The document review considers four types of documents: multilateral organisation documents; internal and external reviews of the organisation’s performance; external assessments such as the Survey on Monitoring the Paris Declaration, the Common Performance Assessment (COMPAS) report, and previous MOPAN surveys; and evaluations, either internal or external, of the achievement of results at various levels.

Document review ratings are based on a set of criteria that MOPAN considers to represent good practice in each area. The criteria are based on existing standards and guidelines (for example, UNEG or OECD-DAC guidelines), on MOPAN identification of key aspects to consider, and on the input of subject-matter specialists. The rating for each micro-indicator depends on the number of criteria met by the organisation.

Interviews

Interviews are conducted with staff based at headquarters and country offices of multilateral organisations who are knowledgeable in areas that relate to the MOPAN assessment. Interview data are not coded or used as a formal data source but rather to help ensure that the Assessment Team has all appropriate and necessary documents to enhance the triangulation of data from various sources and provide contextual information to assist in the analysis of the key performance indicators.

2.3 STRENGTHS AND LIMITATIONS OF THE MOPAN COMMON APPROACH

MOPAN continues to improve methodology based on experience each year. The following strengths and limitations should be considered when reading MOPAN reports.

Strengths

• The MOPAN Common Approach is based on existing bilateral assessment tools with the intent to replace or reduce the need for other assessment approaches by bilateral donors.

• In line with donor commitments to aid effectiveness, it seeks perceptual information from different stakeholder groups.

• It uses multiple sources of data to increase the validity of the assessment, enhance analysis, and provide a basis for discussion of agency effectiveness.

• MOPAN reports are validated and reviewed by MOPAN members and by the multilateral organisation being assessed.

Limitations

• The countries selected for MOPAN assessments comprise only a small proportion of each institution’s operations, thus limiting generalisations.

• Although MOPAN uses recognised standards and criteria for good practice, such criteria do not exist for all indicators. Many document review criteria were developed by MOPAN; these are a work in progress and not definitive standards.

• Survey rating choices may not be used consistently by all respondents. Some respondents may tend to avoid extremes on a scale and respondents in some cultures may be unwilling to criticise or too eager to praise.

• The survey covers a broad range of issues and individual respondents may not have the knowledge to respond to all the questions relating to a given organisation.

3. Main Findings: ADB’s organisational effectiveness

3.1 INTRODUCTION

This chapter presents the findings of the 2013 MOPAN assessment of ADB’s organisational effectiveness, that is its practices and systems that support the achievement of results. Findings are based on respondent survey data and document review.

• Section 3.2 presents overall ratings on the performance of ADB and summarises respondent views on its primary strengths and areas for improvement;

• Section 3.3 provides findings on each of the four areas of performance (strategic, operational, relationship, and knowledge management).

3.2 OVERALL RATINGS

This section provides a summary of overall ratings. It includes: survey respondent ratings of overall organisational effectiveness, survey respondent views on strengths and areas for improvement, and survey and document review ratings for all key performance indicators.

Overall ratings of key performance indicators

Figure 3.1 below shows scores from the document review and the survey on key performance indicators (KPIs) in the MOPAN 2013 assessment of ADB’s practices and systems. The white bar presents the survey score, while the black square presents the document review score. For example, on the first indicator, “providing direction for results”, ADB received a score of 4.58 (strong) in the survey and a score of 5 (strong) in the document review.

In the overall ratings from the survey and document review, ADB was seen to perform adequately or better on the majority of key performance indicators. ADB received scores of adequate or better on all 19 KPIs assessed in the survey and scores of strong or better on 15 of the 18 KPIs assessed by the document review.[4]

The survey and document review ratings differed on eight KPIs – all of which were rated higher by the document review than by survey respondents. The reasons for these differences are discussed in the following sections.

Figure 3.1 | Overall ratings on key performance indicators of ADB’s organisational effectiveness (survey mean scores and document review ratings)

Survey ratings of ADB’s organisational effectiveness

MOPAN has defined “organisational effectiveness” as the extent to which a multilateral organisation is organised to support clients in producing and delivering expected results. Respondents were asked the question: “How would you rate the overall organisational effectiveness of ADB?” As shown in Figure 3.2, MOPAN donors at headquarters gave the largest proportion of ratings on the high end of the scale, with 59% rating ADB as effective or very effective.

Figure 3.2 | Overall ratings of ADB organisational effectiveness by respondent group (n=159)

Respondents’ views on ADB’s strengths and areas for improvement

The survey included two open-ended questions that asked respondents to identify ADB’s greatest strengths and areas of improvement. All 159 respondents[5] provided comments on both questions.[6] Their comments are summarised below with an analysis of the level of homogeneity among respondent groups as well as areas where the respondent groups felt differently about certain issues.

Overall, survey respondents indicated that ADB’s greatest strengths lie in the expertise and quality of its staff and its relationships with clients. The Bank was also noted for its sound financial management and focus on results.

Among all respondent groups, 23 % of respondents (n= 37) commented that ADB’s greatest strength lies primarily in the quality of its staff, who were generally commended for their reliability and professionalism, as well as their technical and sectoral expertise; 19% (n=31) noted ADB’s relationship with client countries as one of its key strengths. Respondents often cited the Bank’s regional presence and close proximity to clients as key to understanding the needs of the region and facilitating dialogue.

Among MOPAN donors at headquarters, 32% (n=12) cited ADB’s focus on results (considered essential for organisational effectiveness) as its greatest strength, 22% (n=8) noted its specific commitment to infrastructure development, and 16% (n=6) noted its strategic focus in select sectors of relevance.

Among MOPAN donors in-country, 14% (n=5) viewed the Bank’s convening capabilities and co- ordinating role as a strength and another 14% (n=5) considered ADB’s sound financial management as one of its greatest strengths. This view was shared by 16% (n=14) of ADB clients who also considered ADB’s mandate/vision of a “region free of poverty” as a strength (9%).

Overall, survey respondents suggested that ADB should delegate more decision-making authority to resident missions, streamline administrative procedures to improve efficiency, and ensure greater consultation and interaction with national stakeholders.

Respondents in all groups indicated that ADB is still a relatively bureaucratic and centralised organ- isation. Taken together, the most frequently noted areas for improvement were related to practices that affect operations at the implementation level. Comments reflected the perception that ADB’s heavy bureaucratic measures hinder the effective- ness and efficiency of its operations, and that this is compounded by the Bank’s centralisation of project management and administration at headquarters.

The need for ADB to further decentralise its decision- making processes in order to grant resident missions more autonomy and allow them to operate more flexibly was noted by 29% of in-country donors (n=10), 16% of donors at headquarters(n=6), and 9% of ADB clients (n=8) commented on.

The need for greater interaction and wider consultation with national stakeholders (i.e. local communities and civil society organisations in particular) was noted by 15% of ADB clients (n=13) and 11% of MOPAN donors in-country (n=4).

Among ADB clients, 23% (n=20) commented that the area most in need of improvement was ADB managerial processes, particularly its procurement processes.

Among donors in-country, 20% (n=7) emphasised the need for better collaboration and co-ordination with other development agencies in-country.

Among donors at headquarters, 16% (n=6) suggested that ADB needs to improve its monitoring and evaluation processes and 16% (n=6) felt that the organisation should place greater strategic focus on its operational areas.

3.3 ADB’S PERFORMANCE IN STRATEGIC, OPERATIONAL, RELATIONSHIP, AND KNOWLEDGE MANAGEMENT

3.3.1 Overview

This section presents the results of the 2013 Common Approach assessment of ADB in four organisational performance areas: Strategic, Operational, Relationship, and Knowledge Management.

The following sections (3.3.2 to 3.3.5) provide the overall survey and document review ratings for the KPIs in each performance area, the mean scores by respondent group, and findings based on an analysis of survey and document review ratings in each performance area.

When there were notably divergent ratings between survey respondent groups or between the survey results and document review ratings, these are noted and the information gleaned from interviews with staff is integrated when it has a bearing on the analysis. Where statistically significant differences among categories of respondents were found, these differences are noted.

The survey data for each KPI and MI by performance area are presented in Volume II, Appendix V. The document review ratings are presented in Volume II, Appendix VI.

3.3.2 Strategic management

ADB has demonstrated increased commitment to instilling a results-oriented culture at both the organisational and country level. The document review and survey respondents rated the Bank as strong in most areas of strategic management, but indicated room for improvement in linking levels of results.

Figure 3.3 shows the overall survey and document review ratings for the five KPIs in the strategic management performance area.

Survey respondents rated the Bank as strong on almost all aspects of strategic management. They commended the Bank’s efforts to promote regional co-operation and integration but gave it a rating of adequate on other cross-cutting thematic areas.

The document review rated the Bank as strong overall in this performance area, noting, in particular, the considerable efforts made in recent years to strengthen MfDR practices and systems at both the organisational and country level. It noted room for improvement in linking outputs to expected outcomes in its results.

Figure 3.3 | Performance area I: Strategic management, survey and document review ratings

Figure 3.4 shows the mean scores for the five KPIs for all survey respondents, and by category of respondent.

Figure 3.4 | Performance area I: Strategic management, mean scores by respondent group

KPI 1: Providing direction for results

Finding 1: Survey respondents rated the ADB as strong overall for providing clear direction for results. They particularly commended ADB’s efforts to ensure the application of results management across the organisation and the document review supported this view.

Overall, survey respondents rated ADB as strong on this KPI.

The document review rated the Bank as strong for its organisational policy on results management and noted the commendable efforts made to mainstream MfDR into all ADB operations. In this regard, documentary evidence indicated that the Bank has developed several tools and guidelines to disseminate knowledge on MfDR across the organisation and amongst clients.

Figure 3.5 | KPI 1: Providing direction for results, ratings of micro-indicators

MI 1.1 – Value system supports results-orientation and client focus

Survey respondents were asked whether ADB’s institutional culture reinforces a focus on results, and whether it is client focused. The majority (82%) rated ADB as adequate or above. Donors at headquarters rated the organisation more positively than other respondent groups on the question pertaining to ADB’s focus on client needs, and the difference was statistically significant.[7]

MI 1.2 – Leadership on results management

MOPAN donors at headquarters were the only respondent group asked about the extent to which ADB’s senior management demonstrates leadership on results management. More than half (62%) rated ADB as strong or above.

MI 1.3 – Organisational policy on results management

When asked whether ADB ensures the application of results management across the organisation, the majority of donors at headquarters (89%) rated the organisation as adequate or above.

The document review rated ADB as strong for promoting an organisation-wide policy on results manage- ment and noted the significant efforts made in the past decade to mainstream Managing for Development Results (MfDR) into all of the Bank’s operations.

Since the 2010 MOPAN assessment, ADB has undertaken several processes to strengthen MfDR practices and systems at both the organisational and country level. Among these are: the refinement of its corporate results framework in 2010; the comprehensive review of its framework in 2012, which led to the adoption of a new results framework in 2013; and the systematic use of the Development Effectiveness Review report (DEfR) as a corporate planning and decision-making tool (see KPI 3 below). ADB’s efforts to strengthen MfDR capacities within the organisation and in developing member countries (DMCs) were also noted. The Bank reports that it has organised several training activities and developed various tools and guidelines aimed at disseminating knowledge on MfDR across the organisation and in client countries.

Despite the considerable progress made in this area and the positive rating, the review also took note of the recommendations of the “Special Evaluation Study on Managing for Development Results” conducted by the Independent Evaluation Department (IED) in 2011, which indicated there is still some work to be done to further instil a MfDR culture within the organisation and strengthen MfDR capacities in DMCs.[8]

KPI 2: Corporate strategy and mandate

Finding 2: ADB received strong survey ratings for its articulation of a long-term strategy (Strategy 2020) that is based on a clear mandate. The document review took positive note of the Bank’s recent decision to review the implementation of Strategy 2020 so as to ensure that the Bank’s strategy continues to meet the needs of DMCs.

The Bank received strong ratings on this KPI from both the document review and surveyed MOPAN donors at headquarters.

Figure 3.6 | KPI 2: Corporate strategy and mandate, ratings of micro-indicators

MI 2.1 – Corporate strategy based on clear mandate

Donors at headquarters were asked two questions on this MI: whether ADB has a clear mandate, and whether ADB’s organisation-wide strategy (i.e. Long-Term Strategic Framework 2008–2020 or Strategy 2020) is aligned with the mandate. The majority of respondents rated the organisation as strong or very strong on both questions (78% and 73% respectively).

The Bank received a rating of very strong on this MI in the document review. Its original mandate is outlined in the 1965 articles of Agreement Establishing the Asian Development Bank. The mandate is complemented by the ADB’s Long-Term Strategic Framework 2008–2020 (i.e. Strategy 2020), which articulates strategic goals, core areas of specialisation, and key drivers of change. The review of documents indicated that relevant revisions have been made to ADB’s Articles of Agreement to ensure continuing relevance of its work. This is further reflected in the Bank’s recent commitment to conduct a review of Strategy 2020 to ensure its continuing relevance and alignment with the changing needs of DMCs.

KPI 3: Corporate focus on results

Finding 3: ADB was rated as strong overall for its focus on results. The Bank has continuously improved the quality of the results framework that guide the implementation of Strategy 2020, with most recent changes adopted in 2013. However, linkages between different levels of results could be strengthened.

All MIs under this KPI were assessed by document review only and were based on an analysis of ADB’s 2008 results framework and the recently approved 2013 results framework.

The document review rated ADB’s performance as strong overall and noted the Bank’s commitment to institutionalising MfDR practices within the organisation. It particularly noted ADB’s efforts to improve the assessment and reporting on corporate performance through continuous refinements of its results framework.

In defining itself as a “results driven” organisation, the Bank adopted a corporate results framework in 2008 to guide the implementation of Strategy 2020 and introduced the Development Effectiveness Review as a corporate reporting tool used to monitor its progress towards the goals set in Strategy 2020. The Bank’s results framework, based on a four-year time frame (2009-2012), was organised in four levels and included four tiers of performance indicators to track development progress and measure the effectiveness of the Bank’s operations. In 2010 the Bank refined the indicators, drawing on lessons learned in previous years. The most recent improvements were made early in 2013 with the approval of a revised corporate framework that not only introduces changes to the structure of the framework, but also proposes further refinements to the indicators, and sets new intermediate targets for 2013-2016.

Figure 3.7 | KPI 3: Corporate focus on results, ratings of micro-indicators

MI 3.1 – Plans and strategies contain results frameworks

ADB was rated as strong on this MI based on a review of the 2013 revised results framework approved in January 2013.

ADB’s results framework contains four levels of results that are divided into two sections: section I includes results that measure development progress in Asia and the Pacific (level 1); and section II specifies three levels of results that aim to measure the development effectiveness of ADB’s operations (level 2) and assess ADB’s operational and organisational effectiveness (level 3 and 4 respectively).

ADB’s results framework includes indicators for both development results (i.e. level 1 and 2) and management results (i.e. levels 3 and 4), as well as statements of expected outputs and outcomes that are appropriate to their results level. The recently revised results framework also includes new outcome indicators in level 2.

MI 3.2 – Results frameworks link outputs to final outcomes/impacts

The document review, which assessed the results chain in ADB’s results framework, rated the Bank as adequate.

ADB, like other multilateral development banks, faces a challenge in ensuring a clear and logical progression from lower-level results to results at the impact level. The 2011 “Special Evaluation Study on MfDR” indicated that this was an area that required improvement; a review of the 2013 results framework shows progress in this area.

The 2012 Review of the ADB Results Framework noted that while the former results framework from 2008 that was revised in 2010 had served ADB well, there were some challenges to be addressed, such as: the need to clarify the causal links between the levels of results and provide a consolidated approach to assessing output and outcome results under level 2. There is some indication that these challenges are being addressed in the revised framework. For example, the 2013 revised results framework clarifies indicators and results so that linkages are more easily made between the different levels. The fundamental assumptions underlying the framework remain the same: a well-managed organisation (level 4) will enhance the effectiveness of the Bank’s operations (level 3) which in turn will increase the Bank’s contribution to country level results (level 2). Causal links cannot be made between levels 1 and 2.

The document review noted some inconsistencies in the sampling methods used to measure core sector output and outcome results.[9] However, it commended the Bank’s efforts to improve the measurement of outcomes by integrating outcome-oriented indicators under level 2 of its results framework.

MI 3.3 – Plans and strategies contain performance indicators

The document review rated ADB as strong for the inclusion of performance indicators in its results framework.

Organisation-wide results presented in the revised 2013 results framework include performance indicators that are adequate, relevant to the results they are associated with, and clear on what is to be measured. In both the development and management results frameworks, performance indicators include targets with clear dates for achievement.

The document review noted the Bank’s recent efforts to introduce new outcome and sex-disaggregated indicators in the revised framework, which should improve ADB’s ability to report on its contributions to development results (i.e. level 2).

KPI 4: Focus on cross-cutting thematic areas

Finding 3: Of the seven thematic areas assessed, survey respondents perceived the Bank as strong in promoting regional co-operation and integration and adequate in all other areas. The Bank received mixed ratings from the document review which indicated the need for better monitoring of results in climate change but noted the Bank’s ongoing initiatives to address this issue.

The assessment of this KPI examined the extent to which ADB has maintained a focus on and/or mainstreamed seven thematic areas that were identified as priorities in the Bank’s strategic framework or of particular importance to MOPAN members. These included thematic areas identified by the Bank as “key drivers of change” or “core areas of specialisation” in Strategy 2020, as well as areas identified as priorities by MOPAN members (i.e. gender equity, good governance, private sector development, environment [including climate change], regional co-operation and integration, and human rights). It should be noted that the Bank does not identify human rights as a priority or cross-cutting theme in either its strategic framework or mandate and this has been taken into account in the assessment.

Overall, all respondents surveyed rated ADB as strong for promoting regional co-operation and integration and as adequate for mainstreaming or placing sufficient focus on the other themes.

The document review rated ADB as adequate or better in all thematic areas but noted room for improvement in the integration and monitoring of results in climate change. It also noted that the Bank has recently developed several operational plans to institutionalise the focus on results in most of the core sector/thematic areas identified in Strategy 2020. Further, three of the Bank’s thematic areas (climate change, governance, and regional co-operation and integration) were being evaluated by IED at the time of the MOPAN assessment. An evaluation of the Bank’s Safeguards Policy Statements is also on-going and expected to be completed in 2014.

Figure 3.8 | KPI 4: Focus on cross-cutting thematic areas, ratings of micro-indicators

MI 4.1 – Gender equality

When asked whether ADB sufficiently mainstreams gender equality in its operations, the majority of survey respondents (70%) rated ADB as adequate or above. Clients were more positive than other respondent groups surveyed with the majority (55%) rating the organisation as strong or above. These differences are statistically significant.

The document review rated the Bank as strong for its mainstreaming of gender equality. In recent years, ADB has placed greater emphasis on gender equality. This has translated into increased budgetary allocations to develop staff skills and increased awareness with respect to gender mainstreaming. An examination of the Development Effectiveness Reviews (DEfRs) shows that the Bank has made significant progress towards its organisational gender targets. Since the 2010 MOPAN assessment, ADB has undertaken numerous initiatives, including: the review and integration of new gender indicators and gender disaggregated data in its revised corporate results framework; the development of the 2013-2020 Gender Operational Plan to guide the Bank’s work and promote its gender equality agenda; and, the recruitment of gender specialists to strengthen in-house capacities and gender expertise in ADB’s regional departments and Resident Missions. The Bank has also indicated that it intends to increase the percentage of completed sovereign operations delivering intended gender equality results to 70% by 2016 (during the period 2010-2012, this represented 54% of ADB projects and 53% of ADF projects).

While the document review rated the Bank as strong in mainstreaming gender equality, it also noted the Bank’s comments in the new Gender Equality and Women’s Empowerment Operational Plan, 2013-2020 (2013), which highlighted the need to ensure better monitoring of gender results and increase the integration of gender issues into country and project planning documents (i.e. country partnership strategies and project-specific gender action plans [GAPs]).

MI 4.2 – Environment

Overall, 81% of survey respondents rated ADB as adequate or above for mainstreaming environment in its operations. Clients were more positive than other respondent groups, with the majority (62%) rating the organisation as strong or above. The differences were statistically significant.

The document review rated ADB as very strong for the mainstreaming of environment into its operations. The review of documents and interviews with Bank staff indicate that the Bank has put into place several systems and developed various tools to enable the integration and mainstreaming of environmental considerations in the Bank’s operations. Mainstreaming is encouraged at all stages of a project cycle but particularly during the appraisal, design and implementation stages of the Bank’s projects so as to avoid adverse impacts and ensure adequate mitigation measures on the environment and project-affected people. According to the Bank’s 2009 Safeguard Policy Statement (SPS), all projects with potential environmental effects are subject to an Initial Environmental Examination (IEE) or an Environmental Impact Assessment (EIA). The Bank has focused on developing and strengthening clients’ capacities to ensure compliance with the safeguards policies and its own internal capacities to guarantee their enforcement. The Bank’s commitment to the environment is also noted in the increased number of safeguard specialists at Headquarters and, more importantly, in Resident Missions. Its commitment is further noted through its increasing levels of financing of i) sustainable infrastructure projects that contribute to environmentally sustainable, low carbon and climate resilient development; (ii) investments in natural capital for the protection and sustainable management of large scale ecosystems (e.g. the Greater Mekong Subregion, the Heart of Borneo, and the Coral Triangle);[10] and (iii) other regional and sub-regional programmes funded by the Poverty and Environment, the Clean Air initiative for Asian Cities, the Cities Development Initiative for Asia, and the Sustainable Transport Initiative, among others. The document review also noted the Bank’s proactive agenda to promote green growth, which is reflected in the Environment Operational Directions 2013-2020 adopted in 2013.

MI 4.3 – Climate change

The majority of survey respondents (78%) rated ADB as adequate or above for its focus on climate change.

The document review rated the Bank’s performance as adequate and moving in the right direction to improve its measurement of climate change results.

Climate change is addressed as a core area of specialisation of the Bank’s environmental sustainability agenda. ADB’s 2010 climate change strategy highlights five key priorities/actions: (i) expanding the use of clean energy; (ii) encouraging sustainable transport and urban development; (iii) managing land use and forests for carbon sequestration; (iv) promoting climate-resilient development; and (v) strengthening policies, governance, and capacities.[11]

ADB’s budget documents illustrate its increased financial commitment to climate change mitigation and adaptation activities, particularly through the financing of the Climate Change Fund (CCF) and co-financing of the Climate Investment Funds (CIF). ADB has also invested in the development of tools and guidance to help staff and clients identify climate change risks and adaptation measures in the key sector areas (i.e., transport, energy, agriculture, and water and sanitation) and ensure adequate climate-proofing of ADB’s investments. The Bank reports that it is pilot testing some technical tools to facilitate the identification and integration of climate risk mitigation and adaptation strategies in the project cycle.[12] In concerted efforts with other IFIs the ADB has also been engaged in a series of initiatives aimed to improve the monitoring of climate financing. Such initiatives have, for instance, led to the adoption of a joint MDB approach to tracking climate financing (including adaptation and mitigation financing) in 2012. Other initiatives have included the adoption of innovative approaches to supporting climate change mitigation activities in ADB’s developing member countries (DMCs).

The review of documents found recent changes made in the revised 2013 corporate results framework, which now incorporates an indicator aimed to enable a better assessment of ADB’s support to climate change mitigation and adaptation measures.[13] Moreover, it noted the development of an operational plan that aims to guide the Bank’s work and is expected to lead to better reporting on results in this area.[14]

At the time of this assessment the review of the climate change strategy was underway. It is expected that the results and recommendations emanating from this review, along with those emerging from the mid- term review of Strategy 2020 (under the thematic area of environment), will provide a greater indication of the Bank’s performance in this area.

MI 4.4 – Private sector development

When asked whether ADB sufficiently focuses on private sector development, the majority of survey respondents (74%) rated the organisation as adequate or above. The document review rated the Bank strong on this MI.

Under the private sector development (PSD) umbrella, ADB generally focuses on initiatives that seek “to improve the business environment, increase private investment in developing member countries (DMCs), develop public–private partnerships (PPPs), and undertake privatization and/or corporatization of publicly held entities.”[15] Similarly, the Bank’s private sector operations provide “direct financial and technical support to entities majority owned by the private sector, as well as to sub sovereign entities majority owned by the public sector that can contract and obtain financing independently.”[16] The Bank’s budget documents and annual reports indicate a steady increase in the financing and co-financing of private sector operations. The Bank has invested in technical assistance to strengthen the capacities of private sector clients, and, according to the 2012 Annual Report, the Bank had invested $5.05 million in technical assistance, mostly for project preparatory and capacity development technical assistance to non-sovereign clients. The document review also noted the Bank’s proactive agenda to promote public private partnerships as reflected in the 2012-2020 Operational Plan (2012).

While the Bank received a strong rating on this MI, the document review also took note of the recommendations of the Bank’s Report on the Development Effectiveness of Private Sector Operations (2011), which highlighted the need for a more streamlined process to monitoring and measuring the Bank’s performance in this area. The report indicated that while there are two mechanisms in place through which the Bank’s support to Private Sector Operation Department (PSOD) is assessed – project ratings in Extended Annual Review Reports (XARRS) and ratings from project validation reports (produced by IED) – the findings generated by these two mechanisms do not always concur.[17]

The Special Evaluation Study on ADB Assistance for Strengthening the Enabling Environment for Private Sector Development (currently on-going) should shed further light on the Bank’s contribution and support to private sector development in the region.[18]

MI 4.5 – Good governance

Overall, 77% of survey respondents rated ADB as adequate or above for promoting the principles of good governance in its operations. Clients were more positive than other respondent groups, with the majority (60%) rating ADB as strong or above. These differences were statistically significant.

The document review rated the Bank as very strong on this MI. Good governance is identified in the Bank’s corporate strategy as a key thematic priority area. ADB identifies four elements of good governance (i.e. transparency, accountability, predictability, and participation) as important building blocks to ensure that ADB resources are used in the most effective, efficient and accountable manner for development. The Bank has developed two action plans aimed at promoting and instilling good governance practices in ADB‘s operations and has developed guidelines that provide ADB staff with guidance on how to identify, manage and mitigate governance and corruption risks at project/programme, sector and country levels.

MI 4.6 – Regional co-operation and integration

When asked whether ADB sufficiently promotes regional co-operation and integration, 57% of survey respondents rated the Bank as strong or very strong.

The ADB was rated as strong on this MI by the document review. ADB’s Regional Co-operation and Integration Strategy (2006) identifies four areas necessary to promote regional co-operation and integration in Asia and the Pacific: i) increase cross-border physical connectivity among DMCs; ii) promote international trade and investment of DMCs with regional and non-regional economies; iii) develop regional financial markets and maintain regional macroeconomic and financial stability; and iv) help protect the regional environment and improve health and social conditions.[19] The implementation of the Regional Co-operation and Integration Strategy is guided by regional co-operation strategies developed by respective regional departments. The roles and responsibilities for each of the four areas noted above are assigned to the regional departments of the, the Regional and Sustainable Development Department (RSDD), and the Office of Regional Economic Integration (OREI).

The Bank reports on the results of its regional co-operation and integration efforts via the DEfR – specifically under Level 2 (which outlines ADB’s contribution to Strategy 2020 thematic areas) and Level 3 (which assesses the extent to which the Bank has financed the core areas of operations identified in its strategic plan).

MI 4.7 – Human Rights

When asked whether ADB is supportive of the human rights of project and programme beneficiaries, 53% of survey respondents overall rated ADB as adequate or above. While 72% of clients rated the Bank adequate or above on this MI, donors at headquarters had mixed views: 41% rated it adequate or above and 41% rated it inadequate or below. These differences were statistically significant. Among donors in- country, 46% rated it adequate or above and 28% answered “don’t know”.

The survey results signal a potential area for discussion with regard to donor expectations of ADB’s role in promoting human rights. Unlike UN organisations, ADB has no clear mandate for adopting human rights- based approaches, nor are human rights part of its mandate or strategic objectives under Strategy 2020. The assessment team took this into account.

The document review found that, although not always explicitly stated, ADB recognises the links between human rights and development and promotes the principles of human rights (i.e. equality and non- discrimination, participation and accountability) in its policies for gender, education, water, accountability mechanism, and safeguards.

For example, ADB’s Policy on Gender and Development (2003) puts forth human rights principles (equality, non-discrimination, participation and inclusion) when acknowledging the necessity, importance and benefits of including women and ensuring their participation in ADB’s development activities, and suggests that this will contribute to advancing the Bank’s agenda of achieving economic and sustainable development.[20]

The Bank’s Education Policy (2003) recognises basic education as a human right and seeks to ensure equity of access.[21]

ADB’s Water for All Policy (2001) puts forth the principles of equality and participation by indicating “that communities need to be empowered, educated, and involved in the process of water management, to provide more equitable access to water for the poor” and that “ADB will promote granting entitlements in water or usage rights”.[22]

In its social protection strategy (2011), the ADB commits to ensuring compliance with the internationally recognised Core Labor Standards (CLS) in the design and implementation of its projects. During project preparation, initial poverty and social analysis (IPSA) require that project officers assess compliance with the CLS or national labour laws. When potential labour concerns arise, one of the tasks of ADB’s project social development specialists is to identify opportunities to mitigate possible negative impacts of project and programme lending on labour.[23]

The Bank’s accountability mechanism and safeguards policies also indicate that ADB has practices, systems and procedures in place to ensure that its investments “do no harm”, and that if they do, risks are mitigated to the extent possible.

For example, following a review of its Safeguards Policy Statement (SPS), which was undertaken in consultation with non-governmental organisations and civil society organisations, ADB adopted a compliance review system in 2010 that requires all projects financed by the Bank to be classified according to their risk level at early stages (i.e. before appraisal and loan negotiations). Projects with a higher level of risk (i.e. in terms of adverse effects on the environment or on people) are more regularly monitored to reduce potential for negative effects on the project’s expected impact. Under the revised policy, the planning and implementation of the safeguards are the responsibility of the borrower/client and ADB’s role is to ensure that the requirements are fulfilled. ADB ensures that adequate disclosure of information, consultations, and participation of stakeholders have taken place and are in accordance with the Bank’s policy requirement. There is now a grievance redress mechanism at project levels which allows, in the case of non-compliance with the Bank’s policies, affected communities to have direct interaction with Bank officials and for corrective action to be taken.

The accountability mechanism policy of the Bank, also revised in 2010 in consultation with relevant stakeholders and approved in 2012, provides a mechanism for persons affected to file a complaint against an ADB-supported project as a last resort.

All of these elements taken together suggest that the Bank implicitly promotes the human rights principles of equality, participation and accountability and de facto, seeks to respect the human rights of persons and communities affected by ADB-assisted projects. Although ADB has not taken a proactive role in integrating human rights principles, its approach is considered adequate given its mandate, strategic direction approved by its Board of Directors, and current practice in multilateral development banks (MDBs).

KPI 5: Country focus on results

Finding 5: Survey respondents perceive ADB’s country partnership strategies (CPS) to have a strong focus on results. The document review concurred in its review of CPS’ completed since 2010.

ADB clients and MOPAN donors in-country rated ADB as strong on most MIs in this KPI and adequate on the extent to which its frameworks include indicators.

The document review, which was based on an analysis of country partnership strategies and associated documents, rated ADB as strong overall. The document review particularly noted recent improvements in the CPS format, such as the quality of its results frameworks and those presented in the sector assessments.[24]

Figure 3.9 | KPI 5: Country focus on results, ratings of micro-indicators

MI 5.1 – Frameworks link results at project, programme, sector and country levels

MOPAN donors in-country and clients were asked whether ADB’s country/sector results frameworks and project design and monitoring frameworks link results at project, sector and country levels. The majority of respondents (73%) rated the Bank as adequate or above.

The document review rated the Bank as strong on this MI. Most of the country results frameworks sampled for the review contained statements of expected results at outcome levels, but lacked information on expected outputs (e.g. Indonesia and Viet Nam). However, their accompanying “linked” documents provided detailed information on expected outputs. This reflects a fairly recent change in practice at ADB; following the 2009 review of the CPS process, ADB shortened and simplified the CPS format to show clear linkages between government goals, ADB strategies at the country level, and Strategy 2020’s core areas of specialisation. Hence, while reporting in the new format now presents results at higher levels (i.e. outcomes), specific information on outputs are now integrated in sector assessments linked to the CPS (i.e. sector result frameworks).

MI 5.2 – Frameworks include indicators at project, programme, sector and country levels

Survey respondents were asked whether ADB’s country/sector results frameworks and project design and monitoring frameworks contain indicators at country, sector, and project levels. The majority of respondents (80%) rated ADB as adequate or above.

The document review rated the Bank as strong on this MI. Two of the three country partnership strategies (CPS) reviewed were developed using the recent guidelines (i.e. the 2010 Guidelines on Preparing Results Frameworks and Monitoring Results - Country and Sector Results frameworks). Most of the CPS’ sampled for the review (along with their linked documents) contained performance indicators that were clear, relevant, and adequate. However, not all performance indicators were accompanied by time bound targets or baseline information. The CPS and sector results frameworks reviewed did not present data sources or data collection methods for monitoring expected results.

MI 5.3 – Expected results consistent with national strategies

MOPAN donors in-country and clients were asked whether ADB’s country partnership strategies (CPSs) contain statements of expected results that are consistent with those in the poverty reduction strategy papers (PRSP) or national development strategies. The majority (83%) rated ADB as adequate or above. The document review provided a rating of very strong on this MI. All CSP’ sampled for the assessment provide clear and explicit links between ADB’s expected results and those identified in national development strategies/plans or PRSPs. Further, they all provide information on how ADB’s strategies are aligned with the national priorities of respective DMCs (in both narrative and graphic form).

MI 5.4 – Expected results developed in consultation with clients/beneficiaries

This MI was assessed by survey only. MOPAN donors in-country and clients were asked whether ADB consults with clients to develop its expected results. The majority (81%) rated the ADB as adequate or above.

MI 5.5 – Results for cross-cutting thematic areas in country frameworks

MOPAN donors in-country and clients were asked whether ADB’s country partnership strategies include results for the thematic areas it identifies as priorities. The majority (75%) rated ADB as adequate or above

The document review rated ADB as strong for its inclusion of cross-cutting priorities in its country partnership strategies. All the CPSs reviewed included references to at least three of the cross-cutting themes assessed. Gender, private sector development, and environment were the most frequently cited.

3.3.3 Operational management

ADB received strong survey and document review ratings for its policies and processes related to financial accountability. The document review noted room for improvement in results-based budgeting and in the delegation of decision-making authority to resident missions. Survey respondents high- lighted the need for the Bank to improve its management of human resources – more specifically, its procedures for staff promotion.

Figure 3.10 below shows the overall survey and document review ratings for the seven KPIs in the operational management performance area.

ADB received survey ratings of adequate for most of the MIs assessed under operational management. Donors at headquarters perceived the Bank as strong for conducting financial audits that provide credible information to its governing bodies; but inadequate in establishing transparent and meritocratic promotion processes. ADB’s use of performance information to plan new interventions was considered strong by respondents in-country (i.e. MOPAN donors in-country and ADB clients).

The document review ratings varied from adequate to very strong.

Figure 3.10 | Performance area II: Operational management, survey and document review ratings

Figure 3.11 shows the mean scores for the KPIs for all survey respondents, and by respondent group.

Figure 3.11 | Performance area II: Operational management, mean scores by respondent group[25]

KPI 6: Resource allocation decisions

Finding 6: ADB was considered adequate overall for its resource allocation decisions.

This KPI examined the extent to which ADB makes transparent and predictable resource allocation decisions.

Overall, survey respondents rated the ADB’s resource allocation performance as adequate.

The document review rated the Bank as strong for having clear, publicly available criteria for the allocation of Asian Development Fund (ADF) resources and adequate in releasing resources according to agreed schedules.

Figure 3.12 | KPI 6: Resource allocation decisions, ratings of micro-indicators[26]

MI 6.1 – Criteria for allocating funding publicly available

All surveyed groups were asked whether ADB makes its criteria for allocating resources readily available. The majority (74%) rated ADB as adequate or above.

ADB was rated as strong by the document review in this area. The criteria for allocating Asian Development Fund (ADF) resources are clearly explained and made available to the public on line. ADB uses a performance-based allocation system (PBA) for concessional funding that is similar to those used by other multilateral development banks. The allocation of the Bank’s resources (loans and grants) is based on country performance (relying on Country Performance Assessments), country population, and national gross income per capita. The PBA system, developed in 2001, was designed to more effectively meet the needs of less developed member countries and has been refined twice since then, most recently in 2008. The application of the PBA policy is evident in the annual Country Performance Assessment reports. Every document relating to the Bank’s allocation of resources is available on line, but only in English.

MI 6.2 – Resource allocations conform to criteria

This MI was assessed by survey only. The majority of respondents (75%) rated ADB as adequate or above for allocating its resources according to established criteria.

MI 6.3 – Resources released according to agreed schedules

This MI was rated by document review only, which gave a rating of adequate. The assessment of ADB’s performance was based largely on findings drawn from the 2011 Survey on Monitoring the Paris Declaration (Indicator 7, aid is more predictable), the 2011 Aid Effectiveness Report, as well as the Bank’s Development Effectiveness Reviews (DEfRs). ADB has demonstrated its commitment to aid effectiveness by conducting assessments of the implementation of the Paris Declaration since 2008. Monitoring data on Indicator 7 show that the Bank has not yet met the target of 89%, as the percentage of predictable aid decreased from 86% in 2005 to 60% in 2010. The 2011 DEfR outlines two main reasons for the Bank’s declining performance in this area. According to the Bank, good performance on this indicator depends not only on the extent to which ADB’s disbursement of assistance is scheduled, but also on the extent to which developing member countries record aid disbursements in their public accounts. It also reports that exogenous events, such as natural disasters and economic crises, have required the Bank to respond through critical, demand-driven lending operations that cannot be anticipated. In the 2012 DEfR, the Bank no longer reports on Paris Declaration indicators but notes that the 2013 Results Framework includes place-holders for post-Busan indicators on aid effectiveness. Nonetheless, the 2012 DEfR reports that ADB has met its own target with regard to predictability, which is based on ensuring that the average time from approval to first disbursement for ADB operations takes less than 10 months.

KPI 7: Results-based budgeting

Finding 7: Donors at headquarters rated ADB as adequate in linking its budgets and expenditures to results. The document review noted application of results-based budgeting at the project level. With regard to administrative budgets, there is room for improvement in data and documentation and some initiatives are underway to strengthen this area.

This KPI examined two dimensions of results-based budgeting: whether ADB links budget allocations to expected results, and whether its reports link expenditures to results achieved.

Donors at headquarters, the only group asked to assess ADB’s performance in this area, rated the Bank as adequate overall. Their level of familiarity seemed low, however, with ‘don’t know’ responses ranging from 22% to 27% on all MIs.

At ADB, budgets are linked to development results through CPS, sector results frameworks, COBPs, and project approval documents. At a corporate level, the application of results-based budgeting in developing the Work Program and Budget is achieved through a results-based programme which, according to the Bank, is cascaded down to department, division and individual work plans for which the budget is provided. The review indicated that there remains room for improvement in linking expenditures to results achieved. It should be noted however that linking resource allocations and/or expenditures to expected development results remains a challenge for all MDBs.

Figure 3.13 | KPI 7: Results-based budgeting, ratings of micro-indicators

MI 7.1 – Allocations linked to expected results

Donors at headquarters were asked two questions on this MI: whether ADB links its operational budget to expected results, and whether it links its administrative budget to expected results. On both counts survey respondents rated ADB as adequate but there were high proportions of ‘don’t know’ responses on both questions (22% and 27% respectively).

The document review rated the Bank as adequate on this MI. At ADB, budgets are linked to results at the project level, where planned outputs are costed and linked to outcomes. The review of project documents, sector results frameworks, and CPS illustrate the linkage between planned results and the allocation of resources. [27]

At a corporate level, its allocation of administrative budgets is country driven and based on departments’ results-based work programs as well as estimates of staff-time coefficients. The coefficients are based on survey data of staff time used for types of activities (such as projects and CPS preparation). In the absence of a cost accounting system, the Bank uses these staff-time coefficients to allocate staff resources to departmental work programs.

Allocations are made to sectors and resident missions based on work programs and department priorities or special initiatives. Since 2011, the Bank has been piloting a time management system that, in principle, could capture staff time spent on different activities and provide a foundation for activity-based costing, which can provide better data on cost of products/activities as input for the budgeting process. Further assessments will be required before this system can be scaled up.

MI 7.2 – Expenditures linked to results

MOPAN donors at headquarters were asked to assess the extent to which ADB’s reports on results provide information on the amounts spent to achieve those results. Just over half (51%) rated the Bank as adequate or above, while 27% answered ‘don’t know’.

The document review rated the Bank as inadequate in this area based on how corporate reports link expenditures to results. Although the Bank’s Annual Reports provide a good overview of the Bank’s strategic directions and priorities as well as the Bank’s operations and institutional reforms, these reports do not clearly align financial disbursements (or any type of financial data) with results achieved. This is also the case for ADB’s Development Effectiveness Review.

KPI 8: Financial accountability

Finding 8: Donors at headquarters perceived ADB’s internal audit processes to be strong while respondents in-country viewed its procurement and contract management processes as adequate. The document review found ADB’s policies and processes for financial accountability to be strong in general.

All MIs under this KPI were assessed by document review and two were complemented by the views of survey respondents.

Overall, donors at headquarters perceived ADB to be strong in providing credible information from internal financial audits, and donors in-country and clients perceived ADB’s procurement and contract management processes to be adequate.

The document review found financial accountability to be one of ADB’s strengths and provided ratings ranging from adequate to very strong. The systems and practices in place for audits and for anti-corruption are very well detailed and there is evidence that policies are followed.

Figure 3.14 | KPI 8: Financial accountability, ratings of micro-indicators

MI 8.1 – External financial audits performed across the organisation

This MI, which was rated by document review only, assessed whether external financial audits are performed across the organisation; ADB was rated as very strong. ADB’s Annual Reports are complemented by a second volume presenting financial performance information related to ordinary capital resources, special funds and grant co-financing. Each year, financial reports are accompanied by a letter from an external auditor certifying that the audit was conducted in accordance with attestation standards established by the American Institute of Certified Public Accountants.

MI 8.2 – External financial audits performed at the regional, country, or project level

This MI was also rated by document review only and assessed whether external financial audits are performed at the regional, country or project level. ADB was rated as strong on this MI. According to ADB’s Operations Manual, all borrowers are required to have their annual project financial statements audited by an independent auditor acceptable to the ADB and the audit must be performed in accordance with acceptable auditing standards. For independent entities, ADB may accept financial statements prepared in accordance with national accrual-based financial reporting standards. The Operations Manual provides guidelines and requirements for the conduct of external audits, as well as rules for the selection of external auditors. According to the Manual, project financial statements and the audit report are disclosed 30 calendar days following receipt by ADB. However, the publication of these reports is also subject to ADB’s public communication policy, under which the Bank may choose not to publicly disclose an audit report that contains sensitive information.

MI 8.3 – Policy on anti-corruption

This MI was rated by document review only and the ADB was rated as very strong for its policy and guidelines to combat fraud and corruption. The ADB Charter establishes that the Bank has a duty to consider and address governance and anticorruption in all its financed activities. The Bank’s two guiding documents on anti-corruption are: i) the Anticorruption Policy, which was approved by the Board in 1998 and subsequently updated in 2004 and 2006; and, ii) the Integrity Principles and Guidelines (IPG) approved in 2006 and subsequently updated in 2010 and 2012.

According to the Bank’s Anticorruption Policy, each staff member has a personal obligation to ensure the integrity of ADB operations within their respective areas of responsibility. This is in line with their Code of Conduct. An independent Office of Anticorruption and Integrity (OAI), established in 1999 as an anticorruption unit in the Office of Auditor General (OAG), is the initial point of contact and investigative office for allegations of integrity violations involving ADB-related activities or ADB staff. OAI also conducts project procurement-related reviews (PPRRs), fraud awareness raising activities, and integrity due diligence. The Office reports on the main activities and outcomes of its work directly to the ADB President and, through the President, to the Board of Directors’ Audit Committee.

The IPG are harmonised with the International Financial Institutions’ (IFIs) Anti-Corruption Task Force’s Uniform Framework for Preventing and Combating Fraud and Corruption, to which ADB and other multilateral development banks (MDBs) are signatories. The Bank has a “zero tolerance” policy when it comes to fraud and corruption. On 9 April 2010, ADB and four other MDBs signed the Agreement for Mutual Enforcement of Debarment Decisions. Under the agreement, sanctions covering fraudulent, corrupt, coercive, and collusive practices that are imposed and made publicly available by any participating MDB may be enforced by other participating MDBs.[28]

The Second Governance and Anticorruption Action Plan (2006) is intended to help ADB country teams strengthen national governance systems and reduce the vulnerability of ADB investments to corruption. In support of this action plan, the OAI provides training workshops and seminars on fraud and corruption detection and prevention to staff at Headquarters and in Resident Missions and to third parties such as Executing Agencies, Supreme Audit Institutions and private auditing firms.

MI 8.4 – Systems for immediate measures against irregularities

The document review, which looked for the actual procedures for response and follow up on irregularities identified during an external financial audit, rated ADB adequate on this MI.

The Bank’s Operations Manual on Project Financial Reporting and Auditing (2012) describes the processes in place at the Bank for conducting an audit of an ADB-supported project. According to the manual, borrowers and/or executing agencies (EAs) are to submit annual audited financial statements to the Bank along with the auditor’s opinion and a management letter that includes any irregularities found during the audit assessment. The Bank has developed several other guiding documents for audit processes that include: Guidelines for the Financial Management and Analysis of Projects (2005), Project Administration Instructions on Financial Reporting and Auditing of Loans and /or Grant Financed projects (PAI 5.07), and the Financial Management Technical Guidance Note on Project Financial Reporting and Auditing (2013). Although the Bank has established clear guidelines for the auditing of ADB-financed loans and grants, the procedures to be followed in cases of non-compliance with ADB’s loan’s financial or audit agreement are not described in detail and do not include timelines for ADB’s management response to irregularities. However, the document review found evidence of management follow up on audit recommendations and their implementation status.

MI 8.5 – Internal financial audits provide credible information

MOPAN donors at headquarters, the only group surveyed on this MI, were asked whether ADB conducts internal financial audits that provide credible information to its governing bodies. Nearly half (49%) rated it as strong or very strong.

The document review rated ADB as very strong. The Office of the Auditor General (OAG) is responsible for undertaking independent reviews of ADB’s operations, encompassing financial, administrative and project- related activities. It has specific terms of reference and is separate from programming areas. It reports to the President of the Bank, and through the President, to the Audit Committee of the Board (ACB).The OAG meets periodically with the ACB to discuss its work programmes, reports, and the status of its recommendations. There is evidence that the ACB reports to the Board on its annual activities; and annual reports of the audit committee generally include its involvement with OAG within the given period.

MI 8.6 – Effective procurement and contract management processes

When asked whether ADB’s procurement and contract management processes for the provision of services are effective, donors in-country and clients (76%) rated the Bank as adequate overall.

The document review rated ADB as adequate on this MI. The Bank has clear guidelines related to procurement and contract management; a Procurement Committee ensures consistent and correct application of procurement practices. Although procurement guidelines emphasise the timely delivery of products and services, there do not seem to be any clear targets or requirements related to this. The guidelines also state that procurement must be carried out with due consideration of economy and efficiency but, again, the document review did not find specific requirements relating to these criteria. A Procurement Governance Review completed in January 2013 provided several recommendations to improve procurement governance. Key recommendations addressed the need for streamlined procurement review processes, increased/ regular assessments of ADB procurement practices, the need to expand ADB capacity development programs for procurement etc. The Bank indicates that it has already initiated the implementation of select recommendations that emanated from the review. For instance, it has begun drafting key guiding documents for staff on preparing procurement risk assessments at country and sector levels and determining project procurement risk classification. These initiatives are considered to be works in progress.

MI 8.7 – Strategies for risk management

This MI was rated by the document review only, which rated the Bank as strong due to the strategies in place for risk identification, mitigation, monitoring and reporting. ADB has developed several tools and frameworks meant to mitigate different types of risk, namely: credit, market, liquidity, and operational risks. In addition, ADB has an Office of Risk Management (ORM) that was set up to identify, measure, manage, and report on risks. There is evidence that risks analyses are presented to the Board. According to the Annual Report of the Audit Committee of the Board 2011-2012, the ORM regularly presented “key portfolio developments, explained their focus for risk issues, and presented updates on policies and methodologies”. The Office is also reported to have provided quarterly updates to the Board on risks linked to portfolio and investments.[29]

KPI 9: Using performance information

Finding 9: Survey respondents rated ADB as adequate overall for its use of performance information and as strong for its use of performance information to plan new interventions. The document review rated the Bank very strong overall.

This KPI examined the extent which ADB uses performance information to revise policies and plan new interventions, acts upon recommendations, and allocates resources based on performance. ADB was rated as adequate by survey respondents on three MIs and as strong on one. The document review considered ADB’s practices to be very strong in most areas but found its management of unsatisfactory operations to be adequate overall.

Figure 3.15 | KPI 9: Using performance information, ratings of micro-indicators

MI 9.1 – Using information to revise and adjust policies

When asked whether ADB uses project/programme, sector and country information on performance to revise corporate policies, 76% of donors at headquarters surveyed rated ADB as adequate or above. The document review provided a rating of very strong on the Bank’s use of information to revise and adjust policies. Information on organisation-wide performance is available in the annual Development Effectiveness Review (DEfR). According to a Special Evaluation Study on Managing for Development Results conducted by the Bank in 2011, findings and recommendations from the DEfR are used by the Bank. The study stated that the DEfR “report has moved from being simply a public accountability tool to also being an important internal planning document, which is integrated into the ADB’s high-level planning processes [...]”.

The assessment team also found several instances in which the Bank made adjustments based on evaluation recommendations. In the 2011 “Special Evaluation Study on MfDR”, for example, IED recommended that the Bank, “improve the corporate results framework by (i) strengthening the links between outputs and outcomes used in the corporate results framework and (ii) review its indicators to serve as a better management tool to gauge development effectiveness.” As indicated in MI1.3, the Bank conducted a comprehensive review of its corporate results framework in 2012 which led to the adoption of a new outcome-oriented framework in 2013. The 2011 study also recommended that the Bank “improve the results orientation of the corporate sector and thematic plans by clearly defining their purpose and the means to monitor their achievement.” In 2013, the Bank issued Staff Instructions on Processing and Monitoring Sector and Thematic Operational Plans in which it notes its intent to update all operational plans that do not contain a results framework and/or time-bound monitoring plans.[30]

MI 9.2 – Using information to plan new interventions

Donors in-country and clients were asked whether ADB uses information on country and sector performance to plan new interventions at the country level. Nearly half of the respondents surveyed (47%) rated ADB as strong or very strong.

The document review also rated the Bank’s performance as very strong. Information about the Bank’s country-level performance is found in several documents: Country Partnership Strategy review missions, Development Effectiveness Country Briefs, and Country Assistance Program Evaluations (CAPE). The Development Effectiveness Country Briefs (DECBs) present an overview of country achievements and CAPEs provide an assessment of the performance of ADB assistance in a particular country in at least two of the most recent Country Partnership Strategy cycles. There is evidence that this performance information is used in planning new interventions. According to the 2010 Revised Guidelines for the Preparation of Country Assistance Program Evaluations, the goal of the CAPE “is to inform ADB Management and the Board of Directors of the CAPE findings to ensure incorporation of the CAPE lessons and recommendations in the design of the subsequent Country Partnership Strategies, together with its corresponding COBPs to improve the performance of future ADB assistance.” There is evidence from all three countries sampled that information from the CAPE has been used to plan new interventions. For instance, Appendix II of the 2009 Viet Nam CAPE highlights the actions taken as a result of the 1999 CAPE recommendations, as well as the actions taken as reflected in the 2005-2006 Country Strategy and Program Update and in the 2007-2010 Country Partnership Strategy.

MI 9.3 – Proactive management of unsatisfactory initiatives

Donors in-country and clients were asked whether ADB manages unsatisfactory projects from previous fiscal years. This MI yielded a high percentage of ‘don’t know’ responses (43%), with a little less than half of respondents (47%) rating ADB as adequate or above, and 10% as inadequate or below. MOPAN donors in-country had the highest proportion of “don’t know” responses (60%). Clients rated the organisation more positively than donors in-country, and the difference was statistically significant.

The document review provided a rating of adequate based primarily on information from the 2012 Development Effectiveness Review (DEfR) report and COMPAS reports on Indicator 8. The COMPAS reports showed that the percentage of projects that were unsatisfactory in 2010 that became satisfactory in 2011 remained constant at 63% and did not improve. However, there is evidence that the Bank has made efforts to improve its project performance management system (PPMS). In 2011, the ADB introduced a new system to assess project performance, i.e. the Project Performance Reporting system (PPR) which serves as a management tool used for monitoring project implementation and identify potential problem projects and/or projects at risk.[31]

Data from the 2012 DEfR indicates that the proportion of projects rated satisfactory at implementation was 89% for both ADB operations and ADF operations, which is above the 80% target set by the Bank. While this is indicative of the progress being made, the Bank also notes that “despite the positive performance, Management remains cautious about the high percentage of projects rated satisfactory by the updated project performance reporting system introduced in 2011”. It further comments on downwards adjustments made to project performance and indicates that in 2013 ADB will “undertake strict quality control over the implementation of the project performance reporting system to ensure the accuracy of performance ratings”.[32]

The DEfR highlights reasons for less successful ratings of projects, including: the inappropriateness of project designs that are either too complex or overambitious, weak project management capacity which can affect implementation, and unforeseen factors such as political and economic issues.

MI 9.4 – Evaluation recommendations are acted upon

MOPAN donors at headquarters, the only respondent group surveyed on this MI, rated ADB as adequate for its tracking of the implementation of evaluation recommendations reported to the Board. While 49% of respondents overall rated ADB as adequate or better, 30% rated it strong or very strong.

The document review rated ADB as very strong in acting upon evaluation recommendations reported to the Board. The Bank’s evaluation policy is embedded in the 2008 Review of the Independence and Effectiveness of the Operations Evaluation Department, and is supported by the Operations Manual on Independent Evaluation (2011) which provides ADB staff with guidance on all aspects of ADB’s evaluation work. The documents reviewed revealed that the evaluations conducted by ADB are generally followed up with a management response and are discussed by the Development Effectiveness Committee Chairs, when applicable. Follow-up on evaluation recommendations has been simplified since 2008 by the establishment of an electronic Management Action Record System, which keeps track of management responses, the status of agreed-upon action plans, and implementation progress. In addition, an Annual Report on Acting on Recommendations (available on line up until 2010) provided consolidated information on progress of management actions on IED recommendations and was discussed with the Board on a yearly basis. The Management Action Record System is accessible to all ADB staff and the Board of Directors through ADB’s intranet portal. According to its TORs, the Development Effectiveness Committee has to review all IED reports and discuss certain major reports, as well as all management responses.

MI 9.5 – Resources allocated based on performance

This MI was rated by document review only, which rated the Bank as very strong for allocating ADF resources to countries and projects based on performance.

ADB has a performance-based allocation system (PBA) that applies to its concessional resources. The criteria are publicly available and based on annual country performance assessments (CPAs). Performance is the main criterion of the PBA system and it is broken down into four components: quality of macroeconomic management, coherence of structural policies, degree to which policies and institutions promote equity and inclusion, and quality of governance and public sector management. Since its inception in 1978, the PBA system has been reviewed on several occasions (2004 and 2008). In 2011, the Board approved revisions to strengthen the link between performance and allocations. There is clear and wide evidence that the system is applied, notably through the annual country performance assessment results presented in the Annual Report on Country Performance Assessment Exercise.

KPI 10: Managing human resources

Finding 10: While the survey ratings suggest that ADB manages its human resources adequately overall, donors at headquarters found its promotion processes to be inadequate. The document review considered the Bank to be strong in managing staff performance but found the system to assess the performance of vice presidents to be less clear.

ADB’s management of human resources received mixed reviews from surveyed stakeholders and the document review.

The Bank has several tools to manage human resources, including the Our People Strategy (2010) a human resource strategy that aligns human resource management to Strategy 2020 by clarifying what ADB offers and seeks in present and future staff, values and competencies. It also has a Human Resource Action Plan (2009) that defines a number of actions to address shortcomings in ADB’s HR policies and service delivery infrastructure.

Figure 3.16 | KPI 10: Managing human resources, ratings of micro-indicators

MI 10.1 – Results-focused performance assessment systems for senior staff

This MI was assessed by document review only, which rated ADB as adequate for having a results-focused performance assessment system for senior staff. The Bank has a formal annual performance review system that captures information on staff performance during the year in three components: 1) key accomplishments and results assessments; 2) competency assessment; and 3) overall assessment. All Bank staff are also requested to participate in a work planning process at the end of the year to develop their own work plans based on departmental plans. Once the staff member and supervisor agree on deliverables and indicators of performance, the information is captured in an online system at the beginning of the year. Given that this is an online process, the Bank relies on its IT system to ensure staff compliance with the work plan and performance review processes. Staff who have not yet completed their assessments are sent alerts, urging them to commence the performance review process. Alerts are also sent to their supervisors to remind them to follow up with staff. This process applies to all staff appointed by the President, including heads of departments and the Managing Director General (who are assessed by vice- presidents). The performance review system for vice presidents was found to be less clear. According to the Bank, separate rules apply for vice presidents. Vice Presidents are appointed by the Board of Directors on the recommendation of the President in accordance with Article 35 of the Charter. The Conditions of Service of Vice Presidents are set out in Section 6 of the By-laws of the Bank. Other terms and conditions are provided in the Employment Contract which is a confidential document.

In addition to these systems and processes, ADB introduced a 360 degree feedback mechanism in 2011 that allows managers to receive feedback from subordinates, peers and supervisors on their managerial competencies. At the moment, this instrument is only used as a developmental tool for managers and is not yet mandatory.

MI 10.2 – Transparent system to manage staff performance

This MI was rated by document review only, which rated the Bank as strong for the transparency of its system to manage staff performance.

The Bank has a performance review system (PR) to manage staff performance and staff promotions are based on performance. According to the Bank, cumulative performance reviews serve as input for assessing and selecting the most suitable candidates for vacant positions and management-driven promotion exercises.

In 2008 the Bank introduced the “Our Voice Survey”, conducted every two years, which allows staff to assess human resource practices and indicate areas that have improved or require further improvements.

In 2011, the Bank introduced a ‘rewards and recognition’ programme to ensure that staff are acknowledged for their work and accomplishments. The programme consists of: 1) spot awards, given at any time during the year to recognise exceptional efforts by individuals or team; 2) annual awards, to recognise special accomplishments of staff. According to ADB staff, ADB management is paying attention to this area and seeking feedback on implementation of the programme from its staff surveys.

MI 10.3 – Country staff deployment supports partnership development

This MI was assessed by survey only. Donors in-country and clients were asked two questions: whether the number of ADB staff at the country level and the quality of staff allow for effective country level partnerships. The majority of respondents rated ADB as adequate or above on both questions (61% and 83% respectively).

MI 10.4 – Staff recruitment and promotion is transparent and based on merit

This MI was assessed by a survey of MOPAN donors at headquarters who were asked two questions. On the question of whether staff recruitment at ADB is transparent and based on the skill requirements for the job, 59% rated the Bank as adequate or above. On the question of whether the promotion of staff is transparent and based on merit, 46% rated the organisation inadequate or below and 27% responded ‘don’t know.’

KPI 11: Performance-oriented programming

Finding 11: Donors at headquarters generally perceive ADB as adequate in the application of benefits/impact analysis prior to the approval of loans and grants. The document review provided an overall rating of strong for ADB’s performance-oriented country programming processes.

This KPI examined the extent to which ADB’s country and regional processes are performance oriented. Donors at headquarters, the only group surveyed, rated ADB as adequate for subjecting new initiatives to benefits/impact analysis.

The document review acknowledged the efforts made by the Bank to ensure that projects are financially and economically sound at the appraisal stage. It also found that the majority of project documents contain milestones and performance targets to monitor progress during implementation.

Figure 3.17 | KPI 11: Performance-oriented programming, ratings of micro-indicators

MI 11.1 – New initiatives subject to benefits/impact analysis

MOPAN donors at headquarters were asked whether ADB subjects new loans and grants to due diligence procedures and analyses, including benefits and impact analyses. Nearly half (49%) rated ADB strong or very strong, 43% as adequate, and 8% as inadequate or below.

The document review rated the Bank as strong on the extent to which it subjected new initiatives to benefits/impact analysis. ADB requires that all projects undergo a financial benefit-cost analysis prior to their initiation. Investment projects are appraised to ensure that they are technically sound and financially and economically viable. Guidelines are provided for the conduct of benefit-cost analyses and there is ample evidence in project data sheets and project documents that these guidelines are implemented. Findings from these analyses are later used to inform Reports and Recommendations of the President to the Board of Directors on Proposed Loans as well as Periodic Financing Request (PFR) reports. A review of a sample of projects indicated that ADB conducts regular economic and financial analyses during the project appraisal stage.

MI 11.2 – Milestones/targets set to monitor implementation

On this MI, which was assessed by document review only, ADB was rated as very strong for setting milestones to track the progress of project implementation.

According to the Bank’s Operations Manual on Project Performance Management System (2011) which outlines the policies and procedures to be followed in the management of ADB-financed projects, each ADB financed and administered project is required to have a design and monitoring framework (DMF) that provides a description of activities and milestones, performance indicators to be measured and targets to be achieved in a specific timeframe, among other requirements. ADB’s Project Administration Instructions on Project Performance Monitoring (PAI 5.08), also provides detailed information of how ADB financed and administered projects ought to be monitored and rated using ADB’s eOperations project implementation module, a system that records country and project level information at different stages, i.e. from concept to ex-post evaluation. According to the PAI, management at the Bank relies on a rating system based on five performance indicators (i.e. technical, procurement, disbursement, financial management and safeguards) to gauge the implementation status of a project and determine actions to be taken in case problems arise.[33]

The Bank also relies on several reporting tools to monitor project implementation. Amongst those are the Project Administration Manuals (PAM) which describe the essential administrative and management requirements of every project. All PAM sampled for this assessment contained milestones and targets for project implementation that seemed appropriate for the activities to be implemented and most of the PAM established dates for the milestones.

KPI 12: Delegating authority

Finding 12: Respondents in-country rated ADB as adequate overall for its delegation of decision- making authority. The document review concurred and noted increased responsibilities given to resident missions in recent years.

ADB clients and donors in-country were the only groups consulted on ADB’s delegation of decision-making authority to the country level. While the majority perceived ADB’s performance as adequate, there was a high proportion of ‘don’t know’ responses from MOPAN donors in-country.

The document review rated ADB as adequate overall and acknowledged the Bank’s efforts to increase the responsibilities of resident mission staff. Resident missions are able to approve minor changes to projects, emergency projects, and have authority to approve technical assistance projects depending on the proposed amount. The Bank has increased its delegation of authority to resident missions and, with its Strategy 2020, has committed to further increase the decentralisation of decision making to resident missions.

The number of fully operational resident missions grew rapidly following the approval of the 2000 Resident Mission Policy (from 13 in 2000 to 26 in 2012). Many evaluations since 2002 have recognised the relevance of the Resident Mission Policy, but also recommended strengthening the capacity and increasing the authority of resident missions to facilitate more efficient and effective decision making. The Bank is currently undergoing an evaluation of its decentralisation process that is expected to be completed in 2013.

Figure 3.18 | KPI 12: Delegating authority, ratings of micro-indicators

MI 12.1 – Aid allocation decisions can be made locally

Donors in-country and clients were asked whether ADB’s programming and project tasks are managed at the country level. The majority of respondents (69%) rated ADB as adequate or above.

The document review rated the Bank adequate based on a review of the authority that is delegated to its resident missions, including, but not limited to, the allocations or adjustments that can be made in projects.

The Bank has delegated a certain amount of authority to its resident missions since 1982. Initially, the tasks were related mostly to project monitoring. Gradually, however, the responsibilities increased and now include: i) government, civil society, and private sector relations; ii) policy dialogue and support; iii) country reporting; iv) aid co-ordination; and v) external relations and information dissemination. Four specific functions that relate to the delivery and implementation of ADB’s products are also identified by the 2000 Resident Mission Policy: i) country programming; ii) project and technical assistance processing; iii) portfolio management and project administration; and iv) economic and sector work and analytical work.

According to interviews with Bank staff, the Bank has increased delegation to the country level over the past three years. A number of tasks are now fully delegated to the country level, including the preparation of CPS, country portfolio reviews, as well as country and sectoral economic work.

Resident missions (RMs) are able to approve minor changes to projects, emergency projects, and have authority to approve technical assistance projects of less than $225,000. In terms of changes to individual projects, the Bank differentiates between major and minor changes. Major changes (including a change that fundamentally affects the scope and project outcomes) must still be approved by the Board. However, minor changes to RM-administered projects (e.g. changes to cost estimates, project outputs, performance targets, implementation arrangements) can be approved by the mission at the country level.

MI 12.2 – New programmes/projects can be approved locally within a budget cap

Country-level respondents (donors in-country and ADB clients) were asked whether ADB has delegated appropriate decision-making authority at the country level. Overall, 58% rated ADB as adequate or above. However, 29% of MOPAN donors in-country answered ‘don’t know’.

The document review rated the Bank as adequate on this MI. The delegation of authority to approve new loans is limited in the case of an IFI because authority rests with the Board of Directors. A review of the Summary of ADB Financial Instruments and Approval Procedures indicates that the majority of the Bank’s operations need approval from the Board. The main exception is technical assistance, which can be approved by head of departments or offices, depending on the proposed technical assistance amount (i.e. less than $225,000). Once a loan has been approved, resident missions have the following authority: procurement and approval of changes to the method of procurement for contracts less than $1 million and approval of minor changes in project scope or implementation arrangements.

3.3.4 Relationship management

ADB was considered adequate or better in all areas of relationship management. Survey respond- ents rated it strong for contributing valuable inputs to policy dialogue and respecting the views of clients. The document review gave it generally strong ratings for using country systems and noted room for improvement in the efficiency of its administrative procedures and use of country procurement systems.

Figure 3.19 below shows the overall ratings for the five KPIs in the relationship management performance area. ADB’s commitment to aid effectiveness is demonstrated through its annual monitoring of and reporting on 9 of the 12 PD indicators since 2008. Once the Busan indicators are finalised and become available, the Bank plans to incorporate these in the 2013 revised results framework.[34]

Figure 3.19 | Performance area III: Relationship management, survey and document review ratings

Figure 3.20 shows the mean scores for the five KPIs for all survey respondents, and by respondent group.

Figure 3.20 | Performance area III: Relationship management, mean scores by respondent group

KPI 13: Supporting national plans

Finding 13: Surveyed respondents in-country rated ADB as strong for ensuring that funding proposals are designed and developed in collaboration with national stakeholders. The document review rated the Bank as strong for its use of indicators or benchmarks for conditionality.

This KPI examined the extent to which ADB co-ordinates and directs its aid programming at the country level in support of agreed national plans or partner plans.

The document review found that ADB’s support to national plans is evident in the processes of preparing Country Partnership Strategies (CPSs) and Country Business Operations Plans (COBPs), which are generally developed in consultation and collaboration with national governments.

Figure 3.21 | KPI 13: Supporting national plans, ratings of micro-indicators

MI 13.1 – Funding proposals developed with national government or clients

This MI was assessed by survey only. ADB clients and MOPAN donors in-country were asked about the extent to which ADB supports funding proposals designed and developed by the national government or other clients. Nearly half of respondents (49%) rated ADB as strong or very strong, 30% as adequate, and 10% as inadequate or below.

MI 13.2 – Conditionality draws on national benchmarks/indicators/results

When asked whether ADB applies conditionality that corresponds with national government goals and benchmarks, the majority of MOPAN donors in-country and clients (69%) rated ADB as adequate or above. Donors in-country appeared to be less familiar with this issue with 32% answering ‘don’t know.’

ADB was rated as strong for ensuring that loan conditions respect the principles of country ownership set out in the Paris Declaration.

ADB introduced programme lending or policy-based lending in 1978. Since then, the Bank has reviewed and updated its policy on a regular basis. It carried out studies on its policy in 1983, 1987, 1996, 1999, and 2011. The 2011 review proposed two major reforms to ADB’s policy: 1) mainstreaming programmatic budget support (with enhanced Board oversight); and, 2) enhancing responsiveness to allow for more flexible lending terms in times of crises. Following the 2011 review, the Bank updated its Operations Manual which provides ADB staff with guidance on the four policy-based lending products offered at the Bank (i.e. stand-alone policy-based lending, programmatic approach, special policy-based lending, and counter- cyclical support facility lending) and outlines the operational procedures for processing policy-based loans. The manual indicates that: “Staff should adopt a flexible approach to conditions in policy-based loans, and apply the following good practice principles in the design and implementation of conditionality in policy- based loans: (i) conditions should reinforce country ownership, (ii) they should be agreed with the government and linked closely to national development strategies, (iii) only a minimum number of actions critical for removing or mitigating binding constraints should be selected as conditions, and (iv) transparent progress reviews that contribute to predictable and performance-based financial support should be conducted regularly.”[35]

The review of the 2010 guidelines on Preparing Results Frameworks and Monitoring Results indicates that ADB has committed to ensuring alignment with government priorities/national development strategies and government benchmarks. Evidence of implementation of the principles of country ownership was also noted in the newly formatted Country Partnership Strategies (CPSs) sampled for this assessment.

KPI 14: Adjusting procedures

Finding 14: Survey respondents perceived ADB to be adequate in adjusting its procedures to local conditions and capacities.

This KPI was assessed solely by survey and only by respondents at the country level (i.e. MOPAN donors in-country and ADB clients). Both groups rated ADB as adequate in ensuring that its procedures take into account local conditions and capacities. There were fairly high percentages of ‘don’t know’ responses on all MIs assessed by donors in-country offices.

Figure 3.22 | KPI 14: Adjusting procedures, ratings of micro-indicators

MI 14.1 – Procedures easily understood and completed by clients

When asked whether ADB uses procedures that can be easily understood and followed by clients, the majority of respondents (63%) rated the organisation as adequate or above. A considerable proportion of MOPAN donors in-country (31%) answered ‘don’t know.’

MI 14.2 – Time for procedures does not affect implementation

When asked whether the length of time it takes to complete ADB procedures significantly delays implementation, 54% of respondents rated the organisation as adequate or above. A high proportion of donors in-country (31%) answered ‘don’t know.’

MI 14.3 – Ability to respond quickly to changing circumstances

ADB was rated adequate overall for adjusting its portfolio to respond to changing circumstances in countries. One-quarter of respondents answered ‘don’t know.’

MI 14.4 – Flexibility in implementation of projects/programmes

Survey respondents were asked whether ADB adjusts its implementation of individual projects/ programmes as learning occurs. The majority (59%) rated ADB as adequate or above and 29% of donors in-country answered ‘don’t know.’

KPI 15: Using country systems

Finding 15: Overall, clients and MOPAN donors in-country perceived ADB to be adequate in its use of country systems. The document review rated the Bank strong in most areas but found room for improvement in the use of procurement systems.

Clients and donors in-country, the only two respondent groups asked to assess this KPI through the survey, perceived ADB to be adequate in its use of country systems. Their level of familiarity seemed low, however, with ‘don’t know’ responses ranging on average from 31% to 39% on all MIs.

The document review of this KPI was based on various sources of data, including the Development Effectiveness Review reports (2011 and 2012), ADB’s Aid Effectiveness Report (2011), OECD’s Survey on Monitoring the Paris Declaration (indicators 3, 5a, 5b and 6) and other relevant external reviews.

ADB was found to have made considerable progress in its use of country systems for disbursement and operations. The Bank has met or exceeded the PD indicator targets for three of the four micro-indicators assessed in this area: aid recorded on budget, use of country public financial management systems, and avoiding use of PIUs. On the issue pertaining to the use of country procurement systems, the Bank noted some challenges related to fiduciary accountability and highlighted the need to first strengthen the capacities of national institutions.

Figure 3.23 | Using country systems, ratings of micro-indicators

MI 15.1 – ODA disbursements/support recorded in annual budget

This MI, assessed by document review only, assessed the percentage of an organisation’s overall Official Development Assistance (ODA) disbursements recorded in the annual budgets of developing member countries (DMC) as revenue, grants or ODA loans (indicator 3 of the Survey on Monitoring the Paris Declaration). ADB received a strong rating based on OECD’s and the Bank’s aid effectiveness reports, which indicated that the percentage of disbursements reflected in DMC budgets has increased steadily over the years. According to the 2011 Aid Effectiveness Report, the proportion of aid disbursement recorded in national budgets increased from 66% in 2005 to 89% in 2009, surpassing the 2010 Paris Declaration target of 85%. ADB’s success in this area is reported to have been largely due to increased alignment between ADB’s Country Partnership Strategies (as well as business plans) and DMC’s priorities since 2005 and, in particular, since 2010.

It should also be noted that in the Global Campaign for Aid Transparency’s 2010 assessment, the ADB ranked 3rd among 30 bilateral and multilateral donors with regard to percentages of aid recorded in partner countries’ budgets.[36]

MI 15.2 – Use of country financial systems for operations

Donors in-country and clients were asked whether ADB uses country financial systems as a first option for its operations where appropriate. Responses were mixed: 46% rated ADB as adequate or above and 42% of MOPAN donors in-country answered ‘don’t know.’

The document review rated ADB as strong on the use of country public financial management systems (i.e. indicator 5a of the Survey on Monitoring the Paris Declaration) as the Bank has either met or exceeded the 2010 PD target. According to OECD’s Aid Effectiveness report (2011), ADB’s use of public financial management (PFM) systems increased from 69% in 2005 to 78% in 2010, which was the 2010 target for this indicator. However, ADB’s Aid Effectiveness Report 2011 indicates that the percentage more than doubled in five years – from 42% in 2005 to 94% in 2010. The report noted that ADB’s strong improvement on this indicator may have been partly due to the “substantial use of PFM systems in large borrowing DMCs, such as Bangladesh, the People’s Republic of China, India, Indonesia, and Pakistan.” It also noted that “countries where ADB did not meet the target in 2010 were Azerbaijan, the Maldives, Mongolia, the Philippines, and Viet Nam”, indicating that structural problems in country systems resulted in ADB’s hesitation to use them for aid disbursement.[37]

MI 15.3 – Use of country non-financial systems for operations

Survey respondents were asked whether ADB uses a country’s non-financial systems as a first option for its operations. Respondents’ views were mixed: 43% rated ADB as adequate or above, 18% as inadequate or below, and 39% overall answered ‘don’t know’ (54% of MOPAN donors in-country and 24% of ADB clients).

ADB was rated as adequate by the document review on the use of country procurement systems (i.e. indicator 5b of the Survey on Monitoring the Paris Declaration). The review drew largely on data from the most recent reports from the OECD and the ADB which noted that the Bank has not made sufficient progress in this area. According to the OECD report, ADB’s use of country procurement systems fell from 45% in 2005 to 37% in 2010. ADB’s internal monitoring documents indicated that its use of country procurement system has fluctuated (from 44% in 2007, to 59% in 2008, to 45% in 2009).[38] According to Aid Effectiveness Report (2011), ADB’s failure to meet the 60% target is in part explained by the fact that in many DMCs, national capacity limitations put the Bank at a level of fiduciary risk that is not acceptable, thereby leading to limited use of country procurement systems.[39] The Bank underlines that it is however, attempting to improve these said procurement systems via technical assistance and knowledge sharing.

According to the Bank staff, ADB has recently undertaken several initiatives to increase the use of national procurement systems, including: i) a Procurement Governance Review (2013), which provides a more proactive approach to assessing national procurement risks and factoring these into the determination of international or national competitive bidding (ICB/NCB) and prior reviews thresholds; ii) a Technical Assistance on Developing Procurement Capacity for Improved Procurement Outcomes (2012), which seeks to assist procuring entities to devise a comprehensive procurement strategy in light of international best practices and the policy/ operating procurement environment within a country, and iii) the recent adoption of the Operational Plan for Enhancing ADB’s Effectiveness in Fragile and Conflict-Affected Situation (FCAS). The operational plan highlights the Bank’s commitment to finding innovative initiatives and processes to better “respond to the unique challenges of FCAS, including weak institutional capacity in procurement, financial management, and safeguards.”[40]

MI 15.4 – Parallel implementation structures avoided

When asked whether ADB avoids the use of parallel implementation units (PIU), clients and donors in- country had mixed views: 48% rated ADB as adequate or above, 17% rated it inadequate or below, and 34% answered ‘don’t know.’ Clients rated the organisation more positively than donors in-country, and the difference was statistically significant.

The Bank was rated as strong by the document review, which considered reporting on indicator 6 (use of parallel implementation units) of the Survey on Monitoring the Paris Declaration. Both OECD’s and ADB’s aid effectiveness reports indicate that the Bank has made considerable progress in reducing the use of PIUs. ADB reported that it was using 95 PIUs in 2005 and only six in 2010, which is significantly lower than the Paris Declaration target of 32. In the 2011 Aid Effectiveness Report, ADB indicates it is using PIUs only in cases where a DMC’s capacity to implement the project or programme is inadequate and that these will be integrated into national systems once the proper capacity has been established.[41]

MI 15.5 – Promotion of mutual accountability in its partnerships

This MI was assessed by survey only. ADB was rated as adequate overall for encouraging mutual accountability assessments of Paris Declaration and subsequent aid effectiveness commitments (Accra Agenda for Action, and Busan High Level Forum). Surveyed respondents seemed to lack familiarity with this issue, with 33% answering ‘don’t know.’

KPI 16: Contributing to policy dialogue

Finding 16: ADB’s contribution to policy dialogue and respect of client views was perceived as strong by in-country respondents.

This KPI was assessed by survey only. ADB was rated as strong for its contribution to policy dialogue.

Figure 3.24 | KPI 16: Contributing to policy dialogue, ratings of micro-indicators

MI 16.1 – Reputation for valuable input to policy dialogue

All respondent groups were asked whether ADB provides valuable input to policy dialogue. The majority (61%) rated ADB as strong or very strong.

MI 16.2 – Policy dialogue respects client views

When asked whether ADB respects the views of clients when undertaking policy dialogue, the majority of respondents (57%) rated ADB as strong or very strong.

KPI 17: Harmonising procedures

Finding 17: Survey respondents and the document review perceived ADB to be adequate in harmonising procedures at the country level.

According to the Bank staff, ADB mainstreams the principle of harmonisation through joint analytical work undertaken with other development partners as part of its project, programme and CPS processes.

Figure 3.25 | KPI 17: Harmonising procedures, ratings of micro-indicators

MI 17.1 – Participation in joint missions

Donors in-country and ADB clients were asked about the extent to which ADB participates in joint missions. The majority (70%) rated ADB as adequate or above; 20% of donors in-country answered ‘don’t know.’

ADB received a rating of strong on this MI in the document review, which was based on indicator 10a of the Survey on Monitoring the Paris Declaration and its target of 40% joint missions. In 2008, ADB’s case study on the Bank’s approaches to partnering and harmonisation pointed out that ADB had begun to put emphasis on joint activities in 2000, before the adoption of the Paris Declaration. These efforts stressed the importance of “complementary arrangements with other donors based on comparative advantage,” and included a review of existing working partnerships in order to issue new guidelines on partnership building. The document review indicated that ADB has used a narrower definition of “joint missions” than the OECD, by considering only the proportion of CPS and Country Portfolio Review Missions (CPRMs) conducted jointly with other development partners. Both OECD and ADB reports concur that the Bank has increased its rate of joint missions, but there are differences. The OECD reported an increase from 5% in 2005 to 11% in 2010, while the Bank reported that in 2010 it had achieved 54%, surpassing the target of 40%, partly due to the definition of “joint missions.”

MI 17.2 – Technical co-operation disbursed through co-ordinated programmes

Among donors in-country and ADB clients surveyed, the majority (74%) rated ADB as adequate or above for disbursing technical assistance through co-ordinated programmes in support of capacity development; 23% of donors in-country answered ‘don’t know.’

The document review rated ADB adequate for its efforts in co-ordinating technical assistance (indicator 4 of the Survey on Monitoring the Paris Declaration).The documents reviewed provided mixed assessments of the Bank’s performance in this area. While the OECD reported that ADB had increased its co-ordination of technical assistance from 37% in 2005 to 42% in 2010 (still below the 50% target), the Bank’s 2011 Aid Effectiveness Report indicated that it had slightly surpassed the target, reaching 53%. It also not however, that the target had been met in almost all 25 countries it had surveyed, with the exception of Bangladesh, Cambodia, India, Indonesia, Kazakhstan, the Lao People’s Democratic Republic, and Uzbekistan.[42]

MI 17.3 – ODA disbursements/support for government-led PBAs

This MI was assessed by document review only, which rated the ADB as adequate based on indicator 9 (programme-based approaches) of the Survey on Monitoring the Paris Declaration. Both OECD and ADB reports indicated that the Bank did not meet the 2010 target of 66% despite some progress in this area. The ADB’s own internal monitoring report revealed that the percentage of disbursements to government- led PBAs had increased only slightly, from 55% in 2005 to 61% in 2010. The Bank attributes this small rate of improvement to the lack of clear guidelines for monitoring PBAs, which is reported to have caused some confusion in tracking the indicator.[43]

3.3.5 Knowledge management

Survey respondents found ADB’s knowledge management adequate overall, while the document review considered it to be strong. The Bank was particularly noted for the independence of its evaluations and for making key documents readily available to the public.

Figure 3.26 below shows the overall survey and document review ratings for the four KPIs in the knowledge management performance area.

Figure 3.26 | Performance area IV: Knowledge management, survey and document review ratings

Figure 3.27 shows the mean scores for the four KPIs for all survey respondents, and by respondent groups.

Figure 3.27 | Performance area IV: Knowledge management, mean scores by respondent group

KPI 18: Evaluating results

Finding 18: ADB was perceived by survey respondents as performing adequately in the evaluation of results. The document review rated ADB as very strong for the independence of its evaluation unit and strong for its evaluation coverage and quality assurance mechanism.

Figure 3.28 | KPI 18: Evaluating results, ratings of micro-indicators

MI 18.1 – Independent evaluation unit

Donors at headquarters rated ADB as strong for ensuring the independence of its evaluation unit. The majority (59%) rated ADB as strong or very strong, 19% as adequate, and only 8% as inadequate or below.

ADB was rated as very strong by the document review for having a structurally autonomous evaluation unit. The Independent Evaluation Department (IED), formerly known as the Operations Evaluation Department, was established in 2004 and clarified its independence in 2008 following recommendations in the report “Enhancing the Independence of the Operations Evaluation Department” (2008). The terms of IED’s independence are detailed in three key documents: 1) Review of the Independence and Effectiveness of the Operations Evaluation Department (2008), 2) Operations Manual Bank Policies – Independent Evaluations (2011), and 3) Guidelines to Avoid Conflict of Interest in Independent Evaluations (2012). The IED reports directly to the Board of Directors through the Development Effectiveness Committee (DEC) to which it submits evaluation work programmes, Annual Evaluation Reviews, Annual Report on Loan and TA Portfolio Performance, and select evaluation reports.[44]

MI 18.2 – Sufficient evaluation coverage of programming activities

This MI was assessed by document review only. ADB was rated as strong for its evaluation coverage, which is delineated in the Operations Manual. The Bank requires a self-evaluation of all completed projects and programmes, via the preparation of project completion reports (PCRs) for public sector projects and through expanded annual review reports (XARRs) for non-sovereign projects. IED then has the responsibility of reviewing and validating all of these reports, but has accumulated significant backlog and has not been able to reach that target in recent years. IED conducts in-depth independent evaluations for some projects (the Operations Manual stipulates about ten each year), preparing project or programme performance evaluation reports or technical assistance performance evaluation reports. There were many independent evaluation reports available for the countries sampled; between 2008 and 2012, there were 37 evaluations for Indonesia, 50 for Pakistan, and 57 for Viet Nam.

MI 18.3 – Quality of evaluations

This MI, assessed by document review only, focused on the existence and application of evaluation quality assurance mechanisms (i.e. it did not assess the quality of actual evaluations); ADB was rated as strong. According to ADB’s Operations Manual, the Bank attempts to ensure the quality of its evaluations by requiring that IED review and validate 100% of self-evaluations carried out for each completed project and programme. As noted above, however, in recent years the IED has been unable to meet this objective. COMPAS 2011 reported that IED validated 82% of PCRs (though it reviewed 100% of completed country partnership strategies). IED’s Self-Evaluation (2012) reported that its validations have fluctuated around 60% each year. It also reported that ADB was in full compliance with 20 of the 26 criteria of the Evaluation Co-operation Group’s Good Practice Standards for the evaluation of individual public and private sector operations, and partly in compliance with six criteria. Since 2008, IED’s Annual Evaluation Reviews (AER) have tracked ADB management’s actions on evaluation recommendations. Over the past three years, management has validated and taken action on an increasing share of IED recommendations on Level 2 and 3 results: 83% in 2009, 84% in 2010, and 94% in 2011. This may suggest increasing management agreement with recommendations and better quality of evaluations. The document review found no evidence of reports on the quality of organisation-wide evaluations; only country and project-level evaluations were reviewed for quality.

MI 18.4 – Use of evaluation findings to inform decisions

This MI was assessed by survey only. Donors at headquarters were asked whether ADB uses evaluation findings in its decisions on programming, policy and strategy. The majority (73%) rated ADB as adequate or strong, and 11% as inadequate or below.

MI 18.5 – Beneficiaries and clients involved in evaluations

This MI was assessed by survey only. Donors in-country and clients were asked whether ADB involves clients and beneficiaries in evaluations of its projects or programmes. The majority (70%) rated ADB as adequate or above, and 11% as inadequate or below. Clients rated the organisation more positively than donors in-country, and the difference was statistically significant.

KPI 19: Presenting performance information

Finding 19: Survey respondents believe that ADB adequately presents performance information related to its effectiveness. The document review considered ADB to be strong overall but noted room for improvement in reporting on outcomes and programming adjustments made at the country level.

Donors at headquarters were the only respondent group asked about the extent to which ADB presents performance information on its effectiveness.

The document review of this KPI was based primarily on the Development Effectiveness Review (DEfR), the Bank’s main document for reporting on achievements. The most recent DEfR is based on the Bank’s 2008 results framework and subsequent DEfRs will be based on the 2013 revised framework.

Figure 3.29 | KPI 19: Presenting performance information, ratings of micro-indicators

MI 19.1 – Reports on achievement of outcomes

The majority of donors at headquarters (78%) rated ADB as adequate or above on whether its reports to the Board provide a clear measure of the achievement of outcomes, while 11% gave ratings of inadequate or below.

The document review rated the Bank as adequate on this MI. In 2008, ADB adopted a results framework and corporate scorecard to guide the implementation of its corporate strategy and to track its progress towards priorities identified in Strategy 2020. The scorecard, along with other performance data, is published annually in the Development Effectiveness Review (DEfR) which examines ADB’s achievements at four levels: development outcomes in Asia and the Pacific, core outputs and outcomes, operational effectiveness, and organisational effectiveness. Although the DEfR describes outputs achieved, the extent to which it provides evidence of contribution to development outcomes is still limited. ADB’s reporting under level 2 of the DEfR does not establish clear linkages between outputs and outcomes achieved.[45]

The Bank has acknowledged this as a challenge, and has introduced improvements in its revised 2013 results framework.

MI 19.2 – Reports on performance using data obtained from measuring indicators

This MI was assessed by document review only which rated the Bank as very strong for reporting on performance using data obtained from measuring indicators. All DEfRs provide information on progress using data from the measurement of indicators defined for all four levels of results, and the indicators respect internationally recognised criteria for quality.[46] In addition to comparing baselines and targets for each indicator, DEfRs also present development trends over time.

MI 19.3 – Reports against corporate strategy, including expected results

Nearly half of donors at headquarters (46%) rated ADB as strong or very strong for reporting against its corporate strategy, 41% rated it as adequate, and 3% as inadequate or below.

ADB was rated as strong on this MI by the document review. The DEfR has been the Bank’s organisation- wide annual performance report since 2007. DEfRs report on progress towards expected results in both the development results framework (Levels 1 and 2), as well as the management results framework (Levels 3 and 4) and also provide a rationale for the achievement (or lack of achievement) of these results.

The ADB has conducted two reviews of its development results framework (one in 2010 and another, more comprehensive review in 2012) and the Independent Evaluation Department conducted reviews of the Bank’s management results framework in 2007 and in 2011. The 2011 IED review noted the quality and use of the DEfR as an effective corporate reporting tool and considered the quality of its analyses (i.e. output and outcomes analyses) as reliable. However, these reviews also noted room for improvement in organisation-wide reporting on results, such as the need to more adequately articulate the links in the results chain and the need for systematic data-updating.

MI 19.4 – Reports on aid effectiveness commitments using indicators and country targets

The majority of donors at headquarters (76%) rated ADB as adequate or above on whether ADB reports to the governing body on performance in relation to its Paris Declaration commitments.

ADB was rated as strong on this MI by the document review. ADB has consistently reported against its Paris Declaration commitments over the past several years. Since 2008, it conducted annual internal monitoring surveys in 25 countries in an effort to monitor nine Paris Declaration indicators (i.e. indicators 3, 4, 5a, 5b, 6, 7, 9, 10a and 10b). ADB’s “Aid Effectiveness Report 2011: Overall Achievements on Paris Declaration Commitments”comments on the extent to which the Bank had met the 2008-2010 Paris Declaration targets, based on these internal monitoring surveys. ADB also discusses its performance against Paris Declaration commitments in annual DEfRs under Level 3. While the Bank does not include country targets or show the extent of achievement towards commitments by country, country level data is available on the OECD/DAC website as well as in country reports produced by the OECD.

MI 19.5 – Reports on adjustments to policies/strategies based on performance information

This MI was assessed by document review only. It rated ADB as strong for reporting on adjustments made to policies and strategies based on performance information. While ADB does not have a policy that specifically defines how annual performance reporting will be carried out, the Bank’s 2008 Results Framework clearly stipulates that the Development Effectiveness Review will be prepared annually, serving as the key monitoring tool for Strategy 2020 and ADF operations. The results framework also delineates how reporting from DEfRs will be used to modify ADB’s work program and budget framework – a practice that was also noted in the 2011 “Special Evaluation Study on Managing for Development Results”. This study noted that performance information has been used to sharpen the discussions about strategy and to focus attention at the highest leadership level on improving future performance. The Bank has also used findings from the DEfR to prepare the work programme and budget framework and to link performance information and corporate planning more clearly.

ADB has also adjusted a number of organisational guidelines and policy- related documents based on performance information. For instance, the Bank adopted the Water Operational Plan 2011-2020 based on the 2010 IED review of the water policy and related operations. All DEfRs also generally contain a section on “actions” taken by management in response to Bank’s reported weaknesses in previous DEfRs. In the 2011DEfR, for instance, the Bank reports to have adopted an action plan to improve outcomes based on findings from the 2010 DEfR. The 2012 DEfR also provides an overview of past actions that ADB management committed to during the 2008-2012 period and the impacts that have resulted. The DEfR notes that “Management actions taken during 2008–2012 increased ADB‘s operational effectiveness through: (i) greater support for Strategy 2020 priorities, including gender mainstreaming; (ii) better project performance management (project readiness and implementation, and TA supervision); (iii) expanded cofinancing; and (iv) more effective knowledge management.(…) Other actions targeted stronger organizational effectiveness by refining ADB‘s human resources strategies and implementing them successfully (Our People Strategy and GAP III), increasing budget adequacy and efficiency, and streamlining business processes.”[47]

MI 19.6 Reports on programming adjustments based on performance information

This MI was assessed by document review only. ADB was rated as adequate for reporting on country-level programme adjustments based on performance information. The Bank does not have a policy that outlines how annual performance reporting at the country level is to be carried out, but, according to the Bank, ADB conducts annual missions to assess country portfolio performance and these reviews provide critical feedback on projects where adjustments have (or have not) been made based on the previous review. Given their confidential nature, portfolio reviews were not available for the MOPAN Assessment. There is evidence that other types of country-level assessments and reviews are used to report on changes and inform future country programming, however, these are not conducted annually. For instance, a new CPS is the main mechanism for reporting back to the Board and other stakeholders on changes made in programming as a result of performance information obtained from Country Assistance Performance Evaluations, Country Partnership Strategy validation reports, and other studies. The CAPEs and CPS validation reports are generally conducted at the end of the CPS cycle and their findings and recommendations are used to review and improve future CPS programming.

In addition to DEfR reporting at corporate level, Development Effectiveness Country Briefs (DECBs) are prepared for individual developing member countries and include a section on Operational Effectiveness: Improving Efficiency and Performance. This section generally describes lessons learned from past interventions and changes that have occurred based on performance information. The Country Brief for Pakistan for instance, mentions that “ADB has been improving the overall country portfolio performance through its quarterly and annual country portfolio reviews jointly conducted with the government. This helps weed out poorly performing projects, restructure and rectify slow-moving projects, and prevent automatic extensions of their loan closing dates.”[48]

However, DECBs are not produced regularly or annually for each developing member country.

The document review did not find evidence of budgetary changes made at the country level based on performance information.

KPI 20: Disseminating lessons learned

Finding 20: Survey respondents rated ADB as adequate overall in disseminating lessons learned; the document review considered the Bank a knowledge-based organisation that puts emphasis on lessons learned.

Figure 3.30 | KPI 20: Disseminating lessons learned, ratings of micro-indicators

MI 20.1 – Reports on lessons learned based on performance information

Donors at headquarters were asked whether ADB identifies and disseminates lessons learned from performance information. The majority (70%) rated ADB as adequate or above.

The document reviewed rated this MI as strong. Through its strategic frameworks dating back to 2001, ADB has consistently positioned itself as an organisation committed to the identification, documentation and dissemination of lessons learned and best practices. Its objective of becoming a “knowledge-based learning institution” was first emphasised in the Bank’s Long Term Strategic Framework 2001-2015, and again in Strategy 2020. ADB has also developed the Knowledge Management Action Plan 2009-2011 and, most recently, Knowledge Management Directions and Action Plan 2013-2015.

IED uses the Evaluation Information System (EVIS) and other evaluation products, such as success rates brochures, learning curves summaries, sector summation series, case studies, and IED’s Inquiry Desk to disseminate lessons learned from evaluations.[49] According to IED, one of the department’s main objectives is to “continuously improve its development effectiveness”; and “this is effected by feeding lessons, findings, and recommendations of past evaluations into on-going and future ADB operations.” The EVIS database is the platform used by IED to share evaluation information with clients.[50] Both the IED’s and the EVIS website were found to be easily accessible and quite resourceful.

Despite having such mechanisms in place, IED reports that evidence of use is often harder to find. Indeed, IED’s Special Evaluation Study on Knowledge Products and Services (2012) indicated that the use and application of ADB’s lessons learned and/or best practices were unclear – so much so that the study left this criterion as “not rated.”

MI 20.2 – Lessons shared at all levels of the organisation

This MI was assessed by survey only. Donors at headquarters were asked whether ADB provides opportunities at all levels of the organisation to share lessons from practical experience. Just over half of the donor HQ respondents (54%) rated ADB as adequate or strong, 16% as inadequate or below, and 27% answered ‘don’t know’.

KPI 21: Availability of documents

Finding 21: The document review considered ADB to be strong for the extent to which it makes its documentation available to the public.

The availability of ADB documents underlines the Bank’s commitment to transparency and information disclosure.

Figure 3.31 | KPI 21: Availability of documents, ratings of micro-indicators

MI 21.1 – Key documents available to the public

This MI was assessed by document review only, which rated the Bank very strong for making key documents available to the public. All of the Bank’s key documents (financial statements, strategies, policies, evaluations, etc.) are available on the organisation’s website. The only documents that are systematically available in languages other than English are project documents, country partnership strategies and the Information Disclosure policy, which is available in 13 languages. This was found to be in line with ADB’s Public Communications Policy (2011). The organisation’s website features a section devoted to Information Disclosure, which includes subsections on: information requests, appeals, simultaneous disclosure, translation framework, and contacts.

4. Main Findings: Evidence of ADB’s development results and relevance to stakeholders

4.1 INTRODUCTION

This section presents the results of the 2013 Common Approach assessment of ADB in measuring and reporting on development results. It includes three key performance areas:

• Section 4.2: Evidence of progress towards organisation-wide outcomes (KPI A)

• Section 4.3: Evidence of contributions to country-level goals and priorities, including relevant Millennium Development Goals (KPI B)

• Section 4.4: Relevance of the organisation’s objectives and programme of work to country-level stakeholders (KPI C)

The assessment of this component uses the same “traffic light” colours used in the organisational effectiveness component but applies a simplified 4-point scale. The methodology is explained in Volume II, Appendix I.

Figure 4.1 provides a summary of the assessment of the three KPIs. The detailed findings on each KPI are presented in the sections that follow.

4.2 EVIDENCE OF ADB’S PROGRESS TOWARDS ORGANISATION-WIDE OUTCOMES

4.2.1 Overview

This section presents the results of the assessment of ADB’s progress towards organisation-wide outcomes. KPI A suggests that an effective organisation should demonstrate progress towards organisation-wide, institutional outcomes.[51] These are usually related to the organisation’s strategic objectives. The assessment draws on the evidence that the organisation has available on its different result areas, primarily its reports on results.

ADB’s mandate and core functions[52]

ADB’s mandate, as articulated in the articles of agreement establishing the Bank, is “to foster economic growth and co-operation in the region of Asia and the Far East and to contribute to the acceleration of the process of economic development of the developing member countries in the region, collectively and individually.” Its overarching vision of an Asia and Pacific region free of poverty is highlighted in the 2008-2020 long-term strategic framework, referred to as Strategy 2020, under which the Bank pledges to support three critical strategic agendas of the region: inclusive economic growth, environmentally sustainable growth, and regional integration.

Strategy 2020 focuses on five core areas[53] that “best support its agenda, reflect DMCs’ needs and ADB’s comparative strengths, and complement efforts by development partners”, namely: infrastructure, environment, regional co-operation and integration, financial sector development, and education. Under Strategy 2020, ADB envisions positioning itself more strategically as a partner and agent of change by promoting issues related to: gender equity, private sector development, good governance and capacity development, knowledge, and partnerships.

Figure 4.1 | Development results component – overall ratings

ADB’s results measurement and reporting at an organisation-wide level[55]

In 2008, ADB became the first multilateral develop- ment bank to adopt a corporate-wide results framework and also introduced the Development Effectiveness Review (DEfR), an organisation-wide report on the Bank’s contributions to development results in Asia and the Pacific based on data from monitoring the performance indicators in its corporate results framework.

The 2008 results framework was organised in four levels, included about 77 indicators to measure the Bank’s performance in the areas outlined in Strategy 2020, and set intermediate targets to be achieved by 2012. The four tiers of results were organised as follows:

• Level 1 performance indicators were used to track overall development progress through selected poverty reduction and development outcome indicators to which ADB aims to contribute. The Bank noted that the outcomes at this level could not be attributed to ADB alone but, rather, resulted from the collective action of develop- ment partners over the long term.

• Level 2 indicators aimed to measure ADB’s contribution to country and regional outcomes by aggregating key outputs delivered to developing member countries (DMCs) through ADB projects in priority sectors.

• Levels 3 and 4 included indicators for monitoring operational and organisational effectiveness, respectively. [56]

As noted in Chapter 3 (KPI 3), ADB refined the framework in 2010 and conducted a comprehensive review in 2012 which led to the adoption of a revised framework in January 2013 (see sidebar).

The Development Effectiveness Review (DEfR) is the Bank’s main vehicle for communicating performance information. Published annually, and now in its 6th year of publication, the DEfR reports on all indicators identified in the Bank’s results framework. It includes a corporate performance scorecard that uses a traffic light system to indicate areas where the Bank has made progress towards targets, areas where performance needs to improve, and areas where corrective measures are needed.

Data used for this assessment

The assessment of KPI A was based on survey data from donors at headquarters and on a review of documents made available by the Bank. The document review considered available performance information (organisation-wide level) from this strategy period (2008-2020). Attention was paid to the following elements: quality of results statements, including indicators, baselines and targets; the quality and consistency of evidence presented to substantiate the results achieved, including an assessment of contribution; and the evidence of progress towards organisation-wide outcomes reported by the organisation.

The assessment drew on results presented in the 2012 Development Effectiveness Review, particularly from level 2, which assesses the Bank’s contribution to development results in the region. It should be noted that the 2012 DEfR is based on the 2008 results framework. Hence while some sections in this chapter acknowledge improvements made in the 2013 results framework, the analysis in this chapter is based on the documentary evidence of results achieved and as such, relies on the most recent DEfR report (2012), which reflects the corporate results framework approved in 2008.[57] The assessment also considered sector reports, thematic evaluations, and a series of operational plans and their results frameworks that have been developed to guide the work of the Bank in water, transport, finance, and education sectors.

For an overview of the key documents consulted for this assessment, please refer to Volume II, Appendix VIII.

4.2.2 Evidence of extent of progress towards organisation-wide outcomes

This section provides an overall rating for KPI A based on documentation made available by the organisation and survey data. It also includes assessments of: the quality and consistency of reports on organisation- wide results and on the extent of progress in ADB core areas, which draws on a review of ADB’s performance information and reviewed survey data from donors at headquarters for each of these areas.

Overall assessment

Under Strategy 2020, the Bank notes that in order to“maximize results, efficiency, and impact”it will“employ its financial and institutional resources”in five core areas: infrastructure, environment, regional co-operation and integration, finance sector development, and education.[58] The assessment of KPI A, therefore, focuses on the extent to which ADB has made progress towards achieving results in these areas.

Figure 4.2 shows the overall rating for this KPI based on evidence of progress towards results in core areas – as expressed in ADB reports and as indicated by surveyed stakeholders.

When questioned about ADB’s performance in the five core areas highlighted in Strategy 2020, MOPAN donors at headquarters rated the organisation adequate overall and had positive views about ADB’s contributions to infrastructure development and regional co-operation and integration (RCI), both of which received strong ratings. ADB’s reporting under level 2 of the DEfR provided adequate evidence of achievement of planned outputs outlined in the Bank’s results framework. Recent evaluation studies also highlighted its contributions to select areas such as infrastructure development, which represent an important proportion of the Bank’s portfolio. That being said, evidence of contribution to higher-level change in the region (i.e. outcomes, impacts) remains scarce.

Figure 4.2 | KPI A: Evidence of the extent of progress towards results in core areas, overall rating

Overall Rating: Adequate

How ADB reports on organisation-wide results

Finding 22: ADB’s reports provide a sound assessment of organisation-wide progress towards core sector outputs and demonstrate the Bank’s efforts to increase reliability of data. Although current reports do not allow for a clear assessment of its contributions to outcomes, ADB has modified its results framework in 2013 to facilitate more outcome- oriented measurement and reporting in the future.

Figure 4.3 presents an assessment of the quality of the organisation’s systems and practices for measuring and reporting on its organisation-wide results. The criteria represent elements of good results reporting (see Volume II, Appendix I for full descriptions). The organisation is assessed according to whether it has met, partially met, or not met the criteria for each core area.

Figure 4.3 | ADB’s measurement and reporting on core areas of specialization

Strategic planning and theory(ies) of change

When assessing whether ADB’s planning documents contain a basic theory of change, the assessment team looked for the presence of specific elements that could constitute an “acceptable” theory of change. This included a thorough discussion of the main problems observed in a given sector/thematic area, an explanation of the actions or activities the organisation proposes to undertake to address those problems, with a description of the changes it hopes to bring about and the assumptions and potential risks related to that change occurring. The assessment team also looked for a robust results framework in which specific outcome statements are linked directly to operations or activities that would contribute to those outcomes.

The Bank presents elements of a theory of change in some of its planning documents (e.g. Strategy 2020 and select operational plans). Strategy 2020, for instance, provides a good overview of its vision for the Asia and Pacific region and clear indications of the directions the Bank intends to take to achieve this vision, which include greater selectivity and focus in areas where the Bank considers that it has a comparative advantage. Strategy 2020 is supported by a results framework that is used to monitor and evaluate corporate performance and that includes appropriate indicators, baseline data and targets.

The Bank has also developed several sector/thematic plans to guide its operations and ensure alignment with Strategy 2020. According to ADB, these plans clarify how the broad directions and targets of Strategy 2020 will be implemented and achieved.[64] The operational plans reviewed[65] provided a good overview of the challenges the region faces in each thematic/sectoral area and presented ways the Bank’s assistance will help to tackle those challenges. Some of the plans (such as the 2013 Environment Operational Directions) contained a fairly good results framework, with solid results logic and assumptions and risks; they contained a basic theory of change. The quality of the plans was inconsistent across the thematic areas, sometimes lacking a results framework (e.g. Financial Sector and Sustainable Transport Initiative Operational Plans).[66]

Therefore, although some of the Bank’s planning documents have elements of a basic theory of change, ADB would benefit from fully articulated theories of change especially at the sector levels; providing an explicit explanation of how a set of low level results or accomplishments (i.e. lending and non-lending activities and outputs) is intended to produce medium to long-term results (i.e. impact and outcomes). This would allow stakeholders to have a better understanding of the rationale behind ADB’s approaches and have a better sense of how expected results are linked from one level to the next (e.g. country to organisation-wide, sector to organisation-wide).

Measuring results

As noted in the assessment of MI 3.2 and MI 19.1 (Chapter 3), until recently, the assessment of ADB’s contribution to development results was hampered by:

• Limited number of outcomes indicators: Until 2013, ADB’s results framework included very few outcome-level indicators in level 2.

• Reporting on outcomes: ADB’s reporting under level 2 of the DEfR (through 2012) was not based on actual outcomes. While the DEfR reporting on core sector outputs aims to“capture the quantity of physical outputs delivered or expected to be delivered, and the number of people benefiting or expected to benefit,”[67] the reporting on progress towards outcomes relies solely on an effectiveness rate attributed to Bank’s projects at completion (through PCRs) by the Bank’ staff in operations departments.[68]

• Inconsistencies in the sampling methodologies: Different project samples were used to assess achievement of outputs and progress towards outcomes. Because projects in the two samples were approved during different periods and were at different stages of implementation, few projects were actually comparable. In the Special Evaluation Study on MfDR conducted in 2011, IED indicates the following:

“There is a disconnect between the methodology adopted for measuring/reporting outputs and measuring/reporting outcomes. Outputs reported may be a subset of the total outputs expected from a project, which at times may reflect mainly physical outputs rather than difficult to achieve–non- tangible outputs—(…).’ However, the DEfR methodology uses as outcomes, evaluation ratings of effectiveness relating of all outputs in the project, which includes both access/coverage-related outputs and other outputs (...). If only the former output is achieved, but other outputs are not achieved, output measure may be high, while the outcome measure may be low for the same project. Another disconnect between outputs and outcomes is that outputs assessment includes anticipation of outputs not yet completed, but outcomes assessment concerns finished projects only.”[69]

According to the 2011 DEfR, such inconsistencies in sampling methodologies affected the Bank’s ability to establish clear linkages between successful delivery of outputs and achievement of outcomes.[70] The Bank has acknowledged and sought to overcome this challenge in measurement and reporting on output and outcomes under level 2; and the 2012 DEfR reports stronger linkages between successful delivery of outputs and achievement of outcomes.[71]

• Credibility of effectiveness ratings: Until recently ADB’s reporting under level 2 of the DEfR relied on an aggregation of results based on assessments of projects (i.e. success ratings) conducted by its own operations departments at the time of project completion (self-reported assessments). In an effort to “improve accountability for achieving results, quality of completion reports, and independence of project/program ratings,”the Bank introduced in 2008 a project validation system that requires PCRs to be reviewed and validated by a quality control reviewer in the IED.[72] The system has been applied over the years and led to the introduction of ‘adjusted success ratings’ of ADB projects. In the 2012 DEfR, the ADB introduced ‘adjusted effectiveness ratings’. The adjusted ratings aim to reflect the differences in effectiveness ratings given by the Bank’s operations department and by IED and take into account findings from IED’s validation and evaluation reports.[73] These changes reflect the Bank’s commitment to improve the data used in corporate results reporting, but has resulted in a complex approach (using adjusted and actual ratings) for making judgments on progress. In the 2012 DEfR, the Bank further commits to “strengthen quality control of project completion reports within operations departments; and clarify expected standards by updating guidelines for project completion reports, IED validation reports and project performance evaluation reports; as well as ensure a consistent approach to rating project success across these three tools”.[74]

Reporting and communicating results

As indicated in KPI 19, the main report used to communicate results and monitor corporate perform- ance at the Bank is the Development Effectiveness Review (DEfR), which has been published on an annual basis since 2008. According to interviews with ADB staff, the DEfR is now considered a key corporate planning and decision-making tool. Since 2010, the DEfR process has become more embedded in the corporate planning process of the Bank which now requires that the DEfR be prepared early in the year (i.e. usually in March) to ensure that its findings feed into ADB Planning Directions, which in turn are used to define /design the three-year Work Programme and Budget Framework (WPBF).

While the DEfR provides a significant amount of information on operations, performance information is not agregated or presented in a way that allows for a clear identification of ADB’s contribution to results at the outcome level. The performance information presented in the DEfR to date aggregates results solely at the output level. This is a challenge that many multilateral organisations, including multilateral development banks, are facing.

In 2013, the Bank introduced a new online system for communicating/ presenting information on its performance. The focus is still at the output level, but aggregates results of ADB-supported operations since 2004 and up to the year 2012. The results are based on self-reported data (i.e. PCRs) and reflect output quantities that have been verified/validated by the respective governments during the production of ADB Project Completion Reports. Navigating the Bank’s website allows one to see how outputs generated by ADB-supported projects are aggregated at the country level and then at the organisational level. Thus, although ADB has limited evidence of contributions towards outcomes, it has a solid evidence base of completed outputs.

Evidence of the extent of progress in ADB’s core areas of specialisation

Finding 23: Surveyed MOPAN donors at headquarters consider that ADB is making progress towards achieving results in select core areas of specialisation particularly in infrastructure development and intraregional co-operation, where it is considered to have made strong contributions. The document review found evidence of the achievement of planned outputs in some of the areas assessed, but little evidence of contribution to outcomes.

The survey asked whether ADB is making contributions to improving/developing sector and thematic areas identified as “core areas of specialisation” in Strategy 2020, in developing member countries (DMCs). A total of 53 MOPAN donors at headquarters responded. The Bank received ratings ranging from adequate to strong in all five areas. Infrastructure and regional co-operation and integration received strong ratings and ADB’s contributions to all sectors under infrastructure development (i.e. transport, energy and water) were considered particularly strong.

Figure 4.4 below shows the mean scores of surveyed MOPAN donors at headquarters. The survey used the same 6-point scale used in the assessment of ADB’s practices and systems for organisational effectiveness.

Figure 4.4 | Extent of progress in ADB’s core areas of specialisation, stakeholder survey

Very weak = 1.00–1.49
Weak = 1.5–2.49
Inadequate = 2.5–3.49
Adequate = 3.5–4.49
Strong = 4.5–5.49
Very strong = 5.5–6.00

The review of ADB’s DEfR provided evidence of progress in outputs delivered in most of the core areas of specialisation where the Bank considers it has a comparative advantage (i.e. infrastructure, finance, and education). Evidence of ADB’s con- tribution to outcomes was also found in the evaluations of select sector/thematic areas con- ducted by the Bank’s independent evaluation department (IED). There was limited evidence of results achieved in multidimensional areas such as environment and regional co-operation and inte- gration, although interestingly the latter is an area where survey respondents perceive that the Bank is making strong contributions.

The following sections provide a discussion of results achieved (mostly outputs) in ADB’s core operational areas as reported in the 2012 Development Effectiveness Review (DEfR).

As pointed out in the 2012 Annual Evaluation Report, the data presented in the DEfR is limited to the number of outputs and beneficiaries benefitting from those outputs, with scant information on the changes that these outputs have brought about (see sidebar).

The sidebars in the following sections provide the 2004-2012 aggregate data on outputs, based on the aggregated output exercise on the Bank’s website.

Please refer to Volume II, Appendix IX for details on outputs delivered in select core operational areas of the Bank, as reported in the 2012 Development Effectiveness Review (DEfR).

MI A.1: Infrastructure

The assessment of ADB’s reported contributions to infrastructure development focused on three main sectors of investment (i.e. energy, transport and water) and the extent to which ADB has contributed to the development of these sectors in the region.

Transport

In the transport sector, ADB provides assistance to DMCs in building transport infrastructure and services that contribute to low-carbon, safe, accessible, and affordable transport systems.[76] In its efforts to align its transport sector investments to Strategy 2020, ADB approved in 2010 the Sustainable Transport Initiative (STI) which focuses on strengthening investment in the subsector areas of urban transport, climate change and transport, cross-border transport and logistics, and road safety and social sustainability.[77]

MOPAN donors at headquarters were asked to assess the extent to which ADB is making contributions to creating safe, affordable and environment friendly transport systems in developing member countries. The majority (59 %) rated ADB as strong or above.

The DEfR reports on four output indicators directly related to infrastructure development in the transport sector. Level 2 reports on outputs delivered in a specific targeted time frame (i.e. 2009-2012). ADB reported that, in comparison to the 2008-2011 period, outputs delivered during 2009-2012 had decreased for all four transport indicators. The DEfR explains the variance in relation to these indicators by indicating that “the decrease was because projects completed in 2012 were smaller in financial and output delivery terms of than those completed in 2008.”[78]

The DEfR reports that during the 2009-2012 period, more than 367.5 million people benefited from roads projects financed by ADB, while the most recent results aggregation exercise, which presents the total outputs achieved since 2004, indicates that as of January 2013 close to 1 billion people had benefited from roads projects financed by ADB (see sidebar).

Although ADB’s reporting is limited to the output level, level 2 of the DEfR also contains brief synopses of ADB’s contribution to development outcomes in select countries based on findings from select project completion reports (self-reported assessments). It reports for example, that “The West Bengal Corridor Development Project in India helped improve connectivity between Bangladesh, Bhutan, India, and Nepal by cutting travel time by about 40% on average for the subprojects and reducing vehicle operating costs by 25%–30%.”It further highlights that “The rehabilitation of rural roads funded by the same project also improved villagers’ access to schools, hospitals, and markets; and helped raise household incomes by 10%–15% per year after the project roads were completed.”[79] However, the credibility of this data remains questionable given that it is based on a self-reported assessment that has not been validated by the IED. The issues of reliability and credibility of data are evident in the discrepancies in PCR ratings given by regional departments and the IED, as noted in the 2012 DEfR.[80]

A recent regional evaluation study conducted by IED provided some evidence of ADB’s contribution to the development of transport infrastructure in Pacific DMCs.[81] The study assessed ADB’s support to the transport sector (particularly in the roads, maritime, and civil aviation subsectors) and highlighted the impacts of a few ADB-financed road projects, including: improved market access (e.g. reduced user costs of travel, better access to schools, medical facilities, and other services); facilitation of trade activities; and impact on transport safety (e.g. decrease in road accidents, fatalities and serious injuries).[82] The evaluation also highlighted the socioeconomic benefits of a road sector improvement project in Timor-Leste that recorded “a 20% increase in visits to health facilities, a 56% increase in food trade volume, and a 2.7% increase in job opportunities.”Although the IED rates ADB’s support to transport infrastructure in Pacific DMCs as successful, it also notes limitations to the study and in particular the lack of data to assess performance.”[83]

In the next DEfR, ADB will report on two new outcome-oriented indicators related to transport (included in the 2013 results framework): use of roads built or upgraded, and use of railways built or upgraded.

Energy

ADB’s work in the energy sector is guided by the 2009 Energy Policy, which places emphasis on three key areas: promoting energy efficiency and renewable energy; maximising access to energy for all; and promoting energy sector reforms, capacity building, and governance.[84] A review of the “Summary of ADB Clean Energy Projects”covering the 2003-2011 period indicated a steady increase in clean energy investments of the Bank which rose from $ 225.9 million in 2003 to $2.1 billion in 2011.[85] In 2012, the Bank reports it invested close to $2.3 billion in clean energy investments, with the largest investment in solar photovoltaic technologies ($247 million).[86]

When asked whether ADB is making contributions to improving access to cleaner, renewable sources of energy in developing member countries, the majority of MOPAN donors at headquarters (65 %) rated ADB as strong or very strong.

The DEfR reports on five output indicators related to infrastructure development in the energy sector. The 2012 DEfR reported that delivered outputs during 2009–2012 increased for two energy indicators (installed energy capacity and greenhouse gas emission reduction) but did not report on any variance with regard to the planned outputs. The 2012 DEfR also highlighted that during 2009-2012, ADB supported the electrification of 2.3 million new households by installing or upgrading 84 000 km of transmission and distribution lines.

The most recent results aggregation exercise conducted by the Bank covering 2004-2012 indicates that as of January 2013 ADB had supported the electrification of close to 5 million new households by installing or upgrading close to 140 000 km of transmission and distribution lines since 2004 (see sidebar). However, this data is not accompanied with information on the actual effects of ADB’s contributions.

Water

MOPAN donors at headquarters were asked two questions on this area. When asked whether ADB is making contributions to improving access to potable water in DMCs, 59% of respondents rated ADB as strong or above. When asked whether ADB is making contributions to improving sanitation and waste management services, 49% rated it strong or above.

The 2012 DEfR reports on five output indicators directly related to the development of water infrastructure. It indicates that the delivered outputs during the 2009–2012 period had increased notably for all water indicators but does not provide an explanation of any variance related to these indicators. It also reports that “ADB projects provided better access to clean water for 6.2 million households” and that “irrigation, drainage, and flood management programs improved 22 million hectares of land.”[87] The Bank does not provide information on outcomes or impacts related to these outputs. The output data available on the Bank’s website indicates that as of January 2013 ADB projects provided better access to clean water for more than 8.6 million households during 2004-2012 (see sidebar).

A recent study conducted by IED noted that the water lending programme had generally been effective in achieving the intended objectives during the 2001-2009 period, particularly in the area of hydropower. However, the sustainability of project results was questioned – in other words, the initial outputs delivered could not be maintained due to operation and maintenance issues. Sustainability issues affect the contributions to desired outcomes.[88]

In its revisions to the corporate results framework 2013, ADB has incorporated several more outcome- oriented indicators for the water sector, reflecting a greater emphasis on improved services (24/7 water supply at right pressure, reliable irrigation services) rather than engineering outputs highlighted in the side bar. The data on these indicators, together with a stronger thematic strategy (Water Operational Plan 2011-2020) and results framework should provide a stronger foundation for assessing performance in the future.

MI A.2 Environment

Donors at headquarters were asked to assess the extent to which ADB is making contributions to promoting sound environmental and natural resource management, and to strengthening environmental safeguard systems of DMCs. Respondents rated ADB as adequate or above on both questions (76% and 70% respectively).

The 2012 DEfR does not report on the Bank’s contributions to this thematic area. Current reporting under level 2 solely analyses the extent to which ADB’s operations integrate environmental sustainability as a priority theme, while reporting under level 3 (on operational management) tracks the proportion of financing allocated to the operational areas of the Bank.

However, as noted in the 2012 Review of ADB’s Results Framework, ADB’s contribution to this multidimensional area is captured in other sectors of operations of the Bank, namely the energy, transport, and water sectors. This has been clarified in the 2013 revised framework, which now includes seven indicators that will be used to assess ADB’s contribution to environment. In addition, ADB has developed the Environment Operational Directions 2013-2020 (2013) to guide the Bank’s work in this area. Both of these developments should improve performance information on environment in the future.

MI A.3 Regional co-operation and integration

The survey highlighted two aspects of regional integration – trade and co-operation. The majority of donors at headquarters (65%) rated the organisation as adequate or above for contributions to promoting intraregional trade in DMCs, and 59% gave ADB strong ratings on contributions to foster intraregional co- operation.

Although donors perceive that the Bank is making contributions to regional co-operation and integration (RCI), there is no formal corporate reporting on results in this area. Current reporting under level 2 looks at the extent to which ADB’s operations integrate RCI as a priority theme, while reporting under level 3 tracks the proportion of financing allocated to this specific area.

The Bank has added two outcome-oriented indicators under level 2 of the 2013 results framework in order to capture results contribution:[89] cross-border transmission of electricity (gigawatt-hours per year) and cross-border cargo volume facilitated (tons per year). It is expected that ADB’s contributions to outcomes in this area will be reflected in the 2013 DEfR. IED’s on-going evaluation of the Bank’s work in RCI (expected for 2014) should help shed some light on the level of contribution in this area (see Chapter 3, KPI 4).

MI A.4 Finance sector development

The survey focused on five areas of financial sector development – basic infrastructure, institutional capacity, access, regulation, and transparency. Overall donors at headquarters rated ADB as strong or above for contributions to developing basic financial infrastructure in DMCs, and the majority rated the organisation as adequate or above in the remaining four areas. Questions related to this sector generated high proportions of ‘don’t know’ responses from donors, particularly questions pertaining to micro and macro prudential regulation of financial institutions, as well as contributions made to promote the principles of transparency and accountability of financial institutions.

The 2012 DEfR reports on two output indicators related to finance sector development. Reporting under level 2 of the DEfR indicates that in comparison to the 2008-2011 period, outputs delivered during 2009-2012 decreased for all finance indicators. The DEfR explains the variation in these indicators and indicates that “the decrease was because projects completed in 2012 were smaller in financial and output delivery terms than of those completed in 2008.”[90] In the DEfR, the Bank reports that it helped close to 272 300 borrowers start their own businesses or improved existing ones between 2009 and 2012. ADB’s reporting does not however provide information on outcomes or impacts related to these outputs.

A recent special evaluation study conducted by IED assessed ADB’s support in developing sustainable and inclusive microfinance systems in DMCs, one area of finance sector development.[91] The study notes that micro-finance operations approved during the 2000–2010 study period were less than effective in achieving results. It explains that “The low score for results was due mainly to the weak development of support institutions and infrastructure, less than effective support in achieving institutional sustainability, and limited outreach to the poor. ADB’s support was focused mostly on addressing macro-level issues on the supply side and much less at institutional development and micro-level issues on the demand side.” Other studies note that affecting client welfare (in other words going beyond the delivery of loan accounts/reaching end borrowers) is still a challenge for micro-finance operations.[92]

As with the other core sectors, this type of performance information does not provide a clear picture of the Bank’s contribution to financial development in DMCs. The improvements in the 2013 results framework include outcome indicators for some of the dimensions of finance sector development. There is now an operational plan for the financial sector (2012), but it does not include a results framework.

MI A.5 Education

ADB views education as a “right, not a privilege” and in the past years has focused on assisting DMCs in achieving the goal of quality education for all. According to a recent publication of the Bank, over the past four decades (1970-2010), ADB has provided USD $8.2 billion in support to DMCs for the development of the education sector. Of this, 39% went to pre-primary and basic education, 21% to technical education and vocational training, 12% to upper secondary education, and 11% to tertiary and higher education.[93]

In an effort to meet the changing needs and priorities of its DMCs in the education sector, ADB has programmed US $2 billion to support education development in the region for the 2012-2014 period.

MOPAN donors at headquarters were asked to assess the extent to which ADB is making contributions to improving 1) the quality of basic and secondary education, 2) the quality of technical and vocational education and training, and 3) the quality of tertiary education in DMCs. ADB received ratings of adequate or above in all areas.

The 2012 DEfR reports on three output and outcome-oriented indicators directly related to the development of the education sector. It reports that during the 2009-2012 period there were greater increases in the numbers of teachers trained and classrooms built than in the number of students benefiting from school improvement programmes. It does not explain the reasons for these differences. According to the 2012 DEfR, more than 11.4 million students had benefited from 265 300 new classrooms and other school improvement programmes. The aggregated outputs for the period 2004-2012 are shown in the sidebar.

Although the DEfR contains a very brief synopsis under level 2 of ADB’s contribution to development outcomes at the country level (based on PCR findings), ADB’s reporting on contribution to the development of this sector in DMCs remains limited to the output level. Level 2 of the DEfR for instance gives a brief overview of ADB’s contribution to development outcomes in Uzbekistan and notes that Bank- supported projects have helped “improve quality and create equitable access to basic education for students in rural and poor areas through greater use of information and communication technology”(…); and “improve the average level of knowledge of students to 81% in mathematics, language, and science against the baseline of 71% before project implementation.”[94] However the reliability of this data remains questionable given that it is based on self-reported assessments that have not been validated by the IED.

The revised 2013 corporate results framework introduces three new sex-disaggregated indicators, of which two are more outcome-oriented. The new indicators in the Education Operational Plan (2011) provide a basis for better measurement and reporting and results.

4.3 EVIDENCE OF ADB’S CONTRIBUTION TO COUNTRY-LEVEL GOALS AND PRIORITIES, INCLUDING RELEVANT MDGS

4.3.1 Overview

This section presents the results of the assessment of evidence of ADB’s contributions to country- level results and relevant MDGs. By separating the KPI at the organisation-wide level from KPIs at the country level, MOPAN recognises the demand-driven nature of many of the activities of a multilateral organisation and the key role that is played by its country programming or strategy document, where expected results at the highest level (outcomes and impact) reflect a shared responsibility between the multilateral organisation and the partner country.

Section 4.3.2 examines evidence of the organisation’s contribution to country-level goals and priorities, including relevant MDGs.

(Note: Section 4.4 examines relevance in terms of the extent to which ADB clients and donors in-country believe the organisation supports country priorities and meets changing needs.

ADB’s results and reporting at country level[95]

ADB’s work at the country level is based on a three to five-year country partnership strategy (CPS) that outlines its strategic approach in client countries. A CPS is prepared in collaboration with each developing member country (DMC) and is accompanied by a 3-year country operations business plan (COBP).[96] The CPS is generally based on and aligned with a country’s own development plan and poverty reduction goals, as well as Strategy 2020 priorities.

As indicated in KPI 5, since the 2010 MOPAN assessment, ADB has revamped the CPS to show: i) clear alignment between national goals and priorities and the Strategy 2020’s five core areas; and, ii) its clear intention to support the five key drivers of change of the Bank.[97] In the CPS results framework, expected results are kept purposefully broad and set at the outcome level. Details on outputs that are meant to lead to the achievement of outcomes are found in the sector assessments linked to the CPS, which contain a detailed results framework of their own.

The Bank communicates its results at the country level through a number of key documents. These include: project completion reports (PCRs)[98], project/programme performance evaluations (PPERs), technical assistance completion reports, country portfolio review missions (CPRMs) and development effectiveness country briefs (DECBs) produced as part of the CPS monitoring process; and the CPS final reviews, country assistance performance evaluations (CAPEs), CPS final review validation report, and sector assistance performance evaluations (SAPEs) conducted at the final evaluation stage of a CPS.

While the CPS final reviews are self-assessments provided by the Bank’s regional departments, the CAPEs and SAPEs are independent evaluations conducted by the Bank’s evaluation department.[99] The primary mechanism for reporting on country-level performance, and the foundation for assessment of this KPI, is the CPS final review, CAPE and SAPE. For a list of documents consulted, please see Volume II, Appendix VIII.

Data used for this assessment

The assessment of ADB’s reported contributions to goals and priorities at the country-level is based on data from the three countries selected for the 2013 MOPAN survey (Indonesia, Pakistan and Viet Nam).

All results-related information provided in documents from the most recently completed programming cycle in the three countries sampled was reviewed. This also included any external reviews, assessments or evaluations carried out during that programming cycle.

Attention was paid to the following elements: quality of the results statements; the relevance of indicators, baselines and targets; the strength of the link between results statements and results achieved; the quality of evidence presented to substantiate the results achieved, including an assessment of contribution; and, the overall performance story.

In the survey, ADB clients and MOPAN donors in-country were asked questions that were tailored to each of the three countries. Interviews with senior ADB staff in Resident Missions also informed the analysis of context and ensured that the assessment team had a full set of documentation with which to conduct the document review and analyse results.

4.3.2 Evidence of contribution to country-level goals and priorities, including relevant MDGs

This section provides an overall rating for KPI B based on documentation made available by the organisation and survey data. It also includes a brief assessment of ADB’s measurement and reporting on country level results and a more detailed assessment of the extent of progress towards results defined by ADB in its country- level results frameworks.

Overall assessment and rating

ADB’s data on country level results achievement is fragmented in different reports and most of these were available for the countries selected for this assessment. To the extent possible, the assessment also drew on performance information collected from evaluations conducted by the Independent Evaluation Department of the Bank (IED). The majority of reports reviewed generally provided evidence of outputs achieved and, in some instances, the link to their intended outcomes.

The review of ADB’s reports provided a mixed picture of its contributions to country-level goals and priorities. While some reports provided a good overview of the Bank’s contribution in country (e.g. Viet Nam), in other reports, it was more difficult to gauge the magnitude of the Bank’s contributions at country levels (e.g. Pakistan).

Surveyed stakeholders also provided mixed views about ADB‘s progress towards its stated results in the selected priority areas from each of the three country partnership strategies CPS.

Figure 4.5 shows the overall rating for this KPI based on the review of ADB’s reports on country-level results and as indicated by surveyed stakeholders.

Figure 4.5 | KPI B: Evidence of the extent of contribution to country-level goals and priorities, rating

Overall Rating: Adequate

Although ADB was found to be adequate overall, this rating is to be interpreted with caution given that the assessment was based on a small sample of ADB countries (3 out of 47 DMCs); and one of the countries (Pakistan) has undergone an entire portfolio restructuring and provided limited data of results achieved in the country.

It should be noted that conclusions from the 2011 Special Evaluation Study (SES) on Managing for Development Results (MfDR) conducted by IED, noted that there is room for improvement in the monitoring and reporting of outcomes and impacts at the country/sector and project levels. This study identified two issues that hampered the results orientation of country-level frameworks: the lack of reliable data, including baselines, in project design and monitoring frameworks (DMF) and the quality of CPS results frameworks.[100]

How ADB reports on country-level results (for all countries sampled)

Finding 24: ADB’s reporting practices provide mixed evidence on its contributions to country- level goals and priorities. Country-level reporting remains largely focused on the output-level and is also hampered by the lack of reliable data (targets and baseline information) against which to measure performance and assess progress.

Figure 4.6 presents an assessment of the quality of the organisation’s systems and practices for measuring and reporting on country-level results. The headings show the criteria MOPAN used to assess the systems and practices.

Figure 4.6 | ADB’s measurement and reporting on country-level results[101]

Strategic planning and theory(ies) of change

As indicated above, ADB’s strategic agenda at the country level are generally highlighted in the country partnership strategy. All CPSs sampled for this assessment provided a basic description/analysis of each country context, indicated the ways in which the Bank interventions were aligned with national priorities, and how it planned to contribute to country-level goals. As was done in the assessment of KPI A, the analysis of this key performance indicator included a review of country level results frameworks when looking for concrete examples of a theory of change in ADB’s planning documents.

Although some improvements were noted in the newest generation of CPS frameworks, those from earlier country strategies, which formed the basis of the analysis for this component of the assessment, contained a number of weaknesses: There were performance indicators at both outcome and output levels, however, some of the indicators and targets were overly ambitious (e.g. Pakistan); performance indicators weren’t always associated with data sources or data collection methods; and frameworks were often lacking baseline information or time-bound targets against which to reliably monitor and report on the progress towards higher-level results.

It is fair to say that, although all CPS and other planning documents reviewed contained some elements of a theory of change, they were not fully developed theories. Key elements, such as underlying assumptions related to expected results, were either missing or not clearly articulated.

Reporting and communicating results

Reporting of country-level results is found in six documents produced by the Bank: country portfolio review missions (CPRM), CPS final reviews (CPSFR), development effectiveness country briefs (DECB), country assistance performance evaluations (CAPE), CPS final review validation reports, and sector assistance performance evaluations (SAPEs). The first three reports are based on self-assessments of performance and are completed by operations departments. The final two documents, and most important sources of results information, are the CAPE and SAPE, which are external assessments of the Bank’s assistance in a given country or sector area generated by the Independent Evaluation Department of the Bank (IED).

The CAPE and SAPE generally cover two CPS cycles and evaluate the Bank’s performance based on six evaluation criteria (i.e. strategic positioning, relevance, efficiency, effectiveness, sustainability, and development impacts)[106] and triangulation of multiple sources of data (i.e. self and independent evaluations of ADB’s country projects and programmes).[107] None of the countries under review had conducted recent CAPEs or SAPEs, thus the information on results included in the following section is sometimes dated.

For every CPS cycle where a CAPE is not scheduled, the IED conducts a CPS final review (CPSFR) validation report to assess the quality of the CPS final reviews. IED highlights instances where achievement indicated by the CPS final reviews are vague and could be improved as well as results achieved that are unclear. It also acknowledges the strengths of the CPS final reviews.

Evidence of the extent of progress toward ADB results at the country level

The assessment of ADB’s contribution to country-level goals and priorities was largely based on the analysis of documents produced for the most recently completed planning cycle. This included an analysis of reports, reviews and evaluations covering past country partnership strategies, namely: the 2006-2009 CPS for Indonesia, 2007-2010 CPS for Viet Nam, and the 2004-2006 CPS for Pakistan. The team also reviewed documents produced by the Bank covering the on-going country partnership strategies: the 2012-2014 CPS for Indonesia, 2012-2015 CPS for Vietnam, and the 2009-2013 CPS for Pakistan. A sample of sectors and subsectors of continuing priority for the Bank were selected from past and current CPS, based on the assumption that sectors of on-going relevance have more investments over the years and are more likely to show evidence of results. The survey questions were based on this assumption and were developed in consultation with Bank staff at headquarters and in resident missions (RMs).

VIET NAM

Finding 25: Survey respondents perceived ADB to have made strong contributions to the development of the water and transport sectors in Viet Nam. The review of the Bank’s documents indicated that ADB’s contribution to the attainment of Viet Nam’s goals and priorities has been mixed.

The assessment of ADB’s contribution to Viet Nam’s goals and priorities was based on the reviews and evaluations conducted for the 2007-2010 Country Strategy and Program. These included the 2007-2010 Country Strategy Mid Term Review, 2007-2010 Country Strategy Final Review, 2009 Sector Assistance Program Evaluation (SAPE) on Urban Services and Water Supply and Sanitation Sector, 2009 Country Assistance Program Evaluation (CAPE), as well as the 2007 and 2011 Development Effectiveness Country Briefs (DECB).

The survey questions were based on an analysis of the 2007-2010 Country Strategy and Program, as well as the on-going 2012-2015 CPS and focused on the following sectors: transport, energy, finance, education and water. ADB clients and donors in Viet Nam were asked whether the Bank had effectively contributed to the development of these sectors in the country and 44 responded. Overall, the surveyed respondents rated ADB strong in two sector areas (transport and water supply/other municipal infrastructure) and adequate in three (energy, finance and education). There were high proportions of ‘don’t know’ responses on questions related to ADB’s work in the energy, finance and educations sectors in Viet Nam.[108] Overall, MOPAN donors in-country seemed less knowledgeable about the Bank’s work in those areas.

The evidence of ADB’s contribution to country-level results was largely provided by independent evaluations conducted by IED. These included the 2009 CAPE, which assessed ADB’s assistance in the country between 1999 and 2008, and the SAPE, which assessed ADB’s assistance to Viet Nam’s urban services and water supply and sanitation sectors during the same period.

The CAPE, which rated ADB’s interventions in Viet Nam as successful,[109] confirmed the relevance of ADB’s strategies and programmes in the country and highlighted ADB’s strategic alignment to Viet Nam’s goals and priorities. The CAPE indicated that progress in meeting ADB’s targets differed among the priority sectors. For every sector where targets weren’t met, an explanation of variance was provided. In the area of transport for instance, the CAPE indicated that progress had been mixed and characterised by delays in planning and implementation. Large-scale projects were considered one of the main causes for these delays. Projects in the energy sector also faced delays in implementation. Results were also mixed for the finance sector, as some of the targets had been achieved and others not. Most targets in education were achieved, except one on gender parity in basic education. Finally, targets for improving access to clean water, sewerage and drainage were met in urban and rural areas.

The CAPE noted that ADB’s overall contribution to development results in Viet Nam during the CAPE period had been substantial. It also noted that although identifying outputs from ADB’s investments was easy, aggregating their contribution to outcomes had proven difficult.[110] Please refer to Volume II, Appendix X for examples of ADB’s contribution to Viet Nam’s goals and priorities, by sector of priority, as described in the 2009 CAPE. The outcome statements listed in Volume II are to be read with caution given that some of the reported outcomes may be overstated and do not reflect the ADB contribution to the outcome, which is usually achieved only through the contributions of many partners.

PAKISTAN

Finding 26: Survey respondents perceived ADB’s contributions to Pakistan’s goals and priorities to be adequate. A significant proportion of MOPAN donors in-country lacked familiarity with the Bank’s work in most of the areas assessed. ADB’s country-level documents report on output-level achievements. Evidence of the Bank’s contribution to Pakistan’s goals and priorities was scarce.

A total of 45 donors in-country and ADB clients responded to the questions on results in Pakistan. They were asked whether ADB has effectively contributed to the development of select sectors of continuing priority for Bank lending over the years (i.e., energy and infrastructure, governance and financial services, development of urban services and agriculture and natural resource management.[111] Overall, the majority of respondents rated ADB’s performance as adequate. MOPAN donors in-country appeared less knowledgeable of the Bank’s work, particularly in the finance, water and agriculture sectors, with 33 to 58% answering “don’t know”.

The Bank has supported the Government of Pakistan by providing sovereign and non-sovereign loans and grants totalling $22.6 billion and $17 billion respectively.[112] In the past few years, ADB’s assistance has been centred on improving the infrastructure, energy and social sectors.

The MOPAN assessment of the Bank‘s contribution to Pakistan’s goals and priorities is based on the most recently completed CPS – the 2004-2006

Country Strategy and Program –and the findings of the 2007 Country Assistance Performance Evaluation covering that strategic cycle. However, given that the CAPE covered two CPS cycles (hence covering the Bank’s assistance to the country from 1985 to 2006), the assessment also drew on findings from other reviews/evaluation reports covering both the past and current country partnership strategies (i.e. 2004-2006 and 2009-2013 CPSs). Therefore, in addition to the 2007 CAPE, the analysis of reported results focuses on other reports on country-level performance, such as the 2009 Development Effectiveness Country Brief (DECB), Sector Assistance Program Evaluations (SAPES), and impact evaluation studies (IES).

According to the 2007 CAPE, ADB contributed the most to the development of the energy and road infrastructure sectors in Pakistan over the past two decades (1985-2006). While its contribution to results in agriculture and natural resources management was reported to have been mixed, it appeared to have made the least contribution in the development of urban services (water and waste management) in the country.

According to the Bank, ADB’s performance in the country was at the time hindered by multiple factors, including political instability (see sidebar). Changes in governments and the high turnover of decision-makers in the country are reported to have prevented the institutionalisation of the Bank’s initiatives. Poor performance was also reported to have been due to inadequate supervision of ADB’s projects in the country. The lack of a good system to monitor national indicators, made it difficult for the Bank to gauge the extent of its contribution to country-level outcomes and impact. One of the challenges the Bank faced (and still faces today) is the massive data gap in national data/statistics, which made it difficult for the Bank to track the achievement of outputs and any contributions to outcomes. The Bank reports it has sought to address this challenge as part of its MfDR process. This has included working closely with the Pakistani government and providing technical assistance needed to strengthen the government’s monitoring practices.

All of these factors led to an entire restructuring of the Bank’s portfolio in Pakistan in 2007, which consisted of the withdrawal of funding and the closing of poorly performing operations, including unfinished projects. This exercise has put into question the development effectiveness of the Bank’s operations in Pakistan. In the 2010 DEfR, the Bank justified the restructuring exercise by indicating that ADB had “approved a highly ambitious portfolio of operations in the early 2000s. Almost half were complex, such as devolved delivery of multiple social services in various provinces, provincial resource management programs, and reforms of the judiciary and police. Only one-quarter of these succeeded. Almost half of the PCRs concerned programme loans. The low success rate in Pakistan was accompanied by low disbursement rates.”[113] The 2012 Annual Evaluation Report further reports that “most of [ADB’s] unsuccessful projects in the past two decades have been accompanied by significant cancellations of loan amounts, and almost a third of unsuccessful projects were in Pakistan”. It highlighted that “Pakistan continues to weigh down overall performance, as is well known from DEfR’s analysis”, but that “there are recent signs of some improvement”.[114]

Evidence of outputs achieved at the country level was found in a recent results aggregation exercise conducted by the Bank, which indicated that between 2004 and 2012, ADB-supported projects in the energy sector had led to the installation or upgrading of 481kilometers of transmission lines, which brought electricity to close to 281 500 households. In the transport sector, ADB reports to have helped build or upgrade nearly 7 000 km of roads and highways, benefiting more than 24 million people. In water supply and sanitation, ADB reports to have installed or upgraded close to 6 500 km of water supply pipes which served more than 332 500 households served with clean water.[115] Given the scant information on outcome achievement in the country, and the limited (but also outdated) number of comprehensive evaluation studies available,[116] an illustration of ADB’s contribution to outcomes/ impacts in the country was drawn from the most recent impact evaluation study conducted by the IED in Punjab Pakistan (see sidebar).

INDONESIA

Finding 27: ADB’s assistance to Indonesia was generally considered adequate by surveyed respondents. In country-level reports, there are discrepancies in the ratings with regard to ADB performance. While internal assessments tended to be positive, independently conducted external reviews of the Bank’s performance in Indonesia tended to be more negative.

A total of 33 donors in-country and ADB clients responded to the questions on results in Indonesia. They were asked whether ADB has effectively contributed to selected result areas (infrastructure, finance, decentralisation, MDG acceleration, and environmental and natural resource management). Survey respondents rated the Bank adequate overall. There were high proportions of ‘don’t know’ responses on questions related to marine resource management, quality of education (i.e. MDG acceleration), and finance sector development.[117]

The assessment of ADB’s contribution to Indonesia’s goals and priorities was based on an analysis of reviews and evaluation reports covering the 2006-2009 Country Strategy and Program. The documents consulted included the 2006-2009 Country Strategy Final Review (self-assessment) and the 2006-2009 Supplementary Country Strategy and Program Final Review (2010), as well as the 2006-2009 Final Review Validation conducted by IED in 2011.

The document review noted discrepancies between the Bank’s internal and external assessments conducted at CPS completion. While both the assessments conducted by the Bank’s regional department and IED agreed on ratings related to programme relevance, efficiency and sustainability of ADB’s interventions at the country level, discrepancies in ratings were noted in terms of strategic positioning and effectiveness of the Bank’s operations during the 2006-2009 CPS period.

ADB in Indonesia has taken a cautious approach in attributing the achievement of development results to the Bank. Both the internal and external assessments of the Bank noted that “while ADB has made a contribution in many areas, it is not possible to isolate and attribute higher-level results to ADB’s country program assistance in an economy as large and complex as that of Indonesia”.[118] Both reviews noted that outcome targets originally set in the CPS had proven to be overly ambitious in a number of areas, especially the time required to enact major policy and institutional changes (e.g. time required to develop capital markets). In other cases, structural constraints, such as land acquisition for large-scale infrastructure projects, proved difficult to overcome. Even though there seemed to be obvious limitations, both the CPS review and validation report assessed the “progress toward achieving development impacts, as specified in the country partnership strategy results framework” as satisfactory. Ratings provided for the overall assessment of the CPS also concurred despite the discrepancies in ratings mentioned above. The CPS Final Review Validation conducted by IED highlighted a few areas of reporting on achievement that were either vague or unclear in the 2010 Indonesia CPS Final Review. For instance, in terms of decentralisation, one of the achievements according to the CPS Final Review was “regional government reserves estimated at 3.2% of GDP as of December 2009”. This result was considered unclear by the IED validation study.

The Supplementary Country Strategy Final Review indicated that, as of 2010, the country program initiated under the 2006-2009 CPS “will not be realized for another 5–10 years, when reforms are fully implemented and projects initiated under the CPS come to fruition.”[119] However, it identified select areas where the Bank had made contributions, mostly in the development of rural infrastructure and roads.

Please refer to Volume II, Appendix X for examples of ADB’s contribution to Indonesia’s goals and priorities, by sector of priority.

4.4 RELEVANCE OF ADB’S OBJECTIVES AND PROGRAMME OF WORK TO STAKEHOLDERS

For this KPI, MOPAN assessed relevance as a measure of the extent to which a multilateral organisation’s partners and donors in-country believe the organisation supports country priorities and meets the changing needs of clients and the target population. The assessment is based exclusively on survey data gathered from clients and MOPAN donors in-country in the countries selected for the assessment.

Overall assessment

Across the three countries selected, respondents rated ADB as strong in responding to DMCs’ key development priorities. They rated it adequate in providing innovative solutions for development challenges and in adapting its work to changing conditions at the country level.

Figure 4.7 shows the overall assessment rating and the mean scores on the three survey questions on which the assessment is based.

Figure 4.7 | KPI C: Relevance of objectives and programme of work to stakeholders, overall rating and survey mean scores by country

Overall Rating: Adequate

5. Conclusions

These conclusions step away from the specific ratings of the MOPAN assessment and look at the major messages that can contribute to dialogue between individual MOPAN members and ADB and its clients. It draws on the survey findings and principal observations of the assessment of ADB’s practices and systems (Key Performance Indicators 1-21) and the assessment of development results component (Key Performance Indicators A-C).

Conclusions on ADB’s organisational effectiveness

ADB has continued to strengthen its implementation of Strategy 2020, with an articulation of core areas of specialisation and key drivers of change that are aligned with its mandate. MOPAN donors at headquarters recognised the Bank’s continued progress in implementing its results agenda and the strong leadership provided by senior management. The positive perceptions of ADB’s results frameworks (both corporate and country level) were confirmed by the review of documents. The most recent revision to the corporate results framework (including the addition of outcome-oriented indicators in level 2) demonstrates an honest intellectual effort to include outcomes in the framework and, in turn, improve corporate reporting. The format of the country partnership strategy (CPS) has improved over the years, and survey respondents accorded strong ratings for the Bank’s country and sector level frameworks. In addition, CPSs are aligned with national development strategies and surveyed stakeholders in-country commended the Bank for its ability to respond to key development priorities of DMCs.

ADB continues to be valued for being a regionally-owned Bank and for playing a critical role in regional co-operation and integration (RCI). This was evident in survey respondents’ perceptions of ADB’s greatest strengths, their views on the Bank’s promotion of this theme through its strategy and operations, and its contributions to results on co-operation and integration in the region. The Bank’s policies and strategies also provide solid direction for achieving results in this area. Interestingly, however, the reporting on the Bank’s contribution to RCI is limited. This should change as the Bank begins to report on its revised results framework, which has new indicators on RCI. Greater evidence of contribution to regional dimensions may be highlighted in IED’s evaluation expected in 2014.

Survey respondents and the document review noted that ADB has made improvements in mainstreaming gender equality in its operations. ADB clients were more positive than other respondent groups about ADB’s work in environment and good governance, which they rated as strong. In the area of climate change, which is key in Strategy 2020, gaps were noted in the frameworks for monitoring results, based on documents reviewed. The Bank reports that it initiated work on this issue in a concerted manner with other MDBs in 2012.

The issue of human rights, which MOPAN included as a cross-cutting issue for all agencies assessed in 2013, emerged as an important point for discussion between MOPAN donors, other shareholders, and the Bank. While human rights is not an identified priority of the Bank, the review of select ADB policy documents (on gender, education and water) indicates that the Bank either implicitly or explicitly recognises the principles of human rights, has adopted adequate operational policies to ensure compliance with safeguards, and has institutionalised a mechanism for accountability to persons affected by ADB projects.

ADB has delegated an increasing amount of responsibility to the country level since 2000, including the ability to approve minor changes to projects, emergency projects, and technical assistance projects under USD 225,000. Based on data provided for this assessment, there appears to have been further progress in the last few years in the Bank’s decentralisation agenda. An IED evaluation planned for 2014 should provide a more in-depth assessment of the improvements and any shortcomings. This will be important to follow given that country-level respondents highlighted the Bank’s centralised procedures as one of its weaknesses.

ADB is considered an organisation with sound financial accountability mechanisms in place (audit, anti- corruption and risk management). This was confirmed by the views of donor respondents who perceived its internal audit processes to be strong. The ratings on ADB’s human resource policies and systems were adequate or higher, based on the review of documents. The organisation has a results-focused performance assessment system to assess staff performance (with the exception of Vice Presidents who are appointed by the Board of Directors and for whom separate rules apply) and inform promotions and other key human resource decisions. However, MOPAN donors either do not know how promotions take place or express concern about the extent to which staff promotion is transparent or based on merit. This suggests that the implementation of the system may either be inconsistent or not well-known among MOPAN donors.

Conclusions on evidence of ADB’s development results and relevance to stakeholders

ADB’s contributions to infrastructure development were noted by surveyed donors at headquarters and the document review. Its contribution to intraregional co-operation was also rated positively by survey respondents but reported evidence of this could not be found in the review of the Bank’s reports and evaluations.

ADB provides ample evidence on the achievement of outputs in level 2 in the DEfR. It is important to note, however, that the type of outputs that the Bank supports (e.g. in infrastructure) can often take many years to complete. The Bank is also making improvements in data quality and developing approaches to better capture how projects are contributing to sector outcomes. The recent changes in the Results Framework will give the Bank more outcome-oriented indicators to track and should improve future corporate reporting on results. However, current (2012) reporting on results is still lacking evidence of the Bank’s contributions to outcomes and there are only a few evaluation studies on core sector areas (often with a sub-regional scope) to confirm that outputs are leading to outcomes in the past years (since 2008).

ADB’s reporting on results achievement in countries is spread across different types of reports, but the key documents are the CPS final review (CPSFR) validation report, the country assistance performance evaluation (CAPE), and the sector assistance performance evaluation (SAPE). The SAPE is conducted at the final evaluation stage of a country partnership strategy (CPS).

Survey respondents in the three countries sampled provided adequate to strong ratings on ADB’s contributions to key outcomes that had been specified in the CPS. A review of country-level reporting, however, provided a mixed picture; the evidence of contributions to outcomes was more robust in Viet Nam than in Pakistan, for example. It is important to note that the review focused on completed CPS and thus considered strategy documents and results frameworks that do not incorporate the most recent improvements in the CPS.

Across all three countries reviewed, there is a lack of reliable baseline data, which makes it difficult for the Bank to set realistic and achievable targets and also to gauge the extent of ADB’s contribution to country- level results. The Bank recognises this as an area that needs improvement and has committed to strengthening national/DMCs monitoring and reporting capacities as part of its MfDR agenda. The Bank is also committed to enhance its quality control of project completion reports (PCRs) and of country partnership strategy (CPS) results frameworks. Improvement in these areas may help to improve the credibility and reliability of data reported at the country/project levels.

 

List of Figures

Figure 1.1 Number of survey respondents and total population for ADB by country and respondent group (n=159)
Figure 1.2 ADB – Distribution of responses (n=159) on all questions related to micro-indicators
Figure 3.1 Overall ratings on key performance indicators of ADB’s organisational effectiveness (survey mean scores and document review ratings)
Figure 3.2 Overall ratings of ADB organisational effectiveness by respondent group (n=159)
Figure 3.3 Performance area I: Strategic management, survey and document review ratings
Figure 3.4 Performance area I: Strategic management, mean scores by respondent group
Figure 3.5 KPI 1: Providing direction for results, ratings of micro-indicators
Figure 3.6 KPI 2: Corporate strategy and mandate, ratings of micro-indicators
Figure 3.7 KPI 3: Corporate focus on results, ratings of micro-indicators
Figure 3.8 KPI 4: Focus on cross-cutting thematic areas, ratings of micro-indicators
Figure 3.9 KPI 5: Country focus on results, ratings of micro-indicators
Figure 3.10 Performance area II: Operational management, survey and document review ratings
Figure 3.11 Performance area II: Operational management, mean scores by respondent group
Figure 3.12 KPI 6: Resource allocation decisions, ratings of micro-indicators
Figure 3.13 KPI 7: Results-based budgeting, ratings of micro-indicators
Figure 3.14 KPI 8: Financial accountability, ratings of micro-indicators
Figure 3.15 KPI 9: Using performance information, ratings of micro-indicators
Figure 3.16 KPI 10: Managing human resources, ratings of micro-indicators
Figure 3.17 KPI 11: Performance-oriented programming, ratings of micro-indicators
Figure 3.18 KPI 12: Delegating authority, ratings of micro-indicators
Figure 3.19 Performance area III: Relationship management, survey and document review ratings
Figure 3.20 Performance area III: Relationship management, mean scores by respondent group
Figure 3.21 KPI 13: Supporting national plans, ratings of micro-indicators
Figure 3.22 KPI 14: Adjusting procedures, ratings of micro-indicators
Figure 3.23 KPI 15: Using country systems, ratings of micro-indicators
Figure 3.24 KPI 16: Contributing to policy dialogue, ratings of micro-indicators
Figure 3.25 KPI 17: Harmonising procedures, ratings of micro-indicators
Figure 3.26 Performance area IV: Knowledge management, survey and document review ratings
Figure 3.27 Performance area IV: Knowledge management, mean scores by respondent group
Figure 3.28 KPI 18: Evaluating results, ratings of micro-indicators
Figure 3.29 KPI 19: Presenting performance information, ratings of micro-indicators
Figure 3.30 KPI 20: Disseminating lessons learned, ratings of micro-indicators
Figure 3.31 KPI 21: Availability of documents, ratings of micro-indicators
Figure 4.1 Development results component – overall ratings
Figure 4.2 KPI A: Evidence of the extent of progress towards results in core areas, overall rating
Figure 4.3 ADB’s measurement and reporting on core areas of specialisation
Figure 4.4 Extent of progress in ADB’s core areas of specialisation, stakeholder survey
Figure 4.5 KPI B: Evidence of the extent of contribution to country-level goals and priorities, rating
Figure 4.6 ADB’s measurement and reporting on country-level results
Figure 4.7 KPI C: Relevance of objectives and programme of work to stakeholders, overall rating and survey mean scores by country
 

---

[1] It is to be noted however that as one of the countries sampled for this assessment, Pakistan had undergone an entire portfolio restructuring which affected results achievement.

[2] For additional information, please consult the MOPAN website: www.mopanonline.org

[3] See Volume II, Appendix I, Section 3.4 for an explanation of the weighing formula and scheme.

[4] While most KPIs and micro-indicators were considered in the document review, not all were rated. See section 2.3.

[5] As indicated in Chapter 1 - section 1.3, the 159 respondents included 37 donors at HQ, 35 donors in-country and 87 clients.

[6] Respondents who wrote “no comment” or the like were filtered out of the analysis. Comments that included more than one element were coded in various categories.

[7] The normal convention for statistical significance was adopted (p≤.05). See section 2.2, which describes the data analysis process.

[8] ADB (2001), Special Evaluation Study on Managing for Development Results (p. 51)

[9] According to the 2011 Development Effectiveness Review report, the Bank essentially uses different project samples to assess output and outcome achievement that renders the linkage between outputs to outcomes reported within level 2 difficult. For more details, please see discussion on “Measuring results” under KPI A (Chapter 4).

[10] ADB (2013), Environment Operational Directions 2013-2020 Promoting Transitions to Green Growth in Asia and the Pacific.

[11] ADB (2010), Focused Action: Priorities for Addressing Climate Change in Asia and the Pacific.

[12] ADB (2013), Adaptation Solution - Building Resilience to Climate Changes: Adaptation Technical Resources.

[13] In the 2012 Review of ADB Results Framework the Bank commits to including a new indicator under level 3 that aims to track the percentage of operations supporting climate change mitigation and adaptation and sets a target of 60% to be met by the year 2016.

[14] 2010 (ADB), Addressing Climate Change in Asia and the Pacific: Priorities for Action.

[15] ADB (2012), Annual Report 2012: Advancing Regional Co-operation and Integration in Asia and the Pacific. (P.36)

[16] Ibid.

[17] The Bank has indicated that while project success ratings are ultimately determined by the Independent Evaluation Department (IED), the Private Sector Operation Department has conducted a review of both reporting documents to identify areas of recurring differences and is actively engaged with IED to ensure greater alignment between assessment approaches.

[18] ADB (2011), Special Evaluation Study on ADB Assistance for Strengthening the Enabling Environment for Private Sector Development.

[19] ADB (2006), Regional Co-operation and Integration Strategy. (P.19).

[20] ADB (2003), Our Framework Policies and Strategies: Gender and Development. (P.2)

[21] ADB (2003), Our Framework Policies and Strategies: Education. (P.55)

[22] ADB (2004), Water for All: the Water Policy of the Asian Development Bank. (P.34)

[23] See also the Bank’s Handbook on Core Labor Standards (2006) developed with the ILO.

[24] The CSPs reviewed during the 2013 MOPAN Assessment covered the following years: Indonesia: 20012-2014; Pakistan: 2009-2013; and Viet Nam: 2012-2015.

[25] Peer organisation respondents were asked questions in KPIs related to humanitarian response and cross-cutting themes.

[26] The document review was designed to draw data from the 2011 Survey on Monitoring the Paris Declaration and ADB’s reporting on its Paris Declaration commitments.

[27] According to OECD/DAC’s definition, results-based budgeting is: “A budgeting method that links appropriations to outcome level performance targets. Expected results justify resource requirements and actual performance in achieving results is measured by predefined performance targets”.

[28] AfDB, ADB, EBRD et. al.(2010), Agreement for Mutual Enforcement of Debarment Decisions.

[29] ABD (2012), Annual Report of the Audit Committee of the Board 2011-2012 (P.6).

[30] ADB (2013), Staff Instructions on Processing and Monitoring Sector and Thematic Operational Plans.

[31] ADB (2011), Operations Manual – Operational Procedures: Project Performance Management System. (P.2).

[32] ADB (2013), 2012 Development Effectiveness Review. (P.39)

[33] ADB (2011), Project Administration Instructions: Project Performance Monitoring. (P6).

[34] ADB (2013), 2012 Development Effectiveness Review. (P.49)

[35] ADB (2013), Operations Manual – Bank Policies: Policy Based Lending. (p.8)

[36] Publish what you Fund (2010), The Global Campaign for Aid Transparency: Aid Transparency Assessment 2010

[37] ADB (2011), Aid Effectiveness Report 2011: Overall Achievements on Paris Declaration Commitments. (P.5)

[38] It should be noted, however, that ADB’s definition of “use of country systems” differs from that of the OECD. According to ADB’s Aid Effectiveness Report (2011), the ADB “applies use of country systems monitoring only to national procurement. It considers the use of national competitive bidding procedures as effectively a use of country systems because this procurement is carried out under the domestic laws and regulations of a DMC. ADB’s approach thus differs from that of the OECD-DAC, which tracks all procurement financed by a development partner in a partner country and sets a target of 60% of overall disbursements”.

[39] Ibid.

[40] ADB (2013), Operational Plan for Enhancing ADB’s Effectiveness in Fragile and Conflict-Affected Situation. (P.15)

[41] ADB (2011), Aid Effectiveness Report 2011: Overall Achievements on Paris Declaration Commitments. (P.4)

[42] ADB (2011), Aid Effectiveness Report 2011: Overall Achievements on Paris Declaration Commitments. (P.4)

[43] Ibid.

[44] ADB (2011), Operations Manual: Independent Evaluation.

[45] Please refer to Chapter 4, Section 4.2 of this report for more details.

[46] MOPAN has used CREAM and SMART criteria to assess the quality of indicators and results statements. CREAM criteria (clear, relevant, economic, adequate, monitorable) are usually used to evaluate indicators. SMART criteria (specific, measurable, achievable, relevant, time bound) are widely used when developing indicators.

[47] ADB (2013), 2012 Development Effectiveness Review. (P.37)

[48] ADB (2009), Development Effectiveness Brief – Pakistan: Making a Difference in the Fight Against Poverty.

[49] ADB (2011), Operation Manual Bank Policies: Independent Evaluations.

[50] Evaluation System Information (EVIS) Website: http://lnadbg4.adb.org/oed001p.nsf/index?OpenForm

[51] Each organisation may use a different term to refer to this level of results.

[52] ADB (2008), Strategy 2020: the Long-Term Strategic Framework of the Asian Development Bank 2008-2020.

[53] The Bank uses several terms interchangeably: core areas of specialisation, core operational areas, and core areas of operations.

[54] This rating should be interpreted with caution given that the assessment was based on a small sample of ADB countries (3 out of 47 DMCs), and one of the countries (Pakistan) has undergone an entire portfolio restructuring and provided limited data of results achieved in the country.

[55] Please refer to the sections on KPIs 3 and 19 for the analysis of ADB’s results-based systems and reporting practices.

[56] ADB (2008), ADB Results Framework.

[57] It should be noted that reporting on the 2013 framework will begin in 2014 with the publication of the 2013 DEfR.

[58] ADB (2008), Strategy 2020: the Long-term Strategic Framework of the Asian Development Bank 2008-2020.

[59] ‘Theory of change’ is understood in the sense defined by Rist and Morra Imas (2009) as, “a representation of how an intervention is expected to lead to desired results” and in the sense defined by Michael Quinn Patton who has stated that a theory of change is more than the sequential order of results statements presented in a logic model; it requires key assumptions related to the results chain and context (e.g. policy and environment), and important influences and risks to be made explicit – Qualitative Research and Evaluation Methods (2002).

[60] This refers to the existence of reports on outputs as defined by the OECD (i.e. lower level results). Some MOs use different terminology for the various levels of results.

[61] This refers to the existence of reports on outcomes as defined by the OECD (i.e. higher level results). Some MOs use different terminology for the various levels of results.

[62] ‘Reporting according to a theory of change is understood to mean the extent to which organisations provide a narrative describing the actual implementation process and results achieved in relation to that foreseen in the initial ‘theory of change’.

[63] According to Rist and Morra Imas, The Road to Results – “Reliability is the term used to describe the stability of the measurement – the degree to which it measures the same thing, in the same way, in repeated tests.” Attention is also given to the quality of the evidence – specifically, whether or not it has been derived from or validated by an external and/or independent source. Please see the section on Measuring Results for greater detail on the changes made between the 2011 and 2012 DEfR.

[64] ADB, (2013), Processing and Monitoring Sector and Thematic Operational Plans: Staff Instructions.

[65] For infrastructure, the assessment relied on the review of the 2010 Sustainable Transport Initiative (STI) and 2011 Water Operational Plans developed by the Bank. No operational plan had been developed for the energy sector at the time of the assessment.

[66] This is a challenge that the Bank hopes to address soon as part of its efforts to “sharpen focus on results.” According to the 2013 Staff Instructions on Processing and Monitoring Sector and Thematic Operational Plans, the Bank intends to update all Operational Plans that do not contain a results framework and/or time-bound monitoring plans; as recommended in the Special Evaluation Study on MfDR conducted in 2011 by IED.

[67] ADB. (2012). 2011 Development Effectiveness Review (p.25)

[68] According to the Bank, the success ratings of ADB projects are based on the following four criteria: relevance, effectiveness in achieving outcome, effciency in achieving outcome and outputs, and likelihood of sustainability. Outcomes reported in the DEfR are therefore based on the “effectiveness” criterion of the success rating. 69. ADB (2011), Special Evaluation Study on Managing for Development Results. (P.41)

[69] ADB (2011), Special Evaluation Study on Managing for Development Results. (P.41)

[70] ADB (2012), Development Effectiveness Review: 2011 Report. (P.22)

[71] ADB (2013), Development Effectiveness Review: 2012 Report. (P.16)

[72] ADB (2008), Guidelines for the Validation of Project Completion Reports and Extended Annual Review Reports.

[73] ADB (2013), Development Effectiveness Review: 2012 Report.

[74] ADB (2013), Development Effectiveness Review: 2012 Report. (P.57)

[75] The survey ratings presented in this table comprise the average mean scores of multiple sub-questions. The rating for infrastructure for instance, covers questions related to the Bank’s work in the transport, water and energy sectors. Please refer to Volume II, Appendix V for an overview of the questions and sub-questions asked to specific survey respondents during this assessment.

[76] ADB (2013), Sustainable Transport for All.

[77] ADB (2010), Sustainable Transport Initiative Operational Plan. (P.15-21)

[78] ADB (2013), Development Effectiveness Review: 2012 Report. (P.15)

[79] ADB (2012), 2012 Development Effectiveness Review. (P.18)

[80] Ibid. (P.34)

[81] The study covered four Pacific DMCs: Papua New Guinea, Fiji, Timor Leste, and the Solomon Islands which accounted for 93.5 percent of ADB transport sector support to the region.

[82] ADB (2011), Transport Sector in the Pacific Developing Member Countries (1995-2010). The evaluation covered loans, grants, and technical assistance completed between 1995 and 2010.

[83] Ibid. (P.28)

[84] ADB (2009), Energy Policy. (P.5)

[85] ADB (n.d.), Summary of ADB Clean Energy Projects (2003-2011)

[86] ADB (2013), 2012 Clean Energy investments: Project Summaries.

[87] ADB (2013), Development Effectiveness Review report 2012. (P.13).

[88] ADB (2010), Water Policy and Related Operations. The study covered water sector operation conducted over the 2001-2009 period and highlighted concerns over long-term sustainability, largely around operations and maintenance. The Bank indicated that the Water Operational Plan approved in 2011 addresses these issues with a drive to increasing focus on 'business' sustainability ( for additional information please refer to the Water Operational Plan 2011-2020).

[89] ADB (2012), Review of the ADB Results Framework.

[90] ADB (2013), Development Effectiveness Review: 2012 Report. (P.15)

[91] According to IED, the study primarily consisted of an in-depth assessment of ADB’s microfinance programmes in six case countries (Cambodia, Pakistan, Papua New Guinea, Philippines, Uzbekistan, and Viet Nam). Together, these countries accounted for 33% of the ADB’s microfinance portfolio

[92] ADB (2012), Microfinance Development Strategy 2000: Sector Performance and Client Welfare. The study covered microfinance operations approved during the 2000-2010 period.93 93. ADB (2011), Education in Asia and the Pacific.

[93] ADB (2011), Education in Asia and the Pacific.

[94] ADB (2013), Development Effectiveness Review: 2012 Report. (P.20)

[95] Please refer to chapter 3 on KPIs 3, 5, 19 and 20 for the analysis of ADB’s results-based systems and practices.

[96] ADB (2013), Operations Manual Operational Procedures (OP): Country Partnership Strategy.

[97] ADB (2010), Preparing Results Frameworks and Monitoring Results: Country and Sector Levels.

[98] Or extended annual review reports (XXARRs) in case of non-sovereign operations

[99] In select cases, the IED also conducts independent evaluations at the project level, i.e. rigorous impact evaluations (RIEs) which aim to assess the attribution aspects of a given project; as well as special evaluation studies (SESs) which cover a specific thematic area.

[100] ADB (2011) Special Evaluation Study on Managing for Development Results

[101] This part of the MOPAN assessment analyses the most recently completed programme cycles, i.e. the 2006-2009 CPS for Indonesia, 2007-2010 CPS for Viet Nam and the 2004-2006 CPS for Pakistan.

[102] ‘Theory of change’ is understood in the sense defined by Rist and Morra Imas (2009) as, “a representation of how an intervention is expected to lead to desired results” and in the sense defined by Michael Quinn Patton who has stated that a theory of change is more than the sequential order of results statements presented in a logic model; it requires key assumptions related to the results chain and context (e.g. policy and environment), and important influences and risks to be made explicit – Qualitative Research and Evaluation Methods (2002).

[103] This refers to the existence of reports on outputs as defined by the OECD (i.e. lower level results). Some MOs use different terminology for the various levels of results.

[104] This refers to the existence of reports on outcomes as defined by the OECD (i.e. higher level results). Some MOs use different terminology for the various levels of results.

[105] ‘Reporting according to a theory of change is understood to mean the extent to which organisations provide a narrative describing the actual implementation process and results achieved in relation to that foreseen in the initial ‘theory of change’.

[106] The ratings for each criterion are (i) for relevance: highly relevant, relevant, partly relevant, and irrelevant; (ii) for effectiveness: highly effective, effective, less effective, and ineffective; (iii) for efficiency: highly efficient, efficient, less efficient, and inefficient; (iv) for sustainability: most likely sustainable, likely, less likely, and unlikely; and (v) for impact: high impact, substantial, modest, and negligible

[107] ADB (2010), Revised Guidelines for the Preparation of Country Assistance Performance Evaluations.

[108] See Volume II, Appendix V for questions asked of survey respondents in Viet Nam.

[109] The evaluation methodology indicates that “the top-down assessment reviewed ADB’s strategic positioning, ADB’s contributions to development results, and ADB’s performance as a development partner. The bottom-up assessment evaluated the performance of ADB lending and nonlending assistance in key sectors by examining its relevance, effectiveness, efficiency, impact, and sustainability”. (p.46)

[110] ADB (2009), Country Assistance Program Evaluation for the Socialist Republic of Viet Nam.

[111] See Volume II, Appendix V for questions asked of survey respondents in Pakistan.

[112] ADB (2013), Asian Development Bank & Pakistan: Fact Sheet.

[113] ADB (2011), Development Effectiveness Review report 2010. (P.33)

[114] ADB (2012), Annual Evaluation Report.

[115] ADB (2012), Results of ADB-Supported Operation: Pakistan.

[116] At the time of the evaluation, the IED was in the process of conducting an evaluation of the Bank’s assistance to Pakistan covering the 2002-2012 period. The Bank was also in the process of developing a DECB.

[117] See Volume II, Appendix V for questions asked of survey respondents in Indonesia.

[118] ADB. Supplementary Country Strategy and Program Final Review: Indonesia, 2006-2009. (P.12)

[119] Ibid

[120] Detailed scores are shown in Volume II, Appendix VI.