Preface

The Multilateral Organisation Performance Assessment Network (MOPAN) is a network of donor countries with a common interest in assessing the organisational effectiveness of multilateral organisations and their measurement and reporting on development and/or humanitarian results. MOPAN was established in 2002 in response to international forums on aid effectiveness and calls for greater donor harmonisation and co-ordination.

Today, MOPAN is made up of 17 donor countries: Australia, Austria, Belgium, Canada, Denmark, Finland, France, Germany, Ireland, The Netherlands, Norway, Republic of Korea, Spain, Sweden, Switzerland, the United Kingdom, and the United States. For more information on MOPAN and to access previous MOPAN reports, please visit the MOPAN website (www.mopanonline.org).

Each year MOPAN carries out assessments of several multilateral organisations based on criteria agreed by MOPAN members. Its approach has evolved over the years, and since 2010 has been based on a survey of key stakeholders and a review of documents of multilateral organisations. MOPAN assessments provide a snapshot of four dimensions of organisational effectiveness (strategic management, operational management, relationship management, and knowledge management). In 2013, MOPAN has integrated a new component to examine the evidence of achievement of development and/or humanitarian results to complement the assessment of organisational effectiveness.

MOPAN 2013

In 2013, MOPAN assessed four multilateral organisations: the Asian Development Bank (ADB), the International Fund for Agricultural Development (IFAD), the World Food Programme (WFP) and the World Health Organization (WHO).

MOPAN Institutional Leads liaised with the multilateral organisations throughout the assessment and reporting process. MOPAN Country Leads monitored the process in each country and ensured the success of the survey.

Acknowledgements

We thank all participants in the MOPAN 2013 assessment of IFAD. IFAD’s senior management and staff made valuable contributions throughout the assessment and document review processes and provided lists of their direct partners to be surveyed. Survey respondents contributed useful insights and time to respond to the survey. The MOPAN Institutional Leads, Canada and Spain, liaised with IFAD throughout the assessment and reporting process. The MOPAN Country Leads oversaw the survey planning process in the field and ensured the success of the survey. Consultants in each country provided vital in-country support by following up with country-level survey respondents to enhance survey response rates.

Roles of authors and the MOPAN Secretariat

The MOPAN Chair was held by Australia in 2013 and worked in close co-operation with the MOPAN Technical Working Group to launch and manage the survey. Once the MOPAN Secretariat was established at OECD in April 2013, the Head of the Secretariat oversaw all related tasks.

MOPAN developed the Key Performance and Micro-Indicators, designed the survey methodology, co-ordinated the preparation of the lists of survey respondents, and approved the final survey questionnaire for each agency. MOPAN also directed the approach to the document review and oversaw the design, structure, tone, and content of the institutional reports.

Universalia and Epinion developed the survey instrument and carried out the survey and analysis. Universalia carried out the document review, conducted the interviews with multilateral organisation staff at headquarters and country levels, and wrote the institutional reports.

Epinion is a consulting firm in Denmark that analyses and evaluates data to support decision making. It conducts specially designed studies for public and private organisations based on data collected among an organisation’s employees, members, customers, partners, and other sources. Epinion has 75 employees and 200 interviewers. Website: www.epinion.dk

Universalia Management Group is a Canadian consulting firm established in 1980 that specialises in evaluation and monitoring for international development. Universalia has made significant contributions to identifying best practices and developing tools in the fields of organisational assessment; planning, monitoring, and evaluation; results-based management; and capacity building. Website: www.universalia.com.

Acronyms

Executive summary

This report presents the results of an assessment of the International Fund for Agricultural Development (IFAD) conducted by the Multilateral Organisation Performance Assessment Network (MOPAN). The MOPAN Common Approach examines organisational systems, practices, and behaviours that MOPAN believes are important for aid effectiveness. It also examines the extent to which there is evidence of an organisation’s contributions to development and/or humanitarian results, and the relevance of its objectives and programme of work to stakeholders at the country level.

IFAD’s mandate is to empower poor rural women and men to achieve higher incomes and improved food security and to strengthen their resilience. IFAD provides loans and grants to agriculture and rural development projects with the perspective that smallholder agriculture can act as both a crucial source of rural income and nutrition and a vector for rural economic growth. Its projects seek to incorporate smallholders into global value chain development.

IFAD has implemented a number of reforms to improve its effectiveness in the areas of managing for development results, quality of projects, strategic planning, innovation, efficiency and human resources. In its Ninth Replenishment Period (from 2013 to 2015), IFAD has committed to increasing its focus on value for money and impact measurement.

MOPAN assessment

In 2013, MOPAN assessed IFAD based on information collected through a survey of stakeholders, a review of documents, and interviews with IFAD staff. The survey targeted IFAD’s direct partners, as well as MOPAN donors based in-country and at headquarters. Six countries were included in the MOPAN survey: Ethiopia, Guatemala, Indonesia, Mozambique, Pakistan and Viet Nam. A total of 193 respondents participated in the IFAD survey: 31 MOPAN donors based at headquarters, 18 MOPAN donors based in-country, and 144 direct partners. The document review assessed IFAD through an examination of over 350 publicly available corporate documents as well as country programming and reporting documents from the six countries, some of which IFAD made publicly available for this assessment. MOPAN also interviewed IFAD staff members (21 from IFAD headquarters and 6 from country offices). This information was not coded or used formally as part of the assessment process, but rather to gain a broader contextual understanding of the organisations systems, practices and results-related reporting.

MOPAN assessments provide a snapshot of four dimensions of organisational effectiveness (strategic management, operational management, relationship management, and knowledge management) and of the organisation’s reporting on its development results. The main findings of the assessment of IFAD in these performance areas and in the section on development results are summarised below.

Key Findings

Strategic management

In the area of strategic management, MOPAN established criteria to determine if a multilateral organisation has strategies and systems in place that reflect good practice in managing for results. Overall, the 2013 assessment found that:

• The majority of survey respondents agree that IFAD provides leadership on results management and is focused on its direct partners. IFAD regularly updates its Results Measurement Framework, with input from its members.

•IFAD has a clear mandate and its Strategic Framework is well aligned with this mandate.

• IFAD has a strong focus on results at an organisation-wide level but there is room for improvement in establishing links between the different levels of its results chain. This is becoming more important in a global development context in which the public, donors and other stakeholders increasingly demand “value for money” in their collaboration with development institutions.

•IFAD is considered strong in incorporating the cross-cutting themes of gender equality, environment, and food security and nutrition. While IFAD does not focus directly on good governance and human rights-based approaches (themes of interest to MOPAN), survey respondents considered IFAD adequate in both of these areas

• IFAD’s country-level strategic documents (country strategic opportunity programmes [COSOPs]) place an adequate focus on expected results. The six COSOPs sampled are well aligned with national development plans, although the links between the levels of results (i.e. outputs, outcomes) are not always evident in project and country results frameworks.

• While IFAD has made provisions for annual reviews, Mid-Term Reviews, and completion reviews for COSOPs, annual reviews are not yet being performed systematically for all COSOPs, midterm reviews are often done late and with varying quality, and COSOP completion reviews have not yet been systematically introduced across all countries.

Operational management

In operational management, MOPAN established criteria to determine if a multilateral organisation manages its operations in a way that supports accountability for results and the use of information on performance. Overall, the 2013 assessment found that:

• IFAD was noted for the transparency of its performance-based resource allocation system (PBAS). The PBAS criteria are publicly available and the majority of survey respondents agree that allocations follow the criteria established.

• Donors at headquarters considered IFAD adequate in linking loans and grants to expected results. The document review noted that IFAD has improved its approach to results-based budgeting since the last MOPAN assessment but still does not have a system in place that provides a clear link between allocated resources and expected outputs.

• IFAD has some strong practices for financial accountability (such as internal and external audits, risk management processes, and policies to combat fraud and corruption). However, in spite of recent improvements, there still appears to be room for improvement in IFAD’s procurement and contract management processes at the corporate level.

• Survey respondents rated IFAD adequate or strong for its use of performance information. The document review rated IFAD as very strong for using performance information to revise and adjust policies and for acting on evaluation recommendations, but adequate for its use of performance information to plan new interventions and for its proactive management of poorly performing initiatives.

• The assessment noted the recent improvements in IFAD’s level of transparency in human resource management, and commended IFAD for implementing a performance evaluation system for all levels of staff. IFAD recently implemented a pay-for-performance model to provide incentives for staff performance.

• IFAD is considered strong in conducting impact analyses before project approval.

• IFAD progress reporting is hindered by the lack of milestones and targets.

IFAD has decentralised some decision-making authority, but programme approval remains centralised (as is the case in all other international financial institutions). Survey respondents had mixed opinions about whether IFAD’s level of country presence was sufficient for improving country performance. While the majority of direct partners (the largest respondent group) found IFAD adequate or above, the majority of donors based in-country rated the organisation as inadequate or below.

Relationship management

In relationship management, MOPAN established criteria to determine if a multilateral organisation is engaging with its clients at the country level in ways that contribute to aid effectiveness. Overall, the 2013 assessment found that:

• Survey respondents gave IFAD an overall rating of strong for its support to funding proposals designed and developed by the national government or other direct partners. The document review rated the Fund as adequate for aligning its loan activities with national priorities.

• When asked whether IFAD adjusts its procedures to reflect local conditions and capacities, most survey respondents considered IFAD to be adequate.

• IFAD is considered adequate in its use of country systems as a first option for its operations. However, the document review found that IFAD provides little information on its progress towards meeting its aid effectiveness commitments.

• IFAD makes an adequate contribution to policy dialogue and respects the views of other stakeholders.

• Surveyed stakeholders considered IFAD’s harmonisation practices adequate while the document review provided more mixed ratings.

• The document review found that IFAD met the Paris Declaration target for participation in joint missions, but noted room for improvement in the extent to which its technical co-operation is disbursed through co-ordinated programmes. Some indicators in the Paris Declaration for Aid Effectiveness may be less relevant for IFAD, such as indicator 6 on the use of parallel implementation structures, indicator 3 on the percentage of aid to the government sector that is reported in partners’ national budgets, and indicator 9 on the use of common procedures or arrangements.

Knowledge management

In knowledge management, MOPAN established criteria to determine if a multilateral organisation has reporting mechanisms and learning strategies that facilitate the sharing of information inside the organisation and with the development community. Overall, the 2013 assessment found that:

• IFAD’s evaluation function was rated strong overall by survey respondents. They considered IFAD strong in its use of evaluation findings to inform its decisions, and adequate overall in involving direct partners and beneficiaries in its evaluations (although direct partners considered it strong in this regard).The document review rated IFAD as very strong for the independence of its Office of Evaluation (IOE) and also commended IFAD for the quality assurance processes in place for high-level evaluations.

• Although IFAD’s external reporting practices were considered strong by survey respondents, the document review found that IFAD does not sufficiently report on its aid effectiveness commitments, uses a complex reporting system, and employs inadequate indicators.

• IFAD has made improvements in its knowledge management system, but this is still a work in progress. Overall, survey respondents believe that IFAD’s knowledge management is adequate.

• IFAD makes most of its corporate documents available to the public and amended its Disclosure Policy in 2010 in response to requests for greater public access to its documentation.

Evidence of Contribution to Development Results and Relevance to Stakeholders: MOPAN assessed IFAD on this component for the first time in 2013. These findings should be considered in conjunction with the findings above on IFAD’s systems and practices for organisational effectiveness.

IFAD was rated inadequate in providing evidence of progress towards organisation-wide outcomes (KPI A) and evidence of contribution to country-level goals and priorities, including relevant MDGs (KPI B). Survey respondents rated IFAD adequate for the relevance of its objectives and programme of work to country level stakeholders (KPI C).

• Evidence of extent of progress towards organisation-wide outcomes: The Report on IFAD’s Development Effectiveness (RIDE) and its Annual Review on Portfolio Performance provide evidence of progress towards planned outputs but limited evidence of IFAD’s contribution to development outcomes. Surveyed MOPAN donors at headquarters hold mostly positive views of IFAD’s progress toward organisation-wide results.

• Evidence of extent of contribution to country-level goals and priorities: IFAD project documents provide information on outputs but little data on outcomes. Despite a generally strong evaluation function and a high degree of alignment between the COSOPs and national development plans, the lack of systematic data collection at the outcome level (apart from periodic project or programme evaluations) makes it difficult to judge the extent to which IFAD is contributing to country-level outcomes. Surveyed stakeholders in the six countries assessed have mixed views on the extent of IFAD’s contribution to country-level goals.

• Relevance of IFAD’s objectives and programme of work to stakeholders: Surveyed stakeholders in-country consider IFAD strong overall in responding to the key priorities of national partners and adequate in the provision of innovative solutions to development challenges and the extent to which IFAD adjusts its strategies and objectives according to the changing needs and priorities of the country.

Changes since the 2010 MOPAN assessment

Survey ratings suggest that IFAD’s performance has steadily improved over the past three years. In strategic management, survey respondents were more positive in 2013 than in 2010, and their perceptions of IFAD leadership for results management improved from adequate to strong. Respondents’ views on IFAD’s resource allocation system and its financial accountability improved from adequate in 2010 to strong in 2013. Surveyed stakeholders provided unchanged or slightly improved ratings in all areas of relationship management. Respondent ratings improved from inadequate to adequate on whether the time required to complete IFAD's procedures delays implementation. In its knowledge management practices, the Fund’s ratings were fairly steady. Stakeholders gave IFAD consistently strong scores for its evaluation of external results.

The micro-indicators used for the document review in 2013 and 2010 have changed, as has the rigour brought to bear on the analysis of documents this year. As such, there has been no formal comparison of the document review ratings.

Conclusions on IFAD’s organisational effectiveness

IFAD remains fully committed to a results agenda and strategic management is one of its strengths. Its mandate is clear and well reflected in its 2011-2015 Strategic Framework.

IFAD continues to be acknowledged for its clear and unique mandate and its commitment to instilling a results-oriented culture throughout the organisation. Indeed, when asked what they considered to be IFAD’s greatest areas of strength, some 40% of survey respondents mentioned its specialised mandate, which focuses almost exclusively on poverty alleviation for the rural poor. Respondents commented that no other international financial institution is mandated to focus on the rural poor, or on agricultural and rural development for smallholders in particular, which gives IFAD a clear niche in the development landscape.

IFAD confirmed its commitment to the results agenda during the negotiations for its Ninth Replenishment Period (2013 - 2015). As part of the negotiations, IFAD committed to moving 80 million poor people out of poverty by 2015, a major promise that will require new tools and practices to monitor progress. To this effect, IFAD has revised its Results Measurement Framework to integrate new indicators that will track its progress in areas such as contribution to poverty reduction, household food security and nutrition, and climate change. IFAD also plans to conduct 30 impact studies during the 2013-2015 replenishment period.

IFAD is noted for having increasingly mainstreamed cross-cutting priorities such as gender, environment, and food security and nutrition into its operations.

IFAD’s country strategies systematically include objectives and results statements related to relevant cross- cutting priorities (gender, environment, and food security and nutrition). This suggests that IFAD’s RB- COSOP Guidelines (2010) are being followed and that cross-cutting priorities are being effectively mainstreamed. IFAD has recently increased its efforts to mainstream environmental and gender equality considerations into its strategies and operations; for example, in 2012 it launched an innovative Adaptation for Smallholder Agriculture Programme (ASAP) that provides financing to scale up and integrate climate change adaptation across IFAD’s new investments. IFAD also approved a new Policy on Gender Equality in 2012 that provides a clear division of responsibility for gender mainstreaming across IFAD’s systems and operations and there is evidence that IFAD is following this policy. The extent to which these policies have resulted in positive development outcomes is not yet possible to assess.

While primary decision-making authority remains centralised, IFAD has increased its country presence significantly in recent years and has also made changes to its policies regarding the delegation of decision making. Some survey respondents feel there is still room for improvement in assuring adequate country presence.

IFAD recently adopted a Country Presence Policy and Strategy in which it formally approved the establishment of up to 40 country offices, of which 38 have already been established. IFAD has made efforts to decentralise some decision-making authority to the country level, but this is still somewhat limited (as it is in most IFIs) and further work is needed to flesh out the more substantive delegation of roles and responsibilities in the organisation.[1] The 2013 corporate-level evaluation of IFAD’s institutional efficiency noted that the limited substantive delegation of authority may conflict with the changing business needs associated with a decentralised institution.

The perspectives of in-country donors in the six focus countries suggest that IFAD’s country presence continues to be an important factor for the organisation’s performance. When asked whether IFAD’s country presence is sufficient for improving country- level performance, the majority of in-country donors rated IFAD as inadequate or lower while the majority of IFAD’s direct partners rated the organisation as adequate or higher.

IFAD has sound policies and practices in place for financial accountability, but its corporate procurement procedures could be more efficient.

The document review found that IFAD is performing strongly in relation to its internal and external audit processes and its financial accountability in general. IFAD is also noted for its strong Anti-Corruption Policy (2005) and its processes and procedures for taking immediate action when irregularities are noted. There is room for improvement in IFAD’s corporate procurement processes, which are still perceived to be time- consuming in spite of recent improvements.

IFAD has significantly improved its HR management policies and practices in recent years and remains committed to ambitious reforms in this area.

IFAD has undertaken a reform of its human resources (HR) practices and is currently working to improve several elements of its HR management system including its staff reward system, recruitment process, promotion and dismissal processes, staff rotation system, and support to staff located in-country. It has implemented a number of initiatives to ensure appropriate levels of compensation and a performance- based staff reward system is in place (e.g. a pay-for-performance model).

The document review noted that IFAD has a functional performance evaluation system (PES) that is used at all levels of the organisation. However, some sources suggest that this system is not yet being fully exploited, that managers are too risk averse in their performance assessments, and that, as a result, staff members are rarely recognised for either outstanding or substandard performance.

IFAD has begun to introduce results-based budgeting at the institutional level but this is still a work in progress.

In 2010, the Eighth Replenishment Report urged the Fund to deepen the implementation of results-based budgeting (RBB). IFAD’s 2013 budget is presented in a results-oriented way, and, according to IFAD’s Update on the Reform Agenda, management, in consultation with the Board, is committed to review and refine its RBB practices in the coming years. The budget is allocated to four business areas (clusters) to attain a set of highly aggregated operational and institutional outcomes and corporate management results. While these results are all aligned with IFAD’s mandate and strategic priorities, the cluster outcomes are aggregated at a high level and there is limited basis for decision making on budget allocations based on expected results. IFAD has not yet introduced a system of time recording and cost accounting, which is essential to reliably capture the full costs of all major business activities performed by staff.

IFAD is recognised for the independence of its Office of Evaluation and for following up on and using performance information from evaluations.

The Independent Office of Evaluation (IOE) is structurally independent from IFAD management and the IOE Director reports directly to the Executive Board. This was recognised as a clear strength by both survey respondents and the document review. There is ample evidence that evaluation recommendations reported to the Board are followed up by the responsible units. The implementation of accepted recommendations is tracked through the President’s Report on the Implementation Status of Evaluation Recommendations and Management Actions (PRISMA), which is presented to the Executive Board on an annual basis. In addition, several corporate-level evaluations over the last few years have resulted in major changes to IFAD’s policies and practices.

IFAD stakeholders surveyed in both 2010 and in 2013 provided mostly adequate ratings of the Fund’s performance, with a slightly improving trend over time.

Survey ratings in 2013 were slightly more positive than in 2010 and suggest that IFAD has made progress in three areas: promoting a results-oriented culture, financial management practices, and supporting funding proposals developed by direct partners. These areas were rated adequate in 2010 and strong in 2013. However, some caution is required in making comparisons between 2010 and 2013 as there were some changes in respondents, countries, and wording of questions.

As in 2010, survey respondents in 2013 generally perceived IFAD’s mandate and its corporate focus on results to be among its areas of strength. In both years they also perceived the independence of the Office of Evaluation to be one of IFAD’s strengths. In spite of an increased country presence since 2010, survey responses from in-country donors in 2013 indicate that the level of staff presence at the country level could still be increased.

Conclusions on evidence of IFAD’s development results and relevance to stakeholders IFAD’s current institutional reporting practices do not allow for a clear assessment of its contributions to actual development outcomes and impact. However, changes in the IFAD9 Replenishment Period are likely to improve these practices.

MOPAN donors at headquarters held positive views about IFAD’s contributions to results and provided ratings of adequate or strong on all areas assessed.

A review of IFAD’s performance reports indicates that the organisation is delivering outputs in its thematic priority areas and reporting on these, but its reporting on higher level development results does not include sufficient data on its contributions to outcomes and impact. In addition, IFAD has not articulated a theory of change that explains the logical or causal links between its expected results at different levels (i.e. outputs and outcomes). In the absence of a theory of change and data on outcomes and impact, it is difficult to appreciate its progress towards achieving the objectives outlined in IFAD’s Strategic Framework 2011-2015. This being said, for the IFAD9 period, the Fund has committed to the ambitious target of moving 80 million people out of poverty. For the purpose of monitoring its progress in this context, it has updated the Results Measurement Framework substantially by introducing a set of indicators at the impact level measuring its contributions to poverty reduction and food security, among others. In addition, IFAD has committed to conducting 30 impact assessments during the 2013-2015 replenishment period.

As IFAD’s planned reporting practices at the country level are not yet fully implemented, it is difficult to fully appreciate the organisation’s contributions to country-level goals and priorities. The document review and survey responses suggest that the documented results of IFAD’s operations in the six countries reviewed have been mixed.

Surveyed stakeholders hold mixed views about IFAD‘s progress towards its strategic objectives at the country level. Among the six countries sampled, surveyed stakeholders rated IFAD most positively in Viet Nam, where it was rated as strong in all results areas, and lowest in Indonesia, where it was rated inadequate in three results areas[2] and adequate in the other three. [3]

IFAD has significantly improved its focus on results at the project and country levels since the introduction of results-based COSOPs at the end of 2006. Its results statements are clearly and explicitly aligned with national goals and priorities, which was seen as a strength by both survey respondents and the document review. IFAD has provisions for annual reviews, Mid-Term Reviews, and completion reviews for country strategies (in addition to other reporting practices). However, annual reviews are not yet being performed systematically for all COSOPs; Mid-Term Reviews are often done late and with varying quality; and COSOP completion reviews have not yet been systematically introduced across all countries. These factors, combined with a general lack of systematic and credible data on project outputs and outcomes, make it difficult, if not impossible, to obtain a complete picture of the nature, magnitude, and relative importance of IFAD’s operations at the country level and the extent to which they support the MDG agenda.

Surveyed stakeholders in the six countries sampled consider IFAD to be responding well to countries’ key development priorities and, to a lesser degree, adjusting its strategies to meet the changing needs and priorities in these countries. IFAD received mixed ratings for its provision of innovative solutions to development challenges.

Survey respondents in Mozambique and Viet Nam gave IFAD strong ratings across all three indicators related to relevance. IFAD received the lowest overall country ratings in Indonesia and Guatemala. The pattern of responses was the same in all countries – that is, the highest ratings in each country were for IFAD’s responsiveness to country priorities while it received more mixed ratings for the level of innovation brought to bear on development challenges.

Overall MOPAN ratings of IFAD

The two charts below show the ratings on the key performance indicators that MOPAN used to assess IFAD in 2013. The first chart shows the ratings on 21 indicators designed to measure organisational effectiveness (practices and systems), and the second chart shows ratings on the three indicators designed to assess IFAD measurement and reporting on development results, and relevance to stakeholders.

Organisational effectiveness– overall ratings

Evidence of IFAD’s contribution to development results and relevance to stakeholders – overall ratings

1. Introduction

This report presents the results of an assessment of International Fund for Agricultural Development (IFAD) that was conducted in 2013 by the Multilateral Organisation Performance Assessment Network (MOPAN).

Background

MOPAN was established in 2002 in response to international forums on aid effectiveness and calls for greater donor harmonisation and co-ordination. The purpose of the network is to share information and experience in assessing the performance of multilateral organisations. MOPAN supports the commitments adopted by the international community to improve the impact and effectiveness of aid as reflected in the Paris Declaration on Aid Effectiveness, the Accra Agenda for Action, and the Busan High Level Forum. MOPAN’s processes and instruments embody the principles of local ownership, alignment and harmonisation of practices, and results-based management (RBM).

MOPAN provides a joint approach (known as the Common Approach) to assess the organisational effectiveness of multilateral organisations and their measurement of and reporting on development results. The approach was derived from existing bilateral assessment tools and complements and draws on other assessment processes for development organisations – such as the bi-annual Survey on Monitoring the Paris Declaration on Aid Effectiveness and annual reports of the Common Performance Assessment System (COMPAS) published by the multilateral development banks. In the long term, MOPAN hopes that this approach will replace or reduce the need for other assessment approaches by bilateral donors.

MOPAN assesses four dimensions of organisational effectiveness, and evidence of contribution to development and/or humanitarian results

MOPAN has defined organisational effectiveness as the extent to which a multilateral organisation is organised to contribute to development and/or humanitarian results in the countries or territories where it operates. Based on a survey of stakeholders, a review of documents, and interviews with multilateral organisation staff, MOPAN assessments provide a snapshot of a multilateral organisation’s effectiveness (see Chapter 3) in four dimensions:

• Developing strategies and plans that reflect good practices in managing for development results (strategic management)

• Managing operations by results to support accountability for results and the use of information on performance (operational management)

• Engaging in relationships with direct partners/clients and donors at the country level in ways that contribute to aid effectiveness and that are aligned with the principles of the Paris Declaration and subsequent related agreements on aid effectiveness (relationship management)

• Developing reporting mechanisms and learning strategies that facilitate the sharing of knowledge and information inside the organisation and with the development community (knowledge management).

In 2012, MOPAN piloted a new component to assess a multilateral organisation’s reported contributions to development results. This component, which has become an integral component of all assessments in 2013, examines three areas of performance: evidence of the extent of the progress towards its institutional/organisation-wide results, evidence of contributions to country-level goals and priorities (including relevant millennium development goals [MDGs]), and stakeholder perceptions of the relevance of its objectives and programme of work. See Chapter 4.

Purpose of MOPAN assessments

MOPAN assessments are intended to:

• Generate relevant, credible and robust information MOPAN members can use to meet their domestic accountability requirements and fulfil their responsibilities and obligations as bilateral donors.

• Provide an evidence base for MOPAN members, multilateral organisations and their partners/clients to discuss organisational effectiveness and reporting on development and/or humanitarian results.

• Support dialogue between individual MOPAN members, multilateral organisations and their partners/clients to build understanding and improve organisational performance and results over time at both country and headquarters level.

The MOPAN methodology is evolving in response to what is being learned from year to year, and to accommodate multilateral organisations with different mandates. For example, in 2011, MOPAN began adapting the indicators and approach for the review of a global fund and organisations with significant humanitarian programming. In 2013, MOPAN is using a modified framework to assess the World Food Programme (WFP), an organisation with a predominantly humanitarian mandate.

1.1 PROFILE OF IFAD

Mission and Mandate

The International Fund for Agricultural Development (IFAD) was created in 1974 during the World Food Conference to mobilise resources for agriculture and rural development in developing countries. Its goal is to empower poor rural women and men to achieve higher incomes and improved food security and to strengthen their resilience.

IFAD provides financial and technical assistance to agriculture and rural development projects with the perspective that smallholder agriculture can act as both a crucial source of rural income and nutrition and a vector for rural economic growth. IFAD takes a market-oriented approach to rural development and its projects seek to incorporate smallholders into global value chain development. IFAD has recently committed to pulling 80 million people out of poverty.

Structure and Governance

IFAD headquarters in Rome is responsible for the management and administration of the organisation. IFAD’s highest decision-making body is its Governing Council, formed by representatives of IFAD’s 172 member states. The Council meets annually to take decisions on such matters as approval of new membership, appointment of the President of IFAD, approval of the administrative budget and adoption of broad policies, criteria and regulations.

The Executive Board is the Fund's second main Governing Body, consisting of 18 elected members and 18 alternate members from three categories (OECD countries, OPEC countries and developing countries) who are elected for a three-year term. The Board meets three times a year to review IFAD’s programme of work; approve its projects, programmes and grants; and make recommendations to the Governing Council concerning policy, budget, membership and staffing within the Fund. The President of IFAD chairs the Executive Board.

Originally conceived as a highly centralised funding agency, IFAD has increased its country presence to enhance its representation and impact in borrowing countries. It has 38 country offices and is in the process of establishing two more.

IFAD currently funds 271 approved projects in 120 countries.

Strategy and services

IFAD’s strategic goals, objectives and principles of engagement are laid out in its Strategic Framework 2011-2015. IFAD’s overarching goals are to improve food security and nutrition, raise household incomes and strengthen resilience is underpinned by five strategic objectives:

• A natural resource and economic asset base for poor rural women and men that is more resilient to climate change, environmental degradation and market transformation;

• Access for poor rural women and men to services to reduce poverty, improve nutrition, raise incomes and build resilience in a changing environment;

• Poor rural women and men and their organisations able to manage profitable, sustainable and resilient farm and non-farm enterprises or take advantage of decent work opportunities;

• Poor rural women and men and their organisations able to influence policies and institutions that affect their livelihoods; and

• Enabling institutional and policy environments to support agricultural production and the full range of related non-farm activities.

IFAD also identifies eight areas in which it will continue to concentrate its efforts: natural resources, climate change adaptation and mitigation, improved agricultural technologies and effective production services, inclusive financial services, integration of poor rural people within value chains, rural enterprise development and non-farm employment opportunities, technical and vocational skills development, and support to rural producers’ organisations.

Finances

IFAD is funded by voluntary contributions provided by member state governments, other multilateral organisations, and foundations. Most of IFAD’s operations involve a significant level of co-financing and, in 2012, domestic co-financing was almost equal to IFAD’s own commitment.

In 2012, IFAD’s total loan and grant operations amounted to USD5.877 billion, a 25% increase over the IFAD8 replenishment period.[4]

Change and reform

In response to an independent external evaluation conducted in 2005, IFAD developed an action plan for improving IFAD’s development effectiveness, which laid out key areas for improvement, including managing for development results, quality of projects, strategic planning, innovation, efficiency and human resources.

IFAD has made a number of significant changes to its practices and systems. One of its most significant changes involves moving to a direct supervision model and increasing its formal country presence in 40 borrowing countries. On the human resource front, IFAD has implemented a human resource reform agenda, including performing a job audit and creating a Strategic Workforce Plan (SWP). IFAD has made changes to its Results-based Management system, including modifying its country strategic opportunity programmes (COSOPs) to integrate RBM requirements. IFAD has also developed a number of new policies to improve its strategic planning, including a Private-Sector Partnership Strategy, a Policy on Gender Equality and a Climate Change Strategy. Finally, following the 2010 Peer Review of IFAD’s Office of Evaluation and Evaluation Function, IFAD has made important changes to its self-evaluation and independent evaluation systems.

In its Ninth Replenishment Period (2013-2015), IFAD has committed to a number of actions in the following four areas: operational effectiveness, institutional effectiveness and efficiency, financial management, and results management.[5] To improve its operational effectiveness, IFAD plans to undertake a series of measures focusing on: aid effectiveness, south-south co-operation, private sector engagement, gender equality and women’s empowerment, project efficiency, advocacy, climate change, fragile states and country-level decentralisation. IFAD intends to adopt new management tools for cost analysis and consolidated human resource practices, in order to enhance its institutional effectiveness. It also seeks to upgrade its financial model in line with industry best practices. Lastly, IFAD is placing more emphasis on impact evaluation in order to better demonstrate the outcomes achieved by the Fund.

The IFAD website is www.ifad.org.

1.3 PREVIOUS ASSESSMENTS

IFAD was first assessed by MOPAN in 2010. IFAD was seen to perform well in the areas of strategic management, operational management, and knowledge management. It received lower ratings in the areas of relationship management and the delegation of decision making. Respondents rated IFAD as inadequate for some aspects of its use of country systems, the length of time required to complete its administrative procedures, and in its monitoring of staff performance and deployment of staff in country offices. The assessment noted room for improvement in linking disbursements to reported development results.

Changes since 2010

The 2013 survey ratings were slightly higher on most key performance indicators and only one KPI registered a very slight decline (contributing to country dialogue). From 2010 to 2013 IFAD’s survey ratings increased from adequate to strong on the following four KPIs:

• Providing direction for results – Survey respondent ratings improved from adequate in 2010 to strong in 2013 on two questions: whether IFAD culture reinforces a focus on results, and whether senior management shows leadership on results.

• Resource allocation decisions – In 2010 and 2013 respondents were asked (in slightly different words) whether IFAD makes publicly available its criteria for allocating funding, and whether resource allocation respects these criteria. The ratings improved slightly, but resulted in an overall rating of strong in 2013.

• Financial accountability – Respondent ratings improved from adequate in 2010 to strong in 2013 on whether IFAD internal financial audits provide credible information to its governing bodies.

• Supporting national plans – Survey respondent ratings improved from adequate in 2010 to strong in 2013. This was based on a single question which asked if IFAD supports funding proposals designed and developed by national government or direct partners.

1.4 IFAD SURVEY RESPONDENTS

While there are variations in the response rates by category and location of respondents, IFAD survey results reflect the views of 193 respondents.[6]

Figure 1.1 | Number of survey respondents and total population for IFAD by country and respondent group (n = 193)

A total of 193 individuals provided their views on IFAD’s performance in the areas of organisational effectiveness and contribution to development results. The distribution of responses on the six survey rating options plus ‘don’t know’ answered are summarised by respondent group across all survey questions in Figure 1.2.

Figure 1.2 | IFAD - Distribution of responses on all questions related to micro-indicators (n = 193)

While there were responses in all six possible choices, relatively few responses overall were at the ‘weak’ end of the scale. More than one quarter of responses from donors in-country were ‘don’t know,’ which is significantly higher than the level of ‘don’t know’ responses provided by other groups. Direct partners responded more positively than other groups overall, with 16% of their responses within the ‘very strong’ category responses.

2. MOPAN methodology 2013

2.1 OVERVIEW

The detailed MOPAN 2013 methodology, which includes rating scales, data analysis criteria, and strengths and limitations of the methodology, can be found in Volume II, Appendix I. The following is a brief summary. MOPAN assessments examine:

• Organisational effectiveness: organisational systems, practices, and behaviours that MOPAN believes are important for aid effectiveness and that are likely to contribute to development and/or humanitarian results at the country level; and

• Humanitarian and development results: the extent to which there is evidence of an organisation’s contributions to development and/or humanitarian results.

As the MOPAN methodology has been refined each year since 2003, comparisons of this year’s assessments and any previous MOPAN assessments of an organisation should take this into consideration.

Data collection methods and sources

Over the years, MOPAN developed a mixed methods approach to generate relevant and credible information that MOPAN members can use to meet their domestic accountability requirements and support dialogue with multilateral organisations that they are funding.

The MOPAN approach uses multiple data sources and data collection methods to triangulate and validate findings. This helps eliminate bias and detect errors or anomalies.

In 2013, the sources of data included surveys of the multilateral organisation’s stakeholders and of MOPAN donors at headquarters and country level and a review of documents prepared by the organisations assessed and from other sources.

Assessment of organisational effectiveness

In the organisational effectiveness component, MOPAN examines performance in four areas: strategic management, operational management, relationship management, and knowledge management. Within each performance area, effectiveness is described using key performance indicators (KPIs) that are measured through a series of micro-indicators (MIs) using data from the survey and document review.

In this component, survey respondent ratings are shown as mean scores and are presented alongside document review ratings based on criteria defined for each micro-indicator. Not all micro-indicators are assessed by both the survey and the document review. The charts show survey scores and document review scores for the relevant KPIs or MIs. The full list of micro-indicators comprising this component is provided in Volume II, Appendix V.

Assessment of development and/or humanitarian results

In the development results component, MOPAN does not assess an organisation’s results on the ground, but examines how it measures and reports on its contributions to development and/or humanitarian results through three key performance indicators:

• Evidence of the extent of the organisation’s progress towards its stated institutional/organisation-wide results (i.e. goals, objectives, outcomes)

• Evidence of the extent of the organisation’s contributions to country-level goals and priorities, including relevant millennium development goals (MDGs)

• Relevance of the organisation’s objectives and programme of work to stakeholders.

In this component, a “best fit approach” is used in determining the ratings for the first two KPIs above. One of four qualitative ratings (strong, adequate, inadequate, or weak) is assigned following an analysis of data from all sources and confirmed in a consensus-based consultation (with of institutional advisors, a panel of experts, and MOPAN members). The rating of the KPI on relevance is based on stakeholder surveys only.

2.2 DATA SOURCES AND RATINGS

Survey

MOPAN gathers stakeholder perceptions through a survey of MOPAN members (at headquarters and in- country) and other key stakeholders of the multilateral organisation. Donor respondents are chosen by MOPAN member countries; other respondents are identified by the multilateral organisation being assessed.

The survey questions relate to both organisational effectiveness and to the achievement of development and/or humanitarian results. Survey respondents are presented with statements and are asked to rate the organisation’s performance on a six-point scale where a rating of 1 is considered “very weak” up to a rating of 6 which is considered“very strong.”Mean scores are calculated based on weighted data and a mean score is calculated for each respondent group (e.g. donors at headquarters). (For more details on the weighting formula, please see Volume II, Appendix I.)

MOPAN aims to achieve a 70 per cent response rate from donors at headquarters and a 50 per cent response rate among respondents in each of the survey countries (i.e. donors in-country and other respondent groups such as direct partners/clients).

Document review

The document review considers four types of documents: multilateral organisation documents; internal and external reviews of the organisation’s performance; external assessments such as the Survey on Monitoring the Paris Declaration, the Common Performance Assessment (COMPAS) report, and previous MOPAN surveys; and evaluations, either internal or external, of the achievement of results at various levels.

Document review ratings are based on a set of criteria that MOPAN considers to represent good practice in each area. The criteria are based on existing standards and guidelines (for example, UNEG or OECD-DAC guidelines), on MOPAN identification of key aspects to consider, and on the input of subject-matter specialists. The rating for each micro-indicator depends on the number of criteria met by the organisation.

Interviews

Interviews are conducted with staff based at headquarters and country offices of multilateral organisations who are knowledgeable in areas that relate to the MOPAN assessment. Interview data are not coded or used as a formal data source but rather to help ensure that the Assessment Team has all appropriate and necessary documents to enhance the triangulation of data from various sources and provide contextual information to assist in the analysis of the key performance indicators.

2.3 STRENGTHS AND LIMITATIONS OF THE MOPAN COMMON APPROACH

MOPAN continues to improve methodology based on experience each year. The following strengths and limitations should be considered when reading MOPAN reports.

Strengths

• The MOPAN Common Approach is based on existing bilateral assessment tools with the intent to replace or reduce the need for other assessment approaches by bilateral donors.

• In line with donor commitments to aid effectiveness, it seeks perceptual information from different stakeholder groups.

• It uses multiple sources of data to increase the validity of the assessment, enhance analysis, and provide a basis for discussion of agency effectiveness.

• MOPAN reports are validated and reviewed by MOPAN members and by the multilateral organisation being assessed.

Limitations

• The countries selected for MOPAN assessments comprise only a small proportion of each institution’s operations, thus limiting generalisations.

• Although MOPAN uses recognised standards and criteria for good practice, such criteria do not exist for all indicators. Many document review criteria were developed by MOPAN; these are a work in progress and not definitive standards.

• Survey rating choices may not be used consistently by all respondents. Some respondents may tend to avoid extremes on a scale and respondents in some cultures may be unwilling to criticise or too eager to praise.

• The survey covers a broad range of issues and individual respondents may not have the knowledge to respond to all the questions relating to a given organisation.

• There are advantages and disadvantages to any weighting formula that is applied. In the IFAD assessment, there were very small numbers of in-country donor respondents in some of the countries and given the weighting formula applied it is important to note that their opinions have particular influence on the mean scores for questions that were only asked at the country level.

3. Main Findings: IFAD’s organisational effectiveness

3.1 INTRODUCTION

This chapter presents the findings of the 2013 MOPAN assessment of IFAD’s organisational effectiveness, that is its practices and systems that support the achievement of results. Findings are based on respondent survey data and document review.

• Section 3.2 presents overall ratings on the performance of IFAD and summarises respondent views on its primary strengths and areas for improvement;

• Section 3.3 provides findings on each of the four areas of performance (strategic, operational, relationship, and knowledge management).

3.2 OVERALL RATINGS

This section provides a summary of survey and document review ratings for all key performance indicators. It also presents survey respondent ratings of IFAD’s overall organisational effectiveness and a summary of their written comments on IFAD strengths and areas for improvement.

Overall ratings of key performance indicators

Figure 3.1 shows scores from the document review and the survey on key performance indicators (KPIs) in the MOPAN 2013 assessment of IFAD’s practices and systems. The white bar presents the survey score, while the black square presents the document review score. For example, on the first indicator, “providing direction for results”, IFAD received a score of 4.55 (strong) in the survey and a score of 6 (very strong) in the document review.

In the overall ratings from the survey and document review, IFAD was seen to perform adequately or better on all of the key performance indicators assessed.[8]

Figure 3.1 | Overall ratings on key performance indicators of IFAD’s organisational effectiveness (survey mean scores and document review ratings)

Survey ratings of IFAD’s organisational effectiveness

MOPAN has defined “organisational effectiveness” as the extent to which a multilateral organisation is organised to support direct partners in producing and delivering expected results. Respondents were asked the question: “How would you rate the overall organisational effectiveness of IFAD?” As shown in Figure 3.2, direct partners gave the largest proportion of ratings on the high end of the scale (54%).

Figure 3.2 | Overall ratings of IFAD organisational effectiveness by respondent group (n= 197)

Respondents’ views on IFAD’s strengths and areas for improvement

Prior to rating IFAD on Common Approach micro-indicators, survey respondents were asked two open- ended questions about IFAD’s greatest strengths and areas for improvement. The 193 survey respondents made 200 comments on IFAD’s strengths and 194 on areas for improvement.[9]

Overall, survey respondents considered IFAD’s greatest strengths to be its specific focus on rural development and agriculture in poor remote areas, and its ability to provide technical and capacity- building support.

• 40% of survey respondents (n=81) cited IFAD’s specialised or “niche” mandate as its greatest strength. Most HQ donor respondents identified IFAD's clear mandate as its primary strength, pointing out that IFAD is the only UN organisation that focuses exclusively on poverty alleviation for the rural poor, particularly smallholder farmers. Respondents stated that no other IFI is mandated to focus on the rural poor and on agricultural and rural development.

• 27% (n=55) commented that IFAD’s open- minded, creative and knowledgeable staff members are able to provide technical and capacity-building support and this is one of its key strengths.

• 17% (n=33) mentioned that one of IFAD’s greatest strength is its close relationship with smallholder/ rural community organisations.

• 17% (n=34) noted IFAD’s alignment with national priorities and collaboration with government ministries.

Direct partners also cited IFAD’s rigorous and professional project planning and management process (7% n=11), and noted that IFAD makes efforts to be participative (5%, n=8).

Survey respondents identified several areas for improvement but no single area stands out overall. A considerable number of respondents commented on the need for greater project efficiency.

Survey respondents noted that IFAD could improve its performance in the following five areas:

• 23% of survey respondents (n=45) cited project efficiency as the most notable area for improvement. This included comments on improving efficiency throughout the project cycle (from project design and planning to project implementation, management and evaluation), length of bureaucratic processes, and efficiency of project management including project preparation and staff recruitment.

• 12% (n=23) commented that IFAD could improve its collaboration and engagement with local stakeholders and community groups (beneficiaries).

• 12% (n=24) suggested the need for increased country presence in remote, poor rural areas, although it is interesting that 16% (n=32) also identified IFAD’s connections with local networks as a strength.

• 11% (n=21) commented on the need for greater outreach and collaboration with other donors, IFIs and Rome-based agencies.

• 11% of respondents (n=21) found room for improvement in IFAD’s knowledge management (e.g. collection and dissemination of lessons learned, more proactive in providing information on its programme results). This could provide IFAD with resources to assist its participation in policy dialogue at the national level. According to one respondent, "IFAD needs to develop a more systematic approach to knowledge management and encourage a culture of open dialogue from senior management level."

There were some differences between respondent groups.

Donors at headquarters: 23% (n=7) noted that IFAD should improve its knowledge management and dissemination, and 19% (n=6) identified human resources (e.g. staff recruitment) as key areas for improvement.

Donors in-country: 30% (n=6) suggested that IFAD improve its co-ordination with other international organisations, particularly with other Rome-based agencies such as FAO and WFP; 20% (n=4) felt that IFAD’s country presence is insufficient; and 20% (n=4) felt that IFAD should improve its evaluation practices and develop better performance indicators.

Direct partners: 28% (n=40) suggested that limited project efficiency (e.g. lengthy loan approval and procurement processes and poor project and financial management) is an impediment to IFAD’s organisational effectiveness; 11% (n=16) mentioned collaboration with village and community institutions as an area for improvement.

3.3 IFAD’S PERFORMANCE IN STRATEGIC, OPERATIONAL, RELATIONSHIP, AND KNOWLEDGE MANAGEMENT

3.3.1 Overview

This section presents the findings of the 2013 Common Approach assessment of IFAD in four organisational performance areas (quadrants): Strategic, Operational, Relationship, and Knowledge Management.

The following sections (3.3.2 to 3.3.5) provide the overall survey and document review ratings for the KPIs in each performance area, the mean scores by respondent group, and findings based on an analysis of survey and document review ratings in each performance area.

When there were notably divergent ratings between survey respondent groups or between the survey results and document review ratings, these are noted and the information gleaned from interviews with staff is integrated when it has a bearing on the analysis. Where statistically significant differences among categories of respondents were found, these differences are also noted.

The survey data for each KPI and MI by performance area are presented in Volume II, Appendix V. The document review ratings are presented in Volume II, Appendix VI.

3.3.2 Strategic management

IFAD’s strategic management is one of its strengths. IFAD was judged by survey respondents to be generally strong or adequate in its strategic focus on results. The document review ratings ranged from adequate to very strong.

Figure 3.3 shows the overall survey and document review ratings for the five KPIs in the strategic management performance area.

The review of key documents such as the Results Measurement Framework 2013-2015 revealed an organisation with a strong results focus. Since the last MOPAN assessment, IFAD has made large strides in addressing cross-cutting priorities, such as gender, environment and food security. IFAD also intends to carry out a number of impact evaluations by the end of 2015, which should strengthen its ability to link its operations with development outcomes.

Figure 3.3 | Performance area I: Strategic management (survey and document review ratings)

Figure 3.4 shows the mean scores for the five KPIs for all survey respondents, and by category of respondent.

Figure 3.4 | Performance area I: Strategic management, mean scores by respondent group

KPI 1: Providing direction for results

Finding 1: Survey respondents considered IFAD to be strong in providing direction for results-based management. The document review rated IFAD as very strong for the implementation of its Results Measurement Framework.

IFAD states that it is fully committed to a Managing for Development Results (MfDR) approach. Examples of this approach can be found in the results frameworks included in most of IFAD’s recent policies and strategies and in country strategy documents, known as results-based Country Strategic Opportunities Programmes (RB-COSOPs).

Figure 3.5 | KPI 1: Providing direction for results, ratings of micro-indicators

MI 1.1 – Value system supports results-orientation and direct partner focus

In this MI, which was assessed by survey only, survey respondents were asked whether IFAD’s institutional culture reinforces a focus on results and if IFAD’s institutional culture is direct-partner focused. On both questions and across all respondent groups 74% rated IFAD as adequate or above.

MI 1.2 – Leadership on results management

This MI was assessed by survey only. Among MOPAN donors at headquarters, the only group asked about this MI, 84% rated IFAD as adequate or above in showing leadership on results management.

MI 1.3 – Organisational policy on results management

In the survey, donors at headquarters were asked about the extent to which IFAD ensures the application of results management across the organisation. The majority (71%) rated IFAD as adequate or above.

IFAD was rated as very strong on this MI by the document review. The Fund has adopted guidelines on results measurement, which are found in the Results Measurement Framework (RMF) 2013-2015.[10]

Although not defined as a policy per se, the RMF presents IFAD’s institution-wide approach to results measurement. It includes five levels of results allowing the organisation to assess and track progress on development outputs and outcomes, as well as reporting on operational and organisational efficiency and effectiveness. IFAD provides guidelines for reporting on different results levels in a handbook on the Results and Impact Management System[11] and in the Practical Guidance for Impact Surveys.[12] During the consultations on IFAD’s replenishment, IFAD members review and suggest improvements to the RMF. During the Third Session of the Consultation on IFAD’s Ninth Replenishment in 2011, members agreed that IFAD should add measures and indicators to:

• Provide both broader and more in-depth assessment of development impact;

• Better monitor performance on environment and natural resource management, partnerships, project- level monitoring and evaluation (M&E), and operations in fragile states; and

• Sharpen the focus on IFAD’s scaling-up objectives.[13]

The Human Resources Division of IFAD offers courses to its staff on results-based management topics, including logframe development and monitoring and evaluation.

KPI 2: Corporate strategy and mandate

Finding 2: IFAD’s mandate is clear and its Strategic Framework is well aligned with this mandate, as noted by both survey respondents and the document review.

Overall, IFAD was perceived as strong by survey respondents who were asked about IFAD’s overall alignment between mandate and organisation-wide strategies. These ratings were echoed in their written comments: 40% of respondents considered IFAD’s specialised mandate focused on providing aid for agriculture and rural development as its greatest strength.

Figure 3.6 | KPI 2: Corporate strategy and mandate, ratings of micro-indicators

MI 2.1 – Corporate strategy based on clear mandate

Donors at headquarters were asked two questions on this MI: whether IFAD has a clear mandate, and whether IFAD’s organisation-wide strategy (Strategic Framework 2011-2015) is aligned with the mandate. The majority (97%) rated IFAD as adequate or above on both questions.

The document review provided a very strong rating on this MI. IFAD’s organisation-wide Strategic Framework 2011-2015[14] and the Agreement Establishing the International Fund for Agricultural Development[15] are key documents that set out IFAD’s mandate and development goals. IFAD’s goals are periodically reviewed at each Replenishment of Resources cycle, and strategic goals and priority areas are revised with each Strategic Framework. There are explicit links between IFAD’s mandate and the eight thematic areas set out in IFAD’s current Strategic Framework.

IFAD also produces a Medium-Term Plan to set out a rolling, three-year corporate workplan. This plan describes how the Fund intends to meet its strategic objectives of rural poverty reduction and food security – for example, through the allocation of financial resources (loans and grants) and human resources for each strategic objective.

KPI 3: Corporate focus on results

Finding 3: IFAD has a strong focus on results at an organisation-wide level. However, the link between outputs and outcomes could be strengthened.

IFAD’s corporate focus on results was assessed by document review only, which rated IFAD as adequate overall. It noted that IFAD’s results measurement framework is regularly revised – the latest RMF includes ten new indicators, including four impact indicators. IFAD states that it is fully committed to a results- focused approach. Each corporate strategy includes a Results Framework that sets out key deliverables against which IFAD reports (e.g. in the Annual Report on IFAD’s Policy on Gender Equality; in the Progress Report on IFAD’s Change and Reform Agenda; etc.).

However, the document review also noted that IFAD’s results frameworks do not provide an explicit Theory of Change – that is, a complete picture of how the organisation as a whole envisions its contribution to global agriculture and poverty reduction objectives.

Figure 3.7 | KPI 3: Corporate focus on results, ratings of micro-indicators

MI 3.1 – Plans and strategies contain results frameworks

The document review rated IFAD as adequate in outlining management and development results frameworks. IFAD’s Results Measurement Framework 2013-2015 contains a five-point results chain, which includes both development and management results.

• Levels 1, 2 and 3 contain Development Results indicators.

− Level 1 indicators represent global development outcomes (i.e. progress towards MDG1) to which IFAD contributes.

− Level 2 indicators measure country-level development outcomes that are delivered by IFAD-supported programmes.

− Level 3 indicators track outputs delivered by IFAD-supported programmes (e.g. in the areas of natural resource management, agricultural technologies, rural financial services, marketing, microenterprise and policies and institutions).

• Levels 4 and 5 track management results. At these levels, IFAD monitors the quality and performance of its projects, as well as its progress on institutional management and efficiency.

IFAD monitors indicators at levels 2 and 3 by comparing current values to targets and baselines using reported changes to show progress towards expected results.

Through quarterly corporate performance reports, IFAD tracks ten management indicators related to its organisational effectiveness. These are called corporate management results and they track four areas of activities from aid and programme harmonisation to financial and human resource management.

IFAD’s Results Measurement Framework is regularly revised. The RMF for 2013-2015 incorporates ten new indicators to ensure a stronger focus on impact measurement and on additional topics of interest to IFAD, such as scaling up, environment and natural resource management, and gender. This RMF includes four impact indicators: household asset ownership, level of child malnutrition, length of hungry season and number of people moved out of poverty.

MI 3.2 – Results frameworks link outputs to final outcomes/impacts

The document review rated IFAD as adequate on this MI. The organisation has established a RMF based on a five-level results chain with links between organisational and operational effectiveness (levels 4 and 5) and project outputs (Level 3) and implicit links between outputs (level 3) and development outcomes and impact (level 2).

While the implicit logic between levels 2 and 3 is relatively clear in the RMF as a whole, IFAD has not clearly articulated the links (anticipated or actual) between project performance (i.e. delivery of activities and outputs) and development outcomes nor has it established a clear theory of change that explicitly and plausibly explains how IFAD-funded activities are intended to lead to intended development outcomes.

While the RMF and IFAD’s Strategic Framework 2011 – 2015 clearly include both development and management results, some of the results under Level 3 labelled as “outputs” more closely resemble “immediate outcomes” (e.g. 3.5 Voluntary savers (with male: female ratio); 3.6 Active borrowers (along with male: female ratio); 3.7 Value of loans and savings mobilised – USD million; 3.8 Value of gross loan portfolio; 3.12 Enterprises accessing facilitated non-financial services).

Most of the results measured at Level 2.1 are based on performance evaluation criteria (e.g. relevance, effectiveness, rural poverty impact) rather than indicators of actual development outcomes. These performance criteria are measured by expert assessment of a project’s performance at project completion using a six-point scale (a rating of 1 to 3 is considered to reflect an overall negative performance, while a rating of 4 to 6 reflects an overall positive performance). Level 2.1 indicators are calculated using different data sources from level 3: the fact that IFAD “outcomes” at level 2 are not based on RIMS data, unlike level 3, makes it difficult to establish links between these results levels. In addition, IFAD’s use of the term “outcomes” does not correspond to standard definitions of the term.[16]

This being said, IFAD should be commended for developing additional indicators in its new RMF (2013- 2015) that will provide a stronger focus on impact measurement, as well as on scaling-up, environment and natural resource management, and gender. In addition, IFAD has committed to conducting 30 impact studies between 2013 and 2015, which will provide more information about how and to what extent projects supported by IFAD contribute to development results.

MI 3.3 – Plans and strategies contain performance indicators

The document review rated IFAD as adequate on this MI. Performance indicators are included in all levels of the development results framework. To track these indicators, IFAD’s RMF has established baselines and targets at the outcome level. Given the demand-driven nature of IFAD’s operations, baselines are established at the output level, but not targets.

IFAD indicators were examined in terms of SMART and CREAM criteria. More than half of the IFAD output indicators are ‘specific’ and ‘measurable’, and progress can be assessed by comparing baselines and results. However, output targets are not set. It is difficult to tell if they are ‘adequate’ (meaning that each indicator provides a sufficient basis to assess performance) or if they are ‘economic’ as no information is provided on the cost of obtaining the output data (see RIDE 2012, table 4, p.8). There also seems to be some confusion between outputs and outcomes; the assessment team considers, for instance, that “number of voluntary savers” should be an outcome rather than a project output.

Outcome indicators partially meet CREAM or SMART criteria.[17] As with the output indicators, it is difficult to determine whether outcome indicators are adequate or economic. While outcome indicators are monitorable, they are not very specific (see RIDE 2012, Charts on pp.10-11).

Level 2.1 indicators do not conform to standard OECD definitions.[18] As noted above, most of the “indicators” at Level 2.1 are in fact performance criteria (relevance, effectiveness, efficiency, rural poverty impact, etc.) that are based on perceptions of performance rather than a clear and unambiguous measure of actual progress towards outcomes and impact. According to Rist & Kusek who first defined the CREAM criteria in a World Bank publication from 2004:

“Public sector management is not just about documenting perceptions of progress. It is about obtaining objective information on actual progress that will aid managers in making more well-informed strategic decisions, aligning budgets, and managing resources.”

This observation is becoming more important in a global development context where the public, donors and other stakeholders increasingly demand “value for money” in their collaboration with development institutions. From this perspective, it is a positive step forward that IFAD’s updated RMF includes new impact indicators that should provide information about actual development impacts.

KPI 4: Focus on cross-cutting priorities[19]

Finding 4: IFAD is considered fairly strong in mainstreaming gender equality, environment, and food security and nutrition. Survey respondents rated the organisation as adequate in promoting good governance and human rights-based approaches, but the assessment team recognises that IFAD does not address these themes directly.

The assessment examined five cross-cutting thematic priorities identified by MOPAN: gender equality, environmental practices, good governance, human rights-based approaches, and strategies to promote the resilience of household food security and nutrition. The document review noted that IFAD is getting better at reporting on progress in its cross-cutting priorities (e.g. reporting on indicators that relate to gender equality, environment and nutrition).

Figure 3.8 | KPI 4: Focus on cross-cutting priorities, ratings of micro-indicators

MI 4.1 – Gender equality

The majority of survey respondents (76%) rated IFAD as adequate or above for the extent to which it mainstreams gender equality in its programmatic work and16% rated it inadequate or lower.

The document review rated it as strong and found that IFAD has made great progress in mainstreaming gender into its strategies, policies and operations since 2010. The organisation has adopted a new Policy on Gender Equality and Women’s Empowerment in response to the recommendations of the 2010 corporate-level evaluation (CLE) on gender.[20] The key findings and recommendations of the CLE were that IFAD had done well in addressing gender issues through country strategies and a results-based approach but should more systematically document lessons learned and should develop an evidence and results- based policy on gender equality.[21]

IFAD’s Policy on Gender Equality presents a clear distribution of roles and responsibilities for gender mainstreaming at IFAD. Furthermore, the policy calls for the allocation of corporate human and financial resources as well as an accountability framework that fully supports gender equality and women’s empowerment. The document review found that IFAD has adequate human resources for this task, but has not yet provided a description of its financial resources dedicated to this work. The gender desk is housed within the Policy and Technical Advisory Division (PTA), two permanent staff members (P-4 and P-5), as well as two consultants provide support and guidance to other staff members implementing gender mainstreaming activities at IFAD. Gender focal points in most of divisions provide gender expertise for the design and implementation of IFAD programmes and projects.

IFAD produced its first Annual Report on its Policy on Gender Equality in 2012. This report will be the primary vehicle for official reporting on IFAD’s progress on meeting gender equality targets and for monitoring and accountability.

In sum, IFAD has made significant progress in its strategies and policies on gender equality. The key challenge will now be to translate these policies into results at the operational level. IFAD indicated in its annual report that it will focus in 2012-2013 on improving capacity-building, skills development and monitoring of gender equality. It will also participate in the launch of a joint programme on women’s economic empowerment with the Rome-based agencies (FAO, WFP) and UN Women.

MI 4.2 – Environment

All survey respondents were asked whether IFAD sufficiently mainstreams environmental considerations in its operations. The majority (83%) rated IFAD as adequate or above and 8% as inadequate or weak.

The document review rated IFAD as very strong for mainstreaming environment in its operations. IFAD has stepped up efforts to improve the integration of sustainable practices across its activities and recently created the Environment and Climate Division (ECD) to co-ordinate these efforts. One of the key responsibilities of this new division is to manage the Adaptation for Smallholder Agriculture Programme (ASAP), a multi-year, multi-donor programme launched in 2012, to which donors have pledged USD 247 million and which is currently expected to co-finance one third of IFAD-assisted projects.

Although IFAD’s latest performance reports do not include specific outcome level reporting on climate change and the environment, its 2013 – 2015 Results Measurement Framework does include a new indicator on environment and natural resources management. In addition, the Climate Change Results Implementation Framework commits IFAD to mainstreaming environment and climate change activities by requiring all new COSOPs to systematically and appropriately reflect climate and environment issues.[22]

In the course of 2012, efforts have been made to integrate the climate change dimension into the IFAD results management system, more specifically new reporting lines on climate change have been included in the Project Completion Report (PCR) template, the Supervision Report template, the Guidelines for Portfolio Review and the sub-categories and themes of the Project Portfolio Management System (PPMS).

IFAD has also invested a great deal in its human and financial resources for environment and climate change recently and the Climate Change Strategy (approved in 2010)[23] defines, in broad terms, how the mainstreaming of these priority areas will be organised across the organisation. The role of the ECD in mainstreaming climate change and the environment is also clearly described in the Climate Change Strategy, and since 2011 the ECD has implemented a new organisational structure designed around the creation of dedicated staff positions with expertise on climate and environment shared with the regional divisions. The funding available for IFAD’s environmental mainstreaming activities is clearly described in the ECD budget. In addition, IFAD’s 2013 programme of work and budget provides details on the additional cost of supporting the ASAP.

However, given the fact that IFAD only recently increased its focus on climate change and the environment it is still too early to assess its actual progress and results achieved related to climate change mitigation and adaptation.

MI 4.3 – Good governance

Survey respondents were asked whether IFAD sufficiently promotes the principles of good governance in its work, and the majority (73%) rated it as adequate and above. Direct partners were more positive than MOPAN donors in-country and at headquarters and this difference is statistically significant.

Strictly adhering to the document review criteria, IFAD received a rating of weak on this MI. IFAD does not have an official policy on good governance, does not consider good governance to be a cross-cutting issue or specific priority area, nor does it allocate specific funds or reports on its progress in the area of good governance. However, due to its specialised mandate and the fact that it deals with recipient member states at a microeconomic rather than macroeconomic level, good governance in IFAD is quite different than in other international financial institutions and the extent to which it can promote principles of good governance is limited. However, the Fund does promote good governance through indirect means, such as its Anti-Corruption Policy. The policy, which covers staff, individuals representing IFAD and borrowers, aims at preventing fraud and corruption in IFAD-supported programmes and projects. Good governance is also promoted indirectly through IFAD’s Performance Based Allocation System (PBAS), which includes a section on governance in the Rural Sector Performance scoring sheet (sector E. Public resources management and accountability). In response to recommendations presented in the “Annual Report on Quality Assurance in IFAD’s Projects and Programmes” (2012),[24] IFAD management also introduced stronger governance and anti-corruption measures from the design stage of projects in countries with a low corruption perceptions index.[25]

IFAD has a strict zero tolerance policy towards fraudulent, corrupt, collusive or coercive actions. Through its Anti-Corruption Policy (2005) and other transparency measures (such as participation in the International Aid Transparency Initiative), IFAD contributes to the promotion of good governance in agriculture and rural development projects.

MI 4.4 – Human rights-based approaches

Donors and direct partners of IFAD were asked whether IFAD sufficiently promotes the principles of human rights in its operations, through its approach to social inclusion. Most survey respondents (73%) rated IFAD as adequate or above on this question, 18% responded ‘don’t know’, and 9% rated it inadequate or lower. The MOPAN assessment defines human-rights-based approaches (HRBA) as a cross-cutting theme.[26] As neither IFAD’s mandate nor its results framework include a specific focus on HRBA, an agreement was reached to focus on IFAD’s promotion of the principle of social inclusion, which can be linked to HRBA. IFAD has a specific Targeting Policy adopted in 2006, which states that IFAD “proactively strives to reach extremely poor people (as defined by MDG 1) who have the potential to take advantage of improved access to assets and opportunities for agricultural production and rural income-generating activities.”[27]

This approach fits well with IFAD’s mandate to enhance food security and reduce poverty in rural areas in the developing world.

However, despite the agreed modification in the focus of the assessment for this MI, the assessment team did not find sufficient documentary evidence upon which to rate this indicator.

MI 4.5 – Food security and nutrition

Survey respondents were asked whether IFAD sufficiently promotes strategies to improve food security and nutrition at the household level. The majority (90%) rated IFAD adequate or above on this MI. Of the five cross-cutting issues assessed, this MI received the highest survey ratings. Among respondent groups, 80% of direct partners rated IFAD strong or very strong, compared to 61% of donors at headquarters and 57% of donors in-country.

The document review rated IFAD as strong on this MI. Food security and nutrition is one of IFAD’s overarching objectives and two of the key components of its mandate. Following this mandate, IFAD’s Strategic Framework 2011 – 2015 sets out its policy and provides overall guidance for its operations relating to food security and nutrition. IFAD currently reports on progress towards its food security and nutrition objectives indirectly through the performance criterion “Rural Poverty Impact” under Level 2 in the RMF. However, the RMF 2013-2015 developed for the IFAD9 Replenishment Period includes two new indicators that directly relate to food security and nutrition, namely “level of child malnutrition” (3 sub-indicators - acute, chronic and underweight), disaggregated for girls and boys; and “length of the hungry season.”[28]

KPI 5: Country focus on results

Finding 5: MOPAN donors in-country and direct partners agreed that IFAD’s country strategic opportunity programmes (COSOPs) adequately focus on results. The six COSOPs sampled were well aligned with national development plans, although the link between results frameworks at the project/programme and the country level was not always evident.

IFAD revised its approach to producing country strategy papers in 2006 and began producing results- based country strategic opportunity programmes (RB-COSOPs) which place more emphasis on results, accountability and country ownership.[29] As IFAD’s main planning instrument at the country level, COSOPs review contextual factors such as the specific economic situation of the country to determine the focus of its operations (geographic as well as thematic) and discuss the political context and other institutional factors that might affect the outcome and impact of its operations. COSOPs also review lessons learned from previous operations and include a results management framework that identifies strategic objectives and indicators. The assessment team understands that the COSOPs are developed for long-term strategic planning purposes and are not meant to provide a detailed results perspective. However, they remain the primary source of evidence for IFAD’s country (rather than project) level planning. As the unit of analysis for this key performance indicator is the country, COSOPs were the main document used.

Figure 3.9 | KPI 5: Country focus on results, ratings of micro-indicators

MI 5.1 – Frameworks link results at project, programme, sector and country levels

MOPAN donors in-country and direct partners were asked whether IFAD’s RB-COSOPs link results from project, sector and country levels. The majority of survey respondents (76%) rated IFAD as adequate or above and 13% inadequate or below on this MI.

IFAD received a rating of adequate from the document review on this MI. All of the COSOPs reviewed include results frameworks that identify performance indicators at the output or“milestone”level and at the outcome level.[30] They also provide explicit links to some of IFAD’s eight priority areas as defined in the 2011 – 2015 Strategic Framework. However, as the COSOPs are developed to provide a broad framework for making strategic choices about IFAD’s operations in a country, they do not provide explicit links between expected project-level results and country-level results. According to IFAD, more detailed log frames are normally found in project planning documents developed against the overall strategic objectives set in the COSOPs.

MI 5.2 – Frameworks include indicators at project, programme, sector and country levels

MOPAN donors in-country and direct partners were asked whether RB-COSOPs include indicators at all levels (country, sector, and project/programme). The majority of respondents (72%) rated this MI as adequate or above, and 22% answered ‘don’t know’.

The document review rated IFAD as adequate on this MI. The unit of analysis for this indicator is the country level. More than half of the performance indicators (referred to as milestone indicators) included in the six COSOP project logical frameworks sampled were adequate, relevant, and monitorable. However, there is still room for improvement as approximately one-third of the indicators did not have clear targets or deadlines for their achievement. Furthermore, the 2012 Viet Nam COSOP was the only COSOP sampled to identify data sources and none of the six COSOPs sampled identified data collection methods for each indicator.

MI 5.3 – Expected results consistent with national strategies

MOPAN donors in-country and direct partners were asked whether RB-COSOPs contain statements of expected results consistent with national development strategies. The majority of respondents (73%) rated this MI as adequate or above, which gave an overall rating of strong, and 22% answered ‘don’t know’.

IFAD was rated very strong on the link between its COSOPs and objectives declared in national development strategies. In the six COSOPs sampled, there are statements linking COSOP objectives with Poverty Reduction Strategy Papers or other national development plans. This is consistent with statements found on IFAD’s website and with IFAD’s guidelines on COSOP completion.

MI 5.4 – Expected results developed in consultation with direct partners/beneficiaries

This MI was assessed by survey only. MOPAN donors in-country and direct partners were asked whether IFAD consults with direct partners to develop its expected results. The majority of respondents (65%) rated this MI as adequate or above, and 21% rated it inadequate or lower.

MI 5.5 – Results for cross-cutting priorities included in country frameworks[31]

MOPAN donors in-country and direct partners were asked whether RB-COSOPs include results related to cross-cutting priorities such as gender, environment, and household food security. The majority of survey respondents (76%) rated this MI as adequate or above (45% as adequate and 31% strong or very strong).

IFAD was rated as strong on this MI by the document review. More than half of the results frameworks in the COSOPs sampled include a description of strategies, operations and results related to the three cross- cutting issues (gender, environment, and food security and nutrition). RB-COSOP Guidelines (2010) specify that new results-based COSOPs must focus on a maximum of three strategic objectives rather than multiple areas of intervention. COSOP Guidelines also suggest that women’s empowerment be considered for inclusion as a strategic objective for all COSOPs. Regardless of whether gender equality is included as a strategic objective, the Guidelines indicate that all COSOPs should mention that IFAD-funded operations will target women, especially women heads of households who are often disadvantaged.

IFAD does not have an official policy on good governance and does not consider good governance as a cross-cutting priority. As a result, the assessment team found little documentary evidence of IFAD’s mainstreaming efforts in this area. This being said, the Fund does promote good governance through indirect means, such as via its Anti-Corruption Policy and the Performance Based Allocation System (PBAS), which is tailored to its specific mandate.

3.3.3 Operational management

IFAD is considered to have sound systems and practices in place in most areas of operational management. Some areas for improvement were identified in IFAD’s results-based budgeting.

Figure 3.10 shows the overall survey and document review ratings for the KPIs in the operational management performance area.

Figure 3.10 | Performance area II: Operational management, survey and document review ratings

Figure 3.11 shows the mean scores for the KPIs for all survey respondents, and by respondent groups.

Figure 3.11 | Performance area II: Operational management, mean scores by respondent group

KPI 6: Resource allocation decisions

Finding 6: Overall, IFAD was considered strong in making its funding criteria publicly available and following the criteria, but inadequate on the predictability and timely disbursement of funding at the country level.

IFAD loans are based on a Performance-Based Allocation System (PBAS) that follows a three-year cycle. Implementation progress reports are presented annually to the Executive Board. Within each three-year cycle, IFAD reviews the ex-ante allocations using information provided in the annual country performance assessments (e.g. changes in country needs and in policy and institutional frameworks). The PBAS distinguishes itself from similar systems in other international financial institutions in two ways: the needs assessment is based on a country’s rural population rather than its total population (given IFAD’s mandated focus on agriculture and rural needs); and, loan allocations are linked to an index of Rural Sector Performance (in addition to a standard index on country policy and institutional assessments).

Figure 3.12 | KPI 6: Resource allocation decisions, ratings of micro-indicators*

*The document review was designed to draw data from the 2011 Survey on Monitoring the Paris Declaration.

MI 6.1 – Criteria for allocating funding publicly available

Survey respondents were asked whether IFAD makes its criteria for allocating resources publicly available. The majority (67%) rated IFAD as adequate or above, and 13% as inadequate or below. Donors at headquarters were more positive than direct partners, rating IFAD strong on this MI, while direct partners rated it adequate. This difference is statistically significant.

The document review rated IFAD as very strong for making its funding allocation criteria publicly available. Loans from IFAD are allocated based on a Performance-Based Allocation System (PBAS) that defines the maximum amount that can be lent to a given country in a given allocation cycle.32 PBAS documents are available on IFAD’s website in the four official IFAD languages: French, English, Spanish and Arabic.

MI 6.2 – Resource allocations conform to criteria

This MI was assessed by survey only. Survey respondents were asked whether IFAD allocates resources according to the established criteria. The majority respondents (74%) rated IFAD as adequate or above resulting in a strong overall rating.

MI 6.3 – Resources released according to agreed schedules

This MI was assessed by document review only, which rated IFAD as inadequate.

The rating on this indicator is based on Indicator 7 of the Paris Declaration on Aid Effectiveness, which measures the gap between aid scheduled and aid both effectively disbursed and recorded in countries’ accounting systems. According to the RIDE 2012,33 IFAD did not meet its 2012 target of 14 months from project approval to first disbursement, although it has improved from a baseline of 21 months to 18 months in 2012.

The 2011 Survey on Monitoring the Paris Declaration34 found that in 2010, only 34% of IFAD disbursements were on schedule (compared with 53% in 2005 and 44% in 2007). While the data indicate that IFAD is failing to deliver most of its disbursements on time, some of these delays may be outside of IFAD’s control and related to the borrower’s capacity. Indeed, according to the 2012 RIDE, “disbursements have increased rapidly, but the time required to start projects is still high on average. An analysis of variation reveals that this is largely the product of the lengthy parliamentary loan approval mechanisms in a number of countries, in the context of which IFAD`s efforts to facilitate and accelerate project start-up may yield few results.”35

KPI 7: Results-based budgeting

Finding 7: Donors at headquarters considered IFAD to be adequate in linking loans and grants to expected results. The document review noted that IFAD has made commitments to strengthen its approach to results-based budgeting in the 9th Replenishment of IFAD’s Resources.

The assessment looked at two different questions for this KPI: whether IFAD links budgetary allocations to expected results, and whether its reports on results include the amount disbursed to achieve those results.

The survey component of this KPI was assessed only by donors at headquarters, who rated IFAD as adequate overall.

The document review found room for improvement in this KPI. While IFAD continues to improve its application of results-based budgeting and has committed to introduce a time recording system to facilitate this process, the document review found little evidence that IFAD’s Annual Reports provide links between expenditures and results achieved.

Figure 3.13 | KPI 7: Results-based budgeting, ratings of micro-indicators

MI 7.1 – Allocations linked to expected results

MOPAN donors at headquarters were asked whether IFAD links loans and grants to expected results. A majority (71%) of respondents rated IFAD as adequate and above, while 13% rated it inadequate or weak, and 16% answered ‘don’t know’ on this MI.

The document review rated IFAD as adequate for its application of results-based budgeting. Projects financed by IFAD are required to develop results frameworks and other tools to monitor the progress of implementation and achievement of their objectives.36 From this perspective, there is a clear link between allocations (in the form of loans and grants) and expected results at the project level. At a corporate level, which was the focus of the document review, IFAD’s 2013 budget is presented in a results-oriented way. According to IFAD’s Update of the Change and Reform Agenda (December 2012), management will continue to review and refine this approach in future years in consultation with the Executive Board.

IFAD’s annual budget is broken down into four business areas (clusters), each of which correspond to a set of high-level development outcomes and corporate management results.37 This allows IFAD to set overall priorities for the deployment of its budgetary resources to operational and non-operational activities. While this assures some level of alignment between IFAD’s Strategic Framework and its annual budget, full implementation of results-based budgeting (RBB) would require a more detailed breakdown of expected results in relation to the budget.

One of IFAD’s key commitments under its Ninth Replenishment includes the introduction of a time recording system that will measure the full costs of performing key business processes and activities (Report of the Consultation on the Ninth Replenishment of IFAD’s Resources. 9). According to the recent corporate-level evaluation of IFAD’s institutional efficiency and efficiency of IFAD-funded operations, the review of IFAD’s Zero-Base Budgeting exercise in the Programme Management Department (PMD) showed that the divisions did not justify their total budget requirements on the basis of expected results and planned outputs, but instead focused on justifying their incremental budget needs for 2011 over 2010. It should be noted, however, that achieving a linear relationship between expected development results (i.e. outputs and outcomes) and budget allocations is admittedly a challenging objective. IFAD has continued to develop and deepen its approach to results-based budgeting through its membership in an IFI Benchmarking Network for Budgeting38 and through its commitments described in the reform agenda and the two latest replenishment reports (IFAD8 and IFAD9).

MI 7.2 – Expenditures linked to results

Donors at headquarters were asked about the extent to which IFAD’s reports on results include the amount disbursed to achieve those results. The majority (74%) rated IFAD as adequate or higher, and 10% rated it inadequate or lower. Direct partners rated this MI more positively than donors at headquarters and the difference was statistically significant.

IFAD was rated as inadequate on this MI by the document review. The assessment team found little documentary evidence at the corporate level that systematically links results (i.e. outputs or outcomes) to expenditures incurred in achieving these results. None of the reports reviewed (including Annual Reports, Annual Reviews on Portfolio Performance, RIDE and ARRI) provide explanations of variations in operational expenditure nor do they establish links between expenditures (disbursements) and results achieved (outputs and outcomes). The Annual Reports do, however, report on results achieved against IFAD’s 2011-2015 Strategic Framework and they also provide a comprehensive overview of the Fund’s finances and programme of work within the latest financial year. In addition, the ARRI and RIDE speak to results and other performance criteria of IFAD’s operations, yet the reports do not link the resources expended with the results achieved.

KPI 8: Financial accountability

Finding 8: Most of IFAD’s practices related to financial accountability are considered strong. However, some areas for improvement include its procurement and contract management processes.

In this KPI, two MIs were assessed by survey and all MIs were assessed by the document review, which found financial accountability to be one of IFAD’s strengths. An external assessment of the internal audit function in 2013 noted that IFAD’s audit functions are well-designed and very effective but that its procurement procedures are cumbersome and overly time- and labour-intensive, which might affect the efficiency of project implementation.

Figure 3.14 | KPI 8: Financial accountability, ratings of micro-indicators

MI 8.1 – External financial audits performed across the organisation

This MI was assessed by document review only, which rated it as very strong. The document review found that external financial audits are performed annually at the organisation-wide level and that they adhere to recognised international standards for auditing. IFAD’s website provides all of the external, organisation- wide financial audit reports prepared since 1997. Each audit report includes a letter from the auditors confirming that the external audit was conducted in accordance with international standards on auditing and that IFAD’s consolidated financial statements were prepared in accordance with International Financial Reporting Standards (IFRS).

MI 8.2 – External financial audits performed at the regional, country, or project level39

This MI was assessed by document review only. IFAD received a strong rating for its external auditing practices at the project level. The Fund has recently updated its Guidelines on Project Audits that describe the detailed procedures and requirements related to project audits. In addition, Article 7 of the Agreement Establishing IFAD40 requires the Fund to make arrangements to ensure that the proceeds of any financing are used only for the purposes for which the financing was intended – with due attention to considerations of economy, efficiency and social equity. The Guidelines on Project Audits require the Lead Project Agency (i.e. the project implementer) to have its annual financial statements audited in accordance with internationally accepted auditing standards. In addition, it stipulates that a full operational audit may be carried out if IFAD considers it necessary based on the audit reports received. Finally, according to the Minimum Requirements for Project Audit, as described in Appendix III in the Guidelines on Project Audits, “The LPA will have the annual financial statements audited in accordance with internationally accepted auditing standards. The auditor’s report should indicate the standards used, and the extent, if any, to which the examination did not conform to those standards.”41

MI 8.3 – Policy on anti-corruption

This MI was assessed by document review only. IFAD’s Policy on Preventing Fraud and Corruption in its Activities and Operations was rated as strong.42 The policy was revised and approved by the Executive Board in 2005 but has not been reviewed since that time. IFAD’s anti-corruption practices are based on a zero-tolerance policy that is applied to its loan and grant activities at the country level and for all IFAD staff members and other individuals representing the fund.

IFAD’s Office of Audit and Oversight (AUO) and its Investigation Section (AUO/IS) are responsible for implementing the Anticorruption Policy. AUO/IS has been mandated to investigate alleged irregular practices, namely: (i) fraud and corruption, in relation to entities, contractors and non-staff individuals applying for or participating in an IFAD-financed project or headquarters-related contract; and (ii) staff misconduct. IFAD monitors and reports on its anti-corruption activities through its Annual Report on Investigation and Anticorruption Activities, which provides an overview of the annual investigation activities, the activities of the Sanctions Committee, examples of investigations closed within the last year, and a general update on the implementation status of the anticorruption policy. The first annual report was produced in 2004.

MI 8.4 – Systems for immediate measures against irregularities

This MI was assessed by document review only and was rated as strong.

The AUO is mandated to handle all matters related to investigations of irregular practices. A Sanctions Committee is established to review investigative findings and determine whether any action should be taken.

The AUO submits annual reports to the President and the Audit Committee of the Executive Board summarising the investigative activities of the AUO, its findings, and the sanctions imposed or other measures taken by the Sanctions Committee or the president. This report is posted on the website. Annual reports are also provided by the Director of AUO to the President and the Audit Committee, summarising significant oversight results, including recommendations for improvements and the status of management actions taken or planned in response to reported results. This report is not publicly disclosed.

The AUO has a database that includes all audit recommendations and their status of implementation; the RIDE also includes an indicator under Level 5 that measures the”percentage of actions overdue on high- priority internal audit recommendations.”43

According to the 201144 and 2012 Annual Reports on Investigation and Anticorruption Activities,45 there have been some staffing constraints in the investigations area due to a high turnover of staff, making it difficult to address the growing caseload. The “Report of the Chairperson on the 125th meeting of the Audit Committee”46 mentions that expiring contracts for associate audit officers may need to be extended given the high investigations workload. Given these constraints, the assessment team is concerned that not all major or systemic irregularities are adequately reported to the Board.

MI 8.5 – Internal financial audit processes provide credible information

Among MOPAN donors at headquarters, the only respondent group asked about this MI, 77% agreed that internal financial audits provide credible information to IFAD management and the Executive Board, and 48% rated IFAD as strong or very strong.

IFAD received a rating of very strong by the document review. IFAD’s guidelines for project and programme audits are set out in the Guidelines on Project Audits (2011) and Operational Procedures for Project and Programme Audits (2011).47 AOU’s internal audit activities adhere to the standards issued by the Institute of Internal Auditors (IIA). In 2012, an external quality assessment of the internal audit function of the Office of Audit and Oversight (AUO) confirmed that IFAD’s level of reporting to senior management and the Board “generally conforms to the standards”, which is the highest possible rating of the IIA.48 The assessment highlighted a number of successful practices, including: high levels of respect for and confidence in the Chief Audit Executive by the audit committee and senior management; clear independence and objectivity; staff and managers of the Office of Audit and Oversight seen as highly competent; well-designed and very effective Quality Assurance and Improvement Programme; comprehensive policies and manuals; highly effective risk assessment process; and, excellent communications between staff members in the Office of Audit and Oversight.

MI 8.6 – Effective procurement and contract management processes

Donors in-country and direct partners were asked whether IFAD’s procurement and contract management processes for the provision of services or goods are effective. The majority (64%) agreed, with 32% rating IFAD as strong or very strong. Direct partners were more positive than donors in-country and the difference is statistically significant.

The document review rated IFAD adequate on this MI. IFAD’s project-related procurement responsibilities lie with the borrower. IFAD does, however, have a number of policies and guidelines that inform that procurement process. At the project level, IFAD follows the principles set out in the Paris Declaration on Aid Effectiveness and the Accra Agenda for Action with respect to the use of existing national procurement systems. IFAD’s revised Procurement Guidelines focus less on the details of procurement methods and more on the general principles, standards and policies that borrowers and recipients must adhere to in implementing IFAD-financed projects. IFAD’s procurement practices at the corporate level are guided by the General Terms and Conditions for Procurement of Services49 and the General Terms and Conditions for Procurement of Goods.50

In 2013 the corporate-level evaluation on IFAD’s institutional efficiency (CLEE) concluded that there was room for improvement in the corporate procurement process. The evaluation reported that, “Procurement of services through institutional or commercial contracts has many steps, is unclear and extremely time and labour-intensive.”51 However, since the formal approval of the CLEE steps have been taken to redress these weaknesses, including the introduction of a new version of the Procurement Manual,52 a new Manual for Institutional Contracts, and the establishment of a number of long-term service agreements.

MI 8.7 – Strategies for risk management

This MI was assessed by document review only. IFAD was rated as strong for its risk identification, mitigation, monitoring and reporting practices. IFAD’s Policy on Enterprise Risk Management53 (ERM), approved by the Executive Board in 2008, defines key roles and responsibilities for all stakeholders in ERM activities. The policy is based on international principles on risk management principles, set out by the Committee of Sponsoring Organizations of the Treadway Commission54 in its Integrated Framework for Enterprise Risk Management, and is aligned with IFAD’s existing management processes, such as results- based strategic planning and internal control. A report on ERM activities and their status is presented to the Executive Board annually for information purposes in addition to periodic reporting by the Audit Committee on risk management matters. The external assessment of the internal audit function of the Office of Audit and Oversight presented to the Audit Committee in March 2013 concluded that IFAD has a “highly effective risk assessment process” (Appendix, p. 2). However, some room for improvement was identified in the RIDE 2012, which noted that IFAD was unable to reduce the percentage of overdue actions on internal audit recommendations to meet its target of 20%.

KPI 9: Using performance information

Finding 9: Survey respondents provided adequate to strong ratings for IFAD’s use of performance information. The document review rated IFAD as very strong for its use of performance information to revise and adjust policies and for acting on evaluation recommendations. However, it found some room for improvement in the use of performance information at the country level.

IFAD was rated as adequate on one MI, inadequate on another and very strong on two of the four MIs assessed by the survey.

Although IFAD seems responsive to performance information on results, and was rated very strong for its ability to follow up on evaluation recommendations, there is still room for improvement in this area. First, an important issue is the quality and validity of its performance information. As noted in the 2013 corporate-level evaluation of efficiency (CLEE), IFAD cannot adequately assess its performance in several areas due to the inadequacy of baseline surveys, complexity of project monitoring and evaluation systems, and cost to project authorities of IFAD’s reporting requirements. Second, while the document review found evidence that the development of new COSOPs are based on performance information, both the CLEE and the recently completed RB-COSOP Evaluation Synthesis (2012) found some room for improvement in terms of systematically linking performance reviews of recently closed IFAD-funded projects to establish new strategic directions and to select new projects.

Figure 3.15 | KPI 9: Using performance information, ratings of micro-indicators

MI 9.1 – Using information to revise and adjust policies

Donors at headquarters were asked whether IFAD uses project/programme, sector and country information on performance to revise corporate policies. The majority (84%) rated IFAD as adequate or above, and only 3% considered the Fund inadequate.

In the document review, IFAD received a rating of very strong for its ability to revise and adjust policies using performance information. IFAD documents reveal that there is a strong culture of evaluation at the Fund; it assesses its corporate performance systematically through annual reports on its development effectiveness and portfolio performance. Furthermore, the Fund undertakes periodic corporate-level evaluations of its key policies. Recent evaluations on gender, IFAD’s country presence, and its effectiveness and efficiency (for instance, in relation to human resources management) have been acknowledged by IFAD management and have led to appropriate actions taken in response to evaluation recommendations. Two IFAD policies were developed in response to recommendations of corporate-level evaluations: the IFAD Policy on Gender Equality55 and the IFAD Country Presence Policy.56

IFAD reviews and evaluations have also led to broad reviews of the Fund’s organisation. The Change and Reform Agenda was launched in 2009 in response to concerns that IFAD was not maximising its organisational performance. As a result, IFAD has improved its human resource management system, reporting on results, and medium-term planning.

MI 9.2 – Using information to plan new interventions

Donors in-country and direct partners were asked whether IFAD uses information on its country and sector performance to plan new interventions at country level. The majority (78%) rated IFAD adequate or above and 51% rated IFAD strong or very strong.

The document review rated IFAD as adequate on this MI. IFAD regularly conducts Country Programme Evaluations (CPEs) in which it assesses the Fund’s performance and progress towards expected outcomes at the country level. CPEs were conducted for four of the six sampled countries within the last four years. Analysis of the six COSOPs suggests that performance information from these CPEs is systematically integrated in the development of new COSOPs. Indeed, all the COSOPs reviewed included sections on lessons learned from past performance and analysis of the implications of these lessons on new strategic directions. However, according to the CLEE and the recent RB-COSOP Evaluation Synthesis (2012), this information is not systematically used to plan all new interventions. In addition, the CLEE noted that:

“There is little evidence that IFAD’s results framework effectively guides planning decisions and accountability, starting from the impact and outcome end of the results chain, or that work programmes are based on adequate strategic selectivity informed by results.”57

MI 9.3 – Proactive management of poorly performing initiatives

Donors in-country and direct partners were asked whether IFAD actively manages ‘unsatisfactory’ projects from the previous fiscal year. The results were quite mixed: 39% replied ‘don’t know’; 30% rated it adequate; 17% rated it strong or very strong, and 15% rated it inadequate or below.

The document review rated IFAD as adequate on this MI. On the one hand, IFAD has shown evidence of progress in managing unsatisfactory projects. IFAD undertook a portfolio clean-up in 2010 and closed eight non-performing or underperforming projects. The 2011 COMPAS report58 and the RIDE 2012 indicated improved levels of satisfaction with project performance and evidence of IFAD’s proactive management of unsatisfactory investments.

On the other hand, IFAD’s annual report on results and impact (ARRI), produced by the Independent Office of Evaluation, noted that most IFAD projects were only ‘moderately satisfactory’ and concluded that IFAD has much to do to increase the proportion of satisfactory projects. IFAD’s proactivity index, which was reported on in the RIDE 2012, also indicated room for improvement given that IFAD did not meet its 2012 target for “percentage of problem projects in which major corrective actions were taken.”59

However, as mentioned in the RIDE, these numbers should be treated with caution. “The ratio of actual problem projects to the total on-going portfolio has stabilised at about 18% in 2012. Because of the increased emphasis on early warning, operational divisions have identified more projects that are potentially problematic.” IFAD has found that setting a quantitative target for major interventions on problem projects is not a useful exercise, because it creates a perverse incentive to rate projects as “less risky” and could reduce rigour in performance rating as a result. For this reason, IFAD will track this indicator during IFAD9 without a target. (RIDE, p. 15)

MI 9.4 – Evaluation recommendations are acted upon

Donors at headquarters were asked whether IFAD regularly tracks implementation of evaluation recommendations reported to the Executive Board. The majority (77%) rated IFAD as adequate or above, and only 3% rated it inadequate, although 19% answered ‘don’t know’.

IFAD was rated very strong by the document review on this MI. There is ample evidence that evaluation recommendations reported to the Board are followed up by the responsible units. IFAD generally acts according to its Evaluation Policy, which requires that IFAD’s management and concerned governments sign an agreement at the completion of each corporate and country programme evaluation, outlining the recommendations they agree to adopt and implement in a specified timeframe.60

The implementation of accepted recommendations is tracked through the President’s Report on the Implementation Status of Evaluation Recommendations and Management Actions (PRISMA), which is presented to the Executive Board on an annual basis. Certain evaluation recommendations are also regularly tracked and reported on outside of the PRISMA. This was the case for the implementation of the 2010 “Peer Review of IFAD’s Evaluation Function”61 which was the object of a separate report of the Evaluation Committee Chairperson to the Board following three consecutive Committee sessions (in 2010 and 2011). This progress report commented in detail on the implementation of each Peer Review recommendation.

KPI 10: Managing human resources

Finding 10: IFAD’s human resource management practices received strong ratings from the document review based on the recent introduction of new performance management systems.

This KPI was assessed by document review only. IFAD has undertaken a reform of its human resources practices and is currently working to improve several elements of its human resource management system, such as: (1) its system linking rewards to staff performance; (2) transparency in its hiring, promotion and firing processes; and, (3) its staff rotation system and support to staff located in country offices (e.g. through the use of e-learning courses). It has piloted a number of initiatives to ensure appropriate levels of compensation and performance-based reward systems are in place for its entire staff (e.g. pay-for- performance model).

Figure 3.16 | KPI 10: Managing human resources, ratings of micro-indicators

MI 10.1 – Results-focused performance assessment systems for senior staff

IFAD was rated strong on this MI. IFAD uses a performance evaluation system (PES) for all staff grades up to the D-2 level (Division Directors) that connects individual staff performance to organisational performance. The most senior staff members (the President, Vice-President and Associate Vice-Presidents) who form the Executive Management Committee participate in a team-based performance evaluation system.

MI 10.2 – Transparent system to manage staff performance

IFAD was rated strong on this MI. IFAD’s new Reward and Recognition Framework, implemented in 2013, clearly explains how staff performance relates to reward and recognition. In addition, the Human Resources Procedures Manual (HRPM) provides details on how staff performance relates to promotion. Promotions are achieved solely on a competitive basis through a posted vacancy announcement for positions at a higher grade. However, IFAD is considering a promotion process based on an objectively demonstrated and professionally graded increase in job responsibilities that are sufficient to warrant a higher grade, as well as the demonstrated ability to perform at that higher level as determined by an independent panel, but this has not yet been implemented.

Although the performance evaluation system (PES) is functional and there is evidence of high compliance, the 2013 corporate-level evaluation on efficiency (CLEE) expressed some concerns about its application:

“IFAD has a cutting edge staff performance evaluation system in terms of design and process, which is well supported by the Human Resources Division. However, the performance evaluation system is not yet assisting the organisation in managing for performance, which is critical to enhance overall efficiency.”62

It noted that managers tend to be too risk averse in their performance assessments and, as a result, staff members are rarely recognised for either outstanding or substandard performance.

KPI 11: Performance-oriented programming

Finding 11: Survey respondents rated IFAD as adequate for its use of benefit/impact analysis and the document review found it to be strong in this area. The document review found that IFAD could do better in setting milestones and targets to monitor implementation.

IFAD has developed a project appraisal process and has taken steps to ensure that its projects are submitted to rigorous environmental and social impact analysis. However, to be consistent with its renewed focus on impact analysis and results-based management, it could improve its collection of and reporting on baseline data to enable comparisons between baseline values and outputs achieved.

Figure 3.17 | KPI 11: Performance-oriented programming, ratings of micro-indicators

MI 11.1 – New initiatives subject to benefit/impact analysis

Donors at headquarters were asked whether IFAD conducts benefit/impact analysis prior to implementation of new initiatives. The results were mixed as 58% gave IFAD a rating of adequate or higher, 13% a rating of inadequate or lower, and 29% replied ‘don’t know’.

IFAD received a strong rating by the document review for its systematic application of environmental impact analyses prior to new project implementation. An Environmental and Social Impact Assessment (ESIA) is undertaken at the project appraisal stage and IFAD’s practices are clearly explained in the Environmental and Social Assessment Procedures.63 The most environmentally challenging projects – those which may have sensitive, adverse or irreversible environmental consequences – are rated “Category A” and are the object of a full ESIA. Recommendations from the ESIA are reviewed and changes are made to the project design, as appropriate. At the end of project implementation, Category A projects may be re-evaluated to assess the effectiveness of any preventative actions and mitigation measures in the project design.

MI 11.2 – Milestones/targets set to monitor implementation

This MI was assessed by document review only, which rated IFAD as adequate. Slightly more than half of all project logical frameworks sampled for this assessment included targets/milestones for project implementation.

Implementation targets are described in the Annual Work Plan and Budget (AWPB) documents produced every year for all IFAD projects. The AWPB forms the basis for monitoring progress at the activity level regarding resource use and allocation and is intended to set clear targets and deadlines for achieving expected activities and outputs.

However, none of the project logframes or AWPBs sampled contained information on baseline values for indicators. This represents a significant challenge for monitoring and reporting on progress.

KPI 12: Delegating authority

Finding 12: IFAD’s delegation of decision-making authority to the country level is considered adequate given that all loans and grants must be centrally approved. However, according to some survey respondents, IFAD’s current presence in the countries sampled may not be sufficient to improve its country-level performance.

As part of IFAD’s ongoing reform process, the organisation has established 40 new country offices in recent years. This development represents a radical shift from its original operating model which, as described in The Agreement of Establishing IFAD64, did not oblige IFAD to have a country presence. Most of IFAD’s country offices are hosted by other United Nations organisations (FAO, in particular, and, in some cases, the World Food Programme) in order to minimise costs and increase efficiency. IFAD’s country offices typically employ between one and four staff members. There were varying types of IFAD staff present in the six country offices sampled for this assessment.65

As is the case with other IFIs, IFAD country programme managers based in country are not authorised to approve any loans or grants without central level approval. The limited delegation of authority may conflict with the changing business needs associated with decentralised institution.

Figure 3.18 | KPI 12: Delegating authority, ratings of micro-indicators

MI 12.1 – Delegation of appropriate decision-making authority at the country level

Donors in-country and direct partners were asked whether IFAD has delegated appropriate decision-making authority at the country level. The results were mixed as 46% rated it adequate or above, 26% rated it inadequate or below, and 27% responded ‘don’t know’.

The document review rated IFAD as adequate on this MI. IFAD’s approach to delegating responsibilities to the country level is guided by its Country Presence Policy. Detailed information about the levels of delegated authority is presented in the Revised IFAD Manual and Framework for Delegation of Authority (2011).66

IFAD Country Programme Managers (CPM) are authorised to submit completed COSOP designs and project designs for quality assurance and to participate in the preparation of draft financing agreements for negotiations, but they are not authorised to approve loans or grants without central level approval. This is standard practice in all IFIs. While IFAD has made efforts to improve delegation of decision making to the country level and other levels in recent years, further work is needed to flesh out the more substantive delegation of roles and responsibilities. The 2013 corporate-level evaluation on IFAD’s institutional efficiency noted that the limited substantive delegation of authority may conflict with changing business needs associated with a decentralised institution and that human resources policies will need to be reviewed for their efficiencies.67

MI 12.2 – Country presence sufficient for improving country-level performance68

This MI was assessed by survey only. In-country donors and direct partners in the six sample countries were asked whether IFAD’s country presence is sufficient for improving its country- level performance. The two respondent groups had different perspectives on this issue. While the majority (63%) of direct partners rated IFAD as adequate or above, the majority of donors (71%) rated it as inadequate or below.

3.3.4 Relationship management

Overall, survey respondents and the document review considered IFAD to be adequate in the area of relationship management.

In assessing those areas of relationship management related to the aid effectiveness agenda, the document review used data from the 2011 Monitoring Survey to assess IFAD performance. However, several of these indicators were considered to be less applicable in the case of IFAD due to the nature of its operations and how it works with client governments, definitions used in the PD Survey, or decisions made by its governing bodies that limit its possibilities to participate in certain modalities of funding, such as programme based approaches. Apart from a general question asked of stakeholders in its annual Client Satisfaction Survey, IFAD does not report on indicators related to the aid effectiveness agenda.

According to the “Aid Quality and Donor Rankings”(2010),69 which examined four dimensions of aid quality (selectivity, alignment, harmonisation, and specialisation), IFAD was rated 7th out of 38 multilateral organisations.

Figure 3.19 shows the overall ratings for the five KPIs in the relationship management performance area.

Figure 3.19 | Performance area III: Relationship management, survey and document review ratings

Figure 3.20 shows the mean scores for the five KPIs for all survey respondents, and by respondent groups.

Figure 3.20 | Performance area III: Relationship management, mean scores by respondent group

KPI 13: Supporting national plans

Finding 13: IFAD was perceived to be strong in its support of national and partner plans.

Respondent groups gave IFAD an overall rating of strong for its support of national and partner plans. The document review rated the Fund adequate for aligning its loan activities with national priorities, but the Fund does not apply policy conditionality as part of its lending practices.

Figure 3.21 | KPI 13: Supporting national plans, ratings of micro-indicators

MI 13.1 – Funding proposals developed with national government or direct partners

This MI was assessed by survey only. Donors in-country and direct partners were asked whether IFAD supports funding proposals designed and developed by the national government or other direct partners. The majority (80%) agreed, rating IFAD as adequate or above.

MI 13.2 – Conditionality draws on national benchmarks/indicators/results

This MI was assessed by document review only and was rated as adequate, based on a broad interpretation of the MI. IFAD does not apply policy conditionality as part of its lending practices (as stated in the 2010 MOPAN assessment of IFAD70); it only applies financial conditions. This is mainly due to IFAD’s small loan portfolio and to its specialised mandate focusing on the agricultural sector. In addition, its grants and loans are not directed towards policy support and the scale of its financing is relatively small compared to other multilateral development banks. Only when IFAD is co-financing projects with the World Bank will it impose conditionality on its loans and in these cases it uses the World Bank’s policies. In December 2012, IFAD’s Executive Board approved the “Review of the Lending Policies and Criteria,”71 which now recognises that IFAD’s lending operations are guided by a series of corporate policies rather than by a single document. While none of these policies specifically deal with country ownership, IFAD’s Medium-Term Plan 2011-2013 states that it will “invest in national capacities for design and implementation and will approve revised procurement guidelines that amplify the use of national systems and joint missions with partnering agencies and government counterparts.”72 IFAD’s Lending Policy and founding documents indicate that IFAD loans must be linked to national priorities.

KPI 14: Adjusting procedures

Finding 14: Survey respondents perceived IFAD to be adequate in the adjustment of its procedures to reflect local conditions and capacities.

This KPI was assessed by survey only.

Figure 3.22 | KPI 14: Adjusting procedures, ratings of micro-indicators

MI 14.1 – Procedures easily understood and completed by partners

Asked whether IFAD uses financial procedures (such as audit and loan withdrawal procedures) that can be easily understood and followed by partners, the majority of direct partners and donors in-country (63%) rated the organisation as adequate or above, 11% as inadequate or below, and 25% replied ‘don’t know’.

MI 14.2 – Time for procedures does not affect implementation

Asked whether the length of time it takes to complete IFAD financial procedures affects implementation, 50% of direct partners and donors in-country rated the organisation as adequate or above, 21% as inadequate or below, and 28% replied ‘don’t know’.

MI 14.3 – Ability to respond quickly to changing circumstances

Asked whether IFAD adjusts its portfolio in-country quickly to respond to changing circumstances on the ground, 59% of direct partners and donors in-country rated the organisation as adequate to very strong, and 26% as inadequate or below.

MI 14.4 – Flexibility in implementation of projects/programmes

Donors in-country and direct partners were asked whether IFAD flexibly adjusts its implementation of individual projects and programmes as learning occurs. The majority (71%) rated IFAD as adequate or above.

KPI 15: Using country systems

Finding 15: Survey respondents rated IFAD as adequate in the use of country systems and in the promotion of mutual accountability. The document review considered IFAD to be adequate in its use of country financial systems but did not have sufficient data to assess the other areas in this KPI.

In this KPI, four MIs were assessed by survey and one by document review.

IFAD provides little recent information on its aid effectiveness commitments. Data from the Survey on Monitoring the Paris Declaration (2011) was not considered to be sufficient to provide a rating on several of the indicators. A corporate-level evaluation on IFAD’s replenishments will consider if and how the principles of the Paris Declaration on Aid Effectiveness are applied.73

Figure 3.23 | KPI 15: Using country systems, ratings of micro-indicators*

*White diamonds indicate that the data required for the assessment was unavailable for IFAD.

MI 15.1 – ODA disbursements/support recorded in annual budget

There was insufficient data for the document review to assess this MI.74

MI 15.2 – Use of country financial systems for operations

Donors in-country and direct partners were asked whether IFAD uses country financial systems (e.g. procurement, public financial management) as a first option for its operations where appropriate. A majority (59%) rated IFAD as adequate or above, 5% as inadequate or weak, and 35% answered ‘don’t know’.

In the document review, IFAD was rated adequate for its use of country financial systems as a first option for its operations. This rating was based on IFAD’s results on indicators 5a and 5b in the 2011 Survey on Monitoring the Paris Declaration. In 2011, the OECD reported that a majority of IFAD’s funds (72%) used national financial management systems, while 85% of loans and grants from IFAD were disbursed using national procurement systems, which was a significantly higher proportion than most other global IFIs.

These percentages have improved significantly since the last OECD surveys were carried out (in 2005 and 2007), but they still do not meet the ambitious target of 90% set by the Paris Declaration for the year 2010. This use of national financial systems varies according to the strength of government systems. IFAD regional directors indicated that national capacity for fiduciary management is weak and slow to respond (for instance, slow to respond to loan withdrawal applications from direct partners in some fragile states which makes it difficult to use these systems).

At the project level, IFAD follows the principles set out in the Paris Declaration on Aid Effectiveness and the Accra Agenda for Action, which state that donors should support the development of national procurement capacity.75 However, responsibility for procurement in connection with projects (i.e. allocation and administration of contracts) lies with the borrower. IFAD does participate in the procurement process as it reviews the borrower’s award recommendations prior to final approval of procurements to ensure that public procurements are in line with internationally accepted practices.

MI 15.3 – Use of country non-financial systems for operations

Donors in-country and direct partners were asked whether IFAD uses a country’s non-financial systems (e.g. monitoring and evaluation) as a first option for its operations where appropriate. Half of survey respondents (53%) rated IFAD as adequate or above, 17% as inadequate or below, and 30% answered ‘don’t know’.

There was insufficient data for the document review to assess this MI.76

MI 15.4 – Parallel implementation structures avoided

MOPAN in-country donors and direct partner organisations were asked to what extent IFAD avoids using parallel implementation structures. Most rated IFAD as adequate or above (60%), 19% rated it inadequate or below, and 20% answered ‘don’t know’.

IFAD projects are mainly implemented by integrated Project Management Units (PMUs) that are usually hosted within governments and not parallel structures established by IFAD. Given definitional issues with the wording of this question in the Survey on Monitoring the Paris Declaration, the PD Survey data – which was the only documentary evidence available – was considered insufficient to provide a rating in the context of this assessment.

MI 15.5 – Promotion of mutual accountability in its partnerships

This MI was assessed by survey only. Donors in-country and direct partners were asked to rate IFAD on the extent to which it has promoted a mutual assessment of progress in implementing its Paris Declaration and subsequent Aid Effectiveness commitments: 58% rated IFAD as adequate or above and 15% as inadequate or weak.

KPI 16: Contributing to policy dialogue

Finding 16: According to survey respondents, IFAD makes an adequate contribution to policy dialogue while respecting the views of its partners in the process.

This KPI was assessed by survey only and was rated adequate overall.

Figure 3.24 | KPI 16: Contributing to policy dialogue, ratings of micro-indicators

MI 16.1 – Reputation for valuable input to policy dialogue

All respondent groups were asked whether IFAD provides valuable inputs to policy dialogue. The majority of survey respondents (61%) rated IFAD as adequate or above, and 29% as inadequate or below.

MI 16.2 – Policy dialogue respects partner views

All respondent groups were asked whether IFAD respects the views of partners when it undertakes policy dialogue. Two-thirds of survey respondents (68%) rated IFAD as adequate or above, 10% rated as inadequate or weak, and 22% responded ‘don’t know’.

KPI 17: Harmonising procedures

Finding 17: Survey respondents rated IFAD as adequate overall for the harmonisation of its procedures with other donors. The document review rated IFAD as adequate for its level of participation in joint missions but inadequate on the extent to which its technical co-operation is disbursed through co-ordinated programmes.77

Figure 3.25 | KPI 17: Harmonising procedures, ratings of micro-indicators.

MI 17.1 – Participation in joint missions

Survey respondents rated IFAD adequate overall for participating in joint missions: 59% of donors in- country and direct partners rated IFAD adequate or above and 22% as inadequate or below. Direct partners were more positive than donors in-country and the difference is statistically significant.

The document review rated IFAD as adequate on this MI. According to the data from the Survey on Monitoring the Paris Declaration, the percentage of co-ordinated missions has declined significantly since the 2007 OECD assessment (from 84% in 2007 to 38% in 2010 – based on the first table 10a in which 26 countries were assessed between 2006 and 2011). Nevertheless, the percentage of joint missions in which IFAD participated was higher than that of any other development bank surveyed apart from the Inter- American Development Bank and met the 2010 target of “40% of all donor missions are joint” (based on the second Table 10a in which 44 countries were surveyed).

MI 17.2 – Technical co-operation disbursed through co-ordinated programmes

Donors in-country and direct partners were asked to assess the extent to which IFAD’s technical assistance is provided through co-ordinated programmes in support of capacity development. The majority (64%) rated IFAD as adequate or above, and 21% provided a rating of inadequate or below.

The document review rated IFAD as inadequate on this MI, based on data from the 2011 Survey on Monitoring the Paris Declaration. In 2007, 76% of its technical assistance was co-ordinated with country programmes. Based on a weighted average of the 26 countries assessed repeatedly between 2006 and 2011, this was reduced to 34% by 2010. However, if data from all 44 countries surveyed in 2010 is taken into consideration, the percentage is somewhat higher with 43% of IFAD’s technical assistance recorded as co-ordinated with country programmes. Regardless of the calculation methods, IFAD was unable to reach the target of 50% established by the Paris Declaration.

3.3.5 Knowledge management

Both survey respondents and the document review considered IFAD to be fairly strong in all areas of knowledge management. The Fund was rated strong for its evaluation of results and very strong for making key documents available to the public. There is room for improvement in the presentation of performance information and dissemination of lessons learned.

Figure 3.26 shows the overall survey and document review ratings for the four KPIs in the knowledge management performance area.

Figure 3.26 | Performance area IV: Knowledge management, survey and document review ratings

Figure 3.27 shows the mean scores for the four KPIs for all survey respondents, and by respondent group.

Figure 3.27 | Performance area IV: Knowledge management, mean scores by respondent group

KPI 18: Evaluating results

Finding 18: IFAD is recognised for the independence of its Office of Evaluation and for its follow- up and use of performance information from evaluations.

Survey respondents rated IFAD’s evaluation function as strong overall. The document review rated IFAD as very strong for the independence of its evaluation unit and quality assurance of its evaluations.

Figure 3.28 | KPI 18: Evaluating results, ratings of micro-indicators

MI 18.1 – Independent evaluation unit

Donors at headquarters were asked if ‘IFAD has an independent evaluation unit that reports directly to its Executive Board’, and rated IFAD as strong overall. More specifically, 87% rated it adequate or above and 71% as strong or very strong. Not a single respondent gave a rating of inadequate or below.

In the document review, IFAD was as rated very strong for the independence of its evaluation unit. According to IFAD’s Evaluation Policy,78 the role and mandate of the Independent Office of Evaluation (IOE) is to conduct independent evaluations of IFAD’s financed policies, operations and strategies to promote accountability and learning within the Fund. IOE is structurally independent of IFAD management and reports directly to the Executive Board. The Executive Board is responsible for appointing and removing the director of IOE. The Director of IOE is entirely responsible for formulating the IOE’s annual work programme and proposed budget appropriations.

The 2010 “Peer Review of IFAD’s Office of Evaluation and Evaluation Function” concluded that “Based on an examination of peer reviews of United Nations agencies, the IFAD evaluation system is arguably the most independent of all United Nations agencies.”79

MI 18.2 – Sufficient evaluation coverage of programming activities

This MI was assessed by document review only and rated IFAD as adequate for its evaluation coverage of projects. This rating is based on both the 2011 COMPAS report and on a review of IFAD’s evaluation policy and IOE documentation. IFAD’s evaluation function has made some significant improvements since 2010. According to the most recent COMPAS report,80 100% of IFAD’s projects were evaluated ex-post in FY 2011, which is consistent with the IOE’s explanation of its evaluation process.81 The IOE conducts desk-based validations of all of IFAD’s project completion reports (PCR) and also carries out project performance appraisals (PPA) on a selective basis (these entail field visits to the projects). However, there is room for improvement in the evaluation coverage of IFAD’s country-level programming: only five country programme evaluations (CPEs) were produced in 2012, and of the six countries sampled for this assessment, only Indonesia, Mozambique and Viet Nam have undergone CPEs in the last four years. Independent project evaluations were produced in Guatemala and Ethiopia, but no independent evaluation has been produced for Pakistan in recent years, apart from a desk-based validation of a project completion report.

MI 18.3 – Quality of evaluations

This MI was assessed by document review only, which rated IFAD as very strong for ensuring quality evaluation process and products. IFAD uses both an internal and external peer review process to monitor the quality of its country programme evaluations (CPE) and corporate-level evaluations (CLE). IFAD staff members review the corporate and country programme evaluations and an independent external quality assessment is conducted for corporate-level evaluations.

The 2010 “Peer Review of IFAD’s Office of Evaluation and Evaluation Function” found that IFAD’s corporate, country and project level evaluations were adequate, relevant and useful. However, it raised some concerns about evaluation costs and consistency of methodology. Most recommendations from the peer review were implemented and led to significant changes in IFAD’s evaluation and self-evaluation practices. For instance, IOE modified its approach to evaluation as a result of the recommendation that it should validate Project Completion Reports (PCR) and base its analysis of portfolio performance on a combined database of IOE project ratings and validated PCR ratings. IOE now ensures that 100% of PCRs are validated through a desk review and that a sample of projects is assessed using field visits, thus ensuring a more cost-effective evaluation process.

MI 18.4 – Use of evaluation findings to inform decisions

This MI was assessed by survey only. Donors at headquarters were asked whether IFAD uses evaluation findings in its decisions on programming, policy and strategy. The majority (71%) rated IFAD as adequate and above, and only 6% as inadequate.

MI 18.5 – Beneficiaries and direct partners involved in evaluations

This MI was assessed by survey only. Donors in-country and direct partners were asked whether IFAD involves direct partners and beneficiaries in evaluations of its projects or programmes. The majority (59%) rated IFAD adequate or above. Direct partners were more positive than donors in-country, rating IFAD strong, while donors in-country rated it adequate. The difference is statistically significant.

KPI 19: Presenting Performance Information

Finding 19: Survey respondents considered IFAD to be strong in reporting on expected outcomes and reporting against its organisational strategy. However, the document review identified several weaknesses in IFAD’s reporting practices, including the use of inadequate indicators and the lack of reporting on aid effectiveness commitments.

IFAD reports annually on its performance in relation to its expected outputs and outcomes. The two main vehicles for external reporting are the Report on IFAD’s Development Effectiveness (RIDE), which reports on performance information using the most current Results Measurement Framework (RMF), and the Annual Report on Results and Impact of IFAD Operations (ARRI), a reporting and verification tool developed by the Office of Evaluation (IOE). Output data for the RIDE is provided by the Results and Impact Measurement System (RIMS) based on quantitative information reported annually by Project Management Units (PMUs), while outcome data is based on project performance ratings (i.e. perceptions on the degree of efficiency, effectiveness, rural poverty impact, etc. of each project) as reported by PMUs at project completion.

IFAD’s reporting practices and systems are generally consistent with OECD and World Bank practices as defined in “Emerging Good Practices in Managing for Development Results.” 82 However, the corporate-level evaluation of IFAD’s institutional efficiency (2013) identified several constraints. It noted that “project monitoring and evaluation systems are often overdesigned, especially in the light of local capacities, and reporting on outcomes and impact levels of the results chain remains weak because of this. The quality of project completion reports varies, and the RIMS indicators are often not properly or fully reflected in project level M&E systems.”83

IFAD has demonstrated an awareness that its reporting system needs to be improved, particularly in order to determine attribution. IFAD has committed to “endeavour[ing] to move to direct and systematic impact measurement and attribution,”84 and in order to do so, the Fund intends to conduct 30 impact evaluations, including three to six randomised control trials between 2013 and the end of 2015 (see: Methodology for Impact Assessments for IFAD9).85

Figure 3.29 | KPI 19: Presenting performance information, ratings of micro-indicators

MI 19.1 – Reports on achievement of outcomes

A strong majority of donors at headquarters (90%) rated IFAD as adequate or above on whether IFAD’s reports to the Executive Board provide clear measures of achievement of outcomes.

The document review rated IFAD as adequate on this MI. IFAD reports annually on outcomes through the RIDE, collecting baseline data and setting targets for each outcome, but there are weaknesses in the results chain as noted in KPI 3. The RIDE 2012 assumes that “if current projects are reaching high output levels and the outputs are the right ones, then they are likely to reach positive project outcomes” (p.6, par 21). However, the links between organisation-wide outcomes and outputs are inadequately explained and the assessment team considers that the implicit link between these two levels (outputs and outcomes) is not a convincing one.

MI 19.2 – Reports on performance using data obtained from measuring indicators

This MI was assessed by document review only and was rated inadequate based a review of the Report on IFAD’s Development Effectiveness. The 2012 RIDE reports on IFAD’s performance against expectations and indicators set out for the Eighth Replenishment of IFAD’s Resources (IFAD8) period, 2010-2012. This includes both data on development effectiveness as well as data on performance against RMF institutional indicators (levels 1 through 5). However, while IFAD reports on outcomes and outputs, it lacks certain key reporting elements, such as output targets and an illustration of trends in measurement of outputs over time. IFAD does provide an illustration of trends for its level 2 results, e.g. the percentage of projects rated moderately satisfactory or above on efficiency over time.86

The quality of the performance data obtained by IFAD depends to a great extent on the quality of the indicators in its results framework, which are often inadequate. As noted in MI 3.3, IFAD indicators do not all meet CREAM or SMART criteria, there is some confusion between output and outcome indicators, and IFAD outcomes are, to some extent, measured on the basis of perceptions rather than actual data. In addition, there is no clear theory of change that explains to stakeholders how IFAD-funded activities are intended to lead to desired development outcomes. This being said, the new impact-level indicators included in the RMF for IFAD9 constitute a clear improvement in its reporting practices.

MI 19.3 – Reports against corporate strategy, including expected results

The majority of donors at headquarters (87%) rated IFAD as adequate or above on reporting against its organisational strategy.

The document review rated IFAD as adequate for reporting against the expected results in its results measurement framework (RMF). Both the 2012 RIDE and IFAD’s Annual Review on Portfolio Performance report on all five levels of results presented in the Results Measurement Framework (RMF) 2010-201287 – which includes both management results (levels 4-5) and development results (levels 1-3). Level 2 results are independently reviewed by the Office of Evaluation (IOE) in the Annual Report on Results and Impact (ARRI). While none of the reports fully explains the extent of achievements on results (particularly in relation to outcomes and impact), the Review on Portfolio Performance does provide some detail about variance of results and explanations of particularly strong or poor performance of recently completed projects.88

However, the reliability of the data can be called into question as one-third of level 3 outputs are based on estimates or extrapolations (see RIDE 2012, par. 27).While extrapolation of results may not be an issue in itself, the 2013 corporate-level evaluation of IFAD’s institutional efficiency commented on this issue as the extrapolation is based on a complex results management system where indicators and results layers are not always fully aligned.89 However, IFAD has made RIMS baseline and completion surveys compulsory, which should lead to better and more harmonised data.

MI 19.4 – Reports on aid effectiveness commitments using indicators and country targets

The majority of donors at headquarters (65%) rated IFAD as adequate, strong or very strong on whether IFAD reports to the Executive Board on performance in relation to its Paris Declaration commitments. However, 19% gave ratings of inadequate or below and 16% answered ‘don’t know’.

In the document review, IFAD was rated very weak on this MI because it does not report on aid effectiveness commitments (e.g. Paris Declaration or Busan) using indicators and targets. The Report on IFAD’s Development Effectiveness (RIDE) uses data from a client satisfaction survey that asks clients about IFAD’s adherence to the aid effectiveness agenda (clients are surveyed on their perception of IFAD’s performance in terms of country ownership, alignment with national priorities, and harmonisation of practices with other donors). However, the RIDE does not present information on aid effectiveness indicators and targets from the country level.

MI 19.5 – Reports on adjustments to policies/strategies based on performance information

This MI was assessed by document review only, which rated IFAD as strong. The Fund defines how it carries out and uses annual performance reporting in its Results Measurement Framework, using a portfolio review process that provides information on the performance of individual projects. According to the Action Plan to strengthen the self-evaluation process,90 this performance information is used to build IFAD’s knowledge base and adjust IFAD’s programmes. Lessons learned reported in the Portfolio Review are intended to be disseminated to internal stakeholders (consultants, country programmes managers, etc.) as well as to external parties (partner institutions, government agencies, other groups receiving loans and grant funding from IFAD) so that they can be internalised in future project designs.

According to the RIDE, IFAD has adjusted a number of its policies and strategies based on performance information. For instance, in May 2011, it adopted a Country Presence Policy and Strategy based on independent findings from the “Corporate-level Evaluation of IFAD’s Field Presence Programme” (2007),91 which indicated that using out-posted country programme managers (CPMs) is a more effective strategy in terms of achieving results than using locally recruited staff to manage the office. IFAD also approved a new Policy on Gender Equality in April 2012 based on recommendations provided in the corporate-level evaluation on gender (2010). This policy relies on information on IFAD’s performance in terms of promoting gender equality and women’s empowerment. Starting in 2012, IFAD’s Executive Board will receive updates on progress towards meeting Gender Policy targets through the Annual Report on IFAD’s Policy on Gender Equality and Women’s Empowerment.

MI 19.6 – Reports on programming adjustments based on performance information

This MI was assessed by document review only, which rated IFAD as strong. There is evidence that IFAD adjusts its project-level goals as a result of Mid-Term Reviews. At the country level, all new country strategic opportunity programmes (COSOPs) are intended to consider findings and recommendations from previous country programme evaluations (CPEs). It is IFAD’s usual practice to modify its strategies at the country level and at the project level based on performance information. However, the corporate-level evaluation (CLE) on efficiency (2013) noted that the COSOP review system could be much more rigorous. Furthermore, there is little information on whether IFAD adjusts its budgets at the project level based on performance information.

KPI 20: Disseminating lessons learned

Finding 20: The document review noted that IFAD has made improvements in its knowledge management system, but still faces challenges in formalising its system for disseminating lessons learned. Overall, survey respondents thought that IFAD’s knowledge management practices are adequate.

The 2005 Independent External Evaluation (IEE) found that IFAD’s management of knowledge (KM) and innovation was “unsystematic and inadequate given its corporate mission.”92 To address this critique, the Fund developed a knowledge management strategy (2007) in which it recognised a number of key areas for improvement, including increasing dedicated human resources. The 2013 corporate-level evaluation on efficiency found “increased attention to KM since the KM strategy was adopted in 2007, but implementation has fallen short of expectations.”93

Figure 3.30 | KPI 20: Disseminating lessons learned, ratings of micro-indicators

MI 20.1 – Reports on lessons learned based on performance information

Donors at headquarters were asked whether IFAD identifies and disseminates lessons learned from performance information. The majority (77%) rated IFAD as adequate or above, and 16% as inadequate or below.

The document review found IFAD to be adequate in reporting on lessons learned based on performance information. In 2007, IFAD approved a Knowledge Management Strategy in which it positioned itself as a ‘knowledge broker’, leveraging its years of experience in financing agricultural and rural development programmes.94 In the strategy, IFAD commits to “drawing on lessons learned” to improve its current strategies, particularly in the areas of scaling up and direct supervision of projects and programmes.

IFAD’s Strategy and Knowledge Management Department (SKM) is responsible for documenting and disseminating lessons learned, as well as championing IFAD’s knowledge management and innovation activities. SKM published the 2010 Rural Poverty Report, IFAD’s flagship publication, which provides a comprehensive overview of the global consequences of and ways to eradicate rural poverty. The SKM department is supported by a Chief Development Strategist, whose role includes leading the formalisation of IFAD`s knowledge management system. The Office of the Chief Development Strategist plans to further develop IFAD’s publications and technical reporting (e.g. Technical Advisory Notes), expand its virtual knowledge-sharing platforms, and offer more opportunities for learning and collaboration across IFAD divisions.

Despite these improvements, IFAD still faces challenges in codifying and formalising its system to collect and disseminate lessons learned. The 2013 evaluation of IFAD’s institutional efficiency reports that IFAD is not doing an adequate job of mining the rich knowledge embedded within project operations. “KM in country programmes has not yet been sufficiently focused on scaling-up IFAD’s impact on the ground. ... [However] The establishment of the Statistics and Studies for Development Division in SKM is useful to strengthen IFAD’s capabilities for generating knowledge through specific development [impact evaluation] studies.”95

MI 20.2 – Lessons shared at all levels of the organisation

This MI was assessed by survey only. Donors at headquarters were asked whether IFAD provides opportunities at all levels of the organisation to share lessons from practical experience. Half of respondents (52%) rated IFAD as adequate or above, 32% answered “don’t know”, and 16% rated IFAD as weak or inadequate.

KPI 21: Availability of documents

Finding 21: IFAD makes most of its key documents available to the public and has amended its Disclosure Policy in response to requests for greater public access to its documentation.

Apart from confidential audit reports, IFAD makes most of its key documents available to the public. These include documents submitted to the plenary sessions of its governing bodies, information notes on projects being developed for Board presentation, agreements for approved loans and grants, and project/programme design documents.

Figure 3.31 | KPI 21: Availability of documents, ratings of micro-indicators

MI 21.1 – Key documents available to the public

IFAD was rated very strong by the document review in making key documents available to the public, which is evidence of IFAD’s commitment to transparency. In 2010, IFAD revised its disclosure policy in response to requests for greater public access to its documentation. The Fund discloses a range of information, including documents submitted to the plenary sessions of its governing bodies, agreements for approved loans and grants, financial information (i.e. audited financial reports), country strategic documents and project/programme design documents. The Consultation on the Eighth Replenishment of IFAD’s Resources requested the Executive Board to amend the IFAD Policy on the Disclosure of Documents,96 to enable project design documents to be disclosed prior to the Executive Board session at which the project is to be considered. The Consultation also directed the Executive Board to review policy provisions with regard to the disclosure of previously undisclosed documents. These changes have been implemented.

The document review noted that the majority of official IFAD documents are available in the four official IFAD languages - French, English, Spanish and Arabic – but that due to cost efficiency concerns IFAD is no longer systematically translating all of its official documents into all four official languages and now requires a rationale for translating certain documents.

4. Main Findings: Evidence of IFAD’s development results and relevance to stakeholders

4.1 INTRODUCTION

This section presents the results of the 2013 Common Approach assessment of IFAD in measuring and reporting on development results. It includes three key performance areas:

• Section 4.2: Evidence of progress towards organisation-wide outcomes (KPI A)

• Section 4.3: Evidence of contributions to country-level goals and priorities, including relevant Millennium Development Goals (KPI B)

• Section 4.4: Relevance of the organisation’s objectives and programme of work to country-level stakeholders (KPI C)

The assessment of this component uses the same “traffic light” colours used in other components but applies a simplified 4-point scale. The methodology is explained in Volume II, Appendix I.

Figure 4.1 provides a summary of the assessment of the three KPIs. The detailed findings on each KPI are presented in the sections that follow.

4.2 EVIDENCE OF IFAD’S PROGRESS TOWARDS ORGANISATION-WIDE OUTCOMES

4.2.1 Overview

This section presents the results of the assessment of IFAD’s progress towards organisation-wide results. KPI A suggests that an effective organisation should demonstrate progress towards organisation-wide, institutional outcomes. These are usually related to the organisation’s strategic objectives. The assessment draws on the evidence that the organisation has available on its different result areas, primarily its reports on results.

IFAD’s mandate and core functions

According to IFAD’s Strategic Framework 2011-2015, its mandate and overarching goal is to: “Enable poor rural people to improve their food security and nutrition, raise their incomes and strengthen their resilience.”

The strategy divides this overarching goal into five strategic objectives, namely:

• A natural resource and economic asset base for poor rural women and men that is more resilient to climate change, environmental degradation and market transformation

• Access for poor rural women and men to services to reduce poverty, improve nutrition, raise incomes and build resilience in a changing environment

• Poor rural women and men and their organisations able to manage profitable, sustainable and resilient farm and non-farm enterprises or take advantage of decent work opportunities

Figure 4.1 | Evidence of Development results, and Relevance to Stakeholders– overall ratings

• Poor rural women and men and their organisations able to influence policies and institutions that affect their livelihoods

• Enabling institutional and policy environments to support agricultural production and the full range of related non-farm activities.

IFAD works towards achieving these objectives by concentrating its efforts in a number of thematic areas linked to its mandate and broader global concerns such as climate change. IFAD’s areas of thematic focus include:

• Natural resources – land, water, energy and biodiversity

• Climate change adaptation and mitigation

• Improved agricultural technologies and effective production services

• A broad range of inclusive financial services

• Integration of poor rural people within value chains

• Rural enterprise development and non-farm employment opportunities

• Technical and vocational skills development

• Support to rural producers’ organisations.

In addition to presenting IFAD’s overarching goal, strategic objectives, and thematic areas of focus, the Strategic Framework 2011 – 2015 also outlines the principles of engagement that guide the organisation’s operations. These include the promotion of gender equality; innovation, learning and scaling-up, targeting and supporting the empowerment of rural people.

IFAD provides financial and technical assistance to agriculture and rural development projects based on the assumption that smallholder agriculture can act as both a crucial source of rural income and nutrition and a vector for rural economic growth. The organisation takes a market-oriented approach to rural development and its projects seek to incorporate smallholders into global value chain development.

Like other development organisations IFAD has committed to supporting the achievement of the MDGs. According to IFAD’s Strategic Framework 2011-2015, its mandate and comparative advantage “position it well to make a decisive contribution to the efforts of its developing member states to achieve MDG1 and to ensure food security for growing populations in the face of ever scarcer resources.”97 This commitment, towards MDG1 in particular, has been further emphasised in IFAD9, where a series of operational, institutional and financial commitments have been agreed to strengthen the Fund’s contribution to eradicating extreme poverty and hunger (MDG1) and to deepen its focus on gender equality and women’s empowerment.98

Among other things, this includes a commitment to move 80 million people out of poverty by 2015.99

IFAD’s results measurement and reporting at an organisation-wide level100

IFAD’s current reporting practices are based on the Results Measurement Framework (RMF) developed for the Eighth Replenishment of IFAD‘s Resources (IFAD8) 2010-2012. The RMF is built around a five-level results chain that captures a continuum of inputs, activities and outputs that are meant to lead to outcomes and impact:

• Level 1 provides an overview of the global development outcomes to which IFAD contributes. These results cannot be directly attributed to the work of IFAD as they result from collective action of countries and development partners as well as contextual factors beyond the control of IFAD or any other development actor

• Level 2 measures outcomes that are delivered by IFAD-supported projects at the country level

• Level 3 captures the outputs produced by projects at the country level

• Levels 4 and 5 include indicators that measure aspects of IFAD’s operational effectiveness and progress on internal reforms with a focus on institutional efficiency.

IFAD’s main vehicle for reporting on performance at the institutional level is the annual Report on IFAD’s Development Effectiveness (RIDE), supplemented by the Office of Evaluation (IOE)’s Annual Report on Results and Impact of IFAD Operations (ARRI), which presents a synthesis of evaluations completed in the previous year, draws general lessons, and covers a specific “learning theme” that is relevant to pro-poor rural development (such as supporting policy dialogue).

The RIDE reports on three main areas, including the:

• Relevance of IFAD’s mandate and operations in the context of the changing framework of international development assistance

• Development effectiveness of IFAD-financed operations in generating development results“on the ground” that support national and global efforts to reduce rural poverty and contribute to achieving the Millennium Development Goals, and the first MDG in particular

• Organisational effectiveness and efficiency of the Fund in delivering those results through improved internal performance management.

The Results Measurement Framework (RMF) used for reporting on results during the IFAD8 period has been updated for the IFAD9 period (2013-2015). The new RMF retains the five results levels, but has a stronger focus on measuring impact, as well as results related to scaling-up, environment and natural resource management (ENRM), and gender. In the RIDE 2013, IFAD will begin to report on four additional impact indicators (under level 2): a household asset ownership index; the level of child malnutrition, disaggregated for girls and boys; the length of the hungry season; and the number of people moved out of poverty as a result of activities supported by IFAD. The addition of these indicators reflects the main preoccupations expressed in the IFAD9 consultations which include impact and value for money. These new impact indicators are directly related to MDG1 and will provide an indication of the extent to which operations supported by IFAD are helping member states make progress towards this goal. However, although committed to supporting their achievement of MDG1, IFAD’s current performance reports, which are based on the RMF developed under IFAD8, do not directly report on its contributions to the MDGs at either the global or country levels.

For a list of the key documents consulted for the development results component of this report, please see Volume II, Appendix VIII.

Data used for this assessment

The assessment of KPI A is based on survey data and a document review that considered all available performance information (organisation-wide level) from the most recent strategic cycle (2011-2015). Attention was paid to the following elements: quality of results statements, including indicators, baselines and targets; the quality and consistency of evidence presented to substantiate the results achieved, including an assessment of contribution; and the evidence of progress towards organisation-wide outcomes reported by the organisation.

The main sources of documentary data include the RIDE 2012, 2011 and 2010, the ARRI 2012, the Corporate Level Evaluation of Gender (2010), and the Corporate Level Evaluation of IFAD’s Institutional Efficiency (2013).

This chapter mainly focuses on results achieved in relation to level 2 and 3 in IFAD’s RMF.

• The level 3 indicators provide quantitative measures of the aggregated key outputs achieved by on-going operations using the data generated by the Results and Impact Management System (RIMS). These output indicators, included in the RMF for the IFAD8 period, were identified by members of the Executive Board as the most relevant to the five strategic objectives contained in the IFAD Strategic Framework 2011-2015.101

• The level 2 indicators reported on in the RIDE are different in nature. Rather than providing a quantitative aggregation of outcomes and impacts achieved by operations supported by IFAD, they provide a measure of the extent to which IFAD’s projects are perceived to achieve their objectives. Level 2 results are based on project completion report ratings102 provided by expert assessments, rather than RIMS data. Results are drawn from the annual portfolio performance reviews, using a sample of around 50 projects (the number of projects which typically close over a two-year period).

4.2.2 Evidence of extent of progress towards organisation-wide outcomes

This section provides an overall rating for KPI A based on documentation made available by the organisation and survey data. It also includes assessments of: the quality and consistency of reports on organisation-wide results and on the extent of progress towards IFAD’s areas of thematic focus which the organisation sees as contributing to the achievement of its five strategic objectives. The assessment draws on a review of IFAD’s performance information and reviewed survey data from donors at headquarters.

Overall assessment

Figure 4.2 shows the overall rating for this KPI based on the review of IFAD’s reports on organisation-wide results in its areas of thematic focus – as expressed in IFAD reports103 and as indicated by surveyed stakeholders.

Overall, the RIDE provides evidence of progress towards outputs based on Level 3 indicators in IFAD’s Results Measurement Framework. While IFAD also reports on outcomes and impact in relation to Level 2 indicators, this level of reporting does not provide a quantitative measure of IFAD’s contributions to reducing rural poverty and improving food security. Also, as indicated above, although IFAD is clearly dedicated to helping member states achieve MDG1 and has developed its Strategic Framework around this objective, the current reporting format does not allow for an analysis of the extent to which IFAD has contributed to the achievement of this goal. However, its promise to move 80 million people out of poverty by 2015 and the introduction of quantitative impact indicators related to poverty reduction and food security provides evidence of the organisation’s strong focus on MDG1. In the current reporting format there is thus limited evidence of IFAD’s contributions to development outcomes and impact.

Overall, donors at headquarters hold positive views about IFAD’s contributions to results in the thematic areas identified for this assessment and provided ratings of adequate or strong on all areas assessed.

Figure 4.2 | KPI A: Evidence of the extent of progress toward organisation-wide outcomes, overall rating

How IFAD reports on organisation-wide results

Finding 22: IFAD’s current institutional reporting practices do not allow for a clear assessment of IFAD’s contributions to outcomes and impact. However, the Executive Board has approved an updated results framework for the IFAD9 period (2013-2015) that includes a set of outcome and impact indicators to be reported on in the RIDE 2013.

Figure 4.3 presents an assessment of the quality of the organisation’s systems and practices for measuring and reporting on its organisation-wide results. The criteria represent elements of good results reporting (see Volume II, Appendix I for full descriptions). The organisation is assessed according to whether it has met, partially met, or not met the criteria in each result or strategic objective area.

The first column shows IFAD’s thematic priorities (A1 to A8) and three themes (A9 to A11) identified by MOPAN members as central to IFAD’s development mission and mandate (gender and women’s empowerment; household food security and nutrition; and poverty reduction overall).

Figure 4.3 | IFAD’s measurement and reporting on it areas of thematic focus

Strategic planning and theory(ies) of change

IFAD’s Strategic Framework 2011 – 2015 provides a thorough description of the changing global context for agriculture and rural development. It also provides an analysis of some of the generic reasons for the persistence of rural poverty across most regions in the world and an account of the changing global architecture for food security. IFAD describes its comparative advantages in relation to tackling these complex issues and lays out its strategic framework in response (i.e. goal, strategic objectives, areas of thematic focus, and projected outcomes). The strategic framework and IFAD’s approach to rural development is founded on a “dynamic vision of small-scale agriculture and rural development”115 which, among other things, is based on and in line with the findings and analyses in IFAD’s “Rural Poverty Report”.116 The Rural Poverty Report provides a comprehensive analysis of the underlying reasons for rural poverty, its global consequences, and the prospects for eradicating it.

IFAD’s Strategic Framework 2011-2015 does not contain elements of a basic theory of change – understood, in this case, as an explicit explanation of how a set of output results (or accomplishments) is intended to produce medium to long-term results. However, it does provide a comprehensive description of the problem (i.e. rural poverty) and how the organisation will contribute to eradicating it based on its comparative advantages and its vision of small-scale agriculture and rural development. From this perspective, the strategy provides a good basis for understanding how IFAD’s interventions are intended to contribute to results.

This being said, IFAD’s results reporting would be strengthened if there was a clearer articulation of the expected results chain at the institutional level as a way of explaining how the organisation is intending to contribute to change. This would also provide a more robust framework for strategic decision making, communicating and reporting. In the absence of this, it is difficult to identify IFAD’s broader contribution towards results in relation to its strategic objectives. This is further exacerbated by the fact that the unit of analysis in its reporting remains at the project level and not the country level. Developing a theory of change at the organisation-wide level for an international financial institution (IFI) is admittedly a challenging task. However, given IFAD’s relatively narrow mandate focusing on rural development, it should be feasible and less complex than for other IFIs with broader mandates. A well-developed theory of change based on solid research and evidence would provide the intermediate steps needed to more credibly link outputs (delivered by projects supported by IFAD) to development outcomes and/or impact.

Measuring results

As noted in the assessment of KPI 3 (in Section 3.3.2 on Strategic Management), the quality of IFAD’s results framework is adequate. It is commended for having established a five-level results chain linking development outputs and outcomes and organisational and operational effectiveness in its Results Measurement Framework. It is also recognised for explicitly linking the different levels of results based on the assumption that if it delivers well at the output level, this will have positive consequences for results achieved at the outcome and impact level. However, it is also noted that there are some weaknesses in this results chain given the lack of a theory of change and an explicit and logical description of the path between outputs, outcomes and impacts. In addition, the current outcome and impact data presented under Level 2 is not based on actual observed outcomes; rather, this information reflects the level of satisfaction with project performance based on expert assessments in Project Completion Reports.

IFAD acknowledges that it is challenging to provide a credible comparison of year-to-year performance based on the relatively small number of projects closing annually (around 25) as a small number of well- performing or poor performing projects can significantly skew results.117 Finally, as indicated in Section 2.3.2 on strategic management, the current outcome indicators under Level 2 are neither SMART nor CREAM (in SMART, neither “achievable” nor “time-bound” are met; in CREAM, it is not possible to judge if they are “adequate”).118 The output indicators fare better in relation to these standards, but it is still difficult to tell if they are “achievable” and “time bound” as there are no targets.

According to the 2013 corporate-level evaluation on IFAD’s institutional efficiency, efforts have been made to strengthen the self-evaluation system in past years. For example, direct supervision and implementation support has been significantly strengthened since 2007 which has ensured a greater focus on the collection of data in key areas of concern to IFAD. The evaluation also concludes that the RIMS surveys, though variable in quality, generate a fair amount of data that is used for reporting project performance.119

However, the evaluation also states that the results measurement system is overly complex and that lack of alignment between indicators and layers in the system makes aggregation of results difficult.120

IFAD’s “Annual Review of Portfolio Performance 2011-2012” also determined that more efforts are needed to increase the quality of Project Completion Reports (PCR).121 According to the review, there have not been any substantial improvements in the overall quality of the PCRs between the 2007-09 and the 2010-12 periods. Indeed, in 2012, no PCR was rated as highly satisfactory.

According to the RIDE 2012, the data reported “are best viewed as being approximate, or ‘more or less’ accurate. It is possible to speculate about finer trends, but only with many caveats – particularly over the shorter term. The RMF gives comfort that IFAD is broadly on course – or alerts that there may be an important issue requiring corrective action. This is important information, but it is not – nor is it intended to be – the basis of ‘microsurgery’.”While this caveat is not unique to IFAD but a premise for most, if not all, IFIs, it is still important to keep in mind when assessing the extent to which IFAD is achieving its intended results.

Reporting and communicating results

As described in previous sections, IFAD produces a number of annual reports to meet its various commitments and its different levels of accountability. The RIDE provides a significant amount of information on the interventions that IFAD is supporting, but it is still challenging to get a clear picture of its contributions to higher-level results and the level of causality between the different results levels. However, as mentioned in previous sections, in the IFAD9 period, IFAD will endeavour to move to direct and more systematic impact measurement and analysis of attribution.122

4.3 Evidence of the extent of progress towards IFAD’s strategic objectives

Finding 23: Surveyed stakeholders consider that IFAD is making progress towards its organisation- wide areas of thematic focus. The review of IFAD’s reports leads to the conclusion that the organisation is making progress towards delivering outputs, but evidence of contribution towards outcomes and impact remains weak.

A total of 31 MOPAN donors at headquarters responded to questions on results (the only respondent group to receive questions in this area). They were asked whether IFAD is making progress towards its eight areas of thematic focus as defined in the Strategic Framework 2011 – 2015. In addition they were asked about three themes identified by MOPAN members as central to IFAD’s development mandate (gender and women’s empowerment, household food security and nutrition, and poverty reduction).

All eleven areas were rated as either adequate or strong by the respondents. The highest rating was provided for IFAD’s support to rural producers’ organisations in borrowing countries.

IFAD reports credibly on its outputs in most of the thematic priority areas identified, but its current reporting on outcomes and impact does not reflect a theory of change or fully explain its contribution to those results. Also, the organisation makes limited reference to baselines and targets when assessing progress in its organisation-wide reports. The outputs reported in RIDE 2012 are based on data generated by the Results and Impact Management System (RIMS). At the end of 2011, IFAD had 259 projects in the on-going portfolio. Of these, 174 reported their outputs directly to IFAD and for the remaining 84 projects the outputs were estimated using extrapolation.

Output-level evidence

Figure 4.4 presents a summary of the extent of progress towards outputs related to IFAD’s thematic priority areas as reported in RIDE 2012 and other relevant documents. The final column shows the mean scores of surveyed MOPAN donors at headquarters who were asked to rate IFAD’s level of contribution in these areas. The survey used the same 6-point scale that was used in the assessment of IFAD’s practices and systems.

Figure 4.4 | Extent of progress towards IFAD’s areas of thematic focus123

Figure 4.5 | Extent of progress towards themes identified by MOPAN, survey ratings

Outcome-level evidence

In addition to reporting on outputs achieved under Level 3, IFAD also assesses outcomes and emerging impact for projects and programmes under Level 2 in the RIDE. Although these do not directly correspond to the areas of thematic focus or all of IFAD’s strategic objectives, they do provide a measure of the extent to which IFAD’s projects are perceived to be relevant, effective, efficient, innovative, and gender sensitive. The table below shows the percentage of projects rated ‘moderately satisfactory’ or above at project completion, based on project completion reports between 2008 and 2012, and reported in the RIDE 2012.126

Figure 4.6 | Projects rated by IFAD as ‘moderately satisfactory’ or above at project completion, 2008-2012127

Keeping in mind the relatively short time interval between the periods, most indicators appear to be close to or above the 2012 targets set for IFAD8. Data for 2012 suggest a positive trend for rural poverty impact, innovation, replicability and scaling up, and gender. The projects completed and reviewed in 2012 show a higher-than-targeted level of performance in terms of their relevance (95%), effectiveness in achieving development objectives (91%), rural poverty impact (95%), innovation and scaling up (91%), and gender (91%).

IFAD’s areas of thematic focus

The following section provides a discussion of results achieved (mostly outputs) in IFAD’s eight areas of thematic focus. See Volume II, Appendix IX for IFAD’s reported contributions to outputs from 2010-2012, as reported in the RIDE 2012.

A1: Natural resources - land, water, energy and biodiversity

MOPAN donors at headquarters were asked whether IFAD is making progress towards enhancing sustainable access to natural resources (land, water, energy and biodiversity) in borrowing countries. The majority (77%) rated IFAD as adequate or above. Indeed, 48% gave ratings of strong and very strong.

The RIDE 2012 reports on two key output indicators directly related to natural resources management. The review of documents found mixed evidence of progress in IFAD’s contributions in this area. While the level of outputs increased steadily until 2011, there seems to have been a drop in 2012. IFAD does not provide an explanation of the variations directly related to these indicators, but states, in more general terms, that “depending on the nature of the projects entering and exiting the on-going portfolio, the level and sector composition of outputs delivered varies over time.”129 While this might explain the drop in outputs achieved, a more specific explanation of this variance would have been helpful. The RIDE 2012 does not report on IFAD’s contributions to higher level results, such as outcome and impact in relation to environment and natural resource management.130

According to the ARRI 2012, two evaluation criteria (natural resources and the environment and climate change) have improved over the decade and do not show a recent decline in performance. These two criteria, which are among the five domains included in evaluations of rural poverty impact,131 have improved from 39% of projects moderately satisfactory or better during 2002-2004 to 79% during 2009-2011. According to the ARRI, this performance level is now similar to the other rural poverty impact domains and represents a major achievement for what used to be a ‘problem’ domain. This being said, the ARRI also states that a significant proportion of projects (53%) are only moderately satisfactory and none are highly satisfactory.132

Information on the actual impact of the amount of land benefiting from improved management practices or constructed/rehabilitated irrigation schemes on rural families is absent in IFAD’s reporting.

A2: Climate change

MOPAN donors at headquarters were asked whether IFAD has effectively contributed to adapting and mitigating climate change impacts in borrowing countries. The majority (77%) rated IFAD as adequate or above and 38% gave ratings of strong and very strong.

The RIDE 2012 does not report on IFAD’s achievement of results related to climate change. Since 2010, however, IFAD has significantly strengthened its focus on environmental and climate issues. In 2010, it established the Environment and Climate Division (ECD) and in 2011 launched its first Climate Change Strategy. In 2011, it put in place a new Environment and Natural Resource Management Policy. According to the RIDE 2012, IFAD has put major effort into expanding the integration of environment and natural resources priorities across its portfolio, but it does not yet include specific outcome level reporting on climate change and the environment. However, its 2013 – 2015 Results Measurement Framework does include a new indicator on environment and natural resources management and the Climate Change Results Implementation Framework commits IFAD to mainstream environment and climate change activities by requiring all new COSOPs to systematically and appropriately reflect climate and environment issues.133 As mentioned under the analysis of MI 4.2 above, recent efforts have been made to integrate the climate change dimension into the IFAD results management system, more specifically new reporting lines on climate change have been included in the Project Completion Report (PCR) template, the Supervision Report template, the Guidelines for Portfolio Review and the sub-categories and themes of the Project Portfolio Management System (PPMS).134

A3: Improved agricultural technologies and effective production services

MOPAN donors at headquarters were asked whether IFAD has effectively contributed to improving agricultural technologies and effective production services. All respondents (apart from two who answered “don’t know”) rated IFAD as adequate or above on this MI. Indeed, a majority (65%) gave IFAD a strong or a very strong rating in this area.

According to results reported in the RIDE 2012, there has been a significant increase in the number of people trained in crop production and livestock practices and technologies. At the same time there has been a clear shift in the balance between men and women receiving these services in the favour of women. This trend might not be surprising given IFAD’s strategic focus on small-scale agricultural production as a key contributor to economic growth and broader food security.135 However, as previously mentioned, neither the RIDE nor any other current report on the effectiveness of IFAD’s operations directly reports on the outcomes or impacts related to these outputs. It is therefore difficult, on this basis alone, to assess the extent to which IFAD has contributed to development results beyond the output level. Further, it is impossible to understand what the shift towards increased numbers of trained women means to rural families, their productivity, the gender division of labour, children’s participation in rural activities, and poverty overall.

A4: Broad range of inclusive financial services

Donors at headquarters were asked whether IFAD has effectively contributed to working with private- sector institutions to provide a broad range of inclusive financial services to small-scale agricultural producers and other poor rural people in borrowing countries. The majority (87%) rated IFAD as adequate or above, including 48% who gave ratings of strong or very strong.

According to RIMS data, the number of active borrowers and voluntary savers has fluctuated over time. The number of active borrowers has remained more or less constant over time (apart from a dip in 2011), while the number of voluntary savers increased sharply in 2010 and remained high in 2011 after which the numbers fell in 2012. It is interesting to observe the significant changes in gender balance. According to the data, women now constitute more than two-thirds of all borrowers and savers involved in IFAD’s operations, but one is unable to discern the impact of this change on the lives of rural women and their families through IFAD’s reports.

A5: Integration of poor rural people within value chains

MOPAN donors at headquarters were asked whether IFAD has effectively contributed to integrating poor rural people within value chains in borrowing countries. The majority (87%) rated IFAD as adequate or above, and 61% thought IFAD performed strongly or very strongly.

Data from the RIDE 2012 shows robust increases in the number of kilometres of road constructed or rehabilitated. The RIDE also provides myriad examples of collaboration between IFAD and the private sector, which aims to create new linkages and market opportunities for poor rural producers. For instance, in Indonesia, IFAD has partnered with Mars Incorporated to provide extension services in Central Sulawesi and Papua. In Guatemala, IFAD continues to provide grant financing to the Guatemalan Exporters’ Association (AGEXPORT) to help integrate small and medium producers into national and international value chains (e.g. supporting SMEs to meet quality standards and fulfil conditions to access international markets).

However, there is little information provided in IFAD’s reports on these activities and medium or longer- term outcomes on the lives of people living in rural areas.

A6: Rural enterprises development and non-farm employment opportunities

Donors at headquarters were asked two questions in this area: Whether IFAD has effectively contributed to developing rural enterprises in borrowing countries, and if IFAD has effectively contributed to developing non-farm employment opportunities in borrowing countries.

Overall, 35% of respondents considered IFAD as adequate and 48% as strong or very strong in developing rural enterprises, while the ratings were mixed in relation to developing non-farm employment opportunities136 (61% rated IFAD adequate or above while 16% rated it inadequate).

Data reported by IFAD shows a strong increase in the number of people trained in business and entrepreneurship and in the number of enterprises accessing business support services (non-financial). The growing number is encouraging and reflects IFAD’s funding priorities – approximately 35% of IFAD’s programme funds go towards training to support smallholders. This level of investment alone warrants an independent evaluation to identify the efficiency, impact and sustainability of these training initiatives.

A7: Technical and vocational skills development

When MOPAN donors at headquarters were asked whether IFAD has effectively contributed to developing technical and vocational skills in borrowing countries, 71% provided a rating of adequate or above and 29% rated it strong or very strong. The RIDE does not report specifically on this target area, but as indicated in A6 above, the number of people trained in business and entrepreneurship has increased significantly during the IFAD8 period.

A8: Support to rural producers’ organisations

Donors at headquarters were asked whether IFAD has effectively contributed to supporting rural producers’ organisations in borrowing countries. All respondents rated IFAD as adequate or above (apart from 6% who replied “don’t know”), with 58% rating it strong or very strong.

From 2008 to 2012, IFAD reported a steady increase in the number of people trained in community management topics. IFAD works with and helps strengthen community-based organisations representing the interests of smallholder farmers, self-help groups, pastoral and small fisher organisations, water users’ associations, savings groups, co-operatives and community banks, road maintenance groups, and village and district development organisations (among others). Also, it is interesting to observe the shift in the gender balance with regard to the number of people trained in community management topics, but one is left questioning the impact of this shift on the lives of men and women in rural communities.

Thematic/cross-cutting priorities of relevance to MOPAN

This section provides a discussion of results achieved (mostly outputs) in three areas of thematic focus established by MOPAN for this assessment.

A9: Promotion of gender equality and women’s empowerment

MOPAN donors at headquarters were asked whether IFAD has effectively contributed to promoting gender equality and women’s empowerment. Overall 45% considered IFAD adequate and 39% rated it strong or very strong.

This is not one of IFAD’s areas of thematic focus, but, rather, a theme reported on as part of its Level 2 results. As such, it reports indirectly on gender equality and women’s empowerment by presenting gender- disaggregated data for 7 of the 13 output indicators in its results framework. On most of the indicators where the gender ratio is measured in Level 3, the gender balance among beneficiaries has tipped to the advantage of women. The only group of indicators where the opposite has happened is in relation to agricultural technologies where the ratio of men and women trained has changed in favour of men.

According to the ARRI 2012, some 80% of projects evaluated in 2010-2011 were rated within the satisfactory zone in promoting gender equality and women’s empowerment.137 Although this reflects only two years of data, it confirms the generally positive findings about the attention devoted to gender issues in IFAD-financed operations. Once again, however, there are further opportunities for improvement, as nearly half the projects evaluated were rated as moderately satisfactory and very few were highly satisfactory.

According to the Annual Report on the implementation of IFAD’s policy on gender equality and women’s empowerment,138 women’s representation among people receiving services from IFAD-supported projects has increased significantly, both numerically (from 19 million to 28 million) and as a percentage of the total (from 45% to 48%). Among economically related activities, outreach to women is strongest in business and entrepreneurship training and rural financial services, where women are more active than men as both savers and borrowers. This demonstrates the significant contribution of women-run microenterprises and small businesses to the rural economy.

Clearly IFAD has improved its promotion of gender equality and, as described in the practices section, several new policies and guidelines have been put in place to better mainstream gender equality in IFAD’s operations over the last few years.

IFAD’s Project Completion Reports provide a rating of the extent to which projects contribute to improving gender equality (using a percentage of projects rated moderately satisfactory or better). In 2012, 91% of projects were rated as moderately satisfactory or better at completion, exceeding the target of 75%. The 2008-2009 baseline was 69%, so there has been a clear increase in levels of satisfaction on gender outcomes over the last four to five years.

A10: Resilience of household food security and nutrition

MOPAN donors at headquarters were asked whether IFAD has effectively contributed to improving the resilience of household food security and nutrition. A clear majority of respondents (90%) rated IFAD of adequate or above and 52% as strong or very strong.

IFAD does not currently report directly on its contributions to improving the resilience of household food security and nutrition (but indirectly through the rural poverty impact indicator under Level 2 in the RMF). However, as part of its enhanced focus on impact measurement in 2013 it has added new indicators to its results framework, including ‘length of hungry season’, and ‘child malnutrition’, as measures of food and nutrition security of target group households and individuals. IFAD will begin to report on these indicators in the 2013 Report on IFAD’s Development Effectiveness (RIDE).

A11: Support to poverty reduction

MOPAN donors at headquarters were asked whether IFAD has effectively contributed to enabling rural women and men to overcome poverty. A clear majority of respondents (94%) provided ratings of adequate or above and 58% rated IFAD strong or very strong.

Currently IFAD does not provide any outcome data related to its support to poverty reduction or its contributions related to the achievement of MDG1. However, as mentioned above, IFAD has modified its results framework, and in 2013 will start reporting on the number of people moved out of poverty as a result of its operations (directly related to MDG1). During its Ninth Replenishment period (2013-2015), IFAD plans to undertake 30 impact studies and conduct randomised control trials to strengthen attribution and understand the causal links between IFAD project outputs and impacts on poverty reduction and food security. In its Report on the Ninth Replenishment, IFAD specified that the methodology used for impact surveys is still experimental but could potentially “yield a high return to the science of impact measurement in the field of poverty reduction.”

At the outcome level (Level 3 in the RMF), IFAD reports on an aggregate indicator called ‘rural poverty impact’.139 IFAD’s target is a rate of 90% moderately satisfactory or above. In 2005 the rate was as low as 55% and has increased significantly since then. In 2012 it was 95%, thus exceeding its target.

4.3 EVIDENCE OF IFAD’S CONTRIBUTION TO COUNTRY-LEVEL GOALS AND PRIORITIES, INCLUDING RELEVANT MDGS140

4.3.1 Overview

This section presents the results of the assessment of evidence of IFAD’s contributions to project level results and relevant MDGs. By distinguishing between the organisation-wide level and the project level, MOPAN recognises the demand-driven nature of many of the activities of a multilateral organisation and the key role that is played by its country programming or strategy document, where expected results at the highest level (outcomes and impact) reflect a shared responsibility between the multilateral organisation and the partner country.

Section 4.3.2 examines evidence of the organisation’s contribution to project-level goals and priorities. (Note: Section 4.4 examines relevance in terms of the extent to which partners and donors in-country believe the organisation supports country priorities and meets changing needs.)

IFAD’s results and reporting at the country and project level141

IFAD’s work at the country level is guided by the country strategy documents, called Results-Based Country Strategic Opportunity Programmes (RB-COSOPs). The COSOPs are prepared in collaboration with each borrowing member country and are generally based on and aligned with national Poverty Reduction Strategy Papers (PRSPs) and planning frameworks. Although there is no clear uniformity in the time period of the different COSOPs, they are typically based on 5 to 6 year time frames. Each country strategy document has its own results framework, which is used to track IFAD’s performance and impact. RB-COSOPs are supposed to be reviewed annually, at mid-term, and at completion. IFAD’s Office of Evaluation (IOE) conducts country programme evaluations of completed COSOPs on a sample basis.

Project-level reporting is founded on the Project Design Reports that include a detailed account of project activities and objectives. During implementation, projects are requested to provide annual progress reports against the objectives and priorities set in the design report and to conduct and Mid-Term Reviews to ensure monitoring of project performance. Self-assessment of project performance at completion is made through project completion reports (PCRs). In addition to these reports, the IOE annually conducts a limited number of project evaluations on a sample basis. IOE also validates all PCRs produced based on desk review.

In addition to progress reports produced by the project, IFAD also produces supervision reports based on annual project supervision missions conducted for all projects.

IFAD’s reporting on results is supported by the Results and Impact Management System (RIMS). All projects are all required to have a logical framework, with clear and measurable indicators, chosen from a list of standard RIMS indicators predetermined by IFAD. Client surveys at the country level are undertaken periodically to collect client perspectives on the performance of IFAD across numerous indicators.

The RIMS includes three levels of results: first level results correspond to the project activities and outputs; second level results relate to project outcomes; and the third level results link to project impact.142 Project management teams are required to include RIMS indicators in logical frameworks and progress reports and IFAD has developed a number of tools to assist project management teams in reporting on RIMS related results, including comprehensive handbooks and guidelines.

Data used for this assessment

The country-level assessment of results achieved is based on data from the six countries included in the 2013 MOPAN assessment (Ethiopia, Guatemala, Indonesia, Mozambique, Pakistan and Viet Nam).

The document review is based on a wide range of documents including Country Programme Evaluations (CPEs), Project Completion Digests, Project Evaluations, COSOP Mid-Term Reviews, Project Supervision Reports, Project Follow-up Mission Aide memoires, Project Progress Reports, and other documents depending on availability. Throughout the document review attention was paid to the following elements: quality of the results statements; the relevance of indicators, baselines and targets; the strength of the link between results statements and results achieved; the quality of evidence presented to substantiate the results achieved, including an assessment of contribution; and, the overall performance story.

In the survey, IFAD’s direct partners and MOPAN members at the country level were asked questions tailored to the strategic objectives of the projects in the current country portfolio. Interviews with senior IFAD country office staff also informed the analysis of context and ensured that the assessment team had a full set of documentation with which to conduct the document review and analyse results.

The following sections present the overall results of this review, with project-specific examples provided to illustrate types of results achieved. However, due to the significant variations with regard to depth, structure and content of the documents reviewed, the following analyses should be seen as indicative rather than conclusive.

4.3.2 Evidence of contribution to country-level goals and priorities, including relevant MDGs

This section provides an overall rating for KPI B based on documentation made available by the organisation and survey data. It also includes a brief assessment of the measurement and reporting on country level results and a more complete assessment of the extent of progress towards results defined by IFAD in its country-level results frameworks.

Overall assessment and rating

IFAD has significantly improved its focus on results at the project and country level since the Independent External Evaluation of IFAD (IEE) in 2005.143 With the introduction of results-based COSOPs at the end of 2006, all COSOPs and projects now have results frameworks and provisions have been made for annual reviews, mid-term reviews and completion reviews for IFAD’s country strategies. In addition, direct supervision and implementation support since 2007 has contributed to improving the focus on collection of data in key areas of concern to IFAD.

However, according to the 2013 corporate-level evaluation of IFAD’s institutional efficiency, annual reviews are not yet being performed systematically for all COSOPs and midterm reviews are often done late and with varying quality. In addition, COSOP completion reviews have not yet been systematically introduced across all countries. This, combined with limitations in the amount of documentation disclosed for this assessment, means that the analysis of results at the project level is based on a mixed portfolio of documents. The variations in format and content of these documents and a general lack of data on outcomes achieved by IFAD-supported projects makes it challenging to measure and report on the overall contribution to outcomes of IFAD’s operations in the six countries assessed.

The document review also looked for evidence of the extent to which IFAD’s operations had contributed to the achievement of relevant MDGs at the country level (MDG1 in particular). IFAD does not directly report on its contributions to the MDGs at the country level as its results measurement systems mainly captures project-level results. However, all the COSOPs assessed make reference to the MDGs in general terms, but only the COSOPs for Ethiopia, Guatemala and Indonesia explicitly refer to MDG1 in the description of their strategic approaches and the overall description of the country context. None of the COSOP results frameworks assessed provides explicit links between their strategic objectives and specific MDGs, but as they all focus on reducing rural poverty and improving food security, they can clearly be associated with MDG1. Further, the key objectives of all the projects assessed focussed on issues such as agricultural productivity, access to financial services, increasing incomes, access to markets, improving household food security that are closely aligned with IFAD’s mandate and can be associated with MDG1. From this perspective it is clear that Fund’s operations support the MDG agenda in its member countries. However, due to IFAD’s project-oriented approach and limited reporting on contributions to country-level development outcomes, it is difficult to appreciate the extent to which outcomes generated by IFAD- supported projects have contributed to the MDG agenda overall at the country level.

The document review suggests that the results of IFAD’s operations in the six countries reviewed have been mixed. There are some examples of projects that have clearly contributed to development outcomes whereas others have experienced delays and have not been able to deliver the intended results. Surveyed stakeholders also hold mixed views about IFAD‘s progress towards its strategic objectives at the country level. The highest rated country was Viet Nam (where surveyed stakeholders rated IFAD strong in all results areas), while Indonesia was the lowest rated (where stakeholders rated it inadequate in three results areas and adequate in the other three).

Figure 4.7 shows the overall rating for this KPI based on the review of IFAD’s reports on country-level goals and priorities and as perceived by surveyed stakeholders

Figure 4.7 | KPI B: Evidence of the extent of contribution to country-level goals and priorities, rating

How IFAD reports on country-level results (for all countries sampled)

Finding 24: IFAD has shown a commitment to improve its reporting practices at the project and country level. However, the intended reporting practices at this level are not yet fully implemented. It is, therefore, difficult to get a full appreciation of the organisation’s contributions to country-level goals and priorities.

Despite recent and on-going work to improve its focus on results, the following strengths and weaknesses in IFAD’s reporting and results frameworks can be identified.

Figure 4.8 presents a summary assessment of the quality of the organisation’s systems and practices for measuring and reporting on country level results. The headings show the criteria MOPAN used to assess the systems and practices.

Figure 4.8 | IFAD’s measurement and reporting on country-level results

Strategic planning and theory(ies) of change

As noted above, strategic planning at the country level is guided by the RB-COSOPs. While the COSOPs provide a thorough description of the specific country context and how IFAD intends to contribute to development results within its areas of comparative advantage, none of the COSOPs, or any other planning document reviewed, include any explicit theory of change.

Measuring results

The core of IFAD’s results measurement at the project level is the Results and Impact Management System (RIMS) that was developed in 2003 and that supports reporting on project outputs and impact. The RIMS system generates a fairly substantial amount of data that is used for reporting on project performance as well as performance at the institution-wide level. Although there is some concern about the quality of the RIMS data, IFAD’s introduction of direct supervision and implementation support since 2007 has helped address some of these issues.

However, according to the 2013 corporate-level evaluation (CLE) of IFAD’s institutional efficiency, and in line with the assessment team’s observations during the document review, systematic evidence of results beyond outputs appears scarce or scattered. The CLE also observed that IFAD’s results measurement system seems overly complex in light of the often limited local capacity to implement the system. It also highlighted some challenges with regard to the alignment of the different layers in the system which makes aggregation of results difficult. Finally it noted that baseline surveys are not always conducted in a timely manner (or not conducted at all) which, combined with the factors above, explain why reporting on outcomes and impact levels at the project and country level remains relatively weak.

Reporting and communicating results

The COSOP review cycle does not seem to be fully or consistently implemented. According to the recent evaluation of IFAD’s institutional efficiency, “measuring and reporting on overall achievements of IFAD activities, in particular country programmes, including the extent to which policy dialogue, KM, partnership building and grants are having a synergistic impact aimed at supporting scaling up, have not received the necessary attention.”149 This assessment is mirrored by IOE’s response to the RIDE 2012 and analyses in the Annual Report on Results and Impact of IFAD Operations (ARRI) 2012, where the same shortcomings are highlighted. However, according to the ARRI 2012, COSOP completion reviews are likely to be introduced in the very near future.

Another challenge related to reporting on results at the country level is discussed in the ARRI 2012. According to the report, most of IOE’s country programme evaluations indicate that IFAD-supported country programmes to a large extent are seen as a compilation of individual investment projects. The ARRI recommends stronger efforts to ensure greater synergy across projects and between projects and grants and non-lending activities (knowledge management, policy dialogue and partnership-building). It also concludes that the integration of all IFAD-supported activities at the country level is critical to ensuring the scaling up of impact and sustainability, as well as to achieving more efficient delivery.

Evidence of the extent of progress toward IFAD results at the country level

In the following sections on all six countries sampled, the assessment examined the strategic objectives of each IFAD country office as well as two areas of interest to MOPAN that are relevant to IFADs mandate: promotion of gender equality and women’s empowerment, and improvement of household food security and nutrition. See Volume II, Appendix X for examples of IFAD’s contribution to country-level goals and priorities, by project strategic objective.

INDONESIA

The document review was mainly based on an assessment of the independent Country Programme Evaluation (CPE) published by IOE in 2013, the Annual Implementation Progress Report from 2012, and the Project Completion Digest of the Post-crisis Program for Participatory Integrated Rural Development in Rain-Fed Areas (PIDRA) from 2011. The CPE assesses the performance of IFAD’s activities, including the project portfolio and non-lending activities, and the COSOP approved in 2008 and ending in 2013. The CPE covers nine years, of which five years (2004 to 2008) are prior to the COSOP, and four years (2009 to 2012) are part of the current COSOP. The analysis is largely based on the independent evaluation’s findings related to results achieved by three on-going projects and two projects that were closed during the 2009-2013 COSOP and, to a lesser degree, on the self-reported data found in the MTR and other internal reports.

Finding 25: Survey respondents provided mixed ratings for IFAD’s achievement of results in Indonesia. The document review found that IFAD-funded projects in Indonesia had been deemed to be ‘moderately satisfactory’ by the CPE and other performance reports in most areas, but did not find strong evidence of contributions in the area of food security and agricultural productivity.

MOPAN donors in-country and direct partners were asked whether IFAD in Indonesia has effectively contributed to each of the strategic objectives and areas of interest to MOPAN. Survey respondents provided mixed ratings on the questions and rated IFAD’s contributions to development results as inadequate on three of the six areas assessed. MOPAN donors in-country provided lower ratings than direct partners in relation to four of the six areas assessed (B1, B2 and B4). These differences in opinion were statistically significant.

The overarching goal of the 2009-2013 COSOP was to empower poor rural women and men to achieve enhanced food security, increased incomes, and poverty reduction with the focus on smallholder agricultural and rural development, the enhancement of the competitiveness of agricultural smallholders and producers, growth of the rural economy, and addressing the key determinants of rural poverty.

The CPE found that IFAD’s performance had been mixed overall. IFAD-funded projects had been moderately satisfactory in terms of increasing household income and assets, strengthening human and social capital and empowerment, improving natural resource management, building capacity of the rural poor to engage in local policy processes, and promoting women’s empowerment and gender equality. However, it did not find strong evidence of contributions by IFAD-supported projects in the area of food security and agricultural productivity.

Evidence of IFAD’s contributions to sustaining the growth of economic activities and increasing household incomes for poor households is mainly based on the results of two completed projects (the “Income- Generating Project for Marginal Farmers and Landless in Indonesia” (P4K) and PIDRA), the CPE and the COSOP Midterm Review from 2012. According to the CPE, P4K enabled poor beneficiaries to gain access to financial services, including savings and loans while PIDRA facilitated an increase in household income for project beneficiaries. The evaluation also found evidence of IFAD’s contributions related to capacity building of the rural poor to engage in local policy and programming processes (see the table above). IFAD’s overall performance related to strengthening Institutions and policies was rated moderately satisfactory by the CPE.

The CPE also rated the performance of IFAD-funded projects for gender equality and women’s empowerment as “moderately satisfactory.” Projects seem to be rather successful overall in encouraging women’s active participation in project formulation, however it remains to be seen whether women are taking a greater leadership role as a result of this. Evidence from one of the on-going projects in the portfolio, the National Programme for Community Empowerment (PNPM), indicates that considerable progress has been made in mobilising women to participate in farmer self-help groups. The CPE also confirmed that self-help groups have been an effective way to enhance the role of women.

However, IFAD’s performance with regard to promoting household food security and agricultural productivity was rated as moderately unsatisfactory. The three current projects only have a limited focus on this area, but as described in the table above (B6) the 2012 Midterm Review of the COSOP provides evidence of contributions in this area based on a recently closed project.

The overall impact of the project portfolio on the natural resources and environment also seems limited given that most of the projects do not focus directly on natural resource management.

Although not directly related to the assessment of the effectiveness of individual projects under the country programme, the extent to which innovative approaches have been scaled up is important given IFAD’s strong focus on the scaling up agenda. According to the evaluation of the COSOP 2009-2013, at least one of the completed projects during the period under review has been scaled up to the national level. The experience of PIDRA has been incorporated in a national programme (the Village Food Sufficiency Programme). The programme is being implemented in 24 provinces, 250 districts in 1,500 villages. Each village is allocated a sum of Rp.50 million. Based on this experience, IFAD’s performance is being rated as moderately satisfactory in this context.

VIET NAM

The document review is based on a set of documents including a Country Programme Evaluation (CPE) from 2012, a Supervision Report of the Developing Business with the Rural Poor project (DBRP) from 2012,150 the Supervision Report for the Sustainable Economic Empowerment of Ethnic Minorities in Dak Nong Province (3EM) from 2012,151 Supervision Reports for the Agriculture, Farmers, and Rural Areas Support Project in the Gia Lai, Ninh Thuan and Tuyen Quang Provinces (2012).152 The evaluation reviewed all projects supported by IFAD over the last 18 years, but only rated projects that were approved from 1999 onwards.

Finding 26: Both the document review and survey respondents provided strong ratings overall for IFAD’s achievement of results in Viet Nam. However, IFAD could develop a more strategic approach to its operations in natural resources management.

MOPAN donors in-country and direct partners were asked whether IFAD in Viet Nam has effectively contributed to the country strategic objectives and to areas of MOPAN interest. Survey respondents provided strong ratings on all six areas.

IFAD has been present in Viet Nam since 1993 and has financed 11 projects, for a total loan disbursement of USD 257 million. Key programming areas include microfinance, capacity-building, support for decentralisation, gender mainstreaming, small-scale infrastructure and market integration.

Viet Nam’s COSOP for the period of 2008 to 2012 (during which most of the on-going projects were formulated) sets out four strategic objectives: (1) Rural poor households in upland areas access markets through increased private-sector partnerships; (2) Poor and vulnerable households take advantage of profitable business opportunities; (3) Poor upland communities secure access to and derive sustained benefits from productive natural assets; and (4) Rural poor people contribute to pro-poor, market driven agricultural policy processes at sub-national levels.

Performance information from the 2012 country programme evaluation is positive overall. IFAD’s performance in terms of establishing a framework for sustainable agro-forestry development has been rated as satisfactory by project progress reports; forest land allocation planning has been completed for a greater-than-targeted number of hectares. Evidence indicates that the provision of forest land-use certificates is an effective way of improving living standards for recipient households.

IFAD’s performance in promoting gender equality is rated as satisfactory by the CPE. Women’s representation is considered to be a very strong component of most IFAD’s projects. Most IFAD-funded projects are conducted in collaboration with the Women’s Union which represents women at all levels of government. Women represent about 70% of all savings and credit groups, which are supported by the Women’s Union. In addition, IFAD encourages participatory processes, e.g. when planning small-scale infrastructure development, poor women are consulted and provide guidance in establishing priority areas to be addressed by the commune development funds (CDFs).

The CPE rates the Viet Nam project portfolio as satisfactory overall. However, one of the few areas where the CPE considers performance to be only moderately satisfactory relates to the programme’s impact on natural resources and the environment and sustainability. Despite the successful implementation of natural resources management programmes at the project level, the CPE concludes that IFAD’s approach has been scattered and not driven by a strategic approach covering the whole portfolio. This has limited the potential for positive synergies between projects and results that are “bigger than the sum of its parts.”

PAKISTAN

The analysis of results of IFAD’s programme in Pakistan is based on the perceptions of surveyed respondents and a set of documents including the Mid-Term Review of the current Results-Based COSOP (MTR) from 2012, a Project Completion Digest of the Microfinance Innovation and Outreach Programme (MIOP) that closed in 2012, a Project Completion Digest of the Restoration of Earthquake Affected Communities and Households project (REACH) that closed in 2010, and a Supervision Report of the Programme for Increasing Sustainable Microfinance (PRISM) from 2012. These three projects constitute half of the on-going projects at the time of approval of the COSOP 2010-2014. During this period three new projects have been designed, but given that they are still in the early phases on implementation, they have not been included in this assessment. IFAD’s financial commitment for the six on-going projects was USD 141.4 million. IFAD contribution in the three projects designed during the current COSOP was expected to be USD 88.6 million.

Finding 27: Survey respondents rated IFAD as strong for providing the rural poor in Pakistan with access to financial services and for increasing their incomes. Respondents also found that IFAD performed strongly in promoting gender equality. The document review found some evidence of results in relation to providing the rural poor with access to finance. However, a general lack of evidence related to the outcomes and impact of IFAD’s contributions in Pakistan makes it difficult to provide a robust analysis.

MOPAN donors in-country and direct partners were asked whether IFAD in Pakistan has effectively contributed to each of the strategic objectives and areas of interest to MOPAN. Survey respondents provided strong ratings on three of the five areas.

IFAD’s 2010-2014 RB-COSOP for Pakistan have two overall areas of strategic focus: (a) to enhance the access of poor rural men and women to productive assets, skills, services and improved technologies, with particular emphasis on productivity enhancement; and (b) to strengthen the capacity of the rural poor to engage in and benefit from local development processes.

As indicated above, assessing IFAD’s contributions to results in Pakistan is challenging due to the lack of reliable outcome-level data. In addition, most reports tend to focus on activities and outputs and do not necessarily report on the types of indicators which are outlined in the COSOP Results Management Framework.153 Indeed, according to the MTR from 2012 “it is unrealistic to expect that IFAD projects could have any significant impact on poverty in the districts in which they were implemented due to their limited coverage and limited funds (...) it is not possible to assess the impact of IFAD investments on participating households in IFAD financed projects due to lack of any systematic data which captures this analysis.”154

As indicated above, the findings and results reported below are therefore indicative rather than conclusive in nature.

According to the MTR of the COSOP, the on-going, completed and planned projects during the current COSOP period were designed to reach 830,000 households which is equivalent of around 5% of the total number of rural households in the country. At mid-term (two years into implementation) the number of households reached was around 3% of the population (72% of its targeted households) which, according to the MTR, is a higher than the average achievement for most of IFAD’s projects. At the time of the MTR, the on-going and completed projects under the current COSOP had thus provided more than 300,000 households access to finance, improved productive asset ownership for 21,694 households, increased productivity for 127,600 households and conducted demonstrations for improved agriculture practices in 1,126 villages.

The overall conclusion of the MTR review is that IFAD’s programme in Pakistan had been successful overall in bringing about the desired impact. It did, however, note that most projects had taken a long time to become effective due to legal and administrative hurdles, slow disbursements and long processing period of withdrawal applications. In addition, it concluded that the key impact indicators identified in the COSOP could not be measured due to the absence of proper baseline and impact evaluation surveys at the project level. It is therefore difficult to provide a conclusive assessment of outcomes and impacts achieved at this level.

However, according to the Project Completion Digest of the Microfinance Innovation and Outreach Programme (MIOP) which was closed in 2012,155 this project had a positive impact on food security indirectly through increased delivery of livestock and agricultural loans. Among the key activities and results quoted, it highlights that all agriculture and livestock specific loan products were associated with management training and in some cases client groups were assisted with establishing market linkages. Reportedly these activities had a positive impact on agricultural productivity and some of its loan products had helped farmers increase wheat and rice yields by 12.5% and 22% per acre respectively.

The Supervision Report of the PRISM concludes that despite the challenges and worsening economic situation in Pakistan, the project is progressing satisfactorily and is achieving its targets and intended results. As opposed to most other IFAD projects in Pakistan, the PRISM project has a sector-oriented focus on the limited involvement of commercial banks in rural microfinance. According to the Supervision Report, the project has substantially contributed to the positive transformation of the microfinance sector in Pakistan where more and more commercial banks have been investing in microfinance institutions across the country.

ETHIOPIA

The analysis of results of IFAD’s programme in Ethiopia is based on the perceptions of surveyed respondents and a set of documents including a Country Programme Evaluation (CPE) from 2008, covering IFAD-funded operations between 1997-2007; a Rural Financial Intermediation Programme – Interim Evaluation (IOE) from 2011; a Project Completion Digest of the Pastoral Community Development Project I (PCDP); a Mid-Term Project Performance Report (2008-2011) of the Pastoral Community Development Project II (PCDP II); a Mid-Term Review of the Agricultural Marketing Improvement Program (AMIP) from 2011; and a Progress Report of AMIP from 2011.

Finding 28: Survey respondents provided strong ratings for IFAD’s contributions to enhancing household incomes and food security and for developing a sustainable farmer-owned and managed model of small-scale irrigated agriculture. They provided adequate ratings overall on the five other indicators. The document review found evidence that IFAD has contributed to development outcomes in the area of rural finance. Lack of data in other areas of IFAD’s work in Ethiopia makes it difficult to provide a conclusive document assessment of IFAD’s overall performance in the country.

MOPAN donors in-country and direct partners were asked whether IFAD in Ethiopia has effectively contributed to each of the strategic objectives and areas of interest to MOPAN. Survey respondents provided strong ratings on two areas and adequate on the remaining five.

Since 1980, IFAD has provided loans and grants of USD398 million for 16 projects and programmes in Ethiopia. The current RB-COSOP of Ethiopia was approved in 2008 and focuses on three strategic objectives aimed at enhancing access by poor rural households to: (a) natural resources (land and water); (b) improved agricultural production technologies and support services; and (c) a broad range of financial services.

A country programme evaluation (CPE) from 2008 covering IFAD-funded operations between 1997 and 2007 concluded that the overall performance of IFAD’s projects in Ethiopia had been better than the average for IFAD operations worldwide. The evaluation noted that IFAD-supported projects in the areas of small-scale irrigation, rural finance, pastoral community development and local governance had performed particularly well and contributed to significant outcomes during the period under review. However, it also found areas for improvement in IFAD’s support to public/private sector partnerships in agricultural marketing and a lack of linkages between research, extension services and farmers. The extent to which IFAD-supported projects had forged linkages and developed synergies was also seen as limited.

Evidence from more recent reports at the project level, particularly related to IFAD’s support to rural finance, seems to confirm some of the trends identified in the 2008 IOE evaluation. For example, according to an interim evaluation from 2011 of the Rural Financial Intermediation Programme I (RUFIP),156 the programme achieved most of its objectives including promoting access to financial services for the rural poor and expanding outreach to well over 1.5 million rural households as targeted. Furthermore, it managed to promote linkages between rural financial institutions and the commercial banking sector and supported expansion of the credit co-operative sub-sector.

Data from another recently completed project, the Pastoral Community Development Project I, indicates that it resulted in some positive outcomes related to humane/social capital and empowerment and the construction of infrastructure. The Rural Financial Intermediation Programme I was initiated in 2003 and ended in 2010, and a Rural Financial Intermediation Programme II has been started which is building on the elements initiated under the first phase of the programme. According to a Project Completion Digest from 2009, pastoral communities supported by the project benefitted strongly from water supply projects which contributed to minimising the rate of mobility and in improving the livelihood of the intended beneficiaries. The project also contributed to the construction of schools in areas without any existing schools which reportedly helped tackling the issue of illiteracy and lack of basic education services. The PCD also refers to other areas of positive outcomes, including longer-term capacity and institutional development at the regional administrative level (Woreda level) in Ethiopia. However, in line with most of the reports on results achieved by IFAD-supported projects in Ethiopia, the PCD states that: “it is quite difficult to assess the project impact due to lack of quantitative as well qualitative information. PCDP impact has been undoubtedly strong on human, social capital and empowerment, due also to the extent of built infrastructures. However, with regard to the other impact domains, the analysis is strongly hindered by lack of data.” It is therefore difficult, on the basis of the PCD alone, to assess the extent to which the project has contributed sustainable development outcomes.

The Pastoral Community Development Project II (which started in 2008) builds on the PCDP I seems to be on track overall in terms of delivering outputs. A mid-term Project Performance Report from 2011 also reported some intermediate outcomes in relation to the Rural Livelihoods Sub-component of the project, but no outcomes on other areas. However, given the nature of data and the date of the report, it is difficult to provide conclusive results on this project.

Finally, data from the 2011 “Mid-Term Review of the Agricultural Marketing Improvement Program (AMIP)” suggests mixed performance of the project at the time of the review (the programme is still on-going). The principal objective of the project is to empower smallholder farmers to engage in and exploit emerging market opportunities. The project provides training to farmers in post-harvest management; improved access to and use of market information, grades and standards and support to organisation and enterprise management. At the time of the Mid-Term Review, the most successful component of the programme was capacity building at the federal level whereas the contribution to results at the regional level was more mixed. Results for the other sub-components (capacity building at regional level; incremental credit for adopting postharvest processing technology; warehouse receipt system; decentralisation of coffee liquoring centres; agricultural; and marketing information system) were poor or non-existing at the time of the Mid-Term Review. The poor performance of the project seems to be correlated with the low level of disbursement. Of the total loan allocated for the implementation of AMIP, only 12.5% was utilised by 2011. In addition, according to the review, a lack of integration and harmonisation of AMIP plans into regional (Woreda) level plans was an obstacle for ensuring sustained ownership in implementation and monitoring of the plans at this level.

However, according to an interview with IFAD staff members in Ethiopia, this project is now on track and is delivering according to plan.

GUATEMALA

The analysis of results of IFAD’s programme in Guatemala is based on the perceptions of surveyed respondents and a set of documents including an Interim Evaluation of the Rural Development Programme for Las Verapaces (PRODEVER, 2009), a Project Completion Report Digest for the PRODERQUI project (2009) and the Latin America and Caribbean region (LAC) portfolio performance report for 2011-2012.157

Finding 29: Survey respondents rated the contributions of IFAD-funded operations in Guatemala as adequate overall. The document review found evidence of contributions to poverty reduction and market access, but room for improvement in terms of targeting the landless poor and promoting gender equality and women’s empowerment.

Survey respondents rated IFAD performance in Guatemala as adequate overall, but a high percentage of the respondents (52%) answered “don’t know” to the question on whether IFAD has effectively contributed to increasing incomes and employment for the rural poor. Nearly three-quarters of survey respondents rated IFAD’s performance as adequate for value chain integration, market access and technical/ management training for the rural poor. 63% of respondents stated that IFAD’s performance was inadequate or weak in terms of incorporating the rural poor into decentralised development structures, and 61% found that IFAD’s performance in promoting the inclusion of poor rural women into the local economy was inadequate or weak.

IFAD has been active in supporting rural development in Guatemala since 1986. The Fund originally financed projects designed to support the peace building process following a 36-year conflict, through building an economic development platform for marginalised communities in rural areas. More recently, IFAD has focused on integrating rural businesses into country-level and international value chains.

Guatemala’s COSOP for 2008 to 2012 sets out three Strategic Objectives (SOs): 1) Promoting the market- driven development of agricultural and non-agricultural rural businesses and microenterprises; 2) Developing social and human capital; and 3) Enhancing pro-poor rural policy dialogue and sectoral planning.

Guatemala’s portfolio currently includes three projects.158 Two projects closed in the last three years: the National Rural Development Programme, Phase I: Western Region (completion: December 2012), and the Rural Development Programme for Las Verapaces, or PRODEVER (completion: September 2011).

The most significant area of investment in Guatemala’s portfolio has been focused on achieving the first COSOP strategic objective (promoting the market-driven development of agricultural and non-agricultural rural businesses and microenterprises). By June 2012, the FONAPAZ implementation agency reported that 1,170 rural families participated in microenterprises with income-generating activities (117% of target), 65 rural microenterprises were still operating after three years (260% of target), and 45 commercialisation establishments were active (150% of target).159

Country staff also noted that IFAD is working on improving local governance and promoting social participation for all, regardless of ethnicity or income. According to interviews with IFAD staff members, the Fund recently conducted a series of case studies about indigenous people issues, where PRODENORTE (continuity of PRODEVER) was selected. These studies showed seven cases of rural organisations led by indigenous women that were contributing to results at the impact level.

Overall, according to the documents assessed, IFAD-funded operations in Guatemala have shown progress towards meeting project objectives in spite of initial delays. The Rural Development Programme in Las Verapaces (PRODEVER) demonstrates moderately satisfactory progress in meeting income and employ- ment objectives (B1), considering that participating farming households increased their incomes from 8% to 146%. However, two IFAD projects performed moderately unsatisfactorily for gender equality and women’s empowerment, though a third project was rated highly satisfactory (B6).

MOZAMBIQUE

The assessment of results achieved in Mozambique is based on perception-based ratings from survey respondents a Country Programme Evaluation (CPE) from 2010, an Aide Memoire of the Artisanal Fisheries Promotion Project (ProPESCA) from 2013 and a Supervision Report of the Rural Finance Support Programme (PAFIR) from 2013. However, as in most other cases, it is challenging to assess IFAD’s contributions to outcomes as the documents mainly include information about the implementation process and planned activities. The extent to which they can be used for an analysis of outcome-level results achieved is limited.

Finding 30: Survey respondents rated IFAD’s performance as strong on all results areas apart from the promotion of gender equality, which was rated as adequate. The document review suggests that the performance of IFAD-supported projects has been mixed.

MOPAN donors in-country and direct partners were asked whether IFAD in Mozambique has effectively contributed to each of the strategic objectives and areas of MOPAN interest. Survey respondents were very positive about IFAD’s contributions overall, but 36% to 39% answered “don’t know” to the questions about IFAD’s contributions to increasing returns from fish sales for artisanal fishers and small market operators (indicators B1 and B2).

IFAD has been present in Mozambique since 1983 and has funded 12 projects and programmes worth more than USD 200 million. IFAD currently provides funding for five projects in Mozambique. The current COSOP (approved in 2011) supports three strategic objectives: i) smallholders and artisanal fishers enjoy greater access to production factors, technologies and resources; ii) the access and participation of smallholders and artisanal fishers to markets that can bring them equitable shares of profit are increased; iii) the availability of and access to appropriate and sustainable financial services in rural areas are increased. These objectives are in line with IFAD’s strategic focus in the country since the mid-1990s. The current COSOP is thus based on previous country strategies, but also draws on lessons learned from a CPE conducted in 2010. According to the COSOP, the CPE confirmed the previous country strategy’s relevance and recommended that the thrust of the strategic focus should be continued in the 2011 COSOP.

IFAD’s country programme in Mozambique was evaluated in 2010. The evaluation included an assessment of seven IFAD-funded projects of which the oldest was approved by the Executive Board in 1993. Of the seven projects covered, four were under implementation during the evaluation and two of these are still active: the Rural Finance Support Programme (PAFIR) and the Agricultural Support Programme (PRONEA).

The country programme evaluation (CPE) had the following two objectives: (i) assess the performance and impact of IFAD operations (including non-lending activities) in Mozambique; and (ii) produce building blocks for the preparation of the new country strategic opportunities programme (COSOP) to be prepared by IFAD’s Eastern and Southern Africa Regional Division (ESA) and the Government of Mozambique following the completion of the CPE (which is the current COSOP).

Overall the evaluation found that the projects had been“moderately satisfactory”in achieving their objectives. More specifically, it found evidence that smallholders to an increasing extent were participating in associations and groups and that their participation in markets had improved in areas supported by IFAD. It also found that IFAD had contributed to an expansion of financial services into rural areas. However, actual evidence (i.e. outcome-level data) of the extent to which projects supported by IFAD had contributed to increasing the income of rural poor people was scarce. According to a recent Implementation Support Mission Aide Memoire of the on-going Rural Finance Support Programme (PAFIR), the activities to support community-based financial institutions have progressed well and are reaching most of the targets agreed for the programme period.

In terms of the programme’s impact on rural poverty, the CPE found evidence of increased levels of income and assets in households that had benefitted from IFAD-supported initiatives related to savings and credit groups, access roads, and improved fishing, processing and marketing methods. According to the evaluation, the most significant direct contributions to raising household income and assets had been obtained from the support to artisanal fisheries, improving farmers’market linkages, establishment of grass- roots financial institutions, small and medium-sized enterprises (SMEs) and business development services.

The CPE noted mixed results in the area of social capital and empowerment. While some positive impacts were noted through the support to various groups and courses (including savings and credit groups, users’ associations for management of water points and social infrastructure, and literacy courses) many groups needed further assistance in order to deliver results. Issues included: (i) difficulties in having the associations legally recognised; (ii) problems in or lack of communication between project authorities/service providers and beneficiary groups; and (iii) some problems in terms of community contributions to social investments.

The impacts related to agricultural productivity and food security also received mixed ratings. Significant impact was for example expected from Niassa Agricultural Development Project (NADP effective from 1994 to 2004), but in spite of large investments in agricultural research and extension, yields did not increase. However, other projects such as the Nampula Artisanal Fisheries Project (NAFP effective from 1994 to 2001) and Sofala Bank Artisanal Fisheries Project (SBAFP effective from 2002 to 2011) had a positive impact on food security particularly through their support to Accumulating Savings and Credit Association (ASCAs).

The effect of IFAD-supported projects related to the improvement of natural resource management and protecting the environment could also have been better according to the CPE. For example, one objective of NADP was to replace slash-and-burn agriculture with more intensive sedentary cultivation, but the cultivation system did not change. Also, reforestation support produced limited results. However, some positive trends were recorded in relation to the sustainability of artisanal fisheries projects although it was noted that many fishers continue using environmentally unfriendly and unsustainable practices.

One area of success seems to be IFAD’s contributions to improving institutions and policies. In agriculture, for example, NADP introduced the concept of district development funds, to be planned and managed jointly with communities, at a time and in a context where public resources were entirely planned and managed by central and provincial governments. In artisanal fisheries, support has facilitated the introduction of the ASCA model, joint management of fisheries resources and the three-mile zone reserved for artisanal fisheries. At the level of community-based organisations, the PAMA Support Project facilitated the important 2006 revision of the law on associations that simplified registration procedures, reducing the costs and time involved in the registration process, directly resulting in a greater number of registered organisations.

Overall, the rural poverty impact of the project portfolio was assessed as moderately satisfactory. Social capital and empowerment, and natural resources and the environment are singled out as the two impact domains that require improvement as they were rated unsatisfactory overall.

4.4 RELEVANCE OF IFAD’S OBJECTIVES AND PROGRAMME OF WORK TO STAKEHOLDERS

For this KPI, MOPAN assessed relevance as a measure of the extent to which stakeholders perceive that the multilateral organisation supports country priorities and meets the changing needs of its partners and the target population. The assessment is based exclusively on perception data obtained from surveyed respondent groups (i.e. MOPAN donors in-country and direct partners) in the six assessed countries.

Overall assessment

Respondents in four of the six countries rated IFAD as strong in responding to countries’ development priorities. Respondents provided mixed ratings for IFAD’s capacity to provide innovative solutions to development challenges: respondents in Mozambique and Viet Nam rated IFAD as strong, respondents in Indonesia rated it inadequate, and respondents in the other three countries rated it adequate. With respect to IFAD’s adaptability to changing conditions at the country level, perceptions were also mixed: while respondents in Mozambique and Viet Nam rated IFAD as strong, their counterparts in Guatemala, Ethiopia and Pakistan rated it adequate and respondents in Indonesia rated it inadequate.

Figure 4.9 shows the overall assessment rating and the means scores on the three survey questions on which the assessment is based.

Figure 4.9 | KPI C: Relevance of objectives and programme of work to stakeholders, overall rating and survey mean scores by country

*Detailed scores are shown in Volume II, Appendix VI.

5. Conclusions

These conclusions step away from the specific ratings of the MOPAN assessment and look at the major messages that can contribute to dialogue between individual MOPAN members, IFAD and its direct partners. It draws on the survey findings and principal observations of the assessment of IFAD’s practices and systems (Key Performance Indicators 1-21) and the assessment of development results component (Key Performance Indicators A-C).

Conclusions on IFAD’s organisational effectiveness

IFAD remains fully committed to a results agenda and strategic management is one of its strengths. Its mandate is clear and well reflected in its 2011-2015 Strategic Framework.

IFAD continues to be acknowledged for its clear and unique mandate and its commitment to instilling a results- oriented culture throughout the organisation. Indeed, when asked what they considered to be IFAD’s greatest areas of strength, some 40% of survey respondents mentioned its specialised mandate, which focuses almost exclusively on poverty alleviation for the rural poor. Respondents commented that no other international financial institution is mandated to focus on the rural poor, or on agricultural and rural development for smallholders in particular, which gives IFAD a clear niche in the development landscape.

IFAD confirmed its commitment to the results agenda during the negotiations for its Ninth Replenishment Period (2013 - 2015). As part of the negotiations, IFAD committed to moving 80 million poor people out of poverty by 2015, a major promise that will require new tools and practices to monitor progress. To this effect, IFAD has revised its Results Measurement Framework to integrate new indicators that will track its progress in areas such as contribution to poverty reduction, household food security and nutrition, and climate change. IFAD also plans to conduct 30 impact studies during the 2013-2015 replenishment period.

IFAD is noted for having increasingly mainstreamed cross-cutting priorities such as gender, environment, and food security and nutrition into its operations.

IFAD’s country strategies systematically include objectives and results statements related to relevant cross- cutting priorities (gender, environment, and food security and nutrition). This suggests that IFAD’s RB- COSOP Guidelines (2010) are being followed and that cross-cutting priorities are being effectively mainstreamed. IFAD has recently increased its efforts to mainstream environmental and gender equality considerations into its strategies and operations; for example, in 2012 it launched an innovative Adaptation for Smallholder Agriculture Programme (ASAP) that provides financing to scale up and integrate climate change adaptation across IFAD’s new investments. IFAD also approved a new Policy on Gender Equality in 2012 that provides a clear division of responsibility for gender mainstreaming across IFAD’s systems and operations and there is evidence that IFAD is following this policy. The extent to which these policies have resulted in positive development outcomes is not yet possible to assess.

While primary decision-making authority remains centralised, IFAD has increased its country presence significantly in recent years and has also made changes to its policies regarding the delegation of decision making. Some survey respondents feel there is still room for improvement in assuring adequate country presence.

IFAD recently adopted a Country Presence Policy and Strategy in which it formally approved the establishment of up to 40 country offices, of which 38 have already been established. IFAD has made efforts to decentralise some decision-making authority to the country level, but this is still somewhat limited (as it is in most IFIs) and further work is needed to flesh out the more substantive delegation of roles and responsibilities in the organisation.160 The 2013 corporate-level evaluation of IFAD’s institutional efficiency noted that the limited substantive delegation of authority may conflict with the changing business needs associated with a decentralised institution.

The perspectives of in-country donors in the six focus countries suggest that IFAD’s country presence continues to be an important factor for the organisation’s performance. When asked whether IFAD’s country presence is sufficient for improving country- level performance, the majority of in-country donors rated IFAD as inadequate or lower while the majority of IFAD’s direct partners rated the organisation as adequate or higher.

IFAD has sound policies and practices in place for financial accountability, but its corporate procurement procedures could be more efficient.

The document review found that IFAD is performing strongly in relation to its internal and external audit processes and its financial accountability in general. IFAD is also noted for its strong Anti-Corruption Policy (2005) and its processes and procedures for taking immediate action when irregularities are noted. There is room for improvement in IFAD’s corporate procurement processes, which are still perceived to be time- consuming in spite of recent improvements.

IFAD has significantly improved its HR management policies and practices in recent years and remains committed to ambitious reforms in this area.

IFAD has undertaken a reform of its human resources (HR) practices and is currently working to improve several elements of its HR management system including its staff reward system, recruitment process, promotion and dismissal processes, staff rotation system, and support to staff located in-country. It has implemented a number of initiatives to ensure appropriate levels of compensation and a performance- based staff reward system is in place (e.g. a pay-for-performance model).

The document review noted that IFAD has a functional performance evaluation system (PES) that is used at all levels of the organisation. However, some sources suggest that this system is not yet being fully exploited, that managers are too risk averse in their performance assessments, and that, as a result, staff members are rarely recognised for either outstanding or substandard performance.

IFAD has begun to introduce results-based budgeting at the institutional level but this is still a work in progress.

In 2010, the Eighth Replenishment Report urged the Fund to deepen the implementation of results-based budgeting (RBB). IFAD’s 2013 budget is presented in a results-oriented way, and, according to IFAD’s Update on the Reform Agenda, management, in consultation with the Board, is committed to review and refine its RBB practices in the coming years. The budget is allocated to four business areas (clusters) to attain a set of highly aggregated operational and institutional outcomes and corporate management results. While these results are all aligned with IFAD’s mandate and strategic priorities, the cluster outcomes are aggregated at a high level and there is limited basis for decision making on budget allocations based on expected results. IFAD has not yet introduced a system of time recording and cost accounting, which is essential to reliably capture the full costs of all major business activities performed by staff.

IFAD is recognised for the independence of its Office of Evaluation and for following up on and using performance information from evaluations.

The Independent Office of Evaluation (IOE) is structurally independent from IFAD management and the IOE Director reports directly to the Executive Board. This was recognised as a clear strength by both survey respondents and the document review. There is ample evidence that evaluation recommendations reported to the Board are followed up by the responsible units. The implementation of accepted recommendations is tracked through the President’s Report on the Implementation Status of Evaluation Recommendations and Management Actions (PRISMA), which is presented to the Executive Board on an annual basis. In addition, several corporate-level evaluations over the last few years have resulted in major changes to IFAD’s policies and practices.

IFAD stakeholders surveyed in both 2010 and in 2013 provided mostly adequate ratings of the Fund’s performance, with a slightly improving trend over time.

Survey ratings in 2013 were slightly more positive than in 2010 and suggest that IFAD has made progress in three areas: promoting a results-oriented culture, financial management practices, and supporting funding proposals developed by direct partners. These areas were rated adequate in 2010 and strong in 2013. However, some caution is required in making comparisons between 2010 and 2013 as there were some changes in respondents, countries, and wording of questions.

As in 2010, survey respondents in 2013 generally perceived IFAD’s mandate and its corporate focus on results to be among its areas of strength. In both years they also perceived the independence of the Office of Evaluation to be one of IFAD’s strengths. In spite of an increased country presence since 2010, survey responses from in-country donors in 2013 indicate that the level of staff presence at the country level could still be increased.

Conclusions on evidence of IFAD’s development results and relevance to stakeholders

IFAD’s current institutional reporting practices do not allow for a clear assessment of its contributions to actual development outcomes and impact. However, changes in the IFAD9 Replenishment Period are likely to improve these practices.

MOPAN donors at headquarters held positive views about IFAD’s contributions to results and provided ratings of adequate or strong on all areas assessed.

A review of IFAD’s performance reports indicates that the organisation is delivering outputs in its thematic priority areas and reporting on these, but its reporting on higher level development results does not include sufficient data on its contributions to outcomes and impact. In addition, IFAD has not articulated a theory of change that explains the logical or causal links between its expected results at different levels (i.e. outputs and outcomes). In the absence of a theory of change and data on outcomes and impact, it is difficult to appreciate its progress towards achieving the objectives outlined in IFAD’s Strategic Framework 2011-2015. This being said, for the IFAD9 period, the Fund has committed to the ambitious target of moving 80 million people out of poverty. For the purpose of monitoring its progress in this context, it has updated the Results Measurement Framework substantially by introducing a set of indicators at the impact level measuring its contributions to poverty reduction and food security, among others. In addition, IFAD has committed to conducting 30 impact assessments during the 2013-2015 replenishment period.

As IFAD’s planned reporting practices at the country level are not yet fully implemented, it is difficult to fully appreciate the organisation’s contributions to country-level goals and priorities. The document review and survey responses suggest that the documented results of IFAD’s operations in the six countries reviewed have been mixed.

Surveyed stakeholders hold mixed views about IFAD‘s progress towards its strategic objectives at the country level. Among the six countries sampled, surveyed stakeholders rated IFAD most positively in Viet Nam, where it was rated as strong in all results areas, and lowest in Indonesia, where it was rated inadequate in three results areas161 and adequate in the other three.162

IFAD has significantly improved its focus on results at the project and country levels since the introduction of results-based COSOPs at the end of 2006. Its results statements are clearly and explicitly aligned with national goals and priorities, which was seen as a strength by both survey respondents and the document review. IFAD has provisions for annual reviews, Mid-Term Reviews, and completion reviews for country strategies (in addition to other reporting practices). However, annual reviews are not yet being performed systematically for all COSOPs; Mid-Term Reviews are often done late and with varying quality; and COSOP completion reviews have not yet been systematically introduced across all countries. These factors, combined with a general lack of systematic and credible data on project outputs and outcomes, make it difficult, if not impossible, to obtain a complete picture of the nature, magnitude, and relative importance of IFAD’s operations at the country level and the extent to which they support the MDG agenda.

Surveyed stakeholders in the six countries sampled consider IFAD to be responding well to countries’ key development priorities and, to a lesser degree, adjusting its strategies to meet the changing needs and priorities in these countries. IFAD received mixed ratings for its provision of innovative solutions to development challenges.

Survey respondents in Mozambique and Viet Nam gave IFAD strong ratings across all three indicators related to relevance. IFAD received the lowest overall country ratings in Indonesia and Guatemala. The pattern of responses was the same in all countries – that is, the highest ratings in each country were for IFAD’s responsiveness to country priorities while it received more mixed ratings for the level of innovation brought to bear on development challenges.

List of Figures

Figure 1.1 Number of survey respondents and total population for IFAD by country and respondent group (n = 193)
Figure 1.2 IFAD - Distribution of responses on all questions related to micro-indicators (n = 193)
Figure 3.1 Overall ratings on key performance indicators of IFAD’s organisational effectiveness (survey mean scores and document review ratings)
Figure 3.2 Overall ratings of IFAD organisational effectiveness by respondent group (n= 197)
Figure 3.3 Performance area I: Strategic management (survey and document review ratings)
Figure 3.4 Performance area I: Strategic management, mean scores by respondent group
Figure 3.5 KPI 1: Providing direction for results, ratings of micro-indicators
Figure 3.6 KPI 2: Corporate strategy and mandate, ratings of micro-indicators
Figure 3.7 KPI 3: Corporate focus on results, ratings of micro-indicators
Figure 3.8 KPI 4: Focus on cross-cutting priorities, ratings of micro-indicators
Figure 3.9 KPI 5: Country focus on results, ratings of micro-indicators
Figure 3.10 Performance area II: Operational management, survey and document review ratings
Figure 3.11 Performance area II: Operational management, mean scores by respondent group
Figure 3.12 KPI 6: Resource allocation decisions, ratings of micro-indicators
Figure 3.13 KPI 7: Results-based budgeting, ratings of micro-indicators
Figure 3.14 KPI 8: Financial accountability, ratings of micro-indicators
Figure 3.15 KPI 9: Using performance information, ratings of micro-indicators
Figure 3.16 KPI 10: Managing human resources, ratings of micro-indicators
Figure 3.17 KPI 11: Performance-oriented programming, ratings of micro-indicators
Figure 3.18 KPI 12: Delegating authority, ratings of micro-indicators
Figure 3.19 Performance area III: Relationship management, survey and document review ratings
Figure 3.20 Performance area III: Relationship management, mean scores by respondent group
Figure 3.21 KPI 13: Supporting national plans, ratings of micro-indicators
Figure 3.22 KPI 14: Adjusting procedures, ratings of micro-indicators
Figure 3.23 KPI 15: Using country systems, ratings of micro-indicators
Figure 3.24 KPI 16: Contributing to policy dialogue, ratings of micro-indicators
Figure 3.25 KPI 17: Harmonising procedures, ratings of micro-indicators.
Figure 3.26 Performance area IV: Knowledge management, survey and document review ratings
Figure 3.27 Performance area IV: Knowledge management, mean scores by respondent group
Figure 3.28 KPI 18: Evaluating results, ratings of micro-indicators
Figure 3.29 KPI 19: Presenting performance information, ratings of micro-indicators
Figure 3.30 KPI 20: Disseminating lessons learned, ratings of micro-indicators
Figure 3.31 KPI 21: Availability of documents, ratings of micro-indicators
Figure 4.1 Evidence of Development results, and Relevance to Stakeholders– overall ratings
Figure 4.2 KPI A: Evidence of the extent of progress toward organisation-wide outcomes, overall rating
Figure 4.3 IFAD’s measurement and reporting on it areas of thematic focus
Figure 4.4 Extent of progress towards IFAD’s areas of thematic focus
Figure 4.5 Extent of progress towards themes identified by MOPAN, survey ratings
Figure 4.6 Projects rated by IFAD as ‘moderately satisfactory’ or above at project completion, 2008-2012
Figure 4.7 KPI B: Evidence of the extent of contribution to country-level goals and priorities, rating
Figure 4.8 IFAD’s measurement and reporting on country-level results
Figure 4.9 KPI C: Relevance of objectives and programme of work to stakeholders, overall rating and survey mean scores by country

---

[1] In this context it should be noted that a Revised Manual for Delegation of Authority relating to the delegation at headquarters level is due to be amended and approved by December 2013.

[2] These areas were: B1 Sustaining the growth of economic activities and reducing the incidence of poverty; B2 Improvement of natural resource management; B4 Building capacity of rural people to engage in local policy and programming processes.

[3] Increasing household incomes for poor households involved in fisheries and marine activities; B5 Promotion of gender equality and women's empowerment; B6: Improvement of household food security and nutrition.

[4] IFAD, Report on IFAD’s Development Effectiveness, December 2012, p.5 par. 16

[5] In total 56 key commitments distributed as follows: Operational Effectiveness: 32 key commitments; Institutional effectiveness and efficiency: 10 key commitments; Financial management: 7 key commitments; Results management: 7 key commitments.

[6] See Volume II, Appendix I, section 3.4 for an explanation of the weighting formula and scheme used in this case.

[7] Despite concerted efforts and the extension of the survey deadline, responses from donors in-country did not meet the MOPAN target of a 50% response rate.

[8] While most KPIs and micro-indicators were considered in the document review, not all were rated. See section 2.3.

[9] Respondents who wrote “no comment” or something similar were filtered out of the analysis.

[10] IFAD, Results Measurement Framework 2013-2015, October 2011.

[11]IFAD, Results and Impact Management System: First and Second Level Results Handbook. February 2011. http://www.ifad.org/operations/rims/handbook/e.pdf

[12] IFAD, Practical Guidance for Impact Surveys, January 2005. http://www.ifad.org/operations/rims/guide/e/part1_e.pdf

[13] IFAD, Results Measurement Framework 2013-2015, introduction pp1-2, paragraph 5. http://www.ifad.org/gbdocs/repl/9/iii/e/REPL-IX-3-R-4.pdf

[14] IFAD Strategic Framework 2011-2015, approved in May 2011. http://www.ifad.org/sf/strategic_e.pdf

[15] IFAD, Agreement Establishing the International Fund for Agricultural Development, Adopted on 13 June 1976 and last amended in February 2006. http://www.ifad.org/pub/basic/agree/e/!01agree.pdf

[16] The OECD-DAC defines “outcome” as the likely or achieved short-term and medium-term effects of an intervention’s outputs. For instance, the output “number of km of roads built” would normally correspond to the outcome “number of people with access to markets”. However, IFAD does not report on this type of outcome. Instead it considers that “outcomes” are: efficiency, sustainability, relevance, innovation, rural poverty impact, etc.

[17] MOPAN has used CREAM and SMART criteria to assess the quality of indicators and results statements. CREAM criteria (clear, relevant, economic, adequate, monitorable) are usually used to evaluate indicators. SMART criteria (specific, measurable, achievable, relevant, time bound) are widely used when developing indicators.

[18] OECD Definitions: “In performance assessment in government, outputs are defined as the goods or services produced by government agencies (e.g. teaching hours delivered, welfare benefits assessed and paid); outcomes are defined as the impacts on social, economic, or other indicators arising from the delivery of outputs (e.g. student learning, social equity)”. Source: http://stats.oecd.org/glossary/detail.asp?ID=7311

[19] IFAD does not report on good governance or human rights-based approaches in its operations, because these are not identified as areas of focus or priority areas for IFAD, whereas IFAD has given emphasis to gender equality, environmental practices and food security in its key strategic documents.

[20] IFAD, Policy on Gender Equality and Women’s Empowerment, http://www.ifad.org/gender/policy/gender_e.pdf

[21] IFAD (December 2010), Corporate-level Evaluation on Gender, Report No. 2324, Foreword.

[22] See IFAD’s Climate Change Strategy, 2010 p. 30.

[23] According to the Climate Change Strategy (p. 26), the Environment and Climate Division (ECD) is responsible for pursuing a common agenda of climate and environment integration throughout IFAD. It will also bring together staff working on climate and the environment to enable (i) continued development of the Global Environment Fund (GEF) pipeline; (ii) more support to the regional divisions on COSOPs and non-GEF programme development; (iii) innovation and knowledge management on the environment and climate change; and (iv) managing external partnerships and global initiatives on climate and the environment.

[24] IFAD (November 2012), Annual Report on Quality Assurance in IFAD’s Projects and Programmes (EC 2012/74/W.P.4/Add.2) https://webapps.ifad.org/members/ec/74/docs/EC-2012-74-W-P-4-Add-2.pdf

[25] Source: please consult IFAD’s webpage on Governance and Corruption. http://www.ifad.org/operations/finance/governance.htm

[26] A human rights-based approach is a conceptual framework for the process of human development that is normatively based on international human rights standards and operationally directed to promoting and protecting human rights. It seeks to analyse inequalities which lie at the heart of development problems and redress discriminatory practices and unjust distributions of power that impede development progress.

[27] IFAD, Targeting Policy, p.6: http://www.ifad.org/pub/policy/target/targeting_e.pdf

[28] The hungry season means the number of months a household does not have enough food because their own stores are depleted and they do not have money to buy food. Source: http://www.ifad.org/operations/rims/question/e.pdf

[29] IFAD, Guidelines for preparation and implementation of a Results-based Country Strategic Opportunities Programme (RB-COSOP). http://www.ifad.org/operations/policy/cosop/guidelines/

[30] The assessment looked at COSOPs for Ethiopia (produced in 2008), Guatemala (2008), Indonesia (2008), Mozambique (2011), Pakistan (2009) and Viet Nam (2012).

[31] This MI did not consider whether results related to human-rights based approaches or good governance are integrated in the COSOPs because IFAD does not consider these to be core cross-cutting issues (and therefore, it would be illogical to consider whether these issues are mentioned in country strategies).

32 See IFAD (September 2003), The Structure and Operation of a Performance-Based Allocation System for IFAD, (EB 2003/79/R.2/Rev.1) as well as IFAD’s webpage on the PBAS for more details: http://www.ifad.org/operations/pbas/

33 IFAD (December 2012), Report on IFAD’s Development Effectiveness, p.14, Table 7 ‘Project performance during implementation’. (EB 2012/107/R.8/Rev.1)

34 OECD-DAC, Aid Effectiveness 2005-10: Progress in Implementing the Paris Declaration, http://www.oecd.org/dac/effectiveness/48742718.pdf

35 IFAD (December 2012), Report on IFAD’s Development Effectiveness, p.iv , paragraph 5 (EB 2012/107/R.8/Rev.1)

36 IFAD’s 2013 results-based programme of work and regular and capital budgets, the IOE results-based work programme and budget for 2013 and indicative plan for 2014-2015, and the HIPC and PBAS progress reports

37 The four clusters (expected operational and institutional objectives) are: (1) Effective national policy, harmonisation, programming, institutional and investment frameworks for rural poverty reduction; (2) Supportive global resource mobilisation and policy framework for rural poverty reduction; (3) An effective and efficient management and institutional service platform at headquarters and in-country for achievement of operational results; (4) Effective and efficient functioning of IFAD’s governing bodies

38 A network composed of 17 members, including all of the major IFIs.

39 As indicated in IFAD’s Policy on the Disclosure of Documents (2010), Audit Committee documents are not disclosed. The assessment team has therefore not had access to audit reports at the project level.

40 IFAD, Agreement Establishing the International Fund for Agricultural Development, amended February 2006, p.12, sect.1(c). http://www.ifad.org/pub/basic/agree/e/!01agree.pdf

41 IFAD (December 2011), Guidelines on Project Audits, Annex III, p.15, paragraph 3. http://www.ifad.org/pub/basic/audit/borrower_e.pdf

42 IFAD (December 2005), Policy on Preventing Fraud and Corruption in its Activities and Operations, (EB 2005/85/R.5) http://www.ifad.org/gbdocs/eb/86/e/eb-2005-86-inf-8.pdf

43 IFAD (December 2012), Report on IFAD’s Development Effectiveness, p.18, Table 10 ‘RMF level 5 – IFAD’s institutional management and efficiency’, (EB/2012/107/R.8/Rev.1)

44 IFAD (2011), Annual Report on Investigation and Anticorruption Activities http://www.ifad.org/governance/anticorruption/report/2012/e.pdf

45 IFAD (2012), Annual Report on Investigation and Anticorruption Activities http://www.ifad.org/governance/anticorruption/report/2013/e.pdf

46 IFAD (2012), Annual Report on Investigation and Anticorruption Activities http://www.ifad.org/governance/anticorruption/report/2013/e.pdf

47 IFAD (2011), Operational Procedures for Project and Programme Audits http://www.ifad.org/pub/basic/audit/operational_e.pdf

48 The external quality assessment is not publicly disclosed.

49 http://www.ifad.org/governance/procurement/procure_11.pdf

50 http://www.ifad.org/governance/procurement/procure_21.pdf

51 IFAD (April 2013), Corporate-level evaluation on IFAD’s institutional efficiency and efficiency of IFAD-funded operations, p. 14, paragraph 71 (EB 2013/108/R.3/Rev.1) https://webapps.ifad.org/members/eb/108/docs/EB-2013-108-R-3-Rev-1.pdf

52 This information is based on personal communication with IFAD staff members. Since the second version of the Procurement Manual had not yet been approved at the time of the MOPAN assessment, it was not included in the document review.

53 IFAD (August 2008), Policy on Enterprise Risk Management (EB 2008/94/R.4) http://www.ifad.org/gbdocs/eb/94/e/EB-2008-94-R-4.pdf

54 Committee of Sponsoring Organizations of the Treadway Commission (September 2004), Integrated Framework for Enterprise Risk Management

55 IFAD (September 2012), Policy on Gender Equality and Women’s Empowerment http://www.ifad.org/gender/policy/gender_e.pdf

56 IFAD (September 2012), Country Presence Policy and Strategy (EB 2011/102/R.10/Rev.2) http://www.ifad.org/gbdocs/eb/102/e/eb-2011-102-R-10-Rev-2.pdf

57 IFAD (April 2013), Corporate-level evaluation on IFAD’s institutional efficiency and efficiency of IFAD-funded operations, p. 15, paragraph 81 (EB 2013/108/R.3/Rev.1) https://webapps.ifad.org/members/eb/108/docs/EB-2013-108-R-3-Rev-1.pdf

58 2011 COMPAS Report: Multilateral Development Banks’ Common Performance Assessment System.

59 IFAD (December 2012), Report on IFAD’s Development Effectiveness, p.14, table 7.

60 IFAD, Evaluation Policy, http://www.ifad.org/pub/policy/oe.pdf

61 IFAD (March 2010), Peer Review of IFAD’s Evaluation Function (EB 2010/99/R.6) http://www.ifad.org/gbdocs/eb/99/e/EB-2010-99-R-6.pdf

62 IFAD (April 2013), Corporate-level evaluation on IFAD’s institutional efficiency and efficiency of IFAD-funded operations, p. 17, paragraphs 92-93.

63 IFAD (March 2009), Environmental management and sustainable development: Environmental and social assessment procedures (EB 2009/96/R.7)

64 Agreement Establishing the International Fund for Agricultural Development, Rome, 1976, http://www.ifad.org/pub/basic/agree/e/!01agree.pdf

65 For example the Viet Nam country office employs three staff members, including the Country Programme Manager. The Pakistan country office employs four staff members, but the Country Programme Manager is based in Rome.

66 IFAD (April 2013), Corporate-level evaluation on IFAD’s institutional efficiency and efficiency of IFAD-funded operations, p.19, paragraph 102. The IFAD Manual and Framework for Delegation of Authority seems to be an internal document, as it was not available on IFAD’s website at the time of writing.

67 IFAD (April 2013), Corporate-level evaluation on IFAD’s institutional efficiency and efficiency of IFAD-funded operations, p.93, paragraph 156.

68 Following discussion with MOPAN and IFAD on early drafts of the report, this MI was moved from KPI 10 on human resources to KPI 12 on delegating authority. This was mainly due to the fact that the wording of the survey question did not refer to “staff”, but to the broader notion of country presence, which is more in line with the focus of KPI 12.

69 World Bank (10 May 2010), Knack, Stephen, F. Halsey Rogers, and Nicholas Eubank, Aid Quality and Donor Rankings

70 MOPAN Common Approach, International Fund for Agricultural Development (IFAD) 2010, January 2011. http://www.mopanonline.org/upload/documents/IFAD_Final-Vol-I_January_17_Issued1.pdf

71 IFAD (January 2013), Review of the Lending Policies and Criteria (GC 36/L.9) https://webapps.ifad.org/members/gc/36/docs/GC-36-L-9.pdf

72 IFAD (May 2011) Medium-Term Plan 2011-2013, p.14, paragraph 42 (EB 2011/102/R.32). http://www.ifad.org/gbdocs/eb/102/e/EB-2011-102-R-32.pdf

73 EC 2013/76/W.P.6/Rev.1 Corporate-level evaluation on IFAD’s replenishments: Approach Paper (April 2013), (paragraph 33, p.6 )

74 This indicator was intended to be assessed by the document review using indicator 3 of the Paris Declaration: Aid flows are aligned on national priorities – Percent of aid flows to the government sector that is reported on partners’ national budgets. However, given the lack of alternative sources and the nature of IFAD’s operations, both the indicator and the data were considered insufficient to provide an assessment of the broader principle of use of country systems.

75 IFAD’s Project Procurement Guidelines (2010) refer to the Agreement Establishing IFAD, which states that: “Procurement of goods, works and services financed by [IFAD] shall be carried out in accordance with the provisions of the Borrower/Recipient’s procurement regulations, to the extent such are consistent with the IFAD Procurement Guidelines. Each Procurement Plan shall identify procedures which must be implemented by the Borrower/Recipient in order to ensure consistency with the IFAD Procurement Guidelines.” (Procurement Guidelines, p.3 par. 2)

76 This indicator was intended to be assessed by the document review using indicator 5b of the Paris Declaration: Use of country procurement systems. MI 15.2 was intended to be assessed by the document review, solely using indicator 5a: Use of country public financial management systems. However, in this assessment, the intention was slightly altered and survey respondents were asked a question on the use of both public financial systems and procurement systems. For clarity’s sake, Paris Declaration survey data on indicator 5b were reported at MI 15.2 rather than MI 15.3.

77 MOPAN and IFAD agreed that the micro-indicator 17.3 on Programme-Based Approaches (based on PD Indicator 9: Use of common procedures or arrangements – Percent of aid provided as programme-based approaches) was not applicable to IFAD in the context of this assessment given its Policy on Sector Wide Approaches in Agriculture and Rural Development (2005) and the Executive Board Directive on no general or sector budget support to its Member States (EB/84, 2005).

78 IFAD (May 2011), Evaluation Policy. http://www.ifad.org/pub/policy/oe.pdf

79 Peer Review of IFAD’s Office of Evaluation and Evaluation Function, March 2010, p.v. (EC 2010/62/W.P.2)

80 2011 COMPAS Report: Multilateral Development Banks’ Common Performance Assessment System, http://www.mfdr.org/COMPAS/documents/2011_COMPAS-Report.pdf

81 IFAD, “IOE’s 2013 programme of work and budget” from: IFAD’s 2013 results-based programme of work and regular and capital budgets, the IOE results-based work programme and budget for 2013 and indicative plan for 2014-2015, and the HIPC and PBAS progress reports. February 2013, (GC 36/L.8/Rev.1) https://webapps.ifad.org/members/gc/36/docs/GC-36-L-8-Rev-1.pdf

82 OECD, Sourcebook: Emerging Good Practices in Managing for Development Results (2008): http://www.mfdr.org/Sourcebook/3rdEdition/SourceBook3FINAL.pdf

83 IFAD (2013),Corporate-level evaluation of IFAD’s institutional efficiency and efficiency of IFAD-funded operations, p.76, par. 111. (EB 2013/108/R.3/Rev.1)

84 IFAD, RIDE 2012, section IV, p.6 par. 19

85 IFAD (December 2012), Methodology for Impact Assessments for IFAD9 (EB 2012/107/INF.7) https://webapps.ifad.org/members/eb/107/docs/EB-2012-107-INF-7.pdf

86 For more information on IFAD’s outcomes and outputs, please consult the Report on IFAD’s Development Effectiveness (2012), p.7-8, tables 3-4.

87 IFAD (August 2009), Results Measurement Framework for the Eighth Replenishment Period (2010-2012), (EB 2009/97/R.2). http://www.ifad.org/gbdocs/eb/97/e/EB-2009-97-R-2.pdf (IFAD reports against the RMF 2010-2012 in the RIDE 2010, 2011 and 2012. During the Ninth Replenishment Period (2013 to 2015), IFAD will report against the RMF 2013-2015). )

88 See, for example, the Annual Review of Portfolio Performance 2011-2012: Level 2: Country programme and project outcomes, Section B: Detailed review.

89 Corporate-level evaluation of IFAD’s institutional efficiency (2013) notes that “Many indicators and different layers of the system are not fully aligned, which makes aggregation of results difficult and casts doubts about the reliability of reporting on performance in some areas.” (p.76, par.111)

90 IFAD, Action Plan to strengthen the self-evaluation process, Appendix to the Results Measurement Framework 2013-2015. http://www.ifad.org/gbdocs/repl/9/iii/e/REPL-IX-3-R-4.pdf

91 IFAD Office of Evaluation (July 2007), Corporate-level Evaluation of IFAD’s Field Presence Programme. http://www.ifad.org/evaluation/public_html/eksyst/doc/corporate/field_2007.pdf

92 IFAD Office of Evaluation(September 2005), An Independent External Evaluation of the International Fund for Agricultural Development, page6.

93 IFAD (April 2013), Corporate-level evaluation on IFAD’s institutional efficiency and efficiency of IFAD-funded operations, p. 52, paragraph 47

94 IFAD Knowledge Management Strategy, April 2007, (EB 2007/90/R.4) http://www.ifad.org/gbdocs/eb/90/e/EB-2007-90-R-4.pdf

95 Corporate-level evaluation of IFAD’s institutional efficiency and efficiency of IFAD-funded operations, p. 52. Paragraphs 47-49

96 IFAD (February 2011), Policy on the Disclosure of Documents (2010), (GC 34/INF.2/Rev.1).

97 IFAD Strategic Framework 2011-2015, p. 26.

98 See p. 58 in the RIDE 2012 for more detail.

99 Source: Report of the Consultation on IFAD’s Ninth Replenishment of IFAD’s Resources.

100 Please refer to Section 2.3.2, ‘Strategic Management’, KPIs 3 and 5 for the analysis of IFAD’s results-based systems and practices.

101 RIDE, 2012, p. 7, paragraph 26.

102 Project completion reports use a six-point scale to assess the degree of relevance, effectiveness, efficiency, rural poverty impact, sustainability, innovation and gender equality for each project. A rating of 6 is equal to highly satisfactory and a rating of 1 is equal to highly unsatisfactory. Charts provided at pp. 10-11 of the RIDE 2012 are based on the percentage of projects which achieve a score of 4 or higher.

103 A list of documents consulted is provided in Vol. II, Appendix VIII.

104 ‘Theory of change‘ is understood in the sense defined by Rist and Morra Imas (2009) as, “a representation of how an intervention is expected to lead to desired results” and in the sense defined by Michael Quinn Patton who has stated that a theory of change is more than the sequential order of results statements presented in a logic model; it requires key assumptions related to the results chain and context (e.g. policy and environment), and important influences and risks to be made explicit - Qualitative Research and Evaluation Methods (2002).

105 This refers to the existence of reports on outputs as defined by the OECD (i.e. lower level results). Some MOs use different terminology for the various levels of results.

106 This refers to the existence of reports on outcomes as defined by the OECD (i.e. higher level results). Some MOs use different terminology for the various levels of results.

107 ‘Reporting according to a theory of change is understood to mean the extent to which organisations provide a narrative describing the actual implementation process and results achieved in relation to that foreseen in the initial ‘theory of change’.

108 According to Rist and Morra Imas, The Road to Results – “Reliability is the term used to describe the stability of the measurement – the degree to which it measures the same thing, in the same way, in repeated tests.” Attention is also given to the quality of the evidence – specifically, whether or not it has been derived from or validated by an external and/or independent source.

109 The new RMF for the IFAD9 replenishment period includes an indicator on environment and natural resource management under Level 2.

110 Ibid.

111 None of the indicators in the RMF directly measure the extent of rural enterprise development and employment opportunities created. However, the indicator “Rural poverty impact” under Level 2 in the RMF indirectly constitutes a result of strengthened rural enterprises and the number of non-farm employment opportunities created.

112 Relevant indicators under Level 3 include gender disaggregated data.

113 None of the criteria are currently met for IFAD’s reporting on its contributions to resilience of household food security and nutrition. However, the revised RMF includes impact indicators that directly relate to this thematic priority (indicator 2.2.2 – “Level of child malnutrition (3 sub-indicators – acute, chronic and underweight), disaggregated for girls and boys”, and; indicator 2.2.3 – “Length of hungry season”).

114 This is part of the core mandate of IFAD and arguably all IFAD’s operations aim to reduce rural poverty. Level 3 in the current RMF measures the number of people receiving services from IFAD, disaggregated by gender. In the revised RMF (2013-2015) this output indicator has been amended to “people moved out of poverty” which is an outcome/impact indicator. Level 2 of the current RMF measures the level of satisfaction with the rural poverty impact of IFAD’s projects based on ratings in Project Completion Reports.

115 See Chapter 5 in IFAD’s Strategic Framework 2011-2015 for more detail.

116 2011 Rural Poverty Report, December 2010, http://www.ifad.org/rpr2011/report/e/rpr2011.pdf

117 RIDE 2012, p. 9.

118 In this context it is important to re-iterate that IFAD’s latest results framework include a set of new indicators at the outcome and impact level that provide better measures of results achieved. In addition in its Ninth Replenishment Period, IFAD has committed to carrying out 30 impact evaluations to demonstrate links between its outputs and developmental impacts.

119 CLE on the Institutional Efficiency of IFAD, 2013, https://webapps.ifad.org/members/eb/108/docs/EB-2013-108-R-3-Rev-1.pdf

120 CLE on the Institutional Efficiency of IFAD, 2013, p. 76, https://webapps.ifad.org/members/eb/108/docs/EB-2013-108-R-3-Rev-1.pdf

121 Annual Review of Portfolio Performance 2011-2012, Annex III

122 RIDE 2012, p. 6.

123 A1 to A8 are identified in IFAD’s Strategic Framework 2011-2015. In addition, areas of thematic focus A9, A10 and A11 were included as themes identified by MOPAN members as central to IFAD’s development mission and mandate.

124 These are outputs reported in the RIDE 2012 and are the results for which IFAD is fully accountable. Currently IFAD only reports on outcome-level results based on ratings of performance criteria such as effectiveness, efficiency, relevance etc. under Level 2. Given the lack of actual data on outcomes and lack of targets at the output level, it is difficult to assess IFAD’s level of “progress” in this area.

125 This rating is the mean score of two sub-questions: i) IFAD has effectively contributed to developing rural enterprises in borrowing countries (mean score= 4.48), and ii) IFAD has effectively contributed to developing non-farm employment opportunities in borrowing countries (mean score = 4.04).

126 According to IFAD’s RMF, a project is rated positively if it has moderately satisfactory or better performance. The percentages represent the number of projects rated as moderately satisfactory or above at completion. A rating of 6 is equal to highly satisfactory; 5 = satisfactory; 4 = moderately satisfactory; 3 = moderately unsatisfactory; 2 = unsatisfactory; and 1 = highly unsatisfactory. A score of 4 or higher reflects overall positive performance. The data in the table is based on the RIDE 2012.

127 The data in figure 4.6 was extracted from the RIDE 2012’s charts on pages 10-11.

128 Based on a sample of 20 projects reviewed for the Independent External Evaluation. Source: Office of Evaluation: Independent External Evaluation of IFAD, 2005. (Table 8, p.133) http://www.ifad.org/evaluation/iee/report/e.pdf

129 RIDE 2012, p. 8, paragraph 28.

130 As previously mentioned, IFAD’s revised RMF for the Ninth Replenishment Period includes a new indicator under Level 2 that measures the level of satisfaction with its contributions to environment and natural resource management a project completion.

131 The other impact domains are HH income and assets, human and social capital, food security and agricultural productivity, natural resources, environment and climate change.

132 ARRI 2012, p. 13.

133 See IFAD’s Climate Change Strategy, 2010 p. 30.

134 RIDE 2012, p. 24.

135 IFAD Strategic Framework 2011-2015, paragraph 49

136 The survey question “IFAD has effectively contributed to developing non-farm employment opportunities in borrowing countries” did not specify whether these employment opportunities were in rural or urban areas. Although IFAD is mandated to work exclusively in rural areas, respondents may have interpreted this to include urban employment opportunities.

137 On the basis of a Corporate Level Evaluation on Gender from 2010, IOE introduced a new gender criterion to be applied in all evaluations undertaken. Since 2011 ARRI has therefore included a separate evaluation criterion on gender.

138 See Annex I in the RIDE 2012.

139 This is a composite outcome indicator which includes elements linked to food security and nutrition. Impact on rural poverty is assessed using five impact domains (household income and assets; food security and agricultural productivity; human and social capital and empowerment; natural resources and environment; institutions and policies), as per IOE methodology, plus a sixth impact domain, which is access to markets.

140 Due to limitations in IFAD’s reporting on results at the country level, project-level reporting has been used as the main source of data for the analysis of results achieved.

141 Please refer to the sections in chapter 3 on KPIs 2, 4, 18 and 20 for the analysis of IFAD’s results-based systems and practices.

142 Second and third level results are measured by a rating scale. A score of 1 corresponds to highly unsatisfactory effectiveness or very weak sustainability. By contrast, a score of 6 corresponds to highly satisfactory effectiveness or very strong sustainability.

143 See http://www.ifad.org/evaluation/iee/docs/index.htm for the evaluation report.

144 IFAD has established country offices in all the countries assessed.

145 Theory of change is understood in the sense defined by Rist and Morra Imas (2009) as, “a representation of how an intervention is expected to lead to desired results” and in the sense defined by Michael Quinn Patton who has stated that a theory of change is more than the sequential order of results statements presented in a logic model; it requires key assumptions related to the results chain and context (e.g. policy and environment), and important influences and risks to be made explicit - Qualitative Research and Evaluation Methods (2002).

146 This refers to the existence of reports on outputs as defined by the OECD (i.e. lower level results). Some MOs use different terminology for the various levels of results.

147 This refers to the existence of reports on outcomes as defined by the OECD (i.e. higher level results). Some MOs use different terminology for the various levels of results.

148 ‘Reporting according to a theory of change is understood to mean the extent to which organisations provide a narrative describing the actual implementation process and results achieved in relation to that foreseen in the initial ‘theory of change’.

149 IFAD (2013), Corporate-Level Evaluation of IFAD’s Institutional Efficiency and Efficiency of IFAD-funded operations, p. 76, paragraph. 112.

150 The project became effective in 2008 and is ending in 2014.

151 The project became effective in 2010 and is scheduled for completion in 2016.

152 The project became effective in 2011 and is ending in 2016.

153 MTR COSOP review 2012, paragraph 118.

154 Pakistan COSOP Mid-Term Review, 2012.

155 The total project cost was USD 30.5 million of which 26.5 million was IFAD financed.

156 Interim Evaluation of the Rural Financial Intermediation Programme I (RUFIP), IOE, 2011.

157 Guatemala’s portfolio currently includes two projects. ProQuiché, the most recent IFAD operation in Guatemala, approved in December 2010, was not ratified on time by Congress, despite the 6-month extension given by IFAD for the signing of the loan approved by IFAD’s Executive Board. The project has been cancelled.

158 It should be noted that IFAD operations in Guatemala were at a standstill at the time of writing the assessment, due to Congress delays in signing the loans approved by IFAD, as well as the closure of the implementing agency. A new government was inaugurated in January 2012, led by President Otto Pérez Molina. A series of Congress Decrees published in July/August 2012 affected the legal status of two of IFAD’s projects (in the Western and Eastern Regions), putting them in a standstill in 2012. Furthermore, on 17 January 2013, the government published a legal agreement to close down FONAPAZ, which had been an executing agency for part of IFAD’s portfolio in the country. ProQuiché, the most recent IFAD operation in Guatemala, approved in December 2010, has not yet been ratified by Congress and a 6-month extension for the signing of the loan has been approved by IFAD’s Executive Board, with a deadline of 15 June 2013. If not signed by the extended date, the project shall be cancelled.

159 PRODEVER, final supervision report, May 2012

160 In this context it should be noted that a Revised Manual for Delegation of Authority relating to the delegation at headquarters level is due to be amended and approved by December 2013.

161 These areas were: B1 Sustaining the growth of economic activities and reducing the incidence of poverty; B2 Improvement of natural resource management; B4 Building capacity of rural people to engage in local policy and programming processes.

162 B3 Increasing household incomes for poor households involved in fisheries and marine activities; B5 Promotion of gender equality and women’s empowerment; B6: Improvement of household food security and nutrition.