Preface

The Multilateral Organisation Performance Assessment Network (MOPAN) is a network of donor countries with a common interest in assessing the organisational effectiveness of multilateral organisations. MOPAN was established in 2002 in response to international fora on aid effectiveness and calls for greater donor harmonisation and coordination.

Today, MOPAN is made up of 16 donor countries: Australia, Austria, Belgium, Canada, Denmark, Finland, France, Germany, Ireland, The Netherlands, Norway, Republic of Korea, Spain, Sweden, Switzerland and the United Kingdom. For more information on MOPAN and to access previous MOPAN reports, please visit the MOPAN website (www.mopanonline.org).

Each year MOPAN carries out assessments of several multilateral organisations based on criteria agreed by MOPAN members. Its approach has evolved over the years, and since 2010 has been based on a survey of key stakeholders and a review of documents of multilateral organisations. MOPAN assessments provide a snapshot of four dimensions of organisational effectiveness (strategic management, operational management, relationship management, and knowledge management). In 2012, MOPAN is piloting a new component to examine an organisation’s development results in addition to its organisational effectiveness.

MOPAN 2012

In 2012, MOPAN assessed six multilateral organisations: the African Development Bank (AfDB), GAVI Alliance (formerly the Global Alliance for Vaccines and Immunisation), the Joint United Nations Programme on HIV/AIDS (UNAIDS), the United Nations Development Programme (UNDP), the United Nations Children’s Fund (UNICEF), and the World Bank.

MOPAN Institutional Leads liaised with the multilateral organisations throughout the assessment and reporting process. MOPAN Country Leads monitored the process in each country and ensured the success of the survey.

Acknowledgements

We thank all participants in the MOPAN 2012 assessment of UNICEF. UNICEF’s senior management and staff made valuable contributions throughout the assessment and document review processes and provided lists of their direct partners and peer organisations to be surveyed. Survey respondents contributed useful insights and time to respond to the survey. The MOPAN Institutional Leads, Austria and Spain, liaised with UNICEF throughout the assessment and reporting process. The MOPAN Country Leads oversaw the process in the field and ensured the success of the survey. Consultants in each country provided vital in- country support by following up with direct partners and peer organisations to enhance survey response rates.

Roles of Authors and the MOPAN Secretariat

The MOPAN Secretariat, led by Ireland in 2012 and co-chaired by Germany, worked in close cooperation with the MOPAN Technical Working Group to launch and manage the survey. MOPAN developed the Key Performance and Micro-indicators, designed the survey methodology, coordinated the development of lists of survey respondents, and approved the final survey questionnaire. MOPAN also directed the design of the approach to document review. MOPAN oversaw the design, structure, tone, and content of the reports.

Universalia and Epinion developed the survey instrument and carried out the survey and analysis. Universalia carried out the document review and wrote the reports.

Epinion is a leading consulting firm in Denmark that analyses and evaluates data to support decision making. It conducts specially designed studies for public and private organisations based on data collected among an organisation’s employees, members, customers, partners, and other sources. Epinion has 75 employees and 200 interviewers. Website: www.epinion.dk

Universalia Management Group is a Canadian consulting firm established in 1980 that specialises in evaluation and monitoring for international development. Universalia has made significant contributions to identifying best practices and developing tools in the fields of organisational assessment; planning, monitoring, and evaluation; results-based management; and capacity building. Website: www.universalia.com.

Acronyms

Figures

Figure 2.1 Respondent Rating Scale
Figure 2.2 UNICEF- Distribution of Responses (n=294) on all Questions Related to Micro- Indicators
Figure 2.3 Number of Survey Respondents and Total Population for UNICEF by Country and Respondent Group
Figure 2.4 MOPAN Ranges and Descriptions
Figure 3.1 Overall Ratings of UNICEF Organisational Effectiveness by Respondent Group
Figure 3.2 Overall Ratings on Key Performance Indicators of UNICEF Practices and Systems (mean scores, all respondents and document review ratings)
Figure 3.3 Quadrant I: Strategic Management, Survey and Document Review Ratings
Figure 3.4 Quadrant I: Strategic Management, Mean Scores by Respondent Group
Figure 3.5 KPI 1: Providing Direction for Results, Ratings of Micro-Indicators
Figure 3.6 KPI 2: Corporate Focus on Results, Ratings of Micro-Indicators
Figure 3.7 KPI 3: Focus on Thematic Priorities, Ratings of Micro-Indicators
Figure 3.8 KPI 4: Country Focus on Results, Ratings of Micro-Indicators
Figure 3.9 Quadrant II: Operational Management, Survey and Document Review Ratings
Figure 3.10 Quadrant II: Operational Management, Mean Scores by Respondent Group
Figure 3.11 KPI 5: Resource Allocation Decisions, Ratings of Micro-Indicators
Figure 3.12 KPI 6: Linking Aid Management to Performance, Ratings of Micro-Indicators
Figure 3.13 KPI 7: Financial Accountability, Ratings of Micro-Indicators
Figure 3.14 KPI 8: Using Performance Information, Ratings of Micro-Indicators
Figure 3.15 KPI 9: Managing Human Resources, Ratings of Micro-Indicators
Figure 3.16 KPI 10: Performance-oriented Programming, Ratings of Micro-Indicators
Figure 3.17 KPI 11: Delegating Authority, Ratings of Micro-Indicators
Figure 3.18 KPI 12: Humanitarian Principles, Ratings of Micro-Indicators
Figure 3.19 Quadrant III: Relationship Management, Survey and Document Review Ratings
Figure 3.20 Quadrant III: Relationship Management, Mean Scores by Respondent Group
Figure 3.21 KPI 13: Supporting National Plans, Ratings of Micro-Indicators
Figure 3.22 KPI 14: Adjusting Procedures, Ratings of Micro-Indicators
Figure 3.23 KPI 15: Using Country Systems, Ratings of Micro-Indicators
Figure 3.24 KPI 16: Contributing to Policy Dialogue, Ratings of Micro-Indicators
Figure 3.25 KPI 17: Harmonising Procedures, Ratings of Micro-Indicators
Figure 3.26 KPI 18: Managing the Cluster, Ratings of Micro-Indicators
Figure 3.27 Quadrant IV: Knowledge Management, Survey and Document Review Ratings
Figure 3.28 Quadrant IV: Knowledge Management, Mean Scores by Respondent Group
Figure 3.29 KPI 19: Evaluating External Results, Ratings of Micro-Indicators
Figure 3.30 KPI 20: Presenting Performance Information, Ratings of Micro-Indicators
Figure 3.31 KPI 21: Disseminating Lessons Learned, Ratings of Micro-Indicators
Figure 4.1 Development Results Component – Overall Ratings
Figure 4.2 KPI A: Evidence of the Extent of Progress toward Organisation-Wide Outcomes, Ratings
Figure 4.3 KPI B: Evidence of the Extent of Contribution to Country-Level Goals and Priorities, Rating
Figure 4.4 Stakeholder Survey, Mean Scores by Focus Area and Country
Figure 4.5 KPI B: Evidence of the Extent of Contribution to Country-Level Goals and Priorities
Figure 4.6 UNICEF Focus Areas and Related MDGs
Figure 4.7 KPI C: Evidence of the Extent of Contributions to Relevant MDGs, Overall Rating and Criteria
Figure 4.8 KPI D: Relevance of Objectives and Programme of Work to Stakeholders, Overall Rating and Survey Mean Scores by Country

Executive Summary

This report presents the results of an assessment of the United Nations Children’s Fund (UNICEF) conducted by the Multilateral Organisation Performance Assessment Network (MOPAN). MOPAN assesses the organisational effectiveness of multilateral organisations based on a survey of stakeholders, a review of documents, and interviews with headquarter- based staff. In past years, MOPAN has not assessed an organisation’s development results, but is testing a component on this with four organisations in this year’s assessment.[1]

According to its mission statement, UNICEF is mandated by the United Nations General Assembly to advocate for the protection of children's rights, to help meet their basic needs and to expand their opportunities to reach their full potential. This implies mobilising political will and material resources to help countries, ensuring special protection for the most disadvantaged children, responding in emergencies to protect the rights of children, promoting the equal rights of women and girls and supporting their full participation in the political, social, and economic development of their communities.

In recent years, UNICEF has implemented a number of reforms to improve its effectiveness, efficiency and governance. Key reforms have included the initiation of an organisational change process with a focus on a revised programme structure, implementation of a new platform to harmonise business tools and performance systems, and the application of the International Public Sector Accounting Standards (IPSAS). Related initiatives include improvements to the regulatory framework, accountability and enterprise risk management systems.

In 2012, MOPAN assessed UNICEF based on information collected through interviews with UNICEF staff, a survey of key stakeholders and a review of documents. The survey targeted UNICEF’s peer organisations, direct partners, and MOPAN donors based in-country and at headquarters. Nine countries were included in the MOPAN survey: Cambodia, Democratic Republic of Congo, Ghana, Honduras, Philippines, Morocco, Niger, Nigeria, and Zimbabwe. A total of 294 respondents participated in the survey. MOPAN’s document review assessed UNICEF through an examination of publicly available corporate documents and country programming documents from five of the nine countries selected.

MOPAN assessments provide a snapshot of four dimensions of organisational effectiveness (strategic management, operational management, relationship management, and knowledge management). The main findings of the assessment of UNICEF in these performance areas and in the development results component are summarised below.

Strategic Management

In the area of strategic management, MOPAN established criteria to determine if a multilateral organisation has strategies and systems in place that reflect good practice in managing for results. Overall, the 2012 assessment found that:

• Strategic management is one of UNICEF’s strengths. Its strongest areas of performance relates to its capacity to provide direction for results and a country focus on results.

• UNICEF is performing well in addressing cross-cutting priorities related to gender equality, human rights-based approaches, and emergency response/humanitarian action, but received slightly lower ratings for its efforts to mainstream the environment, good governance, and conflict management.

• Areas for improvement include the presentation of causal linkages, phrasing of results statements and the selection of indicators in the agency-wide development results framework (DRF) and management results framework (MRF).

Operational Management

In operational management, MOPAN established criteria to determine if a multilateral organisation manages its operations in a way that supports accountability for results and the use of information on performance. Overall, the 2012 assessment found that:

• UNICEF’s financial management is seen as its strongest area of performance in operational management. Within the UN family, UNICEF has well-regarded financial accountability systems. It has steadily improved its audit function and has introduced a new system to assess and manage risks (Enterprise Risk Management).

• UNICEF makes use of performance information to improve its policies and country programmes, as well as its management of human resources.

• The organisation has the appropriate policy documents to guide its humanitarian response and it is well regarded for respecting humanitarian principles while delivering humanitarian/emergency assistance in the countries where it has such programming.

• UNICEF continues to face difficulties in linking aid management and performance. While improvements in some areas are underway – notably in the application of results-based budgeting and new cost classifications for the institutional budget – there remains considerable room for improvement in the linking of disbursements to results achieved.

• Results-based management (RBM) practices have yet to be fully integrated into UNICEF’s programming cycle. This is noted in the limited use of targets/milestones and ex-ante assessments (on impact and benefits) in the design of projects.

• UNICEF is viewed as strong in its adherence to humanitarian principles and work with partners in its cluster leadership.

Relationship Management

In relationship management, MOPAN established criteria to determine if a multilateral organisation is engaging with its partners at the country level in ways that contribute to aid effectiveness. Overall, the 2012 assessment found that:

• UNICEF is seen as strong in its support of national counterparts and national plans, its contributions to policy dialogue, and in its humanitarian cluster leadership.

• Surveyed stakeholders value UNICEF’s technical assistance and engagement with partners.

• There is some room for improvement in the efficiency of its administrative procedures and use of country systems.

• Feedback from MOPAN donors in-country was less positive than from other respondent groups on all six key performance indicators assessed in this quadrant.

Knowledge Management

In knowledge management, MOPAN established criteria to determine if a multilateral organisation has reporting mechanisms and learning strategies that facilitate the sharing of information inside the organisation and with the development community. Overall, the 2012 assessment found that:

• UNICEF was perceived as performing well on the evaluation of external results and the dissemination lessons learned; however, the level of stakeholder familiarity with the content of the lessons learned is low.

• As part of the process of strengthening its evaluation function, UNICEF has invested considerable resources in evaluating external results, and has made progress in the coverage and quality of evaluations.

• Areas identified for improvement include UNICEF’s use of indicators and country targets to report on the Paris Declaration Indicators, and reporting on adjustments to policies/strategies based on performance information.

Development Results Component

The 2012 pilot assessment focused on the evidence of UNICEF’s contributions to development results.

• Evidence of the extent of progress towards organisation-wide results: MOPAN donors at headquarters hold positive views about UNICEF‘s progress towards its stated results in the five focus areas of the Medium Term Strategic Plan (MTSP). UNICEF’s data on results achievement is fragmented. Reports provide evidence of progress towards planned lower level results in UNICEF’s framework, but limited evidence of the overall contribution that UNICEF is making to higher-level change.

• Evidence of the extent of contribution to country-level goals and priorities: In all countries sampled, surveyed stakeholders gave UNICEF positive marks for its contributions to national goals associated with each of its focus areas. The sample reports reviewed, however, do not provide a clear picture of the nature, magnitude, or relative importance of UNICEF contributions to some of the changes reported in national indicators, policy frameworks, and other areas.

• Evidence of the extent of contribution to relevant MDGs: UNICEF’s planned results at country level are often tightly linked to national MDGs and surveyed stakeholders were generally positive about UNICEF’s contribution to the achievement of MDGs. Reports, on the other hand, do not describe UNICEF’s role and contribution in support of the partner country’s efforts to achieve MDGs.

• Relevance of objectives and programmes of work to stakeholders: UNICEF was seen by surveyed stakeholders in-country to be consistently strong in responding to the key development priorities of its partner countries, in providing innovative solutions to help address these challenges, and in adapting its work to the changing needs of partner countries.

Trends since 2009

This is the second time that UNICEF has been assessed by MOPAN using its Common Approach methodology. In 2012, UNICEF was recognised by respondents for the clarity and strength of its mandate, its positive relationships with government partners, and its operational capacity in development and humanitarian contexts. It was seen to be efficient and to get things done on the ground. UNICEF’s strongest ratings in 2009 related to organisational practices and behaviours that had significant repercussions at the country level: delegation of decision making, results focus of country programming, and valued contributions to policy dialogue. UNICEF’s lowest rating was the extent to which the organisation used government systems for procurement, audit, financial reporting, and other procedures.

Conclusions

UNICEF’s organisational development process and related reforms are likely to improve its effectiveness and efficiency.

In response to the 2007 organisational review, UNICEF undertook an organisational development process that is likely to increase the effectiveness and efficiency of its operations. A new system (VISION) for managing financial resources and performance information is being implemented across the organisation and at regional and country levels. Survey respondents expressed confidence in UNICEF’s fiduciary systems, which include external and internal audit and anti-corruption policies, and the document review found that the organisation has very good practices in this area.

Like many other UN organisations, UNICEF is adopting International Public Sector Accounting Standards (IPSAS). There is also evidence of efforts to improve organisational policies, procedures and guidance as part of the regulatory framework, accountability system and enterprise risk management.

UNICEF remains strongly committed to a results-focused reform agenda and is improving its capacity to manage for development results. However, the frameworks that it uses to manage for and report on organisation-wide results need further improvement.

UNICEF’s organisation-wide strategy provides a clear and strong programming focus to a large and decentralised organisation. UNICEF also demonstrates a willingness to consult partners and use reviews, evaluations, and audits (e.g., Mid-Term Reviews) to improve its strategy. However, UNICEF faces challenges in managing for results and applying results-based management principles in its organisation-wide strategy. This shortcoming has an impact on the quality of the results frameworks linked to the Medium-Term Strategic Plan and the related programming tools used across the organisation. The document review highlighted the need to strengthen its application of results-based management through: improving the quality of results statements, strengthening the links between outputs and outcomes, identifying measureable indicators and consistently using baselines and targets, and integrating an analysis of UNICEF’s contribution to higher-level results.

UNICEF’s country programming processes have improved as a result of organisational changes but planning at the project level received low ratings.

Recent programmatic changes have improved the clarity of results frameworks and evaluation of results by specifying two levels of results that are clearly linked: intermediate results leading to programme component results. The country level Summary Results Matrices prepared from 2009 to the present include higher-level results statements, indicators, baselines and targets – all of which meet results-based management standards. UNICEF’s new, web-based monitoring system – VISION – should also improve the organisation’s ability to track progress against a wide range of programmatic and management indicators. It is important to note, however, that the document review found UNICEF to be weak in the development of appropriate indicators, baselines and targets, as well as the application of benefit or impact analyses at the project level.

UNICEF’s has proven particularly effective in mainstreaming gender equality and emergency responses/humanitarian action.

UNICEF’s strategic and programming documents articulate a clear institutional approach to both gender equality and emergency response. UNICEF’s organisational commitment to mainstream gender equality and emergency response/humanitarian action has led to positive results in the vertical and horizontal integration of these cross-cutting themes. For example, in its new Policy on Gender Equality and the Empowerment of Girls and Women 2010, UNICEF commits to mainstreaming gender in its operations and to develop accountability and monitoring mechanisms. Similarly, UNICEF’s approach to Emergency Response/Humanitarian Action is articulated in the 2010 Core Commitments for Children (CCCs) in Humanitarian Action, which includes the framework, principles, programme commitments, and accountabilities relevant to UNICEF’s work in this area.

UNICEF is valued for its field presence and engagement with governmental and non- governmental partners.

UNICEF was praised by respondents for its strong working relationships with and its support to its partners. Its efforts to strengthen government leadership and ownership, foster collaboration in a highly consultative and participatory manner, as well as to enhance national capacities are considered among UNICEF’s greatest assets.

UNICEF is highly valued by its direct partners.

In the four key management areas examined in the assessment of organisational practices and systems, UNICEF’s direct partners provided consistently positive ratings of the organisation, and rated it strong on all KPIs but one (use of country systems). These positive views carry through in the smaller sample of direct partners that were consulted about UNICEF’s results, who gave strong ratings on its contributions to partner country goals and priorities and country efforts to meet the MDGs. UNICEF’s programming is considered to be relevant at the country level. It is seen to respond to country priorities, adapt to changing conditions, and provide innovative solutions.

UNICEF allocates resources in a transparent manner, but has not yet fully implemented results-based budgeting.

UNICEF’s criteria for resource allocations are clear and publicly available. However, there is limited evidence to show the link between allocations or disbursements and expected results at the organisation-wide level. UNICEF is implementing results-based budgeting through a harmonised approach with UNFPA and UNDP.

In contexts where it has significant humanitarian programming, UNICEF is fulfilling its responsibilities as a Cluster Lead and is recognised for its respect for humanitarian principles.

UNICEF’s institutional culture strongly emphasises respect for humanitarian principles and these principles – namely, humanity, impartiality and neutrality – are reflected in the Core Commitments for Children in Humanitarian Action. Documented evidence and survey respondents’ views indicate that UNICEF respects humanitarian principles while delivering humanitarian assistance, and holds on-going policy dialogue with partners on the importance of observing humanitarian principles, particularly in cases of conflict.

Similarly, survey feedback from its peers, direct partners, and MOPAN donors suggest that UNICEF is playing a key role as a cluster lead in the nutrition, water and sanitation and education sectors in the Democratic Republic of Congo, Niger and Zimbabwe by ensuring sufficient analytical inputs and policy engagements, staff time, information flows, and financial forecasting.

UNICEF has strengthened its evaluation function but there is still room for improvement in some areas.

Survey respondents assessed UNICEF as strong in its inclusion of beneficiaries and direct partners in evaluations. With the adoption of a series of new procedures and processes, UNICEF’s evaluation office is reinforcing good evaluation practices. It has strengthened its practice of reviewing and reporting on the quality of evaluations at all levels. Further, it has adopted formal mechanisms to track the implementation of evaluation recommendations. However, the documents reviewed remain vague on the criteria used to determine coverage and prioritisation of evaluations. To address this, UNICEF issued guidance on the prioritisation of major evaluations at the decentralised level as a way of strengthening the relevance and use of evidence generated through evaluations.

Development Results Component

Although UNICEF has provided strong direction for results-based management in recent years, limitations in its frameworks and systems to report on organisation-wide results make it challenging to identify the difference being made in the lives of children and women.

The assessment team took note of the work being done by UNICEF to improve its results- based approaches and reporting – notably, the new VISION system and work to sharpen the programmatic focus on equity. Nevertheless, the organisation-wide reports reviewed showed inconsistencies in the presentation of results achieved and their relationship to expected results defined in the Medium-Term Strategic Plan. Further, the lack of description of the various steps in the results chain (inputs, outputs, outcomes, etc.) made it difficult to see UNICEF’s contributions to noted changes in the lives of children and women. For example, the Annual Report of the Executive Director in 2012 stated that there was a 38 per cent increase in the household use of iodised salt (to 70 per cent) in 67 countries, but did not identify the steps either planned or taken by UNICEF and others to bring about this positive result.

UNICEF’s thematic reports represent UNICEF’s best performance reporting as they link thematic information to the organisation-wide results framework more effectively than other reports produced by the organisation.

At the country level, stakeholders confirm the relevance of UNICEF’s programming and indicate that UNICEF makes contributions to its Country Programme results and to partner country efforts to achieve the MDGs. UNICEF reports do not yet adequately document UNICEF’s contributions in these areas.

According to survey respondents, UNICEF programming is relevant in the countries sampled because it responds to priorities, offers innovative solutions, and adapts to the changing needs of its partners. Across the five countries reviewed, stakeholders also perceived that UNICEF made contributions to each of the focus areas prioritised in the Country Programme Document. Some clear and credible examples of UNICEF’s contribution to national outcomes were found in the document review. The final evaluation of UNICEF Ghana’s Accelerated Child Survival and Development (ACSD) program, for example, was thorough and convincing in its analysis of attribution of UNICEF-supported interventions to improvements to child and maternal mortality levels in certain districts. Further, the reports on water and sanitation interventions supported by UNICEF Zimbabwe during the crisis of 2008/9 and UNICEF Cambodia’s education activities provided convincing evidence of improvements in the lives of women and children.

Unfortunately, however, these examples proved to be the exceptions rather than the rule. Most UNICEF reporting reviewed from previous cycles provided ample data on low-level results but failed to make the link between the interventions supported and the higher-level results achieved.

Overall MOPAN Ratings of UNICEF

The chart below shows the ratings on the 21 key performance indicators that MOPAN used to assess UNICEF in 2012. These indicators were designed to measure organisational effectiveness (practices and systems), not development or humanitarian results on the ground. UNICEF received ratings of strong on 12 of the 21 key performance indicators assessed by survey respondents (9 of the 21 were adequate), and document review ratings ranging from weak to very strong on the 17 KPIs assessed by the document review. The survey and document review ratings differed on 13 KPIs.

1. Introduction

1.1 MOPAN

This report presents the results of an assessment of UNICEF that was conducted in 2012 by the Multilateral Organisation Performance Assessment Network (MOPAN). In 2012 MOPAN assessed six multilateral organisations: the African Development Bank (AfDB), GAVI Alliance (formerly the Global Alliance for Vaccines and Immunisation), the Joint United Nations Programme on HIV/AIDS (UNAIDS), the United Nations Development Programme (UNDP), the United Nations Children’s Fund (UNICEF), and the World Bank.

Background

MOPAN was established in 2002 in response to international fora on aid effectiveness and calls for greater donor harmonisation and coordination. The purpose of the network is to share information and experience in assessing the performance of multilateral organisations. MOPAN supports the commitments adopted by the international community to improve the impact and effectiveness of aid as reflected in the Paris Declaration on Aid Effectiveness, the Accra Agenda for Action, and the Busan High Level Forum. MOPAN’s processes and instruments embody the principles of local ownership, alignment and harmonisation of practices, and results-based management (RBM).

MOPAN provides a joint approach (known as the Common Approach) to assess the organisational effectiveness of multilateral organisations. The approach was derived from existing bilateral assessment tools and complements and draws on other assessment processes for development organisations – such as the bi-annual Survey on Monitoring the Paris Declaration on Aid Effectiveness and annual reports of the Common Performance Assessment System (COMPAS) published by the multilateral development banks. In the long term, MOPAN hopes that this approach will replace or reduce the need for other assessment approaches by bilateral donors.

MOPAN assesses four dimensions of organisational effectiveness

MOPAN has defined organisational effectiveness as the extent to which a multilateral organisation is organised to contribute to development and/or humanitarian results in the countries or territories where it operates.

Based on a survey of stakeholders and a review of documents, MOPAN assessments provide a snapshot of a multilateral organisation’s effectiveness in four dimensions:

• Developing strategies and plans that reflect good practices in managing for development results (strategic management)

• Managing operations by results to support accountability for results and the use of information on performance (operational management)

• Engaging in relationships with direct partners and donors at the country level in ways that contribute to aid effectiveness and that are aligned with the principles of the Paris Declaration (relationship management)

• Developing reporting mechanisms and learning strategies that facilitate the sharing of knowledge and information inside the organisation and with the development community (knowledge management).

In 2012, MOPAN also piloted a new component to assess a multilateral organisation’s contributions to development results. This component was tested with four of the six organisations assessed this year (AfDB, UNDP, UNICEF, and the World Bank).

Purpose of MOPAN assessments

MOPAN assessments are intended to:

• Generate relevant, credible and robust information MOPAN members can use to meet their domestic accountability requirements and fulfil their responsibilities and obligations as bilateral donors

• Provide an evidence base for MOPAN members, multilateral organisations and direct partners to discuss organisational effectiveness and in doing so, build better understanding and improve organisational effectiveness and learning over time

• Support dialogue between MOPAN members, multilateral organisations and their partners, with a specific focus on improving organisational effectiveness over time, both at country and headquarters level.

The MOPAN methodology is evolving in response to what is being learned from year to year, and to accommodate multilateral organisations with different mandates. For example, the indicators and approach for the 2012 MOPAN review of a global fund and organisations with significant humanitarian programming were adapted to reflect the reality of these organisations.

1.2 Profile of UNICEF

Mission and Mandate

The United Nations Children's Fund (UNICEF) was created in 1946 by the United Nations General Assembly to provide emergency assistance to children in countries that had been devastated by World War II. Since then, UNICEF has been mandated by the same body to advocate for the protection of children’s rights, to help meet their basic needs, and to expand their opportunities to reach their full potential.

UNICEF is guided by the Convention on the Rights of the Child (CRC) and works to strengthen capacities of governmental partners to fulfil their obligations towards children as defined in the Convention. In addition to its Mission Statement, which was formally adopted by the UNICEF Executive Board in 1996, UNICEF is guided by commitments made in the Millennium Summit Declaration of the General Assembly and the Declaration and Plan of Action (A World Fit for Children) that was adopted during the General Assembly Special Session on Children in 2002.

Structure and Governance

UNICEF headquarters in New York, USA is responsible for overall management and administration of the organisation. UNICEF is divided into seven regions, each of which has a Regional Office. UNICEF also has specialised offices in Switzerland, Denmark, Italy, Belgium and Japan.

UNICEF is accountable to its Executive Board, which is its governing body and provides inter-governmental support and oversight to the organisation. The Board comprises 36 members who represent the five regional groups of Member States at the United Nations. It meets three times each year to review UNICEF’s activities and approves its policies, country programmes, and budgets.

UNICEF is administered by an Executive Director, who is appointed by the United Nations Secretary-General in consultation with the Executive Board.

As a highly decentralised agency, the heart of UNICEF’s work is in the field and it is currently active in more than 190 countries worldwide. Country offices implement UNICEF’s mission through five-year programmes of cooperation that are developed in collaboration with host governments. UNICEF’s regional offices oversee this work and provide technical assistance to country offices as required. UNICEF works closely with other UN agencies and a multitude of partners to mobilise political will and material resources for the realisation of the rights of children and women in all regions of the world.

UNICEF also has 36 National Committees that are independent, local non-governmental organisations that serve as the public face of UNICEF in higher-income countries – raising funds, engaging in advocacy, and promoting UNICEF’s visibility worldwide.

Strategy and Services

In response to UNICEF’s 2004 mid-term review of the 2002-2005 strategic plan, UNICEF developed the Medium Term Strategic Plan (MTSP) for 2006-2009, which was twice extended by the Executive Board and now covers the period until the end of 2013.

The overall strategic objectives of the MTSP are to realise the rights of children and women by supporting the national and international implementation of the Millennium Summit Declaration and pursuing the Millennium Development Goals, as well as to effectively contribute to poverty reduction through advocacy and partnerships that generate sustained investments in children’s survival, development and protection.

UNICEF accomplishes these objectives by applying human rights-based approaches and promoting gender equality – priorities that cut across all areas of its work. It also accomplishes its objectives through its five priority focus areas, which are the following:

1) Young child survival and development

2) Basic education and gender equality

3) HIV/AIDS and children

4) Child protection

5) Policy advocacy and partnerships for children’s rights

The MTSP includes results matrices that further specify the organisational targets, indicators, areas of cooperation, and key partnerships to be pursued for each focus area, as well as an integrated monitoring and evaluation framework.

Finances

UNICEF is funded entirely by voluntary contributions that derive from governments, corporations, civil society organisations, and more than 6 million donors around the world. In 2010, its annual income from such contributions amounted to US$3.682 million. Although UNICEF reported an increase of 13 percent in its annual income from the previous year, the share of its regular (core) resources income has declined since 2002.

Organisational Improvement Initiatives

In response to an independent organisational review that was commissioned by the organisation in 2007, UNICEF has implemented a number of measures that aim to enhance its capacity to adapt, harness emerging opportunities, and maximise delivery of results for children while applying a rights-based approach to programming. Improvements required to both management and programming were grouped into four categories: improved accountability and effective risk management, programme excellence, operations performance, and effective resource planning and implementation.

The UNICEF website is www.unicef.org.

1.3 Previous Assessments

MOPAN Survey 2006

UNICEF was assessed by MOPAN in 2006 using only a survey-based assessment and with questions different than those that have been standardised since 2009 as part of the MOPAN Common Approach. Nevertheless, it is useful to highlight a few key findings of the 2006 survey.

The survey was carried out with UNICEF’s direct partners and MOPAN donors based in ten countries: Burkina Faso, Colombia, Guatemala, Indonesia, Kenya, Mozambique, Nepal, Pakistan, Sri Lanka, and Uganda. Overall, UNICEF was perceived as a strong voice for children and women, pursuing clear priorities, and determined to demonstrate results and deliver on its promises in very difficult circumstances. This was balanced against the view that UNICEF was not always fully sensitive to the local context or broader development context.

In terms of the quality of UNICEF’s relations with national stakeholders, survey findings indicated that UNICEF contributed positively to policy dialogue, provided good technical expertise, and had a strong comparative advantage in advocacy. Survey respondents had mixed opinions about UNICEF’s performance with regard to capacity development of various national stakeholders and its involvement of NGOs and the private sector in policy dialogue. UNICEF was perceived to require improvement in aligning itself with government modalities and procedures.

The survey findings indicated that UNICEF actively and regularly participated in local donor coordination groups, and had improved its responsiveness to local concerns, particularly as a result of increased decentralisation of decision-making authority to country offices. Survey respondents also noted that UNICEF had rather modest performance in terms of sharing information, and was slow to adapt to local aid harmonisation efforts.

MOPAN Common Approach 2009

In 2009 UNICEF was assessed by the MOPAN Common Approach, which was introduced that year. Using a standardised set of survey questions, UNICEF was assessed by MOPAN donors at headquarters, as well as direct partners and MOPAN donors based in nine countries: Ethiopia, Guatemala, Mozambique, Pakistan, Peru, Senegal, Serbia, Thailand and Uganda.

UNICEF was again recognised by respondents for the clarity and strength of its mandate, its positive relationships with government partners, and its operational capacity in development and humanitarian contexts. It was seen to be efficient and to get things done on the ground. UNICEF’s strongest ratings in 2009 related to organisational practices and behaviours that had significant repercussions at the country level: delegation of decision making, results focus of country programming, and valued contributions to policy dialogue. UNICEF’s lowest rating was the extent to which the organisation used government systems for procurement, audit, financial reporting, and other procedures. 

2. MOPAN Methodology – 2012

2.1 Overview

Background

MOPAN continues to refine its assessment framework. In 2009, the MOPAN Common Approach replaced the Annual MOPAN Survey, which had been conducted since 2003. The Common Approach is broader and deeper than the previous surveys and includes the following components:

• Expanded survey – The MOPAN survey now brings in the views of direct partners or clients of multilateral organisations, peer organisations (or other relevant stakeholder group), and those of donors, that is, MOPAN members at both headquarters and country level.

• Document review – Since 2010, survey data are complemented by a review of documents prepared by the multilateral organisations being assessed and other sources.

• Interviews – In 2012, MOPAN complemented survey data and document review with consultations and interviews at the headquarters of multilateral organisations assessed.

In 2012 MOPAN also tested a new component to assess the evidence of results of multilateral organisations.[2]

As MOPAN’s methodology has changed significantly in the last three years, comparisons of this year’s assessments and previous assessments should take this into consideration.

The following is a summary of the MOPAN methodology in 2012.[3]

MOPAN 2012

In 2012, MOPAN assessed the effectiveness of six multilateral organisations: the African Development Bank (AfDB), GAVI Alliance (formerly the Global Alliance for Vaccines and Immunisation), the Joint United Nations Programme on HIV/AIDS (UNAIDS), the United Nations Development Programme (UNDP), the United Nations Children’s Fund (UNICEF), and the World Bank. The assessment was conducted in Cambodia, Democratic Republic of Congo, Ghana, Honduras, Philippines, Morocco, Niger, Nigeria, and Zimbabwe.[4]

The MOPAN Common Approach examines organisational systems, practices, and behaviours that MOPAN believes are important for aid effectiveness and that are likely to contribute to results at the country level. It groups these organisational capacities in four areas of performance: strategic management, operational management, relationship management, and knowledge management. 

Key Performance Indicators and Micro-indicators – Within each performance area, organisational effectiveness is described using key performance indicators (KPIs) that are measured with a series of micro-indicators (MIs).

The micro-indicators are assessed using data from a survey and document review. The survey collects perception data from a variety of stakeholders (see Section 2.2) and the review of documents relies on a set of criteria that provide a basis for the assessment of each micro-indicator (see Section 2.3). However, not all micro-indicators are assessed by both the survey and the document review; consequently, some charts do not show survey scores and document review scores for each KPI or MI.

UNICEF was assessed using 21 KPIs and 77 MIs. The full list of MIs assessed is provided in Volume II, Appendix V (KPI and MI Data by Quadrant).

2.2 Survey

To gather diverse perspectives on the multilateral organisations being assessed, MOPAN generally seeks the perceptions of the following primary respondent groups:

• Donor Headquarters Oversight (HQ): Professional staff, working for a MOPAN donor government, who share responsibility for overseeing / observing a multilateral organisation at the institutional level. These respondents may be based at the permanent mission of the multilateral organisation or in the donor capital.

• Donor Country Office Oversight (CO): Individuals who work for a MOPAN donor government and are in a position that shares responsibility for overseeing/observing a multilateral organisation at the country level.

• Direct Partner (DP): Individuals who work for a national partner organisation (government or civil society) in a developing country. Respondents are usually professional staff from organisations that receive some sort of direct transfer from the multilateral organisation or that have direct interaction with it at country level (this could take the form of financial assistance, technical assistance, policy advice, equipment, supplies, etc.).

• Peer organisations (PO): Peer organisations were added specifically to answer survey questions related to humanitarian response and cluster coordination. This group includes field-based representatives of UN organisations and international NGOs that make significant investments in humanitarian programming, operate at the country level, and coordinate work with UNICEF. The peer organisations were also asked to respond on questions related to cross-cutting themes.

MOPAN donor respondents are chosen by MOPAN member countries. The other respondents are identified by the multilateral organisation being assessed.

The survey is customised for each organisation assessed and can be completed online in English, French, or Spanish or offline (paper, email, or interview) in these same languages. See Volume II (Appendix II) for the survey. Individual responses to the survey are confidential to the independent consultants managing the online survey or collecting data offline in the field.

Respondent Ratings – Survey respondents are presented with statements describing an organisational practice, system, or behaviour and asked to rate the organisation’s performance on a scale of 1 to 6 as shown below.

Figure 2.1 Respondent Rating Scale

In some cases, not all survey questions are answered, either because: 1) the individual chose not to answer, or 2) the question was not asked of that individual. In these cases, mean scores are calculated using the actual number of people responding to the question. As noted in the methodology (Volume II, Appendix I), ‘don’t know’ survey responses are not factored into the calculation of mean scores. However, when the proportion of respondents answering ‘don’t know’ is considered notable for a micro-indicator, this is indicated in the report. The responses of various categories of respondents on the six choices, plus ‘don’t know’ are summarised across all survey questions in Figure 2.2.

Figure 2.2 UNICEF- Distribution of Responses (n=294) on all Questions Related to Micro- Indicators

While there were responses in all six possible choices, relatively few responses overall were at the ‘weak’ end of the scale. MOPAN donors in-country and at headquarters had higher proportions of ‘don’t know’ responses than the other respondent groups.

Survey Response Rate

MOPAN aims to achieve a 70 per cent response rate from donors at headquarters and a 50 per cent response rate among the population of respondents in each of the survey countries (i.e., donors in-country, direct partners, and peer organisations). The number of respondents targeted in each category (i.e., the total population) and the actual response rates are presented in Figure 2.3 below. Response rates of all categories of respondents exceeded the targets. While there are variations in the response rates by category and location of respondents, UNICEF survey results reflect the views of 294 respondents.

Figure 2.3 Number of Survey Respondents and Total Population for UNICEF by Country and Respondent Group

Converting Individual Scores to Group Ratings

As noted above, individuals respond to survey questions on a six-point scale where a rating of “1” is considered a judgment of “very weak” up to a rating of “6” intended to represent a judgment of “very strong.” A mean score is calculated for each respondent group (e.g., donors at HQ). Since mean scores are not necessarily whole numbers (from 1 to 6) MOPAN assigns numerical ranges and descriptive ratings for each range (from very weak to very strong) as shown below.

Figure 2.4 MOPAN Ranges and Descriptions

The ranges are represented to two decimal places, which is simply the result of a mathematical calculation and should not be interpreted as representing a high degree of precision. The ratings applied to the various KPIs should be viewed as indicative judgments rather than precise measurements.

Data Analysis

First level survey data analysis includes calculations of mean scores, medians, standard deviations, frequencies, (including analysis of ‘don’t know’ and missing responses), as well as content analysis of open-ended questions. The ‘don’t know’ responses are removed from the calculation of mean scores, but the proportion of respondents choosing ‘don’t know’ is retained as potentially useful data.

A weighting scheme is applied to ensure that no single respondent group or country is under-represented in the analysis. The weighting is intended to correct for discrepancies/variation in: the number of individuals in each respondent group, the number of countries where the survey took place, the numbers of donors in-country, direct partners, and other respondent groups within each country where the survey took place. Weighted figures are carefully reviewed and analysed before inclusion in the multilateral organisation reports.

Second level analysis examines differences in the responses among categories of respondents and other variables. When significant differences are found, these are noted in the report.[5]

For a full description of survey data analysis see Volume II, Appendix I.

2.3 Document Review

The document review considers three types of documents: multilateral organisation documents, identified with the help of the organisation; internal and external reviews of the organisation’s performance, found on the organisation’s web site or provided by the organisation; external assessments such as the Survey on Monitoring the Paris Declaration, the Common Performance Assessment (COMPAS) report, and previous MOPAN surveys.

Ratings for key performance indicators (KPIs) are based on the ratings for the component micro-indicators in each KPI. For each micro-indicator, a set of criteria are established which, taken together, are thought to represent good practice in that area. The criteria are based on existing standards and guidelines (for example, UNEG or OECD-DAC guidelines), on MOPAN identification of key aspects to consider, and on the input of subject-matter specialists. The rating on any micro-indicator depends on the number of criteria met by the organisation. In cases where the micro-indicator ratings for one KPI are highly divergent, this is noted in the report.

While the document review assesses most micro-indicators, it does not assign a rating to all of them (when criteria have not been established). Consequently, some charts do not show document review scores for each KPI or MI. Documents are also used to aid in the understanding of the context in which the multilateral organisations work.

The document review and survey use the same list of micro-indicators, but some questions in the document review are worded differently from those in the survey. The document review and survey also use the same rating scale, but scores are presented separately on each chart in the report to show their degree of convergence or divergence.

2.4 Interviews

As of 2012, interviews are conducted at the headquarters of multilateral organisations with individuals who are knowledgeable in areas that relate to the MOPAN assessment.

Interviewees are asked to provide knowledge, insight, and contextual information that could assist the MOPAN Assessment Team in analysing document review data, and to identify other relevant documents for the Assessment Team to consider. This helps ensure that the Assessment Team has all the appropriate and necessary documents, enhances the Team’s ability to triangulate data from various sources, and assists the Assessment Team in the analysis of the key performance indicators by providing contextual information.

Interviews are conducted with a small number of staff who work in the primary units that relate to areas of the MOPAN assessment. Interviewees are identified by the multilateral organisation in conjunction with the Assessment Team and MOPAN. An interview guide is prepared and interviewees are advised of the content areas beforehand.

Data gathered during interviews is used to understand the context in which the agency is working, as well as how decisions are made. In the event that survey data present a picture that is very different from the document review, information from interviews can help clarify how the multilateral organisation approached a certain issue.

2.5 Basis for Judgment

From 2003 to 2009, the basis for judgment in MOPAN assessments was the perceptions of survey respondents. With the introduction of the document review in 2010 and interviews in 2012, judgments now draw on a variety of sources that can be compared and triangulated.

To the extent possible, the assessment standards and criteria are tailored to reflect the nature and operating environment of the multilateral organisations under review.

The MOPAN approach uses multiple data sources and data collection methods to validate findings. This helps eliminate bias and detect errors or anomalies.

The MOPAN reports gain trustworthiness through the multiple reviews and validation processes that are carried out by members of the network and by the multilateral organisations themselves.

2.6 Reporting

Institutional Reports

Individual institutional reports are produced for each multilateral organisation assessed. The results of the document review are presented alongside the survey results and discussed in light of the perception-based scores and interviews in order to further substantiate and contextualise the overall findings. For those agencies that were evaluated in 2009, a brief analysis of trends is included.

Country Data Summaries

A summary of survey results is produced for each multilateral organisation in each of the countries surveyed where sufficient survey data exists. Country Data Summaries provide feedback to those who participated in the MOPAN assessment and provide input for a dialogue process. They are not published and are shared only with individuals who attend the country workshop on the MOPAN assessment findings, which takes place in the first quarter of the year following the assessment.

2.7 Strengths and Limitations of Methodology

MOPAN continues to improve methodology based on the experience of each year of implementation. The following strengths and limitations should be considered when reading MOPAN’s report on UNICEF.

Strengths

• The MOPAN Common Approach is based on the core elements of existing bilateral assessment tools. In the long term, the intent is to replace or reduce the need for other assessment approaches by bilateral donors.

• It seeks perceptual information from different perspectives: MOPAN donors (at headquarters and in-country), direct partners/clients of multilateral organisations, peer organisations, and other relevant stakeholders. This is in line with donor commitments to the Paris Declaration on Aid Effectiveness and the Accra Agenda for Action regarding harmonisation, partner voice, and mutual accountability.

• It complements perceptual data with document review and interviews, thus using multiple sources of data. This should enhance the analysis, provide a basis for discussion of agency effectiveness, and increase the validity of the assessment through triangulation of data.

• The reports undergo a validation process, including multiple reviews by MOPAN members and review by the multilateral organisation being assessed.

• MOPAN strives for consistency across its survey questions and document review for each of the multilateral organisations, while allowing for customisation to account for differences between types of multilateral organisations.

Limitations

MOPAN Framework

• The countries are selected based on established MOPAN criteria and comprise only a small proportion of each institution’s operations, thus limiting broader generalisations.

• The Common Approach indicators were designed for multilateral organisations that have operations in the field. For organisations that have limited field presence or that have regional structures in addition to headquarters and country operations, there have been some modifications made in the data collection method and there will be a need for greater nuance in the analysis of the data.

Data Sources

• The MOPAN Common Approach asks MOPAN members and the organisations assessed to select the most appropriate individuals to complete the survey. While MOPAN sometimes discusses the selection with the organisation being assessed, it has no means of determining whether the most knowledgeable and qualified individuals are those that complete the survey.

• The document review component works within the confines of an organisation’s disclosure policy. In some cases, low document review ratings may be due to unavailability of organisational documents that meet the MOPAN criteria (some of which require a sample of a type of document, such as country plans, or require certain aspects to be documented explicitly). When information is insufficient to make a rating, this is noted in the charts.

Data Collection Instruments

• Three issues potentially affect survey responses. First, the survey instrument is long and a fatigue factor may affect responses and rates of response. Second, respondents may not have the knowledge to respond to all the questions (e.g., survey questions referring to internal operations of the organisation, such as financial accountability and delegation of decision-making, seem difficult for many respondents, who frequently answer ‘don’t know.’) Third, a large number of ‘don’t know’ responses may imply that respondents did not understand certain questions.

• The rating choices provided in the MOPAN survey may not be used consistently by all respondents, especially across the many cultures involved in the MOPAN assessment. One potential limitation is ‘central tendency bias’ (i.e., a tendency in respondents to avoid extremes on a scale). Cultural differences may also contribute to this bias as respondents in some cultures may be unwilling to criticise or too eager to praise.

• Because one of MOPAN’s intentions is to merge previously existing assessment tools into one, and to forestall the development of others, the survey instrument remains quite long.

Data Analysis

• While the document review can serve to evaluate the contents of a document, it cannot assess the extent to which the spirit of that document has been implemented within the organisation (unless implementation is documented elsewhere).

• Mean scores are used in the MOPAN reports to provide central tendency values of the survey results. The mean has the advantage of being the most commonly understood measure of central tendency, however, there is a disadvantage in using the mean because of its sensitivity to extreme scores (outliers), particularly when population samples are small. The assessment team reviewed the median and standard deviations in analysing the survey results. Volume II, Appendix V provides the standard deviations for each survey question.

Basis for Judgment

• Although MOPAN uses recognised standards and criteria for what constitutes good practice for a multilateral organisation, such criteria do not exist for all MOPAN indicators. As a result, many of the criteria used in reviewing document content were developed by MOPAN in the course of the assessment process. The criteria are a work in progress and should not be considered definitive standards.

• The Common Approach assessment produces numerical scores or ratings that appear to have a high degree of precision, yet can only provide general indications of how an organisation is doing and a basis for discussion among MOPAN members, the multilateral organisation, and other stakeholders, including direct partners.

Despite some limitations, the Assessment Team believes that the MOPAN reports generally provide a reasonable picture of systems associated with the organisational effectiveness of multilateral organisations.

2.8 Testing a New Component in 2012: Assessing the Achievement of an Organisation’s Results

Overview

Until 2012, MOPAN assessments focused on the organisational effectiveness of multilateral organisations by examining organisational practices, systems, and behaviours that MOPAN believes are important for managing to achieve development results. In 2012, MOPAN tested a component to assess how multilateral organisations report on development results achieved – with four of the six organisations assessed: AfDB, UNDP, UNICEF, and the World Bank.[6]

Sub-Components

• KPI A – Evidence of extent of progress towards organisation-wide outcomes[7]

• KPI B – Evidence of extent of contributions to country-level goals and priorities

• KPI C – Evidence of extent of contributions to relevant MDGs

• KPI D – Relevance of objectives and programme of work to stakeholders.

The assessments at the institutional/organisational level (KPI A) and at the country level (KPI B) are separated due to differences in focus, scope and reporting on results at these two levels.

KPIs B, C, and D all focus on the country level and are applied in five countries. Each multilateral organisation is asked to identify the countries where they are likely to have the best data on results.

KPI D assesses relevance as a measure of the extent to which surveyed stakeholders consider the multilateral organisation supports country priorities and meets the changing needs of direct partners and the target population.

Methodology

Various types of qualitative and quantitative data are sought to answer a set of questions about the multilateral organisation’s performance in the indicators noted above. Data are collected using three different methods: document review, stakeholder survey, and interviews with staff at HQ and, if feasible, in country offices.

• The document review draws largely on organisational performance reports and country level performance reports and evaluations.

• The stakeholder survey asks donor and direct partner respondents to rate the organisation’s achievement of planned results and the relevance of its activities at the country level. The questions are tailored, as required, to each organisation.

• Interviews are conducted to identify reliable data, identify key staff to consult in country offices, and to help contextualise the analysis of results.

Analysis of institutional level data focuses on the extent to which planned results from the strategic period were achieved, based largely on performance reports and thematic evaluations. Analysis of country level data focuses on the organisation’s contribution to results in the sample of countries selected for the MOPAN assessment.

The judgment of an organisation’s performance on each KPI draws on a set of criteria. The Assessment Team uses a “best fit approach,” a criteria-referenced basis for judgment that is suitable when criteria are multi-dimensional, there is a mix of both qualitative and quantitative data, and it is not possible to calculate a simple sum of the data points.[8] This approach is highly consultative (with institutional advisors, a panel of experts and the MOPAN network) and relies on consensus in the determination of ratings.

The ratings reflect the Assessment Team’s judgment following extensive consultation and analysis of data from all sources. The ratings are qualitative in nature and defined according to a 4-point scale – strong, adequate, inadequate, or weak. As in the 6-point scale used in the survey, a rating of “strong” signals that the organisation is approaching good practice, while a rating of “weak” signals that the organisation still has important limitations in demonstrating progress or contributions.

Limitations to the Methodology

The methodology was designed to draw on the evidence of results achieved, as presented in the reports of a multilateral organisation. However, there is a critical difference between assessing the actual results achieved on the ground and assessing the evidence of results in the organisation’s reports to its key stakeholders. This is a limitation that is inherent in the current approach.

Further, the sample size is small as documentation from only 5 countries is reviewed for this pilot component.

MOPAN will review the experience with this pilot component and make adjustments in the methodology, as required.

3. Main Findings: Practices and Systems that Enable the Achievement of Results

3.1 Introduction

This chapter presents the findings of the 2012 MOPAN assessment of UNICEF’s organisational effectiveness, that is its practices and systems that support the achievement of results. Findings are based on respondent survey data and document review.

• Section 3.2 presents overall ratings on the performance of UNICEF and summarises respondent views on its primary strengths and areas for improvement;

• Section 3.3 provides findings on each of the four areas of performance (strategic, operational, relationship, and knowledge management).

3.2 Overall Ratings

This section provides a summary of overall ratings. It includes: survey respondent ratings of UNICEF’s overall organisational effectiveness, survey respondent views on UNICEF’s strengths and areas for improvement, and survey and document review ratings for all key performance indicators.

Survey Ratings of UNICEF’s Organisational Effectiveness

MOPAN has defined “organisational effectiveness” as the extent to which a multilateral organisation is organised to support direct partners in producing and delivering expected results. Respondents were asked the question: “How would you rate the overall organisational effectiveness of UNICEF?” As shown in Figure 3.1, direct partners gave the largest proportion of ratings on the high end of the scale, with one-quarter of the respondents rating UNICEF as very effective.

Figure 3.1 Overall Ratings of UNICEF Organisational Effectiveness by Respondent Group 

Respondents’ Views on UNICEF’s Strengths and Areas for Improvement

The survey included two open-ended questions that asked respondents to identify UNICEF’s greatest strengths and areas of improvement. All 294 respondents answered both questions.[9] Their comments are summarised below with an analysis of the level of homogeneity among respondent groups as well as areas where the respondent groups felt differently about certain issues.

Overall, survey respondents considered UNICEF’s greatest strengths to be its field presence and engagement with governmental and non-governmental partners. Other areas of strength included UNICEF’s expertise and professionalism, as well as its strategic and programmatic approaches.

Nearly half of the respondents (46 per cent) indicated that UNICEF’s greatest strength lies in its strong working relationships with its partners. Respondents noted that UNICEF has a significant field presence and close proximity to local actors on the ground. Respondents emphasized that UNICEF collaborates with governments and other partners in a highly participatory and consultative manner. The organisation’s support to its partners and capacity building were also highlighted, particularly in relation to governments.

The second most frequently cited strength was UNICEF’s technical expertise and credibility, recognised by 30 per cent of the respondents. Respondents highlighted the organisation’s professionalism, credibility and expertise in the field. Many respondents noted UNICEF’s high public profile and excellent international reputation.

UNICEF’s strategic and programmatic approaches (including thematic focuses) were recognised as a strength in 22 per cent of the respondents. Respondents noted that UNICEF’s interventions rightly focus on children and women, and among these, on the most vulnerable. They highlighted UNICEF’s extensive work in conflict and post-conflict environments and its renewed focus on an equity-based approach to programming. UNICEF’s focus on human rights-based approaches and gender equality were also noted.

UNICEF’s administrative procedures and processes were highlighted as the primary area for improvement, followed by results-based monitoring, evaluation and reporting procedures.

Overall, 39 per cent of the respondents felt that one area in need of improvement was UNICEF’s administrative procedures and processes. Many respondents felt that UNICEF’s administrative procedures are heavy and inflexible, and some also felt that UNICEF is slow to release funds to its partners.

UNICEF’s monitoring, evaluation, and reporting procedures were identified as an area for improvement by 17 per cent of respondents. They indicated that UNICEF could strengthen its monitoring of results on the ground. Some respondents suggested that UNICEF could improve its evaluations of interventions so as to foster a more efficient use of resources as well as to improve the sustainability of results.

Three other areas for improvement, each of which was identified by 15 per cent of respondents, included:

• UNICEF’s engagement with partners in planning processes – Although UNICEF’s working relationships with partners were considered a strength by many respondents, others suggested that UNICEF needs to consult and collaborate more with its partners, particularly civil society organisations. This was mentioned in regard to both the planning and implementation phases of UNICEF’s programmes and projects.

• UNICEF’s coordination and coherence with other actors in the delivery of services – This was most frequently cited in relation to UNICEF’s coordination with other UN agencies, particularly its participation in the ‘Delivering as One’ initiative.

• UNICEF’s strategic and programmatic approaches (including thematic focuses).

Overall Ratings of Key Performance Indicators

Figure 3.2 below shows scores from the document review and the survey on key performance indicators (KPIs) in the MOPAN 2012 assessment of UNICEF’s practices and systems. The white bar presents the survey score, while the black square presents the document review score. For example, on the first indicator, “providing direction for results”, UNICEF received a score of 4.67 (strong) in the survey and a score of 6 (very strong) in the document review.

In the overall ratings from the survey and document review, UNICEF was seen to perform adequately or better on the majority of key performance indicators.

UNICEF received scores of adequate or better on all 21 KPIs assessed in the survey. UNICEF received scores of adequate or better on 13 of the 17 KPIs assessed by the document review.[10]

The survey and document review ratings differed on 13 KPIs – seven of which were rated lower by the document review than by survey respondents, and the opposite for the remaining six. The reasons for these differences are discussed in the following sections.

Figure 3.2 Overall Ratings on Key Performance Indicators of UNICEF Practices and Systems (mean scores, all respondents and document review ratings)

3.3 UNICEF’s Performance in Strategic, Operational, Relationship, and Knowledge Management

3.3.1 Overview

This section presents the results of the 2012 Common Approach assessment of UNICEF in four performance areas (quadrants): Strategic, Operational, Relationship, and Knowledge Management.

The following sections (3.3.2 to 3.3.5) provide the overall survey and document review ratings for the KPIs in each quadrant, the mean scores by respondent group, and findings based on an analysis of survey and document review ratings in each quadrant.

When there were notably divergent ratings between survey respondent groups or between the survey results and document review ratings, these are noted and the information gleaned from interviews with staff is integrated when it has a bearing on the analysis. Where statistically significant differences among categories of respondents were found, these differences are noted.

The survey data for each KPI and MI by quadrant are presented in Volume II, Appendix V. The document review ratings are presented in Volume II, Appendix VI.

3.3.2 Strategic Management

Strategic management is one of UNICEF’s strengths; survey respondents rated UNICEF as strong in this area and the document review concurred with ratings of adequate to very strong on the different MIs.

Figure 3.3 shows the overall survey and document review ratings for the four KPIs in the strategic management quadrant. UNICEF was judged by survey respondents to be strong. The document review ratings ranged from adequate to very strong.

Overall, the evidence indicates that UNICEF has developed a results-oriented culture although it received lower ratings in the document review on its capacity to develop a results-focused organisation-wide strategy. UNICEF is performing well in addressing cross- cutting priorities related to gender equality, human rights-based approaches, and emergency response/humanitarian action, but received slightly lower ratings for its efforts to mainstream the environment, good governance, and conflict management.

Figure 3.3 Quadrant I: Strategic Management, Survey and Document Review Ratings

Figure 3.4 shows the mean scores for the four KPIs for all survey respondents, and by category of respondent.

Figure 3.4 Quadrant I: Strategic Management, Mean Scores by Respondent Group

KPI 1: Providing Direction for Results

Finding 1: Survey respondents felt that UNICEF’s executive management provides strong direction for the achievement of external/beneficiary-focused results and the document review supports this view.

Overall, UNICEF was perceived to be strong in reinforcing a results-orientation and a direct- partner focus, and demonstrating leadership on results management, as well as making key documents available to the public. In general, direct partners gave UNICEF higher ratings than donors in-country.

The Agency’s practice of making key documents available to the public was considered strong by survey respondents and very strong by the document review.

Figure 3.5 KPI 1: Providing Direction for Results, Ratings of Micro-Indicators

MI 1.1 – Value system supports results-orientation and direct partner focus

In this MI, which was assessed by survey only, survey respondents were asked whether UNICEF’s institutional culture reinforces a focus on results, and if UNICEF’s institutional culture is direct-partner focused. The majority of respondents across respondent groups (83 per cent) rated UNICEF as adequate or above to both questions, however, direct partners were more positive than MOPAN in-country donors on both questions. The differences are statistically significant.[11]

MI 1.2 – Leadership on results management

This MI was assessed by survey only. Among MOPAN donors at headquarters, the only group asked about this MI, 90 per cent rated UNICEF as adequate or above in showing leadership on results management.

MI 1.3 – Key documents available to the public

The majority (83 per cent) of survey respondents rated UNICEF as adequate or higher in making key documents readily available to the public, and 14 per cent gave ratings of inadequate. Direct partners were more positive than donors in-country and the differences are statistically significant.

The document review found UNICEF very strong in this area. UNICEF has a disclosure policy that specifies the types of documents that are made available to the public (such as financial information, country programme documents, annual reports, evaluation reports, and information relating to the work of the UNICEF Executive Board) and those which are confidential (such as internal audits and internal communications). This policy was updated in October 2012 to include the publication of internal audits. Most UNICEF documents are available in the working languages of UNICEF: English, French and Spanish.

KPI 2: Corporate Focus on Results

Finding 2: Survey respondents perceived UNICEF’s focus on results as adequate. The document review suggests that UNICEF has a strong focus on results-based management but that there is room for improvement in the quality of its results frameworks.

MOPAN donors at headquarters were the only respondents asked to assess UNICEF’s performance under this KPI. They rated UNICEF strong on having an organisation-wide strategy that is based on a clear definition of mandate. They rated the organisation as adequate in terms of ensuring the application of results management across the organisation.

The document review ratings on this KPI vary from inadequate to strong. It found that UNICEF has taken the initiative to promote results management across the organisation through the development of policies, tools and training. However, some improvements need to be made in presenting causal linkages and in phrasing results statements and selecting indicators in the agency-wide development results framework (DRF) and management results framework (MRF). This may be addressed in the roll-out of VISION, a transaction and performance management system designed to improve the management of monitoring programme performance.

Figure 3.6 KPI 2: Corporate Focus on Results, Ratings of Micro-Indicators

MI 2.1 – Organisational strategy based on clear mandate

Donors at headquarters were asked two questions on this MI: whether UNICEF has a clear mandate, and whether UNICEF’s organisation-wide strategy/strategies are aligned with the mandate. The majority of respondents (99 per cent) rated UNICEF as adequate or above, and 91 per cent gave ratings of strong and very strong on both questions.

While there is no single document that clearly articulates the mandate and all its nuances, UNICEF received a rating of strong from the document review, based on solid evidence that UNICEF’s mandate was revised to ensure its continued relevance. UNICEF’s mandate was established and revised by General Assembly resolutions (e.g., 1946, 1950 and 1953). Although its mandate is dated, UNICEF’s mission has continued to evolve. The Medium- term Strategic Plan (MTSP) 2006-2009 is implemented through a set of four guiding documents[12] that also serve as the basis for the formulation of the organisation’s five focus areas.

MI 2.2 – Organisational policy on results management

In the survey, donors at headquarters were asked if UNICEF ensures the application of results management across the organisation. The majority of respondents (83 per cent) rated UNICEF as adequate or above, and 12 per cent as inadequate or below on this MI.

Based on a review of its policy framework for results-based management (RBM), UNICEF received a rating of strong in the document review. In addition to promoting RBM in its MTSP 2006-2009, UNICEF has a clear and coherent results-based management policy that describes the nature and role of RBM in programme processes such as the Common Country Assessment (CCA), the United Nations Development Assistance Framework (UNDAF), Country Programme Document (CPD), Country Programme Action Plans (CPAP), and the Integrated Monitoring and Evaluation Plan (IMEP). Moreover, UNICEF is implementing VISION, a transaction and performance management information system that includes a “dashboard” on programmes and operations and that allows users to store and generate performance reports. UNICEF is also committed to the implementation of RBM by partners at the country level, as is evidenced in a series of documents completed by partners (e.g., Project Cooperation Agreement and report formats). Evidence was found of updates/improvements in the documentation of and support to staff on RBM through the programme manual, however, insufficient evidence was found of a holistic approach to training on RBM.

MI 2.3 – Plans and strategies contain results frameworks

This MI was assessed by document review only.

According to the document review, UNICEF’s inclusion of a results framework in its organisation-wide plans and strategies is adequate. UNICEF has a development results framework, a related results matrix, and an integrated monitoring and evaluation framework that form an Appendix to the Medium-Term Strategic Plan 2006-2009 and that contain statements at different levels of results – known in UNICEF’s lexicon as ‘areas of cooperation’ and ‘key result areas’. The DRF has been updated every two years since it was approved in 2006 – specifically, in 2008, 2010 and 2012. There are several areas for improvement in the content of the framework. Some results statements are quite complex; they contain elements that could be defined as activities, outcomes and impacts all in one statement. UNICEF does not have a standard management results framework but, rather, a table of key performance indicators that relate to management activities. While the table contains indicators, baselines, and targets, these are not associated with outputs or outcomes.

MI 2.4 – Results frameworks link outputs to final outcomes/impacts

This MI was assessed by document review only.

The document review gave UNICEF a rating of inadequate on this MI. The results chain in UNICEF’s development results framework (results matrix appended to the Medium-Term Strategic Plan) is poor. The Medium-Term Strategic Plan 2006-2009 does not explain how lower level results are linked to higher level results, nor does it explain the assumptions or external factors that may be encountered in the implementation of certain interventions. In short, there is no apparent theory of change or logic model. Instead of presenting a theory of change, the strategy includes a situation analysis, targets, lessons learned, and partners involved in certain activities.

Further, key performance indicators listed in the annex to the MTSP do not provide plausible links between management outcomes and impacts. UNICEF tracks a relatively wide spectrum of management indicators – related to human resources, finance, information and communication technology, supply, programme oversight, risk management practices and evaluations – against baselines and targets without identifying expected results. However, the new Institutional budget for 2012-2013, does include organisational outputs for the key performance indicators contained in the MRF.

MI 2.5 – Plans and strategies contain performance indicators

This MI was assessed by document review only.

The document review rated UNICEF as inadequate on this MI as the majority of UNICEF performance indicators are neither adequate nor relevant. Since there is confusion in the wording and label of results, it is difficult to assess whether the indicators provide a sufficient basis to measure performance. Also, UNICEF’s development results framework only provides one set of indicators which are not associated with baselines and targets. The management results framework includes performance baselines, targets, and indicators that are mostly clear and measurable, but are not associated with any results. The assessment team recognises the improvements that UNICEF has made through updates and fine tuning of indicators following biannual revisions of the annexes to the MTSP (2008, 2010, and 2012).

KPI 3: Focus on Thematic Priorities

Finding 3: UNICEF is seen as adequate or strong in mainstreaming most cross- cutting priorities (gender, environment, human rights-based approaches, and emergency response/humanitarian action) but has put less emphasis on good governance and conflict management.

The assessment examined six cross-cutting thematic priorities identified by MOPAN and UNICEF: gender equality, environmental policy and environmental assessment practices, good governance, human rights-based approaches, conflict management, and emergency response/humanitarian action. All of these were assessed by both the document review and the survey.

Overall, survey respondents rated UNICEF as strong on four MIs and adequate on two. In general, direct partners were more positive than other respondent groups. The document review rated UNICEF adequate to very strong on four themes, but as inadequate in good governance and conflict management. UNICEF has given less emphasis to good governance and conflict management in its key strategic documents and this is reflected in the ratings. UNICEF has been proactive in its gender equality work and in emergency response/humanitarian action and received ratings of strong and very strong in the document review on those thematic priorities. The organisation has made considerable progress on these two cross-cutting issues by developing strategies, actions plans and monitoring tools.

Figure 3.7 KPI 3: Focus on Thematic Priorities, Ratings of Micro-Indicators

MI 3.1 – Gender equality

The majority of survey respondents (86 per cent) rated UNICEF as adequate or above for the extent to which it mainstreams gender equality in its programmatic work. It should be noted that while the mean score for direct partners was strong, almost half (45 per cent) of these respondents gave a rating of very strong, making the average score for this respondent group higher than the average scores of donors in-country and at headquarters. The differences are statistically significant.

The document review rated UNICEF as strong in its efforts at organisation-wide gender mainstreaming. In the 2008 Evaluation of Gender Policy Implementation in UNICEF, it was reported that UNICEF did not systematically implement its 1994 Gender Policy due to institutional weaknesses in that area. Since 2010, UNICEF has taken actions to improve gender mainstreaming. In its new Policy on Gender Equality and the Empowerment of Girls and Women 2010, UNICEF commits to mainstreaming gender in its operations and to develop accountability and monitoring mechanisms. Moreover, the Operational Guidance on Promoting Gender Equality through UNICEF-Supported Programming describes in great detail the various entry points to mainstream gender in UNICEF’s programme cycle. UNICEF also issued a Strategic Priority Action Plan for Gender Equality 2010-2012, which specifies concrete actions to be taken by relevant divisions to increase the organisation’s attention to gender equality. It is important to note that the organisation reports annually to the Executive Board on the progress of its gender equality work in various areas. According to the 2012 Report, UNICEF has made considerable advances in “systems-building” and dissemination of “technical guidance”, but still has to ensure that these mechanisms are applied. Therefore, UNICEF is taking concrete steps to improve its gender-related performance, but it remains to be seen if these organisational improvements will yield results.

MI 3.2 – Environment

Survey respondents were asked whether UNICEF sufficiently mainstreams environment in its work. The majority of survey respondents (66 per cent) rated UNICEF as adequate or above, 15 per cent as inadequate or below, and 19 per cent answered ‘don’t know’. It should be noted that 53 per cent of direct partners gave a rating of strong and very strong on this MI, making the average score for this respondent group higher than the average scores of donors in-country and at headquarters. The differences are statistically significant.

The document review rated UNICEF as adequate in its environmental policy and environmental assessment practices at an institutional level. UNICEF’s Strategy for Sustainable Development dates back to 1989 and still serves as the basis for the organisation’s approach on the environment. UNICEF has committed to environmentally responsible practices in its programming at the country level by conducting Situation Analyses at the country level and Environmental Impact Assessments at the project level. Country Offices are also mandated to report on their experience in promoting environmental sustainability in their annual reports. UNICEF integrates environmental practices into programmes and projects through a series of analytical processes and safeguards, but it is neither explicitly acknowledged as a cross-cutting priority in UNICEF’s organisation-wide strategy, nor monitored at the organisation-wide level to ensure continuous improvement of mainstreaming efforts.

In 2012, the Annual Report of the Executive Director of UNICEF included environmental sustainability as a cross-cutting strategy following self-assessments carried out by Country Offices in their annual reports.

MI 3.3 – Good governance

When asked whether UNICEF sufficiently promotes the principles of good governance in its work, the majority of survey respondents (79 per cent) rated it as adequate and above. Direct partners were more positive than MOPAN donors in-country and at headquarters and this difference is statistically significant.

Strictly adhering to the MOPAN criteria, UNICEF was rated inadequate in this area. The term ‘good governance’ is not mentioned explicitly as a cross-cutting priority in UNICEF’s MTSP 2006-2009 and documents made available do not commit the organisation to develop its institutional capacity to mainstream this cross-cutting theme (e.g., in planning, allocation of resources, monitoring and evaluation). Some elements of good governance – such as the strengthening of capacities in policy, legislative, and budgetary matters – are mentioned in the plan and the results framework, but there is no horizontal or vertical integration of this topic in UNICEF’s programming.

MI 3.4 – Human rights-based approaches

The majority of survey respondents (92 per cent) rated UNICEF as adequate or above on the extent to which it applies human rights-based approaches to development in its work. Direct partners were more positive than MOPAN donors in-country and the difference is statistically significant.

The document review assessed UNICEF’s human rights-based approach (HRBA) practices as adequate. Published in 1998, the Guidelines for Human Rights-Based Programming Approach constitute the foundation of UNICEF’s engagement with human rights-based approaches in its programming. HRBA is identified in the MTSP as a foundational strategy. The documents reviewed also showed that HRBA practices are embedded in various country-level guidelines (Situation Analyses, Country Programme Documents, the Programme Policy and Procedure Manual, and annual reports). Therefore, UNICEF is showing a strong commitment to integrate HRBA horizontally and vertically in its programming. Further, according to the recent Global Evaluation of the Application of a Human Rights Based Approach to UNICEF Programming (2012), UNICEF has played a leadership role in promoting human rights-based approaches at the global level and has made efforts to integrate HRBA principles across its five focus areas. However, the evaluation also noted that UNICEF should update its policy, provide further guidance to its staff and improve programmatic systems (i.e., planning, costing, and accountability) as they relate to the mainstreaming of this approach.

MI 3.5 – Conflict management

Survey respondents were asked whether UNICEF sufficiently considers other priorities such as conflict management in its work. The majority of survey respondents (66 per cent) rated UNICEF as adequate or above, 10 per cent as inadequate or below, and 24 per cent answered ‘don’t know’ on this MI. Direct partners were more positive than in-country donors and the difference is statistically significant.

Strictly adhering to the MOPAN criteria, UNICEF was rated as inadequate on this MI in the document review. Neither the MTSP 2006-2009 nor any other document made available to the Assessment Team explicitly commits UNICEF to develop its institutional capacity in the area of conflict management or to evaluate the application of policies in this area. However, UNICEF does give special attention to the effects of conflict in its programming (particularly in Focus Area 4), and considers conflict management in planning interventions, particularly in the Office of Emergency Programmes.

MI 3.6 – Emergency response/humanitarian action

When asked whether UNICEF sufficiently integrates emergency preparedness and response into its work, the majority of respondents (83 per cent) rated UNICEF as adequate or above.

The document review rated UNICEF as very strong on this MI. UNICEF’s approach to emergency response/humanitarian action is articulated in the 2010 Core Commitments for Children (CCCs) in Humanitarian Action. This document includes the framework, principles, programme commitments and accountabilities relevant to UNICEF’s work in this area. At an organisation-wide level, UNICEF integrates its humanitarian activities into each focus area by articulating specific key result areas related to humanitarian work and emergencies in its development results framework. To improve its emergency response, it has functioning monitoring systems and has also published a series of global evaluations on large-scale humanitarian responses, such as the earthquake in Haiti and Pakistan’s floods.

KPI 4: Country Focus on Results

Finding 4: Survey respondents agreed that UNICEF’s country strategies reflect a strong focus on results. The document review confirmed this assessment in its review of Country Programme Documents completed since 2009.

Survey respondents rated UNICEF strong on all five MIs assessed by the survey. Overall, direct partners were more positive than donors in-country.

The review of documents included the most recent Country Programme Documents and Country Programme Action Plans in the nine countries that were part of the assessment. The KPI received an overall rating of strong in the document review, which acknowledged the inclusion of results and performance indicators at the project, programme, sector and country levels and results consistent with UNDAF. However, it noted some shortcomings in including cross-cutting priorities in results framework.

Figure 3.8 KPI 4: Country Focus on Results, Ratings of Micro-Indicators

MI 4.1 – Frameworks link results at project, programme, sector and country levels

MOPAN donors in-country and direct partners were asked whether UNICEF's Country Programme Documents (CPDs/CPAPs) link results from project, sector and country levels. The majority of survey respondents (79 per cent) rated UNICEF as adequate or above and 17 per cent answered ‘don’t know’ on this MI. Direct partners were more positive than donors in-country, even though their ratings all fall within the range of strong. This difference is statistically significant.

The document review rated UNICEF’s performance as strong in ensuring that country programming frameworks link results from project/programme, sector and country levels, based on the review of the most recent Country Programme Documents, Country Programme Action Plans and Annual Work Plans from the nine countries participating in the 2012 MOPAN assessment. The newest country-level results matrices contain statements of expected results articulated at output and outcome levels – known as Intermediate Results (IR) and Programme Component Results (PCR) respectively. The various results matrices present a clear results chain and are a marked improvement on the previous frameworks in place prior to the organisational changes implemented in 2008/9. The country documents also explicitly link PCR and IR to the organisation-wide development results framework. The sample of CPDs/CPAPs reviewed contained strategies developed using updated sets of guidelines and different programme structures. The document review found that these new results frameworks represent a notable improvement over earlier frameworks in the identification of and linkages between outputs, outcomes and organisation–wide focus areas.

MI 4.2 – Frameworks include indicators at project, programme, sector and country levels

MOPAN donors in-country and direct partners were asked whether UNICEF's Country Programme Documents (CPD/CPAPs) include indicators at all levels (country, sector, and project/programme). The majority of respondents (77 per cent) rated this MI as adequate or above, and 17 per cent answered ‘don’t know’. Direct partners were more positive that donors in-country and the difference is statistically significant.

The document review rated UNICEF as strong on this MI. More than half of the sampled Country Programme Documents and Country Programme Action Plans have frameworks that include indicators at relevant levels. (This represents an improvement over documents from earlier programming cycles.) While most of the indicators are relevant, clear, and measurable, many performance indicators did not have data sources.

MI 4.3 – Expected results consistent with national development strategies and UNDAF

MOPAN donors in-country and direct partners were asked whether UNICEF's Country Programme Documents (CPDs/CPAPs) contain statements of expected results consistent with national development strategies and the United Nations Development Assistance Framework. The majority of respondents (80 per cent) rated this MI as adequate or above, and 17 per cent answered ‘don’t know’. Direct partners were more positive that donors in- country and the difference is statistically significant.

The document review rated UNICEF very strong. In all Country Programme Documents and Country Programme Action Plans reviewed, there is a clear and explicit link between UNICEF’s expected results and those identified in both national development strategies and the UNDAF. In most cases, the results framework contained rows or columns illustrating links to related national development outcomes, the UNDAF, and UNICEF’s expected results. CPDs and CPAPs also include direct references to the UNDAF. It is worth noting that UNICEF also undertakes external assessments of Country Programme Documents every year to ensure that they are aligned with the UNDAF.

MI 4.4 – Expected results developed in consultation with direct partners/beneficiaries

This MI was assessed by survey only. MOPAN donors in-country and direct partners were asked whether UNICEF consults with direct partners to develop its expected results. The majority of respondents (81 per cent) rated this MI as adequate or above. Direct partners were more positive than donors in-country and the difference is statistically significant.

MI 4.5 – Results for cross-cutting priorities included in results frameworks

MOPAN donors in-country and direct partners were asked whether UNICEF's Country Programme Documents (CPDs/CPAPs) include results related to cross-cutting priorities such as gender and environment. The majority of respondents (79 per cent) rated this MI as adequate or above, and 17 per cent answered ‘don’t know’. Direct partners were more positive than donors in-country and the difference is statistically significant.

The document review rated UNICEF as adequate on this MI. It revealed that all country documents (either CPDs or CPAPs) referred to at least four of the organisationally relevant cross-cutting themes (the same assessed in KPI 3), and all country documents incorporated at least two of these into their results frameworks. All countries sampled have evidence of strategies and approaches to address relevant cross-cutting issues and themes, although no country strategy has done so for all cross-cutting themes. Many of the strategies lack clarity in terms of their application. Human rights-based approaches and environment were two cross-cutting themes that were most often missing from country documents.

3.3.3 Operational Management

UNICEF is strong in some areas of operational management, most notably financial management, and has begun a process of improving systems that link aid management to performance.

Figure 3.9 below shows the overall survey and document review ratings for the KPIs in the operational management quadrant.

According to survey results, UNICEF performs adequately or above on all KPIs in operational management. In the document review, UNICEF was rated as adequate or better on six of the eight KPIs reviewed and inadequate on two.

UNICEF’s strongest performance in operational management relates to financial management. As part of the UN family, UNICEF has well-regarded financial accountability systems. It has steadily improved its audit function and has introduced a new system to assess and manage risks (Enterprise Risk Management). Major reforms to the cost classification and budgeting models have yet to be fully implemented. According to the implementation schedule, UNICEF will present a fully integrated budget by 2014.

UNICEF has proven very strong in its respect for humanitarian principles and work with partners in response to humanitarian crises. The organisation has also fared well in the use of performance information to improve its operations and the management of human resources (UNICEF has recently introduced a new performance appraisal process). However, it continues to face difficulties in linking aid management and performance. There is considerable room for improvement in the planning of projects at the field level, setting targets for monitoring, and in the application of results-based management practices.

Figure 3.9 Quadrant II: Operational Management, Survey and Document Review Ratings

Figure 3.10 shows the mean scores for the KPIs for all survey respondents, and by respondent groups.

Figure 3.10 Quadrant II: Operational Management, Mean Scores by Respondent Group*

* Peer organisation respondents were asked questions in KPIs related to humanitarian response and cross-cutting themes.

KPI 5: Resource Allocation Decisions

Finding 5: Overall, UNICEF was considered adequate in its allocation of resources and strong on the predictability and timely disbursement of funding at the country level.

Three MIs were assessed by the survey under this KPI. UNICEF was rated strong for the predictability of its resources; it was considered adequate in making its criteria for allocating resources publicly available and for complying with these criteria.

The document review found that UNICEF has criteria for the allocation of funding to Country Offices, but these are not made public. As a result, UNICEF was rated adequate on this MI. UNICEF was considered strong in releasing resources according to agreed schedules.

Figure 3.11 KPI 5: Resource Allocation Decisions, Ratings of Micro-Indicators*

* The document review was designed to draw data from the 2010 Survey on Monitoring the Paris Declaration.

MI 5.1 – Criteria for allocating resources publicly available

Survey respondents, except peer organisations, were asked whether UNICEF makes readily available its criteria for allocating resources. The majority of survey respondents (61 per cent) rated UNICEF as adequate or above, and 21 per cent as inadequate or below, and 18 per cent answered ‘don’t know’ on this MI. Direct partners were more positive than donors in-country and at headquarters, even though their ratings all fall within the range of adequate. The differences are statistically significant.

Strictly adhering to the MOPAN criteria, the document review assessed UNICEF’s practices as adequate on this MI. In 1997, UNICEF’s Executive Board approved a new system for allocating general resources to Country Programmes. This system follows a clear set of criteria and a multi-step approach to the allocation of funding to reflect the specificity of each country. Following Executive Board decisions, the system was reviewed with only minor modifications in 2003 and 2008 (another is planned for 2012). Although UNICEF’s formula for allocating resources is not clearly identified on its website, UNICEF publishes an annual Programme Planning Levels for Regular Resources document and a Report on Regular Resources. Further, it is possible to find these documents in multiple languages on the UN website.

MI 5.2 – Resource allocations conform to criteria

This MI was assessed by survey only. Survey respondents, except peer organisations, were asked whether UNICEF allocates resources according to the criteria. The majority of survey respondents (59 per cent) rated UNICEF as adequate or above, and 15 per cent as inadequate or below, and 25 per cent answered ‘don’t know’ on this MI. Direct partners were more positive than donors in-country and the difference is statistically significant.

MI 5.3 – Predictable resource commitments

This MI was assessed by survey only.

Survey respondents, except peer organisations, were asked whether UNICEF makes predictable commitments of core and programme funds at the global and country levels. The majority of survey respondents (70 per cent) rated UNICEF as adequate or above, and 23 per cent answered ‘don’t know’ on this MI. Direct partners were more positive than donors in-country and at headquarters, even though their ratings all fall within the range of adequate. This difference is statistically significant.

MI 5.4 – Resources released according to agreed schedules

This MI was assessed by document review only.

This indicator is based on Indicator 7 of the Paris Declaration on Aid Effectiveness, which measures the gap between aid scheduled and aid effectively disbursed and recorded in countries’ accounting systems. The document review has interpreted this indicator more broadly to consider the whole process for allocating and releasing funds at the country level and rated UNICEF as strong on this MI.

The planning of the Country Programming cycle and the approval of a 5-year budget by the Executive Board ensures a high level of predictability of funding at the country level. These budgets are made up of core/regular resources and other resources. Whereas the core/regular funding is guaranteed, the latter is often dependent on fundraising efforts and, as such, is not as secure as the core resources. Any changes to Country Programme budgets must be approved by the Executive Board following the Mid-Term Review or when developing a new Country Programme. Further, detailed budget allocations by sector and subsector are completed as part of the CPAP, which is formally approved/signed by the main government counterpart. Annual work plans include annual budgets and are completed and approved by the partner ministry responsible for activity in the sector.

Financial directives exist to govern the transfer of funds to government and non- governmental partners. UNICEF has adopted the ‘harmonised approach to cash transfers’ (HACT) and all country offices are required to assess the financial capacities and risk level of their partners following which they establish the most appropriate cash transfer modalities.

KPI 6: Linking Aid Management to Performance

Finding 6: Donors at headquarters considered UNICEF adequate in linking aid management to performance; the document review found room for improvement in this area and noted a number of initiatives underway to address weaknesses in results-based budgeting.

The assessment looked at two different dimensions related to this KPI: whether UNICEF links budget allocations to expected results, and whether its reports on results include the amount disbursed to achieve those results.

The survey component of this KPI was assessed only by donors at headquarters, who rated UNICEF as adequate overall.

The document review found UNICEF less than adequate in this KPI. While improvements in some areas are underway – notably in the application of results-based budgeting and new cost classifications for the institutional budget – there remains considerable room for improvement in linking disbursements to results achieved.

Figure 3.12 KPI 6: Linking Aid Management to Performance, Ratings of Micro-Indicators

MI 6.1 – Allocations linked to expected results

MOPAN donors at headquarters were asked whether UNICEF links budget allocations to expected results. Just under 60 per cent (59 per cent) of respondents rated UNICEF as adequate and strong, while 22 per cent gave ratings of inadequate or below, and 20 per cent answered ‘don’t know’ on this MI.

Strictly adhering to the MOPAN criteria for the document review, UNICEF was rated as inadequate in linking budget allocations to expected programmatic results because the budget in the MTSP 2006-2009 and the subsequent planned financial estimates are not yet linked to expected results. However, UNICEF, along with UNDP and UNFPA, is implementing new cost classifications and results-based budgeting (RBB). As part of this process, UNICEF published its first Institutional Budget for 2012-2013 (formerly known as the support budget). This RBB document is a step towards the presentation of one consolidated budget in 2014 that will that integrate resources and results. Also, in 2012

UNICEF is formally launching 2012 VISION (Virtual Integrated System of Information) which, according to the Programme Policy and Procedures Manual (PPPM) will “process transactions following standard business processes and in compliance with International Public Sector Accounting Standards (IPSAS) requirements in relation to programme planning, budgeting and financial management and reporting” at the country, regional and headquarters levels. These reforms, if implemented as planned, represent important steps towards becoming a more performance-oriented organisation.

MI 6.2 – Disbursements linked to reported results

Donors at headquarters were asked about the extent to which UNICEF’s reports on results include the amount disbursed to achieve those results. Just over half (51 per cent) rated UNICEF as adequate or higher, while 24 per cent rated it inadequate to very weak.

The document review rated UNICEF as very weak on this MI. Financial reports do not link disbursements with reported results. Since the new results-based budgeting system has not been fully implemented, annual reports based on the new system are still not available. The document review found no information on future guidelines concerning results-based financial reports. It may take some time for the new system to become fully operational to the level of being able to link aid disbursements with reported results in annual reports.

KPI 7: Financial Accountability

Finding 7: Surveyed stakeholders and the document review found that UNICEF has strong policies and processes for financial accountability.

Survey respondents rated UNICEF strong on four of the six MIs assessed in the survey and adequate on two. UNICEF received the highest scores on MIs related to external and internal audits. It is worth noting that approximately one-third of respondents answered ‘don’t know’ to four of the MIs (MIs 7.2, 7.4, 7.6 and 7.7).

The document review found the area of financial accountability to be one of UNICEF’s strengths. The systems and practices in place for external and internal audits are very well detailed and there is evidence that policies are followed. UNICEF has shown a renewed commitment to risk assessment and management since 2009 with the introduction of the Enterprise Risk Management initiative.

Figure 3.13 KPI 7: Financial Accountability, Ratings of Micro-Indicators

MI 7.1 – External financial audits performed across the organisation

MOPAN donors at headquarters were asked whether UNICEF's external financial audits are meeting the needs of donors. The majority (83 per cent) rated UNICEF as adequate or above.

According to UNICEF Financial Regulation and Rules, UNICEF’s accounts were externally audited by the UN Board of Auditors (UNBOA) on a biennial basis until the end of 2011. As of January 2012, external audits will take place on an annual basis in accordance with International Public Sector Accounting Standards.

Documents reviewed confirmed that external audits of UNICEF’s financial statements were conducted by the UNBOA every two years. All Annual Financial reports reviewed (i.e., 2005, 2007 and 2009) included letters from an external auditor confirming that they were conducted in accordance with international standards and in conformity with article VII of the Financial Regulations and Rules of the United Nations. The document review rated UNICEF very strong on this MI.

MI 7.2 – External financial audits performed at the regional, country, or project level

In-country donors and direct partners were asked whether UNICEF's programmes and projects are appropriately audited at a country level. The majority of respondents (56 per cent) rated UNICEF as adequate or above, and 37 per cent answered ‘don’t know’. Direct partners were more positive than donors in-country and the difference is statistically significant.

The document review, which rated UNICEF as strong on this MI, found that external audits are conducted at headquarters, regional and country levels. External financial audits are conducted by the UNBOA. Although UNBOA has clear rules and procedures for conducting audits, it remains unclear how it determines its own work plan. UNICEF’s very strong internal audit function[13] (noted below in MI 7.5) also audits at headquarters, regional and country levels.

To ensure ample audit coverage across the organisation, which is one of the criteria for this micro-indicator, UNICEF’s Office of Internal Audit and Investigation collaborates with UNBOA and adjusts its risk-based audit planning according to UNBOA planned audits. UNBOA only discloses audited financial statements and the report of the Board of Auditors. Until recently, UNICEF made internal audit reports available upon written request and on a confidential basis to Member States and approved non-Member State donors. As of October 2012 internal audits reports will be made available to the public. Both external and internal audits are required to meet international standards.

MI 7.3 – Policy on anti-corruption

This MI was assessed by document review only and was rated as strong. UNICEF has a zero-tolerance corporate Anti-Fraud Policy which ensures that all cases of fraud are investigated by the Office of Internal Audit. Apart from general training and advocacy on fraud and corruption, this 2006 policy is complemented by UNICEF’s Whistleblower Protection Policy, which includes protection against retaliation. Further, UNICEF established an Ethics Office in 2007 which produces an annual report to ensure the highest standards of integrity within the organisation. The current Anti-Fraud Policy does not include an anti- corruption component, although the Policy is under review and will be expanded to include a stronger and more explicit anti-corruption component. 

MI 7.4 – Systems for immediate measures against irregularities

All survey respondent groups, except peer organisations, were asked whether UNICEF appropriately follows up on financial irregularities, including fraud and corruption. The majority of survey respondents (65 per cent) rated UNICEF as adequate or above, and 29 per cent answered ‘don’t know’ on this MI. Direct partners and donors at headquarters were more positive than in-country donors. The differences are statistically significant.

In the document review, UNICEF was rated as very strong as it has a policy and procedures on measures to be taken against irregularities identified by an external financial audit. While the Financial Regulations and Rules of the United Nations include the terms of reference governing audit procedures and an overall framework for financial auditing within the UN, UNICEF has an internal system to follow up on the UN Board of Auditors recommendations, which include any irregularities. The 2011 Progress report on implementation of the recommendations of the Board of Auditors on the UNICEF accounts for the biennium 2008-2009 demonstrates UNICEF’s capacity to implement the recommendations. Of the 21 main recommendations made in 2010, UNICEF had fully implemented 10 and 10 were under implementation.

While guidelines from the Office of Internal Audit and Investigation do not provide detailed information about the process to be followed in the case of internal audit recommendations, there is an exhaustive list of completed engagements and related recommendations (which are rated as either medium or high priority). Major or systemic irregularities are reported to the Executive Board through the report from the Board of Auditors and the Office of Internal Audit annual report to the Executive Board.

MI 7.5 – Internal financial audit processes provide credible information

Among MOPAN donors at headquarters, the only respondent group asked about this MI, 88 per cent felt that internal financial audits provide credible information to UNICEF management and the Executive Board, and 50 per cent rated UNICEF as strong.

Based on evidence of consistent internal audit processes, the document review rated UNICEF as very strong on this MI. UNICEF Financial Regulations and Rules 2011 states that UNICEF’s Office of Internal Audit and Investigation (OIAI) provides assurance and advisory services that are objective, independent, and in conformity with international standards. The Charter of the UNICEF Office of Internal Audit[14] provides detailed information on its mission, scope of work, independence and responsibility. The review of documents indicated that the OIAI is structurally and operationally independent from UNICEF programming. The OIAI reports to the Executive Director and submits its reports independently and on an annual basis directly to the Executive Board.

MI 7.6 – Effective procurement and contract management processes

This MI was assessed by survey only. Donors in-country and direct partners were asked whether UNICEF's procurement and contract management processes for the provision of services or goods are effective. The majority of survey respondents (67 per cent) rated UNICEF as adequate or above, and 37 per cent of donors in-country answered ‘don’t know’. Direct partners were more positive than donors in-country and the difference is statistically significant.

MI 7.7 – Strategies for risk management

Donors at headquarters were asked whether UNICEF has appropriate strategies and plans for risk management. The majority (71 per cent) rated UNICEF as adequate and strong, 10 per cent as inadequate or below, and 20 per cent answered ‘don’t know’. 

The document review rated UNICEF as strong on risk management. UNICEF has made important progress in this regard with the launch of Enterprise Risk Management (ERM) in 2009, an approach that aims at managing opportunities and risks that could affect UNICEF’s programme and management objectives. It is made up of a comprehensive, corporately approved Risk Management Policy and Risk Reference Guide. The policy is comprehensive in that it identifies various types of risks (related to finance, hazard, operation and strategy), and outlines roles and responsibilities of key actors. ERM is implemented using a Risk and Control Self-Assessment (RCSA) – UNICEF’s risk identification and management tool – at country, regional and headquarters levels. RCSAs are used to develop Risk Profile and Risk and Control Library at various levels. An internal survey of Country Offices on the use of these tools found high levels of application for the purposes of planning, reporting, oversight and streamlining internal processes.

KPI 8: Using Performance Information

Finding 8: Overall, survey respondents rated UNICEF’s use of performance information on results as adequate; the document review found it to be strong in this area.

UNICEF was rated adequate on three and strong on one of the four MIs assessed by the survey.

UNICEF has shown a commitment to improvement and a strong desire for learning. The document review found that the use of performance information to revise, plan, and manage projects and programmes is strong at both the organisational and country levels. The guidelines and procedures related to the management of projects/programmes at the country level are clear. However, the evidence of how performance information has been used to revise or adjust programmes is not always easily identifiable in UNICEF reports.

Figure 3.14 KPI 8: Using Performance Information, Ratings of Micro-Indicators

MI 8.1 – Using information for revising and adjusting policies

Donors at headquarters were asked whether UNICEF uses project/programme, sector and country information on performance to revise organisational policies. The majority (73 per cent) rated UNICEF as adequate or above, and 20 per cent answered ‘don’t know’.

The document review rated UNICEF strong in using performance information to revise and adjust policies and strategies. Although UNICEF implements a Medium-term Strategic Plan (MTSP) every four years, the MTSP 2006-2009 was extended twice to 2013 and revisions have been outlined in additional annexes to the report. MTSPs systematically build on recommendations outlined in (organisation-wide) Mid-term/In-depth/End-of-cycle Reviews conducted effectively every two years after a new MTSP. UNICEF’s system also entails an analysis of the situation of children, findings from recent evaluations, and a vast consultation process (at UN, organisation-wide and country level). The 2008 Mid-term Review (MTR) and the 2010 In-depth Review are examples of key sources of performance information that were used to inform revisions to strategy and policy. UNICEF is also committed to implementing recommendations from audits (see MI 7.4) and/or evaluations (see MI 8.4) if specific performance-related problems are identified and accepted by management. It is worth noting, however, that although systematic processes are in place to review performance using various standardised documents (e.g., MTRs and evaluations), it remains unclear how information from the Annual Report of the Executive Director of UNICEF is being used to revise and adjust policies.

MI 8.2 – Using information for planning new interventions

Donors in-country and direct partners were asked whether UNICEF uses information on its projects/programmes or initiatives to plan new areas of cooperation at country level, and the majority (85 per cent) rated UNICEF adequate or above. Direct partners were more positive than donors in-country, even though their ratings both fall within the range of strong. The difference is statistically significant.

The document review rated UNICEF strong on this MI. UNICEF’s procedure of planning new interventions based on its performance in country is initiated through country-level Mid-Term Reviews conducted mid-way through the country programme cycle and the development of the new Country Programme. The MTR is based on a systematic and in-depth review of progress measured against the original country programme objectives and planned results. According to UNICEF’s 2012 Programme Policy and Procedure Manual, the purpose of the MTR is to identify and make provisions for mid-course adjustments in key elements of the country programme design. Also, during the development of a new Country Programme Document, UNICEF’s Country Offices prepare a Situation Analysis, aggregate results from the various reports, and consult with partners and governments. Also, most CPDs present lessons learned in response to the MTR, but do not document how these lessons learned were specifically applied to the design of new interventions. UNICEF would benefit from a more detailed and systematic approach that demonstrates how programme interventions in Country Programme Documents are aligned with performance assessments in MTRs.

MI 8.3 – Proactive management of poorly performing initiatives

Donors in-country and direct partners were asked whether poorly performing programmes and projects are subject to proactive management. The majority of survey respondents (58 per cent) rated UNICEF as adequate or above, and 11 per cent as inadequate or below, and 31 per cent answered ‘don’t know’.

The document review rated UNICEF strong on its capacity to identify and find solutions to poorly performing programmes or projects. UNICEF reviews the performance of its programmes and projects through Annual Programme Reviews (conducted by UNICEF’s sectoral units at the country level) and Mid-Term Reviews. Detailed guidelines on these processes are available. However, there is limited evidence of the implementation of these processes and their effects on poorly performing programmes or projects. No annual programme review documents were provided for the document review.

In 2012, UNICEF launched VISION, a transaction and performance management system designed to improve the management of monitoring programme performance. The Manager’s Dashboard (MDB), one of the components of this system, will allow for a review of programme performance by providing easily accessible information on key programme and operational indicators for each business area (i.e., all HQ divisions, Regional and Country Offices). This system is still in a pilot phase and is being rolled out in 2012.

MI 8.4 – Evaluation recommendations are acted upon

Donors at headquarters were asked whether UNICEF appropriately tracks the implementation of evaluation recommendations reported to its governing bodies. The majority (71 per cent) rated UNICEF as adequate and strong, and 17 per cent as inadequate or below, and 12 per cent answered ‘don’t know’.

The document review rated UNICEF as strong in ensuring evaluation recommendations are acted upon by the responsible units. UNICEF Evaluation Policy (2008) requires a management response to evaluations conducted by the organisation, as well as an action plan that is made public. In 2009, UNICEF published its guidance for management response to evaluations (i.e., procedures, accountabilities and template). According to the 2012 Annual report on the evaluation function and major evaluations, completed management responses were uploaded to the tracking system for 65 per cent of evaluation reports in 2011 (an increase from 12 per cent in 2009 and 47 per cent in 2010). Although the document review could not find any evidence in the documents submitted to the Executive Board, there are procedures under the responsibility of the Evaluation Office for tracking the implementation of recommendations and then reporting on them annually to the Evaluation Committee and the Executive Board.

KPI 9: Managing Human Resources

Finding 9: The survey ratings and document review suggest that UNICEF is adequately managing its human resources, but its staff performance system is not widely known.

Overall, survey respondents rated UNICEF adequate and strong on the two MIs assessed under this KPI, although there was a high percentage of ‘don’t know’ responses (59 per cent) on MI 9.2.

The document review found UNICEF’s practices to be quite strong in the area of human resource management. It has made organisation-wide improvements in recent years through, for example, the introduction of a new electronic Performance Appraisal System, a new competency framework, e-recruitment, and leadership development.

Figure 3.15 KPI 9: Managing Human Resources, Ratings of Micro-Indicators 

MI 9.1 – Results-focused performance assessment systems for senior staff

This MI was assessed by document review only which rated the organisation as very strong. Following the Organisational Review (2007) and the Strategic Review of Human Resources, UNICEF launched an electronic Performance Appraisal System (e-PAS) in 2010 that covers individual performance, learning, career management, and HR planning. This online performance assessment system, which is to be used by all international professional staff in UNICEF, is outlined in the Performance Appraisal and Rebuttal Process and the e-PAS User Guide. The Performance Evaluation Report includes sections on objectives, key assignments, a training plan, and performance planning and ratings that are entered in the e-PAS. It is a comprehensive system for managing staff performance. UNICEF reports on the submission of Performance Evaluation Reports across the organisation in the Annual Report of the Executive Director.

MI 9.2 – Transparent system to manage staff performance

Donors at headquarters were asked whether UNICEF uses a transparent system to manage staff performance. The majority (59 per cent) answered ‘don’t know’ and 34 per cent rated UNICEF as adequate or above.

This MI received a rating of adequate from the document review. The Performance Appraisal and Rebuttal Process clearly states that “UNICEF has no options for financially rewarding well-performing staff; rewards will most often take the form of recognition in performance appraisals, wider recognition within the office, and assignment of increasingly challenging work”.[15] However, there is evidence in e-PAS guidelines (UNICEF launched the electronic Performance Appraisal System (e-PAS) in 2010) that staff performance is “considered when reviewing staff for rotation, recruitment, consideration for placement on rosters”.[16] Staff performance management was identified as a weakness in the 2007 Organisational Review. Since then, UNICEF has created new systems to address some of the identified weaknesses.

MI 9.3 – Staff rotation adequate for the development of effective partnerships

This MI was assessed by survey only.

In-country donors and direct partners were asked whether UNICEF keeps deployed international staff in country offices for a sufficient time to maintain effective partnerships at country level. The majority or survey respondents (79 per cent) rated UNICEF as adequate or above, 12 per cent as inadequate or below, and 9 per cent answered ‘don’t know’.

KPI 10: Performance-oriented Programming

Finding 10: While survey respondents considered UNICEF’s country and regional programming processes as adequately performance-oriented, the document review ratings suggest that UNICEF has considerable room to improve in this area.

The two MIs assessed by the survey under this KPI received ratings of adequate and strong. Overall, direct partners were more positive than other respondent groups.

The document review found little evidence that new initiatives or activities are subject to a benefit or impact analysis. Further, there were few examples of project-level monitoring frameworks that track progress according to a specific set of indicators that include baselines, targets and milestones. While UNICEF country offices follow the United Nations Development Group (UNDG) guidelines on annual work plans they do not include monitoring plans. 

Figure 3.16 KPI 10: Performance-oriented Programming, Ratings of Micro-Indicators 

MI 10.1 – New initiatives subject to benefits/impact analysis

All survey respondents, except peer organisations, were asked whether UNICEF subjects new programming initiatives to impact analysis. The majority of survey respondents (63 per cent) rated UNICEF as adequate or above and 27 per cent answered ‘don’t know’. Direct partners were more positive than donors in-country and at headquarters. The differences are statistically significant.

Strictly adhering to MOPAN criteria, the document review rated UNICEF very weak as there is no evidence that formal benefits/impacts analysis of new initiatives are being conducted although there are some procedures and safeguards to ensure project quality.

According to the documentation reviewed, neither programmes nor projects are subject to formal benefit/impact analyses. The Annual Work Plan outlines the activities to be undertaken by UNICEF and its partners to achieve the results described in the Country Programme Action Plan, which is the formal agreement between the Government and UNICEF. When formulating an annual work plan, Country Offices base their planning on the CPAP, annual reviews and regular discussions with partners.

UNICEF’s approach to managing the collaboration between UNICEF and civil society organisations is based on the Project Cooperation Agreement (PCA). Under this agreement, partners develop their project plan and budget. The agreement is then reviewed and approved according to a set of programmatic and financial criteria. Importantly, the PCA also outlines procedures for exploring the project impact, outcome and outputs, budget, monitoring, and evaluations.[17]

MI 10.2 – Milestones/targets set to monitor implementation

Donors in-country and direct partners were asked whether UNICEF sets targets to enable monitoring of progress in project/programme implementation at country level. The majority of survey respondents (84 per cent) rated UNICEF as adequate or above. Direct partners were more positive than donors in-country and at headquarters. The differences are statistically significant.

The document review rated UNICEF as inadequate in setting targets to rate the progress of implementation at the country-level. In reviewing a sample of Annual Work Plans, the document review found that they did not contain indicators, baselines and targets to rate the progress of UNICEF’s activities. However, they all contained established milestones (i.e., time frame for undertaking the planned activities). UNICEF’s Annual Work Plans generally followed guidelines established by the United Nations Development Group.

KPI 11: Delegating Authority

Finding 11: Survey respondents rated UNICEF as strong on its delegation of decision-making authority. The document review, however, found that some aspects of the delegation process remain unclear.

Survey respondents perceived UNICEF strong in delegating authority. In general, direct partners were more positive than donors in-country on this KPI.

The document review found evidence of UNICEF’s decentralised organisational structure but insufficient clarity of the criteria for financial decision making by country offices. Further, no information was found on the thresholds for financial approvals by Representatives or other senior Country Office officials. Revisions to these systems are underway as part of the Accountability Initiative launched in 2009.

Figure 3.17 KPI 11: Delegating Authority, Ratings of Micro-Indicators 

MI 11.1 – Project/programme tasks are managed at the country level

In-country donors and direct partners were asked whether UNICEF's project/programme tasks are managed at a country level. The majority of survey respondents (81 per cent) rated UNICEF as adequate or above, and 14 per cent answered ‘don’t know’. Direct partners were more positive than donors in-country, even though their ratings both fall within the range of strong. The difference is statistically significant.

The document review rated UNICEF inadequate on this MI. The 2007 independent Organisational Review of UNICEF pinpointed the “inadequate clarity and coherence in the distribution of roles and accountabilities within and among different levels of the organisation.”[18] UNICEF is a decentralised, field-based organisation and its country offices have a certain level of autonomy to make adjustments and changes to activities – most formally, following a Mid-Term Review. However, the extent of UNICEF’s financial decision- making authority at the country level is not clear in the documentation available.

MI 11.2 – New programmes/projects can be approved locally within a budget cap

Donors in-country and direct partners were asked whether funding for new areas of cooperation can be approved locally, within a budget cap. Half of survey respondents (50 per cent) rated UNICEF as adequate or above, and 42 per cent answered ‘don’t know’. Direct partners were more positive than donors in-country. The difference is statistically significant.

UNICEF received a rating of adequate from the document review. New programmes/projects can be approved at country level if they are an integral part of the plans (financial and programmatic) described in the CPD and CPAP, which are approved by the Executive Board and the recipient government, respectively. The outline of UNICEF’s activities at country level is contained in the CPAP and then the annual/rolling work plans. Planning of activities is done with the recipient government and civil society organisations at the country level. For instance, there is detailed guidance for collaboration between UNICEF and non- governmental organisations and community-based organisations. It contains information on all aspects of the collaboration from the proposal to the plan, the agreement, and the monitoring and evaluation of the project. There are also clear instructions on the requirements for government partners.

KPI 12: Humanitarian Principles

Finding 12: UNICEF’s adherence to humanitarian principles in its field operations was considered to be a strength by survey respondents based in countries with significant humanitarian programming. The document review found that UNICEF’s strategic frameworks include most key humanitarian principles.

Survey respondents based in the Democratic Republic of Congo, Zimbabwe and Niger rated UNICEF as strong on the two MIs assessed by the survey under this KPI. This KPI received the highest score of the survey on UNICEF’s practices and systems.

The document review assessed some of the key strategic documents that UNICEF has produced to guide its humanitarian response. Although there is considerable evidence of UNICEF’s strong commitment to humanitarian principles in its response, there was no consideration of the principle of ‘independence’ in these core documents.

Figure 3.18 KPI 12: Humanitarian Principles, Ratings of Micro-Indicators

MI 12.1 – Strategic framework includes humanitarian principles

This MI was assessed by document review only and was rated as adequate according to the documents reviewed. In its Core Commitments for Children in Humanitarian Action published in 2010, UNICEF defines the principles of humanity, impartiality, and neutrality.[19]

In its Emergency Field Handbook, UNICEF provides guidance on how these three principles should be applied by UNICEF staff in the humanitarian context. While there was no evidence found that UNICEF includes the principle of ’independence’ specifically in its guidance, there is a high level of adherence to humanitarian princes and an operational rigour in UNICEF’s humanitarian response activities.

MI 12.2 – Humanitarian principles respected while delivering humanitarian/emergency assistance

This MI was assessed by survey only. All respondent groups in the Democratic Republic of Congo, Zimbabwe and Niger were asked whether UNICEF's Core Commitments for Children in Humanitarian Action are respected in the delivery of emergency assistance and/or in humanitarian response. The majority of survey respondents (91 per cent) rated UNICEF as adequate or above, and 43 per cent gave ratings of very strong. Importantly, the survey respondents on this question included peer organisations who are UNICEF’s closest collaborators in humanitarian response in these countries.

MI 12.3 – Ongoing policy dialogue with partners on observing humanitarian principles

This MI was assessed by survey only. All respondent groups, except donors at headquarters, were asked whether UNICEF maintains on-going policy dialogue with partners on the importance of observing humanitarian principles in delivering emergency assistance, particularly in cases of protracted crises and complex emergencies. The majority of survey respondents (88 per cent) rated UNICEF as adequate or above, and 33 per cent gave ratings of very strong.

3.3.4 Relationship Management

UNICEF is seen as strong in its support to national counterparts and in its humanitarian cluster leadership. There is some room for improvement in the efficiency of UNICEF’s administrative procedures and use of country systems.

Figure 3.18 below shows the overall survey review ratings for the six KPIs in the relationship management quadrant.

In the survey, UNICEF was rated as adequate or strong in all aspects of relationship management and its contribution to policy dialogue received the highest total score of the survey.

Figure 3.19 Quadrant III: Relationship Management, Survey and Document Review Ratings*

*The document review for KPI 17 was designed to draw on data from the 2011 Survey on Monitoring the Paris Declaration. The white diamond indicates that the data required for the assessment was unavailable for UNICEF.

Figure 3.19 shows the mean scores for the six KPIs for all survey respondents, and by respondent groups.

Figure 3.20 Quadrant III: Relationship Management, Mean Scores by Respondent Group

KPI 13: Supporting National Plans

Finding 13: UNICEF was perceived to be strong in its support of national and partner plans.

This KPI was assessed by survey only. Respondent groups gave UNICEF an overall rating of strong for its support of national and partner plans. However, direct partners had a much more positive view on this area of performance than in-country donors.

Figure 3.21 KPI 13: Supporting National Plans, Ratings of Micro-Indicators

MI 13.1 – Funding proposals developed with national government or direct partners

This MI was assessed by survey only. Donors in-country and direct partners were asked about the extent to which UNICEF supports funding proposals designed and developed by the national government or other direct partners. The majority of survey respondents (84 per cent) rated UNICEF as adequate or above. Direct partners were more positive than donors in-country. The difference is statistically significant.

KPI 14: Adjusting Procedures

Finding 14: UNICEF was perceived by respondents to have clear procedures; however, they were rated as adequate on the timeliness of these procedures.

This KPI was assessed by the survey only. Direct partners and donors in-country provided two ratings of strong and two adequate. Overall, direct partners were more positive than donors in-country.

Figure 3.22 KPI 14: Adjusting Procedures, Ratings of Micro-Indicators

MI 14.1 – Procedures easily understood and completed by partners

Asked whether UNICEF uses procedures that can be easily understood and completed by partners, the majority of direct partners and donors in-country (71 per cent) rated the organisation as adequate or above, and 15 per cent as inadequate or below. Direct partners were more positive than donors in-country. The difference is statistically significant.

MI 14.2 – Length of time for procedures does not affect implementation

Asked whether the length of time it takes to complete UNICEF procedures affects implementation, the majority of direct partners and donors in-country (60 per cent) rated the organisation as adequate or above, and 26 per cent as inadequate or below. Direct partners were more positive than donors in-country and this difference is statistically significant.

MI 14.3 – Ability to respond quickly to changing circumstances

Asked whether UNICEF adjusts its work at country level quickly to respond to changing circumstances on the ground, the majority of direct partners and donors in-country (77 per cent) rated the organisation as adequate or above, and 17 per cent as inadequate or below. Direct partners were more positive than donors in-country. The difference is statistically significant.

MI 14.4 – Flexibility in implementation of projects/programmes

Donors in-country and direct partners were asked whether UNICEF flexibly adjusts its support to programmes and activities based on learning and issues related to budget. The majority of survey respondents (74 per cent) rated UNICEF as adequate or above. Direct partners were more positive than donors in-country. The difference is statistically significant.

KPI 15: Using Country Systems

Finding 15: The survey and document review rated UNICEF as adequate in the use of country financial systems, but survey respondents have a low level of familiarity with UNICEF’s use of country systems.

Both MIs assessed by the survey under this KPI received ratings of adequate. Direct partners were generally more positive than donors in-country.

The document review found that UNICEF’s use of country financial systems is adequate. The document review for two of the MIs in this KPI was designed to draw on data from the 2011 Survey on Monitoring the Paris Declaration. These could not be assessed as the data required for the assessment was unavailable for UNICEF.

Figure 3.23 KPI 15: Using Country Systems, Ratings of Micro-Indicators*

*White diamonds indicate that the data required for the assessment was unavailable for UNICEF.

MI 15.1 – ODA disbursements/ support recorded in annual budget

This MI was intended to be assessed by the document review, but there was insufficient data available to conduct the review for UNICEF.

MI 15.2 – Use of country financial systems

Donors in-country and direct partners were asked whether UNICEF uses country financial systems (e.g., procurement, public financial management, etc.) as a first option for its operations where appropriate. Less than half of survey respondents (45 per cent) rated UNICEF as adequate or above, 22 per cent as inadequate or below, and 33 per cent answered ‘don’t know’. Direct partners were more positive than donors in-country. The difference is statistically significant.

The indicator is based on Paris Declaration Indicator 5, which applies a narrow definition of the use of country procurement and public financial management systems based on the percentage of ODA that uses those systems. In the case of UNICEF, the document review applied a broader notion of these concepts and rated UNICEF as adequate on this MI.

UNICEF has a history of providing funds directly to governmental partners but there is less evidence of their use of country procurement systems. UNICEF has used a cash transfer modality with government partners for many years. This involves the transfer of funds to government partners according to agreed priorities and approved proposals. This system has been strengthened with the introduction of a harmonised approach to cash transfers (HACT), whereby government (and non-governmental partner) systems are assessed/audited to determine their levels of efficiency and transparency. Risk levels are defined based on this assessment and increased compliance measures are put in place when risk levels are high.

No evidence was found of UNICEF’s use of country procurement systems, which could be explained by the volume and nature of the type of supplies provided by UNICEF. The assessment team did find evidence that UNICEF works to reduce transaction costs and harmonise its work with other UN agencies, as well as strengthening national capacity in supply planning, procurement and in-country logistics.

MI 15.3 – Use of country non-financial systems

This MI was assessed by survey only. Donors in-country and direct partners were asked whether UNICEF uses a country's non-financial systems (e.g., monitoring and evaluation) as a first option for its operations where appropriate. Half of survey respondents (50 per cent) rated UNICEF as adequate or above, 16 per cent as inadequate or below, and 35 per cent answered ‘don’t know’. Direct partners were more positive than donors in-country, even though their ratings both fall within the range of adequate. The difference is statistically significant.

MI 15.4 – Parallel implementation structures avoided

This indicator draws on Indicator 6 of the Paris Declaration and was to be assessed by document review only. The data on Indicator 6 was not available for UNICEF, so this MI could not be assessed.

KPI 16: Contributing to Policy Dialogue

Finding 16: According to survey respondents, UNICEF makes a strong contribution to policy dialogue while respecting the views of its partners in the process.

This KPI was assessed by the survey only and UNICEF was rated strong. Overall, direct partners held more favourable views of UNICEF in this area of performance than other respondent groups.

Figure 3.24 KPI 16: Contributing to Policy Dialogue, Ratings of Micro-Indicators

MI 16.1 – Reputation for valuable input to policy dialogue

All respondent groups, except peer organisations, were asked whether UNICEF provides valuable inputs to policy dialogue. The majority of survey respondents (89 per cent) rated UNICEF as adequate or above. Direct partners were more positive than donors in-country, even though their ratings both fall within the range of strong. The difference is statistically significant. 

MI 16.2 – Policy dialogue respects partner views

All respondent groups, except peer organisations, were asked whether UNICEF respects the views of partners when it undertakes policy dialogue. The majority of survey respondents (85 per cent) rated UNICEF as adequate or above. Donors at headquarters were less positive than donors in-country and direct partners. The differences are statistically significant.

KPI 17: Harmonising Procedures

Finding 17: UNICEF was recognised by survey respondents as strong overall for the harmonisation of its procedures with other actors. The document review was unable to assess UNICEF on these MIs due to insufficient data.

In-country donors and direct partners rated UNICEF as strong on this KPI. Direct partners were generally more positive than other respondent groups.

The document review was unable to assess this KPI due to lack of disaggregated data from the Survey on Monitoring the Paris Declaration.

Figure 3.25 KPI 17: Harmonising Procedures, Ratings of Micro-Indicators

MI 17.1 – Participation in joint missions

This MI was intended to be assessed by the document review, but there was insufficient data available.

MI 17.2 – Technical cooperation disbursed through coordinated programmes

Donors in-country and direct partners were asked whether UNICEF's technical assistance is provided through coordinated programmes in support of capacity development. The majority of survey respondents (91 per cent) rated UNICEF as adequate or above. Direct partners were more positive than donors in-country. The difference is statistically significant.

The document review was unable to obtain data on this MI which draws on Indicator 4 of the Paris Declaration.

MI 17.3 – ODA disbursements/support for government-led PBAs

Donors in-country and direct partners were asked whether UNICEF participates in programme-based approaches (other than through budget support). The majority of survey respondents (86 per cent) rated UNICEF as adequate or above. Direct partners were more positive than donors in-country. The difference is statistically significant.

While this MI was intended to be assessed by the document review, there was insufficient data available.

KPI 18: Managing the Cluster

Finding 18: UNICEF’s performance in fulfilling its responsibilities as a Cluster Lead Agency in the Democratic Republic of Congo, Niger and Zimbabwe was perceived to be strong by in-country donors, direct partners and peer organisations.

This KPI was assessed only in the survey by country-based respondents (i.e., donors in- country, direct partners, and peer organisations) in the Democratic Republic of Congo, Niger and Zimbabwe – the three MOPAN survey countries in which UNICEF has significant humanitarian programming. It mainly sought to assess UNICEF’s ability to fulfil Cluster Lead Agency responsibilities at country level. UNICEF was perceived as strong in all areas of this KPI.

Figure 3.26 KPI 18: Managing the Cluster, Ratings of Micro-Indicators

MI 18.1 – Sufficient analytical resources and policy-level engagement dedicated to the cluster

Donors in-country, direct partners and peer organisations were asked whether UNICEF dedicates sufficient analytical resources and policy-level engagement to strategic activities within the cluster. The majority of respondents (80 per cent) rated UNICEF as adequate or above.

MI 18.2 – Dedicated staff for coordination of the cluster

When asked whether UNICEF dedicates sufficient analytical resources and policy-level engagement to strategic activities within the cluster, the majority of country-based respondents (82 per cent) rated UNICEF as adequate or above.

MI 18.3 – Pertinent information circulated within the cluster

When asked whether UNICEF ensures that pertinent information is circulated within the cluster, the majority of country-based respondents (85 per cent) rated UNICEF as adequate or above.

MI 18.4 – Reliable forecasting of financial needs of the cluster

When asked whether UNICEF generates reliable forecasts of financial need for the cluster, the majority of country-based respondents (73 per cent) rated UNICEF as adequate or above, and 12 per cent as inadequate or below, and 13 per cent answered ‘don’t know’.

3.3.5 Knowledge Management

Survey respondents and the document review found UNICEF’s evaluation of external results to be adequate. One area for improvement is in the presentation of performance information.

Figure 3.27 below shows the overall survey and document review ratings for the three KPIs in the knowledge management quadrant.

According to survey results, UNICEF performs adequately in all areas of knowledge management. The document review ratings ranged from inadequate to strong.

Figure 3.27 Quadrant IV: Knowledge Management, Survey and Document Review Ratings

Figure 3.28 shows the mean scores for the three KPIs for all survey respondents, and by respondent groups.

Figure 3.28 Quadrant IV: Knowledge Management, Mean Scores by Respondent Group

KPI 19: Evaluating External Results

Finding 19: UNICEF was perceived by survey respondents as performing adequately in the evaluation of external results. The document review rated UNICEF as strong in the independence, coverage and quality of its evaluations.

Five MIs were assessed by document review only and two by both the document review and the survey under this KPI. UNICEF was rated as adequate on one and strong on the other.

The document review provided ratings range from adequate to strong. UNICEF has invested a lot of resources in this area, as part of the process of strengthening its evaluation function. The document review found that UNICEF has made considerable progress in terms of coverage and quality of evaluations.

Figure 3.29 KPI 19: Evaluating External Results, Ratings of Micro-Indicators

MI 19.1 – Independent evaluation unit

This MI was assessed by document review only. It found UNICEF’s performance to be strong in this area. The role and mandate of the Evaluation Office are described in the UNICEF Evaluation Policy (2008). UNICEF emphasises the integrity and independence of the evaluation function and the reporting structure in place for the evaluation function in its Evaluation Policy. While UNICEF’s Evaluation Office is not structurally independent from management in terms of budget or work programme, it negotiates funding and interacts with the rest of the organisation effectively to determine and carry out its agenda. Unlike evaluation offices at other agencies, the Evaluation Office also plays a crucial advisory role for other sections at UNICEF (methodology, quality assurance, terms of reference) and systematic strengthening of monitoring and evaluation across the organisation (workshops, seminar, guidance, help desk). The Director of the Evaluation Unit reports to the UNICEF Executive Director rather than the Executive Board but does have considerable autonomy in the content and finalisation of evaluations.

MI 19.2 – Sufficient evaluation coverage of programming activities

This MI was assessed by document review only. It rated UNICEF as adequate regarding the coverage of its evaluations. UNICEF Evaluation Policy requires that the organisation engage in broad consultations in order to determine those evaluations that are the most strategic and that ensure an adequate coverage of all major programme components during a programming cycle. The policy also draws attention to the need for adequate resources to be allocated to conduct the prioritised evaluations. This information is compiled in the Integrated Monitoring and Evaluation Framework and presented to the Evaluation Committee for approval. While the Evaluation Policy states that three to five percent of programme expenditure should be dedicated to evaluation, studies and research, UNICEF’s documentation remains vague on the criteria used to determine coverage and prioritisation of evaluations. This has been noted by both the Executive Board and the Office of Internal Audit. To address this, UNICEF issued guidance on the prioritisation of major evaluations at the decentralised level as a way of strengthening the relevance and use of evidence generated through evaluations.

MI 19.3 – Quality of evaluations

This MI was assessed by document review only, which gave UNICEF a rating of strong for its systematic approach to ensuring the quality of both centralised and decentralised evaluations. UNICEF has issued guidance and established procedures to enhance the quality control of evaluations. In 2010, a series of evaluation-related tools and processes were introduced: guidance notes on UNEG’s Evaluation Reports Standards and on UNEG’s Quality Checklist for Evaluation Terms of Reference, and a Global Evaluation Reports Oversight System. The 2010 report from the Oversight System noted that only 36 per cent of reviewed evaluation reports met the UNICEF standards to a degree that could be considered satisfactory. Despite this poor result, evidence was found indicating that UNICEF is putting a considerable amount of resources and effort into improving the quality of its evaluations and reporting.

MI 19.4 – Use of evaluation findings to inform decisions

This MI was assessed by survey only. Donors at headquarters were asked whether UNICEF uses evaluation findings in its decisions on programming, policy and strategy. The majority (76 per cent) rated UNICEF as adequate and above, and 20 per cent as inadequate or below.

MI 19.5 – Beneficiaries and direct partners involved in evaluations

This MI was assessed by survey only. Donors in-country and direct partners were asked whether UNICEF involves direct partners and beneficiaries in evaluations of its projects or programmes. The majority of survey respondents (80 per cent) rated UNICEF as adequate or above. Direct partners were more positive than donors in-country. The difference is statistically significant.

KPI 20: Presenting Performance Information

Finding 20: Survey respondents believe that UNICEF adequately presents performance information on its effectiveness, but evidence from the document review and survey indicate that improvement is needed in the presentation of performance information.

Donors at headquarters were the only respondent group asked about the extent to which UNICEF presents performance information on its effectiveness. The majority perceived UNICEF to perform adequately in this area.

The document review found that UNICEF’s presentation of performance information was one of its main areas for improvement and provided scores from very weak to adequate. It provided low ratings on UNICEF’s reporting on Paris Declaration commitments and reporting on programming adjustments at the organisation-wide level.

Figure 3.30 KPI 20: Presenting Performance Information, Ratings of Micro-Indicators

MI 20.1 – Reports on achievement of outcomes

The majority of donors at headquarters (73 per cent) rated UNICEF as adequate or strong on whether UNICEF's reports to the Executive Board provide clear measures of achievement of outcomes, while 17 per cent gave ratings of inadequate or below.

UNICEF’s performance in reporting on the achievement of outcomes was rated inadequate by the document review as UNICEF does not analyse in detail the link between outputs and outcomes.

UNICEF documents results achieved in myriad reports. The Annual report of the Executive Director of UNICEF is the formal report on achievement of the results identified in the Medium-Term Strategic Plan 2006-2009. In this report, UNICEF presents results and analyses of progress in each focus area. In the narrative, the report provides information on the situation, its approach and lessons learned. It also lists a number of organisation-wide outcomes where UNICEF is deemed to have contributed significantly. Interestingly, the report is accompanied by a Data Companion that presents data on a large number of indicators. However, information is scattered throughout the report, which makes it difficult to understand where UNICEF has made progress against its development results framework.

MI 20.2 – Reports on performance using data obtained from measuring indicators

This MI was assessed by document review only. Based on the review of the most recent Annual Report of the Executive Director of UNICEF and its Data Companion (2012), UNICEF’s use of data obtained from measuring indicators to report on performance is inadequate. There is little evidence of improvement over time. In the section associated with the management results framework (MRF), the report presents trends on all indicators, but does not compare the information against baselines or targets. As for development results, the report compares indicator measurements over time and includes baselines and estimated targets in some cases. The narrative of the report provides very little analysis of the progress towards targets or how the targets link to outcomes.

MI 20.3 – Reports against corporate strategy, including expected results

The majority of donors at headquarters (85 per cent) rated UNICEF as adequate or above on reporting against its organisational strategy.

UNICEF was rated as inadequate by the document review. The Annual Report of the Executive Director of UNICEF makes reference to the development results framework identified in the annexes to the Medium Term Strategic Plan 2006-2009 (revised in 2008, 2010, 2012). The report contains data on most key performance indicators, but UNICEF’s description of progress towards its development results is not clear. UNICEF does report on a number of indicators, but does not report in a systematic and organised way on its organisation-wide outcomes, nor does it explain variances between actual results and planned results identified in the results framework.

MI 20.4 – Reports on Paris Declaration commitments use indicators and country targets

The majority of donors at headquarters (59 per cent) rated UNICEF as adequate and strong on whether UNICEF reports to the governing body on performance in relation to its Paris Declaration commitments. However, 17 per cent gave ratings of inadequate or below and 24 per cent answered ‘don’t know’.

This MI, however, was rated very weak by the document review. Although there is evidence that UNICEF is committed to the principles of the Paris Declaration on Aid Effectiveness, it does not consolidate data from its country offices on an organisation-wide level in conformity with its commitments to the Paris Declaration. Further, no related data was found in other UNICEF reports.

MI 20.5 – Reports on adjustments to policies/strategies based on performance information

This MI was assessed by document review only. Strictly adhering to MOPAN criteria, it rated UNICEF very weak in country-level reporting on programme adjustments based on performance information. While annual performance reporting is carried out, no evidence was found of a policy describing the procedures in this regard. There is evidence of an organisational commitment to strengthening the use of evaluation findings and recommendations (see MI 8.4).

MI 20.6 – Reports on programming adjustments based on performance information

This MI was assessed by document review only. UNICEF was rated as adequate in country- level reporting on programme adjustments based on performance information. Country Office Annual Reports describe, inter alia, programme adjustments being made as a result of the Annual Programme Reviews. The Country Office Annual Reports (COAR) provide input into the Executive Director’s Annual Report to the Executive Board.

KPI 21: Disseminating Lessons Learned

Finding 21: Survey respondents rated UNICEF as adequate overall in disseminating lessons learned, but their level of familiarity with the content of these lessons was low. The document review noted that UNICEF has made improvements in reporting on lessons learned.

In the survey for this KPI, which included donors at headquarters only, UNICEF was perceived to perform adequately in encouraging the identification, documentation and dissemination of lessons learned and/or best practices. The document review concurred.

Figure 3.31 KPI 21: Disseminating Lessons Learned, Ratings of Micro-Indicators

MI 21.1 – Reports on lessons learned based on performance information

Donors at headquarters were asked whether UNICEF identifies and disseminates lessons learned from performance information. The majority of survey respondents (61 per cent) rated UNICEF as adequate or above, 20 per cent as inadequate or below, and 20 per cent answered ‘don’t know’.

The document review rated UNICEF as adequate on this MI. There is evidence that UNICEF has made significant improvements in identifying, documenting and disseminating innovations, lessons learned and good practices as part of its organisational learning and programme improvement processes. However, UNICEF does not have an easily accessible system to collect and disseminate both internal and external lessons learned and/or best practices.

MI 21.2 – Lessons shared at all levels of the organisation

This MI was assessed by survey only. Donors at headquarters were asked whether UNICEF provides opportunities at all levels of the organisation to share lessons from practical experience. Just over half of survey respondents (51 per cent) rated UNICEF as adequate or above, 15 per cent as inadequate or below, and 34 per cent answered ‘don’t know’. 

4. Main Findings: Development Results Component

4.1 Overview

This section presents the results of the 2012 Common Approach assessment of UNICEF in the pilot component to assess the development results of multilateral organisations. It includes four key performance areas:

• Evidence of extent of progress towards organisation-wide outcomes (KPI A)

• Assessment of country level results and relevance:

- Evidence of extent of contribution to country-level goals and priorities (KPI B)

- Evidence of extent of contribution to relevant Millennium Development Goals (KPI C)

- Relevance of objectives and programme of work to stakeholders (KPI D)

Figure 4.1 provides a snapshot of the findings of this assessment. Please note that the pilot applied a simplified 4-point scale that uses the same “traffic light” colours used elsewhere in the report. As noted in Section 2.8 of this report, the scale was simplified to reflect the methodological approach used in the pilot of the development results component – in which various sources of data are considered together when rating the organisation’s performance on each KPI. The methodology is explained in Volume II, Appendix I.

Figure 4.1 Development Results Component – Overall Ratings

4.2 Progress towards Organisation-Wide Results

4.2.1 Overview

This section presents the results of the assessment of UNICEF’s progress towards organisation-wide results. KPI A suggests that an effective organisation should demonstrate progress towards organisation-wide, institutional outcomes.[20]These are usually related to the organisation’s strategic objectives. The assessment draws primarily on the evidence that the organisation has available on its different result areas, namely its reports on results.

UNICEF’s results and reporting at organisation-wide level[21]

According to its mission statement, UNICEF is mandated by the United Nations General Assembly to “advocate for the protection of children's rights, to help meet their basic needs and to expand their opportunities to reach their full potential”.[22] Following these principles, UNICEF’s work at organisation-wide level is based on the Medium-Term Strategic Plan (MTSP) 2006-2009 (as noted in KPI 2).

UNICEF’s organisational plan is structured around a four-year cycle. Exceptionally, the MTSP 2006-2009 has been extended twice until 2013 to align its strategic planning cycle with the quadrennial comprehensive policy review of operational activities for development of the United Nations system. An annex, which contains a results framework and key performance indicators, is developed in conjunction the MTSP.

The MTSP 2006-2009 is organised around five focus areas (Young child survival and development, Basic education and gender equality, HIV/AIDS and children, Child protection and Policy advocacy and partnerships for children’s rights). Each focus area contains high- level results statements, known in UNICEF’s lexicon as key results areas, in which UNICEF will engage with partners and provide support through programmes of cooperation.

UNICEF communicates annually its results at organisation-wide level through a number of key documents: the Annual report of the Executive Director of UNICEF, the Annual Report, Thematic Reports, Regional Analysis Reports and the Annual report to the Economic and Social Council. UNICEF also takes stock of achievements and shortfalls in Mid-term Review and In-depth/End-of-cycle Reviews which are conducted effectively every two years. For an overview of the different documents used and their primary audiences, please see Volume II, Appendix VIII.

Data used for this assessment

For this pilot test in 2012, all performance information (organisation-wide level) provided from the most recent cycle (2006-2013) was reviewed. Attention was paid to the following elements: quality of results statements; relevance of the indicators, baselines and targets; the strength of the link between the result statement and results achieved; and the quality of evidence presented to substantiate the results achieved, including an assessment of contribution.

4.2.2 KPI A: Evidence of the Extent of Progress towards Organisation-Wide Outcomes

The assessment of KPI A reviewed survey data from donors at headquarters and documentation made available by the organisation.

Overall assessment

Figure 4.1 shows the overall rating for this KPI based on the review of UNICEF’s focus areas and their contribution to organisation-wide goals and priorities – as expressed in core UNICEF reports[23] and as indicated by surveyed stakeholders. The headings show the criteria that MOPAN used to assess each focus area and determine the overall rating (criteria met are indicated in blue). The final column shows the mean scores of the surveyed donors at headquarters, based on the same 6-point scale that was used in the assessment of UNICEF’s practices and systems.

Figure 4.2 KPI A: Evidence of the Extent of Progress toward Organisation-Wide Outcomes, Ratings

Evidence of the extent of UNICEF’s contributions

Finding 22: UNICEF has shown a commitment to improve its reporting practices at the organisational level. However, the reports being produced still do not allow for a clear assessment of UNICEF’s contributions to results in its defined focus areas.

Despite the work underway to improve its focus on results, the following weaknesses in UNICEF’s reporting and results frameworks can be identified.

Reporting and Communicating results: UNICEF produces a number of annual reports to meet its various commitments and respect different levels of accountability. While these provide a significant amount of information on the interventions that UNICEF is supporting, performance information is not aggregated or presented in such a way that one can easily measure its contribution to results at various levels.

Performance information presented often quantifies results at the lower level (i.e., outputs). Therefore, it is difficult to clearly see the effect that UNICEF interventions are having on children and women throughout the world. UNICEF’s thematic reports represented UNICEF’s best performance reporting. The assessment team found that they link thematic information to the organisation-wide results framework more effectively than other reports being produced by the organisation.

Theory of change: When analysing the narrative of the MTSP 2006-2009, it is difficult to appreciate the link between outputs and outcomes, or the lower and higher level results in UNICEF’s results framework. Therefore, it is fair to say that the narrative does not contain a basic theory of change – understood, in this case, as an explanation of how a set of low level results (or accomplishments) is intended to produce medium to long-term results. UNICEF would benefit from articulating a theory of change so that stakeholders could understand the rationale for its approach and the significance of the types of results that may have been achieved. In the absence of this, it is difficult to identify UNICEF’s contribution towards broad focus areas and its progress towards expected results.

Measuring Results: As noted in the assessment of KPI 2 (in section 3.3.2 on Strategic Management), the quality of UNICEF’s results framework is poor; the results statements do not meet either the SMART or the CREAM criteria.[24] In addition, the absence of baselines and targets, as well as the quality of indicators, do not allow for adequate monitoring of organisation-wide results.

Furthermore, UNICEF reports against a limited number of indicators in the Data Companion of the Annual Report of the Executive Director. The data shown in this report are not clearly linked to information in the results framework. To understand the data, the reader must explore, compare and review the data with the results framework.

Evidence of the extent of progress in UNICEF Focus Areas

Finding 23: Whereas surveyed stakeholders consider that UNICEF is making progress towards its organisation-wide outcomes, the review of UNICEF’s reports led to mixed conclusions. There is evidence of progress towards lower level results in all five focus areas; however, evidence of contribution to higher level results remains weak.[25]

A total of 41 MOPAN donors at headquarters responded to questions on results. They were asked whether UNICEF is making progress towards its institutional/organisation-wide outcomes. All five focus areas were consistently rated as strong, and the highest rating was for UNICEF’s progress towards its stated goals in young child survival and development.

UNICEF reports with credible documentation on certain low-level results, but its reporting on high-level results does not reflect a theory of change or explain its contribution to those results. Also, the organisation makes limited reference to baselines and targets when assessing progress in its organisation-wide reports.

Further, performance information is listed by indicator in the Data Companion to the Annual Report of the Executive Director, making it difficult to link performance information with the results framework by focus area. Similarly, UNICEF’s contributions are often reported on using national data. Consequently, UNICEF’s contribution at each level (indicators, organisational targets, key result areas) is difficult to assess. The following results, which are drawn UNICEF’s five organisation-wide strategic objectives/focus areas, provide some good examples of evidence that demonstrate positive changes to the lives of women and children as a result of UNICEF interventions.

Young child survival and development

MOPAN donors at headquarters were asked whether UNICEF is making progress towards its stated goals in the area of young child survival and development. The majority (90 per cent) rated UNICEF as adequate or above, and 68 per cent gave ratings of strong and very strong.

When reviewing UNICEF’s documents in the area of young child survival and development, the assessment team found some evidence of progress towards stated results. For example, in the Key Result Area 1 of Focus Area 1, the proportion of households consuming adequately iodized salt has increased. In Key Result Area 2, which focuses on support for national capacity to achieve MDGs 4 and 5, selected indicators on breastfeeding and vitamin A (coverage) have also shown small increases. While these data are valuable and show overall progress against certain metrics, UNICEF does not provide a clear explanation of its contribution to these results or to those of a higher level. This assessment was unable to discern a clear picture of UNICEF’s achievements from the documents available.

Basic education and gender equality

MOPAN donors at headquarters were asked whether UNICEF is making progress towards its stated goals in the area of basic education and gender equality. The majority (85 per cent) rated UNICEF as adequate or above, and 71 per cent gave ratings of strong and very strong.

The review of documents in the area of basic education and gender equality found that UNICEF achieves lower-level results based on evidence found in the Annual Report of the Executive Director. However, information about the effects of, for example, the Child Friendly Schools model in different countries is lacking. It is very challenging to discern the contribution that the Child Friendly Schools approach has had on girls’ education or increased enrolment and retention.

HIV/AIDS and children

MOPAN donors at headquarters were asked whether UNICEF is making progress towards its stated goals in the area of HIV/AIDS and children. The majority (80 per cent) rated UNICEF as adequate or above, and 66 per cent gave ratings of strong and very strong.

UNICEF’s Thematic Report 2011 on HIV/AIDS and Children contains performance information on outputs achieved in some countries but this is not linked to the indicators or baselines in UNICEF’s performance framework. It is worth noting that the Data Companion to the Annual Report of the Executive Director reports on only three indicators out of fifteen contained in the 2012 results framework.[26] Therefore, it is very difficult to assess progress against specific organisation-wide results.

Child protection from violence, exploitation and abuse

MOPAN donors at headquarters were asked whether UNICEF is making progress towards its stated goals in the area of child protection from violence, exploitation and abuse. The majority (80 per cent) rated UNICEF as adequate or above, and 56 per cent gave ratings of strong and very strong.

The review of documents in the area of child protection found evidence of improvement on some indicators. For example, there has been improvement in the proportion of children (aged 0-59 months) whose births have been registered and a decrease in the proportion of children (aged 5–14 years) involved in child labour.

Policy advocacy and partnerships for children's rights

MOPAN donors at headquarters were asked whether UNICEF is making progress towards its stated goals in the area of policy advocacy. The majority (83 per cent) rated UNICEF as adequate or above, and 51 per cent gave ratings of strong and very strong.

MOPAN donors at headquarters were asked whether UNICEF is making progress towards its stated goals in the area of partnerships for children's rights. The majority (71 per cent) rated UNICEF as adequate or above and 51 per cent gave ratings of strong and very strong; 27 per cent answered ‘don’t know’.

The review of documents for policy advocacy and partnerships for children’s rights has shown that there has been an important increase in countries and organisations using DevInfo over the years, which has led to strengthened national capacities for evidence- based policy-making. The Annual Report of the Executive Director and the 2011 Thematic Report (Policy Advocacy and Partnerships for Children’s Rights) provide contextual information on selected indicators (e.g., number of countries supporting DevInfo or comparable tools for monitoring and reporting on data relevant to children).

4.3 Evidence of the Extent of Contribution and Relevance at Country Level

4.3.1 Overview

This section presents the results of the assessment of evidence of UNICEF’s contributions to country level results and its relevance to stakeholders. By separating the KPI at the organisation-wide level from KPIs at the country level, MOPAN recognises the demand- driven nature of many of the activities of a multilateral organisation and the key role that is played by its country programming or strategy document, where expected results at the highest level (outcomes and impact) reflect a shared responsibility between the multilateral organisation and the partner country.

UNICEF’s results and reporting at country level[27]

As noted in the analysis of KPI 4, UNICEF’s work at the country level is based on a five-year programming cycle.

UNICEF communicates its results at country level through a number of key documents produced throughout the programme cycle. These include the Country Programme Document (CPD), the Country Office Annual Report (COAR) produced December 31 each year, and a Consolidated Results Report (CRR) at the end of a programme cycle. For an overview of the documents used and their primary audiences, please see Volume II, Appendix IX on Mapping UNICEF’s Country Level Reports.

In conjunction with the CPD, a Summary Results Matrix is developed. Before each programme component, this matrix briefly lists the related Millennium Development Goals, articles of the Convention on the Rights of the Child, national development priorities, and UNDAF outcomes.

Higher-level results (i.e., outcomes or impacts) are listed along with key progress indicators, baselines and targets. High level results statements are generally formulated in terms of the expected change in the situation of children or women and are categorised by programme component according to the organisation’s five strategic objectives (focus areas). An analysis of expected lower level results does not appear in these frameworks.

Data used for this assessment

For this pilot test in 2012, the country-level assessment is based on data from a sample of five of the nine MOPAN survey countries (Cambodia, Ghana, Niger, Philippines, and Zimbabwe), which were proposed by UNICEF based on the availability of results data.

All results-related information provided in documents from the most recently completed programming cycle in the five focus countries was reviewed. More specifically, UNICEF’s Country Programme Document, the Country Programme Action Plan, the Summary Results Matrix, the Mid-Term Review, Country Programme Evaluation, Consolidated Results Report, and any external reviews, assessments or evaluations carried out during that programming cycle. Attention was paid to the following elements: quality of the results statements; the relevance of indicators, baselines and targets; the strength of the link between results statements and results achieved; the quality of evidence presented to substantiate the results achieved, including an assessment of contribution; and, the overall performance story.

In the survey, direct partners and MOPAN donors in-country were asked questions that were tailored to each of the five countries. Interviews with senior UNICEF country office staff also informed the analysis of context and ensured that the assessment team had a full set of documentation with which to conduct the document review and analyse results.

The following sections on country-level KPIs present the overall results of this five-country review, with country-specific examples provided to illustrate types of results achieved.

4.3.2 KPI B: Evidence of the Extent of Contribution to Country- Level Goals and Priorities

This KPI indicates that an effective organisation would demonstrate contributions to country- level goals and priorities. The assessment reviewed survey data from in-country stakeholders (donors and direct partners) and documentation provided by five UNICEF country offices following interviews with senior country office staff.

Overall assessment

UNICEF’s results statements are meant to be “closely aligned to national priorities, refer to MDGs, to the mandate of the organisation and to the priorities of the Medium-Term Strategic Plan.”[28] This was the case in the results matrices of the five UNICEF country offices reviewed. Therefore, the assessment of this KPI focused on how UNICEF is demonstrating its contributions to the higher level planned results (outcomes and goals) associated with its five focus areas: young child survival and development; basic education and gender equality; HIV/AIDS and children; child protection from violence, exploitation and abuse; and policy advocacy and partnerships for children’s rights. As gender equality and human rights- based approaches are mainstreamed in all programming, they may not appear in results statements in the programming documents for these five countries.

Figure 4.3 shows the overall rating for this KPI based on the review of UNICEF’s contribution to country level goals and priorities – as expressed in UNICEF reports and as indicated by surveyed stakeholders’ views. It also shows the criteria that MOPAN used to assess each country and determine the overall rating (criteria met are indicated in blue).

Figure 4.3 KPI B: Evidence of the Extent of Contribution to Country-Level Goals and Priorities, Rating

* One example is the multi-intervention Accelerated Child Survival and Development model that was adapted to specific country contexts.
** Reporting on higher level results was notable in UNICEF’s water and sanitation sector and in their work with orphans and vulnerable children.

Stakeholder views (survey data)

As illustrated in Figure 4.4, survey respondents in each country gave high ratings to UNICEF for its contributions to result areas. Since some UNICEF country offices define their programming areas slightly differently, particularly in Focus Areas 1 and 5, these are shown in the table below.

Figure 4.4 Stakeholder Survey, Mean Scores by Focus Area and Country

Evidence of the extent of UNICEF’s contributions

Finding 24: Despite the existence of considerable performance information, it is difficult to understand the extent to which UNICEF contributes to development results at the country level from their core reports.

UNICEF country offices engage in highly consultative processes to develop their programming priorities and expected results. There is considerable evidence of strong links to government priorities. Further, UNICEF’s country-level reports are rich in data on many levels. Yet, despite the participatory approach and the wealth of data, it is challenging to develop a clear picture of how UNICEF is contributing to development results on the ground and of the scale and magnitude of that contribution. The few examples of where UNICEF’s contribution to higher-level results was well documented came from external sources.

This section highlights a few factors related to UNICEF’s practices and systems that limit UNICEF’s ability to present a full picture of the results achieved and its contribution to those at a higher level.

Theory of change: UNICEF has recently added a section in its Programme Policy and Procedures Manual on the ‘theory of change’ and its use in programme planning and reporting. The adoption of this approach seems to be in its infancy, which might explain why there were no examples found of a clearly described theory of change in any planning documents. As a result, only few examples were found of programming areas where the links between UNICEF’s outputs and higher-level national results seemed credible.

Results and Monitoring Framework: One of the primary weaknesses in UNICEF’s country- level reporting has been the lack of a single results framework and related monitoring system to track and report on performance against expected results.

Cohesive Planning and Reporting: The Country Programme Action Plan[29] results matrices (prepared at the beginning of the programming cycles) and Consolidated Results Report (prepared at the end of cycle) present different categories of information, which has made it difficult to understand what was planned and what actually happened. Also, the Consolidated Results Reports include many important elements (key results expected, key progress indicators, a description of results achieved, constraints and facilitating factors) but do not provide a description of inputs or interventions or any notion of how these contribute to the higher-level results that are also reported.

Quality of results framework: The quality of results content in country-level documents varies considerably.[30]

• Results statements are not always measurable, but even when they are, the results are often not measured.

• The results described in the Consolidated Results Report are often a mixture of completed activities, UNICEF outputs, and national outcomes.

• There is often a very weak correlation between the results statement and what is reported as results.

• Baselines, when they are provided, are often national figures. Since UNICEF interventions are often regional in scope, it is very difficult to assess its contribution when no analysis is made of the link between a sub-national intervention, the results achieved, and national-level data.

Measurement of results: Measuring the results of advocacy/upstream work is particularly challenging. The absence of theories of change is particularly problematic for upstream work, which has become a much greater percentage of UNICEF’s assistance, particularly in middle income countries. Results are often framed in terms of capacity building support that led to the finalisation of a plan, strategy or policy – but there is insufficient evidence of how this support affected the broader enabling environment, reduced risks to implementation, or enhanced sustainability.

Evidence of results achieved, by country

The following assessment of results achieved by country is drawn from both the survey results and an analysis of documentation consulted as part of the document review. Specific examples are given of results where the UNICEF country office provided particularly strong documented evidence that a result was achieved.

Cambodia

Generally, the majority of MOPAN donors in-country and direct partners rated UNICEF Cambodia as strong and above in all programming areas. MOPAN donors in-country, however, demonstrated a notable degree of unfamiliarity with certain programming areas: 42 per cent answered ‘don’t know’ regarding UNICEF Cambodia’s contributions to HIV prevention, care, treatment and support, and 33 per cent on UNICEF’s work in the areas of water, sanitation and hygiene practices, and in expanded basic education.

The document review found evidence of progress on many of the 22 results identified at the beginning of the programming cycle (2006-2010). The programme that provided the clearest plans and the strongest evidence documenting results from externally verified sources was the Seth Koma programme in which UNICEF Cambodia supported a community-based, multi-sectoral initiative that reached 1.4 million people in six provinces.

Regional data was often higher than the national average (e.g., the coverage of measles vaccination stalled between 2005 and 2010 at the national level but increased from a baseline of 78 per cent to 87 per cent in the six focus provinces; the national rates of household use of iodised salt remained the same between 2005 and 2008 while the six UNICEF-supported provinces registered an 8 per cent increase in consumption of iodised salt). The impact of the Seth Koma programme on these changes is difficult to quantify given that coverage may have improved across other provinces as well.

One other sector where UNICEF’s support did contribute to positive results at the national level was in the expanded basic education programme (EBEP). A review commissioned by SIDA and UNICEF clearly documents areas where UNICEF-supported activities lead to sustainable policy change, capacity development and knowledge acquisition. The evidence of contribution that was provided was clear and convincing. This is a model that could be adopted in UNICEF’s internal reporting.

Although there is evidence of UNICEF Cambodia’s progress on many indicators identified in internal documents at the beginning of the programming cycle, the performance story is quite difficult to discern in the documentation on programmes other than Seth Koma and in an external review of education activities. In many cases, the country-level documentation provides little analysis of UNICEF’s contribution to higher-level results. Further, there are many examples in UNICEF’s end-of-cycle reporting where “good progress” was noted and the data supports this, but UNICEF’s contributing role remained unclear. The Seth Koma programme and work in education seem to be the exception.

Ghana

When asked about UNICEF Ghana’s contributions to country-level goals and priorities, the majority of survey respondents (86 per cent) provided ratings of adequate or above. The document review focused on the 21 results to be achieved in UNICEF Ghana’s most recently completed Country Programme cycle (2006-2010) which was extended to 2011.

An analysis of the key programming documents, sector reviews and evaluations showed that UNICEF Ghana is achieving results at the output level and that some of the supported activities have led to higher-level results, particularly in the area of young child survival and development. In UNICEF’s main programme documents, the description of results achieved generally provides a clear performance story and the data used to measure results is credible. While the Consolidated Results Report presents progress against key indicators, which is largely positive, UNICEF Ghana’s contribution to any higher-level change is not always clear.

The greatest evidence that UNICEF’s interventions in Ghana have led to results for children and women is in a package of interventions based on UNICEF’s Accelerated Child Survival and Development model that was implemented in six districts in the upper east region of Ghana. These were adopted by the Ghanaian government in its national strategy for MDGs 4 and 6 and named ‘High Impact Rapid Delivery’.

Other evidence of higher-level results and UNICEF’s contribution was found in the eradication of guinea worm disease, which was considered eradicated in 2010.

Niger

When asked about UNICEF Niger’s contributions to country-level goals and priorities, the majority of survey respondents (71 per cent) provided ratings of adequate or above.

UNICEF Niger’s contributions to the national goal of child survival were rated strong or very strong by 65 per cent of respondents. 

There were high levels of ‘don’t know’ responses in Niger: 50 per cent on UNICEF Niger’s contribution to national goals in the area of planning, monitoring, evaluation and social policy, and in the area of advocacy and communication; 30 per cent on UNICEF contributions to the areas of basic education and the promotion of child rights and child protection.

The document review focused on the results achieved in UNICEF Niger’s current programming cycle (2009-2013). As in most UNICEF country offices, the expected results in the young child survival, education and child protection sectors are all defined at the level of national outcomes. In other intervention areas, UNICEF results are defined at the output level.

There is evidence of significant progress on some national indicators that UNICEF incorporated in its results matrix at the beginning of the UNICEF programming cycle – notably, reductions in the under-5 mortality rate and the maternal mortality rate and an increase in the primary school enrolment rate. There is some evidence that UNICEP has supported activities in the health and education sectors but these are often descriptions rather than measurable results.

The greatest evidence of UNICEF Niger’s contribution to results is in the area of young child survival and development – in particular, the emergency response to the recent nutritional crisis. Activities in support of community-led total sanitation have also contributed to positive results.

The Philippines

When asked about UNICEF’s contributions to country-level goals and priorities, the majority of respondents (80 per cent) provided ratings of adequate or above. In the areas of communication and local policy and institutional development, 85 per cent of respondents rated UNICEF Philippines contributions as adequate or above and 39 per cent as very strong (55 per cent of direct partners and 20 per cent of MOPAN donors in-country).

There was a high level of ‘don’t know’ responses (29 per cent) on UNICEF Philippines’ contributions to the prevention, care and treatment of HIV and AIDS.

Direct partners were more positive than MOPAN donors in-country on whether UNICEF Philippines is making appropriate contributions to national goals related to the prevention, care and treatment of HIV and AIDS, and the difference was statistically significant.

The document review examined the 17 expected results in six programming areas that UNICEF Philippines identified for the 2005-2009 programming cycle. Some of the “results achieved” presented in the Consolidated Results Report were quantified but they were almost exclusively lower-level results. Some of the strongest evidence of results for children in the Philippines was the emergency response in the education sector where over 81,000 pre-school and school-aged children benefited from an improved learning environment as a result of UNICEF’s interventions. UNICEF’s work to strengthen the child protection system through various initiatives seems to have benefited a large number of vulnerable children.

By and large, however, results achievement was difficult to assess in many areas as key progress indicators often lacked baselines, targets, or recent data. There was little correlation between indicators and the results achieved, and scant evidence provided to substantiate the few causality statements.

Zimbabwe

Both direct partners and MOPAN donors in-country were very positive in their assessments of UNICEF Zimbabwe’s contributions in different programming areas. Notably, 71 per cent of direct partners rated UNICEF Zimbabwe very strong in the area of young child survival and development, and 57 per cent in the areas of water, sanitation and hygiene; basic education and gender equality; and child protection. Among MOPAN donors in-country, more than 80 per cent rated UNICEF strong or above in: young child survival and development; water, sanitation and hygiene; child protection; and strategic planning, social policy and communication.

The document review examined the achievement of results identified in UNICEF’s Zimbabwe’s 2007-2011 programming cycle. The majority of the 18 results defined at the beginning of the cycle were higher-level and had to be revised mid-cycle due to the political crisis in 2008/9. UNICEF Zimbabwe’s mid-term review of 2009 stated that, as a result of the crisis, there was not much to measure and some of their original targets required significant revision.

Nevertheless, there is evidence of results achieved in a challenging context including a good self-assessment of UNICEF Zimbabwe’s contribution in some programming areas (e.g., in the water and sanitation sector) and external evaluations of the multi-sectoral support provided to orphans and vulnerable children. The most robust results-related data was found in external evaluations rather than in UNICEF’s own reports.

Examples

The document review focused first on UNICEF’s planning documents and reports and then consulted external reports and evaluations to complete its document review. The information in Figure 4.5 below represents the few examples of evidence of UNICEF’s contribution to country-level goals and priorities.

Figure 4.5 KPI B: Evidence of the Exent of Contribution to Country-Level Goals and Priorities

1 ‘Stunting’ refers to low height for age ratios and is caused by long-term insufficient nutrient intake and frequent infections.
2 ‘Wasting’ refers to low weight for height ratios and is a strong predictor of mortality of children under 5 years of age.
3 UNICEF Ghana Consolidated Results Report 2006-2011.
4 Final Report – The Retrospective Evaluation of ACSD: Ghana, p. 68.
5 Outputs included: 814 bicycles, 18 motorcycles and one vehicle to the high-impact districts (HIDs) over the course of the project for outreach and supervision activities; equipped health facilities with 553 refrigerator units for cold chain; supported local and national campaigns for vaccination and vitamin A supplementation; and, supplied commodities, including vitamin A supplements, anti-helminths, oral rehydration salts, insecticide treated nets and re-treatment chemicals for the prevention of malaria.
6 Outputs included: over 200,000 ITNs were distributed in the HIDs between 2002 and 2005; ACSD-supported retreatment efforts at the community and facility levels, as well as through campaigns starting in 2004; all health workers and volunteers involved in ITN distribution and retreatment received training.
7 Outputs included: 48 clinicians and 3 regional staff received standard 11-day training on Integrated Mother and Childhood Illnesses in 2005.
8 Outputs included: ACSD-Ghana provided support for the training and supervision of over 1,900 community-based agents (CBAs)in 600 communities to deliver messages to promote infant feeding, care-seeking and treatment of childhood illnesses, insecticide-treated nets and immunisation; CBAs received health kids containing chloroquine, oral rehydration salts and education materials on hand-washing; training and educational materials to community-based mothers’ groups for the promotion of infant feeding practices.
9 Outputs included: supplemental vaccination and support for facility and community-based distribution of vitamin A; regional scale-up of Intermittent preventative treatment for malaria in pregnancy (IPTp).

4.3.3 KPI C: Evidence of the Extent of Contributions to Relevant MDGs

This KPI recognises that multilateral organisations have made commitments to the Millennium Development Goals (MDGs) and assumes that they explicitly articulate or make links to the MDGs that they contribute to at the country level. The MDGs are collective, global targets that have, in many cases, been adapted by partner countries in defining their priorities While partner countries are responsible for making progress toward the MDGs, multilateral organisations ensure that their aid, knowledge, and other types of support facilitate achievement of these goals.

Links between UNICEF’s objectives and the MDGs

At an organisation-wide level, UNICEF has identified five strategic objectives on which it has focused since 2006. Each strategic objective has clearly defined links to specific MDGs and targets.

For example, the strategic intent of UNICEF’s Focus Area 1 on young child survival and development is that, “countries acquire the capacities and systems to ensure the right of the child to survival, growth and development and to achieve the highest attainable standard of health.” This is then linked to the Priority Goal – MDG 4: Reduce child mortality, and, more specifically, “to reduce under-5 mortality by two-thirds between 1990 and 2015.” Each UNICEF Focus Area makes similar links to related MDGs and targets at various levels.

Figure 4.6 shows the formal links between UNICEF’s Focus Areas and related MDGs, as stated in UNICEF’s Medium-Term Strategic Plan 2006-2009. The framework offers UNICEF country offices the ability to link the programming priorities agreed to with their government counterparts to specific MDG and targets. Evidence was found of these linkages in all results- based planning frameworks from the five countries included in this assessment.

Figure 4.6 UNICEF Focus Areas and Related MDGs

Overall assessment

Figure 4.7 shows the overall rating for this KPI based on the review of UNICEF’s contribution to relevant MDGs – as expressed in UNICEF reports and as indicated by stakeholders surveyed. It also shows the criteria that MOPAN used to assess each country and determine the overall rating.

Figure 4.7 KPI C: Evidence of the Extent of Contributions to Relevant MDGs, Overall Rating and Criteria

Evidence of the extent of UNICEF’s contributions

Finding 25: Whereas UNICEF’s results frameworks explicitly link expected results to relevant MDGs, there is little documented evidence of UNICEF’s contributions to particular MDG goals or targets.

As noted above, all UNICEF country offices make explicit links to the MDGs in their planning frameworks. In reporting, however, they often do not provide the link back to the MDG in the Consolidated Results Reports, nor do they explain the extent of their contribution to the MDG goal or target. For example, often the contribution in a particular MDG area is related to direct- service work being done at a local/district level (but only in a small number of districts), as well as actions at the policy level. Yet the reports do not articulate how these actions come together to bolster the partner country’s efforts, and, for most MDG areas, there is no direct association to UNICEF’s role and contribution. In Ghana, for example, UNICEF’s focus has been on quality of pre-school education, with the intent of increasing the percentage of children enrolled in Grade 1 with pre-school experience. This investment in children’s readiness to learn is associated with Ghana’s efforts to achieve net enrolment and primary school completion targets, yet the extent of UNICEF’s facilitative role in this area is not described.

In several of the MDG areas, UNICEF’s expected results are virtually the same as national priorities and national MDG targets. This makes it challenging to assess separately UNICEF’s contribution to national goals, as assessed in KPI B, and its contribution to national efforts to meet MDG targets as captured in this KPI.

Overview of survey data, by country

Finding 26: Surveyed stakeholders were generally positive about UNICEF’s contribution to the achievement of MDGs. On average, respondents in all countries rated UNICEF adequate or above.

Cambodia

When asked about UNICEF Cambodia’s contributions to MDGs at the country level, the majority of respondents (72 per cent) rated UNICEF adequate or above.

The highest mean scores were for UNICEF contributions to Cambodia’s efforts in Goal 4 (Reduce child mortality) and Goal 5 (Improve maternal health).

There were high levels of ‘don’t know’ responses from survey respondents on some MDGs: 47 per cent in relation to Cambodia’s contributions to Goal 7 (Ensuring Environmental Sustainability) and 50 per cent in relation to Goal 9, which is specific to Cambodia and focuses on the efforts to reduce casualties from land mines and explosive remnants of war.

Ghana

When asked about UNICEF Ghana’s contributions to MDGs at the country level, most respondents (83 per cent) rated UNICEF adequate or above.

The highest mean scores were for UNICEF contributions to Ghana’s efforts in Goal 4 (Reduce child mortality) and Goal 5 (Improve maternal health)

Among MOPAN donors in-country, 50 per cent responded ‘don’t know’ in relation to Goal 2 (Achieve universal primary education) and 25 per cent in relation to Goal 1 (Eradicate extreme poverty and hunger) and Goal 7 (Ensure environmental sustainability).

Niger

When asked about UNICEF Niger’s contributions to MDGs at the country level, most respondents (75 per cent) rated UNICEF adequate or above.

The highest mean score was for UNICEF contributions to Niger’s efforts in Goal 5 (Improve maternal health), and the lowest mean score was for its contributions to Goal 7 (Ensure environmental sustainability).

Although UNICEF’s contribution to Niger`s efforts to Goal 2 (Achieve universal primary education) and Goal 7 (Ensure environmental sustainability) were both seen as adequate by survey respondents, they also both received 22 per cent ratings of inadequate or below.The highest level of ‘don’t know’ responses (37 per cent) from both in-country donors and direct partners were in relation to Goal 7 (Ensure environmental sustainability).

Philippines

When asked about UNICEF Philippines contributions to MDGs at the country level, most respondents (75 per cent) rated UNICEF adequate or above.

The highest mean scores were for UNICEF contributions to the Philippines’s efforts in Goal 2 (Achieve universal primary education) and Goal 3 (Promote gender equality and empower women).

Although UNICEF’s contributions to the Philippines’s efforts towards Goal 7 (Ensure environmental sustainability) and Goal 3 (Promote gender equality and empower women) were seen as adequate and strong respectively, they received respectively 19 and 14 per cent ratings of inadequate or below. There were high levels of ‘don’t know’ responses (29 per cent) from both in-country donors and direct partners in relation to to Goal 2 (Achieve universal primary education), Goal 3 (Promote gender equality and empower women), and Goal 6 (Combat HIV/AIDS, malaria and other diseases).

Zimbabwe

When asked about UNICEF Zimbabwe’s contributions to MDGs at the country level, most respondents (88 per cent) rated UNICEF adequate or above.

The highest mean scores were for UNICEF contributions to Zimbabwe’s efforts in Goal 5 (Improve maternal health) and Goal 2 (Achieve universal primary education).

Although UNICEF’s contribution to Zimbabwe’s efforts towards Goal 7 (Ensure environmental sustainability) was perceived as adequate by stakeholders, it also received 24 per cent ratings of inadequate or below.

There were high levels of ‘don’t know’ responses (31 per cent) from both in-country donors and direct partners in relation to Goal 7 (Ensure environmental sustainability).

4.3.4 KPI D: Relevance of Objectives and Programme of Work to Stakeholders

For this KPI, MOPAN assessed relevance primarily as a measure of the extent to which a multilateral organisation supports country priorities and meets the changing needs of direct partners and the target population. The assessment is based exclusively on survey data gathered from direct partners and MOPAN donors in-country in the five countries selected for the test of this component focused on UNICEF results.

Overall assessment

Across the five countries, UNICEF was seen to be consistently strong in responding to its partner countries’ key development priorities, in providing innovative solutions to help address these challenges, and in adapting its work to the changing needs of its partner countries.

Figure 4.8 shows the overall assessment rating and the means scores on the three survey questions on which the assessment is based.

Figure 4.8 KPI D: Relevance of Objectives and Programme of Work to Stakeholders, Overall Rating and Survey Mean Scores by Country

Overall assessment: Strong

* Detailed scores are shown in Volume II, Appendix VI.

5. Conclusion

UNICEF’s organisational development process and related reforms are likely to improve its effectiveness and efficiency.

In response to the 2007 organisational review, UNICEF undertook an organisational development process that is likely to increase the effectiveness and efficiency of its operations. A new system (VISION) for managing financial resources and performance information is being implemented across the organisation and at regional and country levels. Survey respondents expressed confidence in UNICEF’s fiduciary systems, which include external and internal audit and anti-corruption policies, and the document review found that the organisation has very good practices in this area.

Like many other UN organisations, UNICEF is adopting International Public Sector Accounting Standards (IPSAS). There is also evidence of efforts to improve organisational policies, procedures and guidance as part of the regulatory framework, accountability system and enterprise risk management.

UNICEF remains strongly committed to a results-focused reform agenda and is improving its capacity to manage for development results. However, the frameworks that it uses to manage for and report on organisation-wide results need further improvement.

UNICEF’s organisation-wide strategy provides a clear and strong programming focus to a large and decentralised organisation. UNICEF also demonstrates a willingness to consult partners and use reviews, evaluations, and audits (e.g., Mid-Term Reviews) to improve its strategy. However, UNICEF faces challenges in managing for results and applying results-based management principles in its organisation-wide strategy. This shortcoming has an impact on the quality of the results frameworks linked to the Medium-Term Strategic Plan and the related programming tools used across the organisation. The document review highlighted the need to strengthen its application of results-based management through: improving the quality of results statements, strengthening the links between outputs and outcomes, identifying measureable indicators and consistently using baselines and targets, and integrating an analysis of UNICEF’s contribution to higher-level results.

UNICEF’s country programming processes have improved as a result of organisational changes but planning at the project level received low ratings.

Recent programmatic changes have improved the clarity of results frameworks and evaluation of results by specifying two levels of results that are clearly linked: intermediate results leading to programme component results. The country level Summary Results Matrices prepared from 2009 to the present include higher-level results statements, indicators, baselines and targets – all of which meet results-based management standards. UNICEF’s new, web-based monitoring system – VISION – should also improve the organisation’s ability to track progress against a wide range of programmatic and management indicators. It is important to note, however, that the document review found UNICEF to be weak in the development of appropriate indicators, baselines and targets, as well as the application of benefit or impact analyses at the project level.

UNICEF’s has proven particularly effective in mainstreaming gender equality and emergency responses/humanitarian action.

UNICEF’s strategic and programming documents articulate a clear institutional approach to both gender equality and emergency response. UNICEF’s organisational commitment to mainstream gender equality and emergency response/humanitarian action has led to positive results in the vertical and horizontal integration of these cross-cutting themes. For example, in its new Policy on Gender Equality and the Empowerment of Girls and Women 2010, UNICEF commits to mainstreaming gender in its operations and to develop accountability and monitoring mechanisms. Similarly, UNICEF’s approach to Emergency Response/Humanitarian Action is articulated in the 2010 Core Commitments for Children (CCCs) in Humanitarian Action, which includes the framework, principles, programme commitments, and accountabilities relevant to UNICEF’s work in this area.

UNICEF is valued for its field presence and engagement with governmental and non- governmental partners.

UNICEF was praised by respondents for its strong working relationships with and its support to its partners. Its efforts to strengthen government leadership and ownership, foster collaboration in a highly consultative and participatory manner, as well as to enhance national capacities are considered among UNICEF’s greatest assets.

UNICEF is highly valued by its direct partners.

In the four key management areas examined in the assessment of organisational practices and systems, UNICEF’s direct partners provided consistently positive ratings of the organisation, and rated it strong on all KPIs but one (use of country systems). These positive views carry through in the smaller sample of direct partners that were consulted about UNICEF’s results, who gave strong ratings on its contributions to partner country goals and priorities and country efforts to meet the MDGs. UNICEF’s programming is considered to be relevant at the country level. It is seen to respond to country priorities, adapt to changing conditions, and provide innovative solutions.

UNICEF allocates resources in a transparent manner, but has not yet fully implemented results-based budgeting.

UNICEF’s criteria for resource allocations are clear and publicly available. However, there is limited evidence to show the link between allocations or disbursements and expected results at the organisation-wide level. UNICEF is implementing results-based budgeting through a harmonised approach with UNFPA and UNDP.

In contexts where it has significant humanitarian programming, UNICEF is fulfilling its responsibilities as a Cluster Lead and is recognised for its respect for humanitarian principles.

UNICEF’s institutional culture strongly emphasises respect for humanitarian principles and these principles – namely, humanity, impartiality and neutrality – are reflected in the Core Commitments for Children in Humanitarian Action. Documented evidence and survey respondents’ views indicate that UNICEF respects humanitarian principles while delivering humanitarian assistance, and holds on-going policy dialogue with partners on the importance of observing humanitarian principles, particularly in cases of conflict.

Similarly, survey feedback from its peers, direct partners, and MOPAN donors suggest that UNICEF is playing a key role as a cluster lead in the nutrition, water and sanitation and education sectors in the Democratic Republic of Congo, Niger and Zimbabwe by ensuring sufficient analytical inputs and policy engagements, staff time, information flows, and financial forecasting.

UNICEF has strengthened its evaluation function but there is still room for improvement in some areas.

Survey respondents assessed UNICEF as strong in its inclusion of beneficiaries and direct partners in evaluations. With the adoption of a series of new procedures and processes, UNICEF’s evaluation office is reinforcing good evaluation practices. It has strengthened its practice of reviewing and reporting on the quality of evaluations at all levels. Further, it has adopted formal mechanisms to track the implementation of evaluation recommendations. However, the documents reviewed remain vague on the criteria used to determine coverage and prioritisation of evaluations. To address this, UNICEF issued guidance on the prioritisation of major evaluations at the decentralised level as a way of strengthening the relevance and use of evidence generated through evaluations.

Development Results Component

Although UNICEF has provided strong direction for results-based management in recent years, limitations in its frameworks and systems to report on organisation-wide results make it challenging to identify the difference being made in the lives of children and women.

The assessment team took note of the work being done by UNICEF to improve its results- based approaches and reporting – notably, the new VISION system and work to sharpen the programmatic focus on equity. Nevertheless, the organisation-wide reports reviewed showed inconsistencies in the presentation of results achieved and their relationship to expected results defined in the Medium-Term Strategic Plan. Further, the lack of description of the various steps in the results chain (inputs, outputs, outcomes, etc.) made it difficult to see UNICEF’s contributions to noted changes in the lives of children and women. For example, the Annual Report of the Executive Director in 2012 stated that there was a 38 per cent increase in the household use of iodised salt (to 70 per cent) in 67 countries, but did not identify the steps either planned or taken by UNICEF and others to bring about this positive result.

UNICEF’s thematic reports represent UNICEF’s best performance reporting as they link thematic information to the organisation-wide results framework more effectively than other reports produced by the organisation.

At the country level, stakeholders confirm the relevance of UNICEF’s programming and indicate that UNICEF makes contributions to its Country Programme results and to partner country efforts to achieve the MDGs. UNICEF reports do not yet adequately document UNICEF’s contributions in these areas.

According to survey respondents, UNICEF programming is relevant in the countries sampled because it responds to priorities, offers innovative solutions, and adapts to the changing needs of its partners. Across the five countries reviewed, stakeholders also perceived that UNICEF made contributions to each of the focus areas prioritised in the Country Programme Document. Some clear and credible examples of UNICEF’s contribution to national outcomes were found in the document review. The final evaluation of UNICEF Ghana’s Accelerated Child Survival and Development (ACSD) program, for example, was thorough and convincing in its analysis of attribution of UNICEF-supported interventions to improvements to child and maternal mortality levels in certain districts. Further, the reports on water and sanitation interventions supported by UNICEF Zimbabwe during the crisis of 2008/9 and UNICEF Cambodia’s education activities provided convincing evidence of improvements in the lives of women and children.

Unfortunately, however, these examples proved to be the exceptions rather than the rule. Most UNICEF reporting reviewed from previous cycles provided ample data on low-level results but failed to make the link between the interventions supported and the higher-level results achieved.

---

[1] The AfDB, UNICEF, UNDP, and the World Bank

[2] This component was tested in 2012 with the African Development Bank, UNICEF, UNDP, and the World Bank.

[3] The full methodology is presented in Volume II, Appendix I.

[4] MOPAN criteria for country selection include: multilateral organisation presence in-country, presence and availability of MOPAN members, no recent inclusion in the survey, the need for geographical spread, and a mix of low income and middle income countries (middle income countries being subdivided into lower middle and upper middle).

[5] The normal convention for statistical significance was adopted (p≤.05).

[6] These organisations were selected because they were assessed by MOPAN in 2009. The 2009 assessment focused on organisational effectiveness and was based only on survey data.

[7] Different organisations use different terms to refer to their planned results – they may be called goals, objectives, outcomes, etc.

[8] The “best fit” approach is used in public sector institutions (see Ofsted, 2011: Criteria for making judgments)

[9] Respondents who wrote “no comment” or the like were removed from the analysis. Comments that included more than one element were coded in various categories.

[10] While most KPIs and micro-indicators were considered in the document review, not all were rated. See section 2.3.

[11] The normal convention for statistical significance was adopted (p≤.05). See section 2.2, which describes the data analysis process.

[12] The UNICEF Mission Statement, adopted by the UNICEF Executive Board in 1996; the Convention on the Rights of the Child; the Millennium Summit Declaration of the General Assembly; A World Fit for Children adopted by the General Assembly Special Session on Children in 2002.

[13] Contrary to external financial audits conducted by the UNBOA, internal audits do not include financial statements.

[14] Prior to June 2012, the Office of Internal Audit and Investigation was referred to as the Office of Internal Audit.

[15] UNICEF. (2011). Performance Appraisal and Rebuttal Process. (p. 1).

[16] UNICEF. (2011). E-PAS User Guide. (p. 5).

[17] UNICEF. (2002). Guidance for Collaboration with NGOs and CBOs EXD2001-013. (p. 9).

[18] UNICEF. (2007). UNICEF Organizational Review (Synthesis Report). (p. 10)

[19] UNICEF. (2010). Core Commitments for Children in Humanitarian Action. (p. 6).

[20] Each organisation may use a different term to refer to this level of results.

[21] Please refer to the section of this assessment on KPIs 2, 19 and 20 for the analysis made of UNICEF’s results-based systems and practices.

[22] UNICEF. (1996). Report of the Executive Board of the United Nations Children’s Fund on the Work of its First Regular Session of 1996. (p. 1).

[23] A list of documents consulted is provided in Vol. II, Appendix X.

[24] CREAM (clear, relevant, economic, adequate, monitor-able) and SMART (specific, measurable, achievable, relevant, time bound).

[25] Reference is made in this chapter to “lower-level results” (usually ‘outputs’) and “higher-level results” (usually ‘outcomes’ or ‘impact’) rather than to standard OECD-DAC definitions. This has been done purposefully as UNICEF changed its results nomenclature in 2008/9 from ‘activities, outputs, outcomes and impact’ to ‘intermediate results’ and ‘programme component results’. Therefore, it proved challenging to discuss results in different plans or reports when the nomenclature was different. Further, these simplified designations of “low” and “high” have facilitated a focus on the results chain and its logic, rather than whether the terminology used was correct or appropriate.

[26] Indicator “Risk behaviours among young people” which has been absent of the 2010 and 2012 results frameworks, is still reported on in the Data Companion to the Annual Report of the Executive Director.

[27] Please refer to the sections in chapter 3 on KPIs 2, 4, 18 and 20 for the analysis of UNICEF’s results-based systems and practices.

[28] UNICEF (2012). Programme Policy and Procedure Manual. (p. 144).

[29] The new CPAP results frameworks (including Programme Component and Intermediate Results) are clearer than their predecessor but it is too early to determine if these will help UNICEF in the analysis and presentation of results achieved more clearly.

[30] Documentation from previous programming cycles (usually 2006-2010/2011) was reviewed as part of this assessment of UNICEF’s results. In some cases, weaknesses in the definition and measurement of results in completed programme cycles have been addressed in the results matrices of new programme cycles, some of which were analysed in KPI 4.