Preface

The Multilateral Organisation Performance Assessment Network (MOPAN) is a network of donor countries with a common interest in assessing the organisational effectiveness of multilateral organisations and their measurement and reporting on development and/or humanitarian results. MOPAN was established in 2002 in response to international forums on aid effectiveness and calls for greater donor harmonisation and co- ordination.

Today, MOPAN is made up of 17 donor countries: Australia, Austria, Belgium, Canada, Denmark, Finland, France, Germany, Ireland, The Netherlands, Norway, the Republic of Korea, Spain, Sweden, Switzerland, the United Kingdom, and the United States. For more information on MOPAN and to access previous MOPAN reports, please visit the MOPAN website (www.mopanonline.org).

Each year MOPAN carries out assessments of several multilateral organisations based on criteria agreed by MOPAN members. Its approach has evolved over the years, and since 2010 has been based on a survey of key stakeholders and a review of documents of multilateral organisations. MOPAN assessments provide a snapshot of four dimensions of organisational effectiveness (strategic management, operational management, relationship management, and knowledge management). In 2013, MOPAN has integrated a new component to examine the evidence of achievement of development and/or humanitarian results to complement the assessment of organisational effectiveness.

MOPAN 2013

In 2013, MOPAN assessed four multilateral organisations: the Asian Development Bank (ADB), the International Fund for Agricultural Development (IFAD), the World Food Programme (WFP) and the World Health Organization (WHO).

MOPAN Institutional Leads liaised with the multilateral organisations throughout the assessment and reporting process. MOPAN Country Leads monitored the process in each country and ensured the success of the survey.

 

Acknowledgements

We thank all participants in the MOPAN 2013 assessment of WHO. WHO’s senior management and staff made valuable contributions throughout the assessment and document review processes and provided lists of their direct partners, technical partners and peer organisations to be surveyed. Survey respondents contributed useful insights and time to respond to the survey. The MOPAN Institutional Leads, Belgium and the Netherlands, liaised with WHO throughout the assessment and reporting process. The MOPAN Country Leads oversaw the survey planning process in the field and ensured the success of the survey. Consultants in each country provided vital in-country support by following up with country-level survey respondents to enhance survey response rates.

Roles of authors and the MOPAN Secretariat

The MOPAN Chair was held by Australia in 2013 and worked in close co-operation with the MOPAN Technical Working Group to launch and manage the survey. Once the MOPAN Secretariat was established in April 2013, the Head of the Secretariat oversaw all related tasks.

MOPAN developed the Key Performance and Micro-Indicators, designed the survey methodology, co- ordinated the preparation of the lists of survey respondents, and approved the final survey questionnaire for each agency. MOPAN also directed the approach to the document review and oversaw the design, structure, tone, and content of the institutional reports.

Universalia and Epinion developed the survey instrument and carried out the survey and analysis. Universalia carried out the document review, conducted the interviews with multilateral organisation staff at headquarters and country levels, and wrote the institutional reports.

Epinion is a consulting firm in Denmark that analyses and evaluates data to support decision making. It conducts specially designed studies for public and private organisations based on data collected among an organisation’s employees, members, customers, partners, and other sources. Epinion has 75 employees and 200 interviewers. Website: www.epinion.dk

Universalia Management Group is a Canadian consulting firm established in 1980 that specialises in evaluation and monitoring for international development. Universalia has made significant contributions to identifying best practices and developing tools in the fields of organisational assessment; planning, monitoring, and evaluation; results-based management; and capacity building. Website: www.universalia.com.

Acronyms

AfDB - African Development Bank
CCA - Common Country Assessment
CCS - Country Co-operation Strategy
COMPAS - Common Performance Assessment System
ERF - Emergency Response Framework
FWC - Women's and Children's Health Cluster
GER - Gender, Equity and Human Rights
GPW - General Programme of Work
GSM - General Management System
HQ - Headquarters
IFAD - International Fund for Agricultural Development
IHP - International Health Partnership
IPSAS - International Public Sector Accounting Standards
JIU - Joint Inspection Unit
KPI - Key Performance Indicator
MDG - Millennium Development Goal
MI - Micro-indicator
MOPAN - Multilateral Organisation Performance Assessment Network
MRF - Management results framework
MTR - Mid-Term Review
MTSP - Medium Term Strategic Plan
NHPSP - National Health Policy, Strategy or Plan
ODA - Overseas Development Assistance
OECD-DAC - Organisation for Economic Co-operation and Development – Development Co-operation Directorate
OIOS - Office of Internal Oversight Services
OSER - Office and Country Specific Expected Result
OWAP - Action Plan for Gender, Equity and Human Rights Mainstreaming
OWER - Organisation-wide Expected Result
PBA - Programme-based approach
PD - Paris Declaration
PMDS - Performance Management Development System
RBB - Results-based budgeting
RBM - Results-based management
UNDAF - United Nations Development Assistance Framework
UNDG - United Nations Development Group
UNDP - United Nations Development Programme
UNEG - United Nations Evaluation Group
UNICEF - United Nations Children's Fund
WFP - World Food Programme

Executive summary

This report presents the results of an assessment of the World Health Organization (WHO) conducted by the Multilateral Organisation Performance Assessment Network (MOPAN). The MOPAN Common Approach examines organisational systems, practices, and behaviours that MOPAN believes are important for aid effectiveness. It also examines the extent to which there is evidence of an organisation’s contributions to development and/or humanitarian results, and relevance to stakeholders at the country level.

To achieve the attainment by all peoples of the highest possible level of health, WHO provides leadership on global health matters; shapes the health research agenda; sets norms and standards; articulates evidence-based policy options; provides technical support to countries; and monitors and assesses health trends. Globally, WHO directs and co-ordinates health-related activities within the UN system.

In 2013, WHO was engaged in an organisational reform process in which new strategies and policies were adopted to improve its organisational effectiveness. The three main objectives of the reform process are improved health outcomes, greater coherence in global health, and to be an organisation that pursues excellence. The reform process includes three areas of focus: programmes and priority setting, governance reform, and managerial reform. In addition, the 12th General Programme of Work and Programme Budget 2014-2015, recently approved by the Board, include commitments to improving organisational alignment; enhancing performance, accountability and transparency; and strengthening results-based planning and performance measurement.

As with all MOPAN assessments conducted in 2013, the assessment was based on data collected in the first half of 2013 and on the systems, policies and practices in place at the time. The report recognises actions that were underway at WHO the time of the assessment (some of which may be consolidated as institutional practice by December 2013), but these could not be used as evidence in the assessment.

MOPAN assessment

In 2013, MOPAN assessed WHO based on information collected through interviews with WHO staff, a survey of key stakeholders, and a review of documents. The survey targeted WHO’s direct partners, technical partners and peer organisations, as well as MOPAN donors based in-country and at headquarters. Six countries were included in the MOPAN survey of WHO: Ethiopia, Indonesia, Guatemala, Mozambique, Pakistan and Viet Nam. A total of 394 respondents participated in the survey - 49 MOPAN donors based at headquarters, 39 MOPAN donors based in-country, 169 direct partners, 115 technical partners, and 22 representatives of peer organisations. MOPAN’s document review assessed WHO through an examination of close to 1300 publicly available corporate documents and internal country programming and reporting documents from all six countries. MOPAN also interviewed WHO staff members (21 from WHO’s headquarters, 19 from its regional offices, and 23 from its country offices). This information was not coded or used formally as part of the assessment process, but rather to gain a broader contextual understanding of the organisations systems, practices and results-related reporting.

MOPAN assessments provide a snapshot of four dimensions of organisational effectiveness (strategic management, operational management, relationship management, and knowledge management) and of the organisation’s reporting on its development results. The main findings of the assessment of WHO in these performance areas and in the section on development results are summarised below.

Key Findings

Strategic management

In the area of strategic management, MOPAN established criteria to determine if a multilateral organisation has strategies and systems in place that reflect good practice in managing for development results. Overall, the 2013 assessment found that WHO has not yet fully developed these systems at all levels of the organisation, but has initiated promising reforms:

• WHO is performing well in providing direction for results. Although WHO does not have an organisation- wide policy that describes the nature of results-based management, its organisation-wide results framework is outlined and described in a suite of documents and, through its reform process, the organisation is putting emphasis on improving its results orientation.

• WHO’s organisation-wide strategy is based on a clear mandate that is revised periodically to ensure continuing relevance. Survey respondents consider WHO’s role in technical co-operation and in setting global norms and standards as key strengths.

• To promote and facilitate the institutional mainstreaming of gender, equity and human rights (GER), WHO established performance standards for GER in May 2012. WHO is effectively promoting the principles of good governance and the document review rated WHO as very strong for the promotion of environmental health as a focus area.

• There is room to strengthen WHO’s corporate focus on results as expressed in the MTSP 2008-2013 (amended draft), particularly the quality of its results frameworks and performance indicators, as well as the strength of the causal linkages from activities and outputs to higher level results.

• WHO’s country co-operation strategies and workplans are well aligned with national strategies and the UNDAF, but the country focus on results was identified as an area for improvement, particularly with regard to the quality of its results framework and results-based management practices.

• WHO has taken steps to address these shortcomings. The organisation is in the midst of a major reform process that should lead to considerable improvements in its focus on results (including new results chain and theory of change). These changes have been approved by the Executive Board and will be fully implemented in 2014.

Operational management

In operational management, MOPAN established criteria to determine if a multilateral organisation manages its operations in a way that supports accountability for results and the use of information on performance. Overall, the 2013 assessment found that:

• While WHO makes its criteria for allocating funding publicly available, improvement is needed in the consistent use of criteria and validation mechanisms, as well as in the level of transparency related to the actual allocation of resources to countries. Transparency in relation to resource allocation is a particular concern for donors at headquarters.

• WHO’s reports to its stakeholders do not yet demonstrate the link between budget allocations and expenditures and expected programmatic results, although some of these indicators are tracked through internal management instruments. As part of its reform process, WHO will implement a new results-based budgeting system (RBB) based on a revised results chain with a methodology for costing of outputs and an approach to assess contribution of outputs to outcomes.

• Financial accountability is seen as one of WHO’s strengths. The systems and practices in place for external and internal audits are well detailed and there is evidence that policies are followed.

• The organisation received mixed ratings for its use of performance information. It was perceived as doing well in using performance information to revise and adjust policies and manage poorly performing initiatives, but using performance information to plan new interventions as well as acting upon evaluation recommendations were identified as areas for improvement.

• The document review noted that WHO has systems in place to conduct performance assessments and reward staff. However, the recent review conducted by JIU signals human resource management as the most complex and problematic area of the WHO administration and suggests that these systems have been applied inconsistently. As part of its reform agenda, WHO is making efforts to review its practices and make further improvements in this area.

• WHO adequately uses milestones and targets to monitor the implementation of its activities, but could improve its use of benefit/impact analyses to plan new initiatives.

• WHO is performing well in the delegation of authority to the country level with decentralised proce- dures in which country offices have a certain level of autonomy in making adjustments and changes to activities, including revising budget allocations. WHO’s country offices are also responsible for defining activities, products and services, determining costing, and setting indicators, baselines and targets for the results planned.

• WHO received high ratings from survey respondents for its adherence to humanitarian principles in its field operations due to continued improvements in its organisational practices for humanitarian action.

Relationship management

In relationship management, MOPAN established criteria to determine if a multilateral organisation is engaging with its clients at the country level in ways that contribute to aid effectiveness. Overall, the 2013 assessment found that:

• WHO was perceived as adequate overall for its support of national plans and taking into account local conditions and capacities. WHO’s procedures were found to be easily understood by partners and the time for procedures did not seem to delay implementation.

• WHO was rated strong for ensuring that ODA disbursements/support are recorded in national budgets and for avoiding parallel implementation structures. It makes adequate use of country systems consid- ering that it mostly provides technical co-operation to countries, rather than direct project funding.

• WHO makes a strong contribution to policy dialogue while respecting the views of its partners in the process. This aspect of WHO’s work received some of the highest survey ratings, highlighting the nature of WHO as a technical organisation.

• WHO was recognised as adequate for the harmonisation of its procedures with other actors. All the countries sampled for this assessment reported high levels of compliance on the extent to which WHO’s technical co-operation is disbursed through co-ordinated programmes.

• WHO’s management of the Global Health Cluster was perceived as adequate by survey respondents.

Knowledge management

In knowledge management, MOPAN established criteria to determine if a multilateral organisation has reporting mechanisms and learning strategies that facilitate the sharing of information inside the organisation and with the development community. Overall, the 2013 assessment found that:

• WHO has an independent evaluation unit, but needs to increase its evaluation coverage and improve the quality of the evaluations conducted. As part of its reform agenda, WHO approved an Evaluation Policy in 2012. In addition, the Office of Internal Oversight Services plans to establish a Global Network on Evaluation, disseminate an Evaluation Handbook, present an annual workplan for evaluation, develop a web-based inventory of evaluations, and recruit additional staff to improve the capacity of the unit. While WHO’s commitment to evaluation is evident and appears to be bringing positive changes, it is too early to assess the full effects of the reform in this area.

• WHO is doing well in using performance information in reporting against its corporate strategy, on aid effectiveness, and on adjustments to policies, strategies and budgets. There is room for improvement in using performance data to report on the achievement of outcomes and on programming adjustments made during implementation.

• WHO continues to be committed to solidifying its role as a provider of knowledge on health (e.g. practices, statistics and research), but room for improvement was noted in reporting on how lessons learned and best practices are transforming the organisation’s programming.

• WHO makes many documents available to the public, but does not yet have a disclosure/access to information policy.

Development results

In the 2013 development results component, WHO was rated inadequate in providing evidence of progress towards organisation-wide outcomes (KPI A) and evidence of contribution to country-level goals and priorities, including relevant MDGs (KPI B). Survey respondents rated WHO adequate for the relevance of its objectives and programme of work to country level stakeholders (KPI C). These findings should be considered in conjunction with the findings above on WHO’s systems and practices for organisational effectiveness.

• Evidence of extent of progress towards organisation-wide outcomes: WHO’s Performance Assessment Reports for 2008-2009 and 2010-2011 provide some evidence of progress towards planned activities and outputs in WHO’s framework, but unclear and limited evidence of the results and contributions that WHO is making to organisation-wide outcomes (i.e. higher-level change).Some of these shortcomings may be resolved by the current reform and the introduction of a new results chain that links the work of the Secretariat (outputs) to the health and development changes to which it contributes, both in countries and globally (outcomes and impacts). In addition, the emphasis on theories of change will help WHO to present more compelling evidence of its contributions to the health sector in the countries where it works.

• Evidence of extent of contribution to country-level goals and priorities, including relevant MDGs: The document review highlighted both strengths and shortcomings of WHO’s reporting. While the organisation consistently reports on the achievement of Country and Office Specific Expected Results (OSERs), the performance information reported does not provide sufficient evidence of the extent of contribution to country-level goals and priorities as it does not capture the actual progress achieved in relation to targets. Moreover, WHO reports, which are based primarily on self-reported data, do not consistently provide a clear picture of the nature, magnitude, or relative importance of WHO’s contributions to some of the changes reported at the country level. The relatively poor quality of these documents (inconsistent use of indicators, baselines and targets) also limited the extent to which the work of WHO could be assessed.

• Relevance of objectives and programme of work to country level stakeholders: Surveyed stakeholders in-country considered WHO strong overall in responding to the key development priorities of client countries and adequate in providing innovative solutions to help address challenges and in adapting its work to changing country needs.

Changes since the 2010 MOPAN assessment

Survey data suggest that WHO’s performance has remained steady over the past three years. The 2010 and 2013 survey data differed slightly in only four instances. Survey respondents in 2013 were less positive than respondents in 2010 on three micro-indicators (expected results developed in consultation with direct partners/beneficiaries; new programmes/projects can be approved locally within a budget cap; and participation in joint missions). However, they were more positive on the independence of the WHO evaluation unit.

Conclusions on organisational effectiveness

The MOPAN assessment provides a snapshot of WHO’s organisational effectiveness based on the practices and systems in place at the time of the assessment.

WHO’s commitment to organisational development and its related reform agenda are likely to improve its effectiveness and efficiency, although it is too early to assess the effects of the process.

This MOPAN assessment took place during the early stages of WHO’s implementation of an ambitious Reform Agenda. As part of this reform, the organisation is aiming to develop a set of agreed global health priorities that will guide the organisation, achieving greater coherence in global health and resolving the relative lack of clarity on the roles and functions at the country, regional and global levels. The assessment found that positive changes in systems and practices have already resulted from this process; some are well underway and others have yet to be initiated. The reform agenda is being monitored and the Board receives updates on its progress.

WHO’s mandate and comparative advantages provide a good foundation for its focus on results.

WHO is committed to revising its mandate to ensure continuing relevance. Together, the 11th General Programme of Work and the Mid-Term Strategic Plan 2008-2013 articulate the organisation’s goals and priorities and provide a clear indication of the manner in which WHO will implement the mandate during this period. WHO has also made significant improvements in defining and addressing the organisation’s priorities in developing the 12th General Programme of Work.

There is room to further strengthen WHO’s results-based management practices and tools used to manage for and report on organisation-wide results.

The MOPAN assessment found that Mid-Term Strategic Plan 2008-2013 results statements inconsistently labelled activities, outputs and/or outcomes. In addition, the results-based framework is missing levels of results between the organisation’s activities and outputs and the intermediate outcomes it aims to achieve (results chain). This discrepancy trickles down to most related performance indicators. The lack of a clear chain of plausible results from one level to the next limits the organisation’s ability to monitor and report on performance.

WHO has committed to strengthening results-based management across the organisation and is working to improve planning, monitoring, and reporting at all levels. The Programme Budget 2014-2015 includes an improved results framework.

WHO was rated as inadequate with regard to results-based budgeting, but it is introducing a new results- based budgeting system (RBB) based on a revised results chain with a methodology for costing of outputs and an approach to assess contribution. These reforms, if implemented as planned in 2013-2014, represent important steps towards becoming a more performance-oriented and accountable organisation.

WHO is commended for its technical assistance, staff expertise, normative and standard-setting work, and its convening and regulatory functions.

WHO’s technical assistance and country-level operations, staff expertise, and normative and standard setting role were seen as key organisational strengths in the 2013 MOPAN assessment. This was reflected in comments to open-ended questions, in which survey respondents highlighted WHO’s support in the development of national health strategies and plans.

Both survey respondents and the document review also commended WHO for its convening and regulatory functions, as well as its knowledge management function in the health sector. Its convening role in the negotiation of health regulations and treaties is identified as a key facet of this normative and standard-setting work. WHO performs various critical functions in the health sector, such as translating global science and evidence into products for policy-making purposes in countries, co-ordinating surveillance and response to international health threats, and gathering and disseminating the best information available on appropriate health practices.

WHO has sound policies and processes for financial accountability but does not yet have strong practices for risk management.

WHO has strong systems in place for internal and financial audits (including organisational audits), strong policies for anti-corruption, systems for immediate measures against irregularities, and effective procurement and contract management processes. The organisation is working on an organisation-wide common framework and harmonisation of risk management practices.

WHO has strengthened its evaluation function but there is still room for improvement in the coverage and quality of evaluations.

WHO has invested considerable resources in this area and is in the process of strengthening its evaluation function. While it is making progress in systems and practices, the MOPAN assessment found that there is room for improvement in the coverage and quality of evaluations. When fully implemented, the 2012 Evaluation Policy and related procedures for quality control could help to address some of the weaknesses noted by the assessment.

In contexts where it has significant humanitarian programming, WHO is fulfilling its responsibilities as a Cluster Lead and is recognised for respecting humanitarian principles.

WHO has improved its institutional capacity with regard to the application of its humanitarian mandate. WHO’s Emergency Response Framework (ERF) and the Inter-Agency Standing Committee’s Global Health Cluster Guide articulate its humanitarian mandate. Survey respondents felt that WHO adequately respects humanitarian principles and maintains on-going policy dialogue with partners on the importance of observing humanitarian principles in delivering emergency assistance. They also perceived the organ- isation performing adequately in managing the Global Health Cluster.

Conclusions on evidence of WHO’s development results and relevance to stakeholders

Limitations in WHO’s frameworks and systems to report on organisation-wide expected results make it challenging to fully understand WHO’s performance story and identify its contribution to each of its strategic objectives.

The assessment noted the work being done by WHO, under its 11 strategic objectives, to fulfil its mandate: “to achieve the attainment by all peoples of the highest possible level of health”. [1]

Surveyed stakeholders consider that WHO is making progress towards its organisation-wide strategic objectives and the document review found evidence of progress towards organisation-wide expected results in some strategic objectives. However, the data presented was largely self-reported and did not include data collected systematically and verified by a robust evaluation function.

In the absence of a clear results chain or theories of change, WHO’s organisation-wide reporting provides limited links between activities, outputs and outcomes and does not allow for an assessment of WHO’s contributions at the outcome level.

Country-level stakeholders confirm the relevance of WHO’s work and indicate that it makes contributions to its office and country-specific expected results and to partner country efforts to achieve the MDGs. However, despite considerable normative and technical investments and support to countries, WHO fails to provide strong evidence or a clear picture of the nature, magnitude or relative importance of its contributions to changes at the country level.

Although stakeholders see WHO’s work as relevant to country priorities, WHO reported limited progress towards achieving its office and country-specific expected results in the six countries sampled for the assessment.

While WHO does good work at the country level, the extent to which its contribution to country-level goals and priorities can be assessed is limited by the design of its results-based management systems and tools and by the poor quality of its performance and results-related data. The document review found limited performance information by which to understand WHO’s performance story in the six countries sampled for the assessment. It is difficult to understand how WHO’s interventions in each country contribute to achieving national goals and priorities as there is no clearly articulated chain of results. In fact, there is considerable disconnect between the national goals and priorities included in the NHPSP, the strategic priorities and interventions in the Country Co-operation Strategy, the MTSP OWERs, and WHO’s country workplans.

WHO provides consistent data on performance indicators across programme budgets, but data reliability is compromised by the absence of independent and external sources, such as evaluations.

WHO’s performance measurement system relies almost exclusively on self-reported data from Country Offices. The MOPAN assessment found very few independent evaluations that could validate the reported results achieved; the evaluations that have been conducted were in very specific, technical areas that were not relevant to this assessment. WHO’s reporting on its progress towards organisation-wide expected results would benefit from performance information provided through independent evaluations of sectors, strategic objectives, specific themes and/or regions.

Overall MOPAN ratings of WHO

The two charts below show the ratings on the key performance indicators that MOPAN used to assess WHO in 2013. The first chart shows the ratings on 23 indicators designed to measure organisational effectiveness (practices and systems), and the second chart shows ratings on the three indicators designed to assess WHO measurement and reporting on development results. The indicators were adapted to the work of the World Health Organization to encompass its normative role and its reform agenda.

Organisational effectiveness– overall ratings

Evidence of contribution to development results and relevance to stakeholders – overall ratings

Background

MOPAN was established in 2002 in response to international forums on aid effectiveness and calls for greater donor harmonisation and co-ordination. The purpose of the network is to share information and experience in assessing the performance of multilateral organisations. MOPAN supports the commitments adopted by the international community to improve the impact and effectiveness of aid as reflected in the Paris Declaration on Aid Effectiveness, the Accra Agenda for Action, and the Busan High Level Forum. MOPAN’s processes and instruments embody the principles of local ownership, alignment and harmon- isation of practices, and results-based management (RBM).

MOPAN provides a joint approach (known as the Common Approach) to assess the organisational effectiveness of multilateral organisations and their measurement of and reporting on development results. The approach was derived from existing bilateral assessment tools and complements and draws on other assessment processes for development organisations – such as the bi-annual Survey on Monitoring the Paris Declaration on Aid Effectiveness and annual reports of the Common Performance Assessment System (COMPAS) published by the multilateral development banks. In the long term, MOPAN hopes that this approach will replace or reduce the need for other assessment approaches by bilateral donors.

MOPAN assesses four dimensions of organisational effectiveness, and evidence of contribution to development and/or humanitarian results

MOPAN has defined organisational effectiveness as the extent to which a multilateral organisation is organised to contribute to development and/or humanitarian results in the countries or territories where it operates. Based on a survey of stakeholders, a review of documents and interviews with multilateral organisation staff, MOPAN assessments provide a snapshot of a multilateral organisation’s effectiveness (see Chapter 3) in four dimensions:

• Developing strategies and plans that reflect good practices in managing for development results (strategic management)

• Managing operations by results to support accountability for results and the use of information on performance (operational management)

• Engaging in relationships with direct partners/clients and donors at the country level in ways that contribute to aid effectiveness and that are aligned with the principles of the Paris Declaration and subsequent related agreements on aid effectiveness (relationship management)

• Developing reporting mechanisms and learning strategies that facilitate the sharing of knowledge and information inside the organisation and with the development community (knowledge management).

In 2012, MOPAN piloted a new component to assess a multilateral organisation’s reported contributions to development results. This component which has become an integral component of all assessments in 2013, examines three areas of performance: evidence of the extent of the progress towards its institutional/organisation-wide results, evidence of contributions to country-level goals and priorities (including relevant millennium development goals (MDGs)), and stakeholder perceptions of the relevance of its objectives and programme of work. See Chapter 4.

Purpose of MOPAN assessments

MOPAN assessments are intended to:

• Generate relevant, credible and robust information MOPAN members can use to meet their domestic accountability requirements and fulfil their responsibilities and obligations as bilateral donors

• Provide an evidence base for MOPAN members, multilateral organisations and their partners/clients to discuss organisational effectiveness and reporting on development and/or humanitarian results

• Support dialogue between individual MOPAN members, multilateral organisations and their partners/clients to build understanding and improve organisational performance and results over time at both country and headquarters level.

The MOPAN methodology is evolving in response to what is being learned from year to year, and to accommodate multilateral organisations with different mandates. For example, in 2011, MOPAN began adapting the indicators and approach for the review of a global fund and organisations with significant humanitarian programming. In 2013, MOPAN is using a modified framework to assess the World Food Programme (WFP), an organisation with a predominantly humanitarian mandate.

1.1 PROFILE OF WHO

Mission and Mandate

The World Health Organization (WHO) was created in 1948 as a specialised agency of the United Nations (UN) within the terms of Article 57 of the Charter of the United Nations. WHO’s mission is the attainment by all peoples of the highest possible level of health. The organisation directs and co-ordinates health within the UN system. It also provides leadership on global health matters; shapes the health research agenda, sets norms and standards; articulates evidence-based policy options; provides technical support to countries; and monitors and assesses health trends.

Structure and Governance

WHO headquarters in Geneva, Switzerland is responsible for overall management and administration of the organisation. WHO is divided into six regions, each of which has a Regional Office. The Regional Offices play an important role in WHO’s organisational and management structure; they are the link between headquarters and the country offices for all of the policy setting, planning, results and data-related functions.

The World Health Assembly is the decision-making body for WHO, and is attended annually by all 194 member states. The Assembly is supported by the Executive Board, comprised of 34 members required to be technically qualified in the field of health. The Board advises the Assembly, facilities its work, and gives effect to the Assembly’s decisions and policies. The Board meets in January to agree upon the agenda for the annual Health Assembly and adopt resolutions to be forwarded to the Assembly; it also meets for administrative purposes in May, following the Assembly.

WHO is administered by the Director-General appointed by the Health Assembly on the nomination of the Board. The Director-General, subject to the authority of the Board, is the chief technical and administrative officer of the organisation.

WHO is currently active with offices in 150 countries, territories and areas. Through a renewed country focus, WHO seeks to improve performance at the country level according to the country’s needs. Each country office develops a Country Co-operation Strategy to guide its work. WHO’s regional offices oversee this work and provide technical assistance to country offices as required. WHO works closely with other UN agencies and a multitude of partners to mobilise political will and material resources.

Strategy and services

WHO’s 2008-2013 Medium Term Strategic Plan (MTSP) outlines WHO’s results-based management framework and covers a six-year period encom- passing three biennial budget periods. The MTSP reflects the Director-General’s agenda for action and is presented through 13 strategic objectives (see sidebar).

These strategic objectives reflect different, but inter- dependent actions to achieve the organisation’s agenda for action. The MTSP includes results matrices that contain various organisation-wide expected results for each strategic objective; results with indicators, baselines and targets to be achieved by 2009, 2011 and 2013; dedicated resources; and justifications for their inclusion. This framework establishes the basis for WHO’s operational planning and reflects country priorities, as outlined in country co-operation strategies.

Finances

WHO is funded by voluntary contributions from both state and non-state actors, as well as through assessed contributions provided by member states. In 2012, its annual income from these contributions amounted to USD 3 959 million.[2]

Organisational improvement initiatives

WHO is engaging in an ambitious, on-going reform process that is member-state driven, with three objectives that were defined at the 64th World Health Assembly (2011) and the Executive Board’s 129th session: improved health outcomes, greater coherence in global health, and to be an organisation that pursues excellence. The reform process includes three areas of focus: programmes and priority setting, governance reform, and managerial reform. A 12th General Programme of Work (GPW) and Programme Budget 2014-2015 (PB) were approved by the Executive Board in the May 2013 session. The new GPW includes commitments to improving organisational alignment; enhancing performance, accountability and transparency; and strengthening results-based planning and performance measurement. It also details a clear results chain and explains how WHO’s work will be organised over the next six years; how the work of the organisation contributes to the achievement of a defined set of outcomes and impacts; and the means by which WHO can be held accountable for the way resources are used to achieved specified results. The results chain links the work of the Secretariat (outputs) to the health and development changes to which it seeks to contribute, both in countries and globally (outcomes and impacts). These changes are noted throughout the report, but since they have only recently been adopted and full implementation is not expected until 2014, they were not considered in the document review ratings.

The WHO website is www.who.int.

1.2 PREVIOUS ASSESSMENTS

Since its establishment in 2003, MOPAN has conducted three assessments of WHO (2007, 2010, and 2013). Although the MOPAN methodology has changed over time, findings from previous MOPAN surveys can provide insight into the evolution of perceptions of the organisation’s surveyed stakeholders.

Changes since 2010

In 2010, WHO received scores of adequate or strong on 18 of the 19 key performance indicators (KPI) assessed and an inadequate rating on only one (management of human resources). Out of 57 micro- indicators (MI) assessed, 6 received scores of less than adequate: organisational policy on results management, performance agreement systems for senior staff, transparent incentive / reward system for staff performance, staff recruitment and promotion is meritocratic and transparent, reports against its Paris declaration commitments using indicators and country targets, and shares lessons at all levels of the organisation.

In 2013, WHO received survey scores of adequate or strong on 20 of the 21 key performance indicators assessed, and an inadequate rating for results-based budgeting. Out of 54 micro indicators assessed, 3 received scores of less than adequate: organisational policy on results management, allocations linked to expected results, and expenditures linked to results.

From 2010 to 2013, there was a slight decrease in the survey ratings on most KPIs and three that registered very slight increases: corporate strategy and mandate, evaluating results, and disseminating lessons learned. Survey respondents were more positive in 2013 than in 2010 on the independence of WHO’s evaluation unit, but less positive on three other MIs: expected results developed in consultation with direct partners/beneficiaries, new programmes/projects can be approved locally within a budget cap), and participation in joint missions.

1.3 WHO SURVEY RESPONDENTS

Figure 1.1 | Number of survey respondents for WHO by country and respondent group

1. Despite considerable effort on the part of the MOPAN Secretariat and others, many donor HQ respondents failed to respond to the survey even after numerous extensions. The survey was closed following agreement with the Secretariat even though the target response rate of 70% had not been reached.

WHO survey results reflect the views of 394 respondents who provided their views on WHO’s performance in the areas of organisational effectiveness and contribution to development results.[3]

The distribution of responses on the six survey rating options and ‘don’t know’ responses are summarised by respondent group across all survey questions in Figure 1.2.

Figure 1.2 | WHO – Distribution of responses (n=394) on all questions related to micro-indicators

While there were responses in all six possible choices, relatively few responses overall were at the ‘weak’ end of the scale. Direct partners responded more positively than other groups overall, with nearly four- fifth of their responses on the high end of the scale. Nearly one-third of responses from donors in-country were ‘don’t know,’ which is slightly higher than the level of ‘don’t know’ responses provided by other groups.

2.1 OVERVIEW

The detailed MOPAN 2013 methodology, which includes rating scales, data analysis criteria, and strengths and limitations of the methodology, can be found in Volume II, Appendix I. The following is a brief summary.

MOPAN assessments examine:

• Organisational effectiveness: organisational systems, practices, and behaviours that MOPAN believes are important for aid effectiveness and that are likely to contribute to development and/or humanitarian results at the country level; and

• Humanitarian and development results: the extent to which there is evidence of an organisation’s contributions to development and/or humanitarian results.

As the MOPAN methodology has been refined each year since 2003, comparisons of this year’s assessments and any previous MOPAN assessments of an organisation should take this into consideration.

Data collection methods and sources

Over the years, MOPAN developed a mixed methods approach to generate relevant and credible inform- ation that MOPAN members can use to meet their domestic accountability requirements and support dialogue with multilateral organisations that they are funding.

The MOPAN approach uses multiple data sources and data collection methods to triangulate and validate findings. This helps eliminate bias and detect errors or anomalies.

In 2013, the sources of data included surveys of the multilateral organisation’s stakeholders and of MOPAN donors at headquarters and country level and a review of documents prepared by the organisations assessed and from other sources.

Assessment of organisational effectiveness

In the organisational effectiveness component, MOPAN examines performance in four areas: strategic management, operational management, relationship management, and knowledge management. Within each performance area, effectiveness is described using key performance indicators (KPIs) that are measured through a series of micro-indicators (MIs) using data from the survey and document review.

In this component, survey respondent ratings are shown as mean scores and are presented alongside document review ratings based on criteria defined for each micro-indicator. Not all micro-indicators are assessed by both the survey and the document review. The charts show survey scores and document review scores for the relevant KPIs or MIs. The full list of micro-indicators comprising this component is provided in Volume II, Appendix V.

Assessment of development and/or humanitarian results

In the development results component, MOPAN does not assess an organisation’s results on the ground, but examines how it measures and reports on its contributions to development and/or humanitarian results through three key performance indicators:

• Evidence of the extent of the organisation’s progress towards its stated institutional/organisation-wide results (i.e. goals, objectives, outcomes)

• Evidence of the extent of the organisation’s contributions to country-level goals and priorities, including relevant millennium development goals (MDGs)

• Relevance of the organisation’s objectives and programme of work to stakeholders.

In this component, a “best fit approach” is used in determining the ratings for the first two KPIs above. One of four qualitative ratings (strong, adequate, inadequate, or weak) is assigned following an analysis of data from all sources and confirmed in a consensus-based consultation (with of institutional advisors, a panel of experts, and MOPAN members). The rating of the KPI on relevance is based on stakeholder surveys only.

2.2 DATA SOURCES AND RATINGS

Survey

MOPAN gathers stakeholder perceptions through a survey of MOPAN members (at headquarters and in- country) and other key stakeholders of the multilateral organisation. Donor respondents are chosen by MOPAN member countries; other respondents are identified by the multilateral organisation being assessed.

The survey questions relate to both organisational effectiveness and to the achievement of development and/or humanitarian results. Survey respondents are presented with statements and are asked to rate the organisation’s performance on a six-point scale where a rating of 1 is considered “very weak” up to a rating of 6 which is considered “very strong.” A mean score is calculated for each respondent group (e.g. donors at headquarters).

MOPAN aims to achieve a 70% response rate from donors at headquarters and a 50% response rate among respondents in each of the survey countries (i.e. donors in-country and other respondent groups such as direct partners/clients).

Document review

The document review considers four types of documents: multilateral organisation documents; internal and external reviews of the organisation’s performance; external assessments such as the Survey on Monitoring the Paris Declaration, the Common Performance Assessment (COMPAS) report, and previous MOPAN surveys; and evaluations, either internal or external, of the achievement of results at various levels.

Document review ratings are based on a set of criteria that MOPAN considers to represent good practice in each area. The criteria are based on existing standards and guidelines (for example, UNEG or OECD-DAC guidelines), on MOPAN identification of key aspects to consider, and on the input of subject-matter specialists. The rating for each micro-indicator depends on the number of criteria met by the organisation.

Interviews

Interviews are conducted with staff based at headquarters and country offices of multilateral organisations who are knowledgeable in areas that relate to the MOPAN assessment. Interview data are not coded or used as a formal data source but rather to help ensure that the Assessment Team has all appropriate and necessary documents to enhance the triangulation of data from various sources and provide contextual information to assist in the analysis of the key performance indicators.

2.3 STRENGTHS AND LIMITATIONS OF THE MOPAN COMMON APPROACH

MOPAN continues to improve methodology based on experience each year. The following strengths and limitations should be considered when reading MOPAN reports.

Strengths

• The MOPAN Common Approach is based on existing bilateral assessment tools with the intent to replace or reduce the need for other assessment approaches by bilateral donors.

• In line with donor commitments to aid effectiveness, it seeks perceptual information from different stakeholder groups.

• It uses multiple sources of data to increase the validity of the assessment, enhance analysis, and provide a basis for discussion of agency effectiveness.

• MOPAN reports are validated and reviewed by MOPAN members and by the multilateral organisation being assessed.

Limitations

• The countries selected for MOPAN assessments comprise only a small proportion of each institution’s operations, thus limiting generalisations.[4]

• Although MOPAN uses recognised standards and criteria for good practice, such criteria do not exist for all indicators. Many document review criteria were developed by MOPAN; these are a work in progress and not definitive standards.

• Survey rating choices may not be used consistently by all respondents. Some respondents may tend to avoid extremes on a scale and respondents in some cultures may be unwilling to criticise or too eager to praise.

• The survey covers a broad range of issues and individual respondents may not have the knowledge to respond to all the questions relating to a given organisation.

• The MOPAN assessment was designed to consider an organisation’s results and its practices and systems from a corporate/institutional perspective and a country perspective. The assessment team incorporated interviews with WHO staff at regional level and the report refers to regional differences, where appropriate. However, MOPAN’s framework and methodology at the time of the assessment did not facilitate in-depth analysis of the regions.

3.1 INTRODUCTION

This chapter presents the findings of the 2013 MOPAN assessment of WHO’s organisational effectiveness, that is its practices and systems that support the achievement of results. Findings are based on respondent survey data and document review.

• Section 3.2 presents overall ratings on the performance of WHO and summarises respondent views on its primary strengths and areas for improvement;

• Section 3.3 provides findings on each of the four areas of performance (strategic, operational, relationship, and knowledge management).

3.2 OVERALL RATINGS

This section provides a summary of survey and document review ratings for all key performance indicators. It also presents survey respondent ratings of WHO’s overall organisational effectiveness and a summary of their written comments on WHO strengths and areas for improvement.

Overall ratings of key performance indicators

Figure 3.1 below shows scores from the document review and the survey on key performance indicators (KPIs) in the MOPAN 2013 assessment of WHO’s practices and systems. The white bar presents the survey score, while the black square presents the document review score. For example, on the first indicator, “providing direction for results”, WHO received a score of 3.71 (adequate) in the survey and a score of 4 (adequate) in the document review.[5]

In the overall ratings from the survey and document review, WHO was seen to perform adequately or better on the majority of key performance indicators. WHO received scores of adequate or better on 20 of the 21 KPIs assessed in the survey. WHO received scores of adequate or better on 14 of the 19 KPIs assessed by the document review.

The survey and document review ratings differed on nine KPIs – six of which were rated lower by the document review than by survey respondents (KPIs 5, 7, 13, 20 and 22), and the opposite for the remaining four (KPIs 2, 8, 10 and 12). The reasons for these differences are discussed in the following sections.

Figure 3.1 | Overall ratings on key performance indicators of WHO’s organisational effectiveness (survey mean scores and document review ratings)

Survey ratings of WHO’s organisational effectiveness

MOPAN has defined “organisational effectiveness” as the extent to which a multilateral organisation is organised to support direct partners in producing and delivering expected results. Respondents were asked the question: “How would you rate the overall organisational effectiveness of WHO?” As shown in Figure 3.2, direct partners gave the largest proportion of ratings on the high end of the scale (57%).

Figure 3.2 | Overall ratings of WHO organisational effectiveness by respondent group (n=394)

Respondents’ views on WHO’s strengths and areas for improvement

The survey included two open-ended questions that asked respondents to identify WHO’s greatest strengths and areas of improvement. Of the 394 respondents who responded to the survey, 384 commented on the organisation’s strengths and 360 on areas for improvement.[6]

There were 453 comments on WHO’s strengths and 401 comments on areas for improvement. The comments are summarised below with an analysis of the level of homogeneity among respondent groups as well as areas where the respondent groups felt differently about certain issues.

Respondents in all categories considered WHO’s greatest strengths to be its technical assistance and country-level operations, its staff expertise, as well as its strong mandate and normative role.

Of the 453 comments on WHO’s strengths, 32% (n=143) identified WHO’s technical assistance and country- level operations as the organisation’s greatest strength (43% among direct partners (n=80)). They noted that WHO’s technical assistance supports the development of national health strategies and plans.

Across all respondent groups, 17% of the comments (n=76) recognised WHO’s staff technical expertise as another of its strengths.

Two other strengths were identified by survey respondents:

• 11% of comments acknowledged WHO’s strong mandate and leadership role (n=50)

• 11% of comments cited WHO’s normative and standard setting work (n=38).

WHO’s operations at country-level were identified by most respondent groups as its main area for improvement – although, as noted above, this was also identified as one of WHO’s strengths. The efficiency of WHO’s administration (including human resource management) was identified as another area where WHO could improve.

WHO’s technical assistance and country-level operations were identified as both its greatest strength and main area for improvement. In fact, 20% (n=80) of the 401 comments from survey respondents indicated a need for improvement in the work that WHO is doing at the country level. This area for improvement was identified by approximately one-fifth of all respondent groups in-country, but by only 9% of responses from donors at headquarters.

Another area for improvement was WHO’s management and administrative efficiency (including human resources). Of the 401 responses, 16% (n=65) identified the cumbersome administration and bureaucracy inside the organisation.

Corporate governance, transparency and accountability was highlighted as an area for improvement in 11% (n=45) of all comments, but this area received the highest number of comments from donor groups, with 31% (n=18) of all donors at headquarters and 22% (n=8) of all donors in-country identifying this area as the main weakness.

Respondents had a range of opinions about other areas for improvement: 10% of comments from all respondent groups identified work done in specific areas.

3.3 WHO’S PERFORMANCE IN STRATEGIC, OPERATIONAL, RELATIONSHIP, AND KNOWLEDGE MANAGEMENT

3.3.1 Overview

This section presents the results of the 2013 Common Approach assessment of WHO in four performance areas: Strategic, Operational, Relationship, and Knowledge Management.

The following sections (3.3.2 to 3.3.5) provide the overall survey and document review ratings for the KPIs in each performance area, the mean scores by respondent group, and findings based on an analysis of survey and document review ratings in each performance area.

When there were notably divergent ratings between survey respondent groups or between the survey results and document review ratings, these are noted and the information gleaned from interviews with staff is integrated when it has a bearing on the analysis. Where statistically significant differences among categories of respondents were found, these differences are noted.

The survey data for each KPI and MI by performance area are presented in Volume II, Appendix V. The document review ratings are presented in Volume II, Appendix VI.

3.3.2 Strategic management

WHO’s strategic management was perceived by survey respondents as adequate. The document review rated the organisation very strong for its corporate strategy and mandate, and strong for its focus on cross-cutting priorities, but found that its corporate and country focus on results were areas for improvement. WHO is in the midst of a major reform process that should lead to considerable improvements in its focus on results (including new results chain and theory of change). These changes have been approved by the Executive Board and full implementation is planned by the end of 2014.

Figure 3.3 shows the overall survey and document review ratings for the five KPIs in the strategic management performance area. Overall, evidence shows that WHO has not yet fully developed its results culture at all levels of the organisation. Whereas WHO’s Medium Term Strategic Plan 2008-2013 is based on and aligned with its strong technical mandate, the quality of WHO’s results framework and results- based management practices need improvement. The organisation is moving towards a more results- oriented culture and has undertaken initiatives as part of its reform agenda to address these shortcomings.

Figure 3.3 | Performance area I: Strategic management, survey and document review ratings

Figure 3.4 shows the mean scores for the five KPIs for all survey respondents, and by category of respondent.

Figure 3.4 | Performance area I: Strategic management, mean scores by respondent group

Very weak = 1.00–1.49
Weak = 1.5–2.49
Inadequate = 2.5–3.49
Adequate = 3.5–4.49
Strong = 4.5–5.49
Very strong = 5.5–6.00

KPI 1: Providing direction for results

Finding 1: Survey respondents rated WHO as adequate for its results orientation and leadership on results management. WHO’s organisational policy on results management was considered adequate by the document review but MOPAN donors at headquarters suggest that more can be done to ensure the application of results-based management across the organisation.

WHO embarked on an ambitious reform process in 2011, which included results management as a central element. Work carried out since that time has and will continue to strengthen WHO’s accountability and results-based framework, as well as ensure effective engagement with its stakeholders. Most recently, WHO has defined its priorities in the 12th General Programme of Work (GPW) 2014 – 2020 and the Programme Budget 2014-2015. A new and strengthened results framework was included in this package and was approved by the Executive Board in May 2013 for implementation in 2014.[7]

Although recent changes are likely to improve the organisation’s results management in the medium and long term, they were not considered in this assessment as implementation is in its early stages and it is too early to assess their effects.

Figure 3.5 | KPI 1: Providing direction for results, ratings of micro-indicators

MI 1.1 – Value system supports results-orientation and partner focus

In this MI, which was assessed by survey only, survey respondents (MOPAN donors in-country and at headquarters, direct partners and technical partners) were asked whether WHO’s institutional culture reinforces a focus on results and whether it is direct-partner focused. The majority of respondents (67%) rated WHO as adequate or above on both questions. MOPAN donors at headquarters and donors in- country were less positive than direct partners on both questions and the difference was statistically significant.[8]

MI 1.2 – Leadership on results management

This MI was assessed by survey only. MOPAN donors at headquarters were asked whether WHO’s senior management shows leadership on results management. The majority (69%) rated WHO as adequate or above in showing leadership on results management.

MI 1.3 – Organisational policy on results management

MOPAN donors at headquarters were asked whether WHO ensures the application of results management across the organisation. The majority of survey respondents (61%) rated WHO as inadequate or below on this MI.

The document review found that WHO adequately promotes a results-based management culture within the organisation. Although WHO does not have an organisation-wide policy that describes the nature of results-based management, the organisation-wide results framework is outlined and described across a suite of documents.[9] The 11th General Programme of Work (GPW) 2006–2015 describes the long-term perspective on challenges to health and sets forth a global health agenda. The Medium-Term Strategic Plan (MTSP) 2008-2013, which includes the organisation-wide results framework, guides the three biennial Programme Budgets and operational plans. These documents are supported by operational guidelines[10] that present overall instructions for operational planning such as developing country-specific expected results (OSERs), products and services, activities, and associated indicators, baselines and targets. While these documents list commitments of the entire Secretariat (countries, regions and headquarters) and reflect their collective accountability for results, they do not contain up-to-date information on the nature and role of RBM within the organisation. In terms of holding its partners accountable for results, WHO’s template for technical proposals from partners includes results-based expectations, but the document review found no evidence of the application and use of this template.

KPI 2: Corporate strategy and mandate

Finding 2: WHO was recognised by survey respondents and the document review as having an organisation-wide strategy based on a clear mandate.

Figure 3.6 | KPI 2: Corporate strategy and mandate, ratings of micro-indicators

MI 2.1 – Corporate strategy based on clear mandate

In the survey, MOPAN donors at headquarters were asked whether WHO has a clear mandate and whether its organisation-wide strategy/strategies are aligned with the mandate. They rated WHO adequate or above on both questions (94 and 78% respectively) – resulting in a rating of strong overall.

The document review rated WHO very strong on this MI. WHO’s mandate has been revised over the years to ensure continuing relevance.[11] While there is no single document that clearly articulates all the nuances of WHO’s mandate, together, the 11th General Programme of Work and the MTSP 2008-2013 articulate the organisation’s goals and priorities and provide a clear indication of the manner in which WHO will implement the mandate during this period.

KPI 3: Corporate focus on results

Finding 3: The document review found room for improvement in the quality of WHO’s results frameworks in use as of 2013 but took positive note of recent improvements to the new results frameworks that will guide the 12th General Programme of Work.

This KPI was assessed by document review only, which focused on the MTSP 2008-2013 (amended draft) as the strategy in place at the time of the assessment.

The document review rated WHO as inadequate on the inclusion of good quality results frameworks and performance indicators in its plans and strategies and weak on the linkages between outputs and higher results levels. It is worth noting, however, that the draft proposed programme budget for 2014-2015 is based on an improved results chain where the Secretariat will be clearly accountable for inputs, activities and outputs and where the responsibility for outcome and impact results will be shared with member states and partners.

Figure 3.7 | KPI 3: Corporate focus on results, ratings of micro-indicators

MI 3.1 – Plans and strategies contain results frameworks

This document review rated WHO as inadequate on this MI. The MTSP 2008-2013 comprises both a development results framework (strategic objectives 1-11) and a management results framework (strategic objectives 12 and 13). While the organisation-wide expected results (OWER) are outlined under each of these objectives with associated indicators, baselines and targets, the structure of the framework allows for only one results level. More importantly, the results statements do not differentiate between activity, output and outcome levels.

The WHO programme management and operational guidelines define strategic objectives as “medium- term impacts” and OWERs as “desired outcome of the work of the Secretariat, in terms of change or achievement, over a medium-term period”.[12] However, most of the MTSP’s results statements are labelled as activities or outcomes. This discrepancy trickles down to all related performance indicators.

MI 3.2 – Results frameworks link outputs to final outcomes/impacts

The document review rated WHO as weak for the quality of the results chain in its frameworks, based on the review of the MTSP 2008-2013 (amended draft). As noted in the previous MI, the MTSP OWERs, which are meant to be organisation-wide outcomes, are not always stated appropriately to their results level. This incoherence in the structure and the way results statements are worded does not allow for the identification of a clear and logical progression from outputs to outcomes to impact.

In addition, the results-based framework is missing levels of results between the organisation’s activities and outputs and the intermediate outcomes it aims to achieve (results chain). This absence is a significant gap in the results chain from lower-level results to higher-level results and ultimately to goals. This makes it difficult to see the clear and plausible links from one result level to the next and to monitor WHO’s performance.

There is evidence that that these shortcomings are being addressed by the reform process.[13] The Programme Budget 2014–2015 includes an improved results chain in which each priority setting and programme category has clear outcomes and outputs and these are appropriate to their results levels (including key deliverables). The new results chain links the work of the Secretariat (outputs) to the health and development changes to which it contributes, both in countries and globally (outcomes and impact).

MI 3.3 – Plans and strategies contain performance indicators

The document review rated WHO inadequate on this MI as the majority of the performance indicators in the MTSP 2008-2013 are neither adequate nor relevant. The nature of the OWERs, as well as confusion and incoherence around their wording, make it difficult to assess whether the indicators provide a sufficient basis to measure performance. These are generally output indicators.[14] Although most performance indicators are clear on what needs to be measured and are associated with targets and baselines, the majority of these indicators do not meet all SMART or CREAM criteria.[15] While there is evidence of improvements in the organisation’s results-based management system under the reform, the Programme Budget 2014–2015 includes indicators at the outcome level only, some of which do not yet meet all of the SMART and/or CREAM criteria.

KPI 4: Focus on cross-cutting priorities

Finding 4: WHO was rated adequate or better on the cross-cutting areas assessed. The document review found WHO very strong in promoting environmental health.

Overall, survey respondents rated WHO as adequate on all four cross-cutting thematic areas identified by MOPAN (gender equality, environmental health, principles of good governance, and human rights-based approaches). In the document review WHO was rated very strong for promoting environmental health through its work on Strategic Objective 8 of the MTSP 2008-2013.

MOPAN donors at headquarters and donors in-country were less positive than direct partners with regard to good governance and human rights-based approaches.

As part of its reform process, WHO launched a new approach to promote and facilitate the institutional mainstreaming of gender, equity and human rights (GER) in May 2012. This Organization Wide Action Plan (OWAP) should establish performance standards to which WHO will adhere in relation to GER. It will include an action plan to implement the performance standards, to increase synergy between gender, equity and human rights, to develop technical guidelines for implementation of the performance standards, to clarify what mainstreaming GER means in practice in relation to health policies and programmes, and finally to establish an accountability framework tied to implementation of the GER performance standards. A GER policy should also be developed and closely monitored in the near future.

Figure 3.8 | KPI 4: Focus on cross-cutting thematic areas, ratings of micro-indicators

MI 4.1 – Gender equality

All survey respondents were asked whether WHO sufficiently mainstreams gender equality in its work. The majority (71%) rated WHO as adequate or above for the extent to which it mainstreams gender equality in its programmatic work.

The document review rated WHO as adequate for its efforts to integrate gender into programming. Strategic Objective 7 of the Medium-Term Strategic Plan for 2008-2013 includes gender as one of the main focus areas: “to address the underlying social and economic determinants of health through policies and programmes that enhance health equity and integrate pro-poor, gender-responsive, and human rights- based approaches.” In 2007, WHO issued a strategy for integrating gender analysis and actions into its work. A monitoring and evaluation framework for the strategy was also developed and implemented, one of the first in the UN system. As part of this effort, the organisation conducted an extensive baseline exercise to track its strategy. WHO followed with a Mid-Term Review in 2011 which indicated that while WHO has implemented its gender mainstreaming programme, little change has occurred since it began. This process allowed for the development of a myriad of tools to support effective gender mainstreaming across the organisation, such as tools to improve responsiveness to women’s needs and to assess policy coherence in health sector strategies.

In May 2012, WHO launched a new approach to mainstreaming gender, equity and human rights (GER) at the institutional level, with a particular focus on improving country office capacity to support member states in incorporating these themes within their national laws, strategic health plans, policies, activities, as well as monitoring efforts. The responsibility for mainstreaming gender now resides within WHO’s Family, Women’s and Children’s Health cluster (FWC), which hosts a Gender, Equity and Human Rights Mainstreaming Team. The latter supports and co-ordinates institutional mainstreaming at all levels of WHO. A budgetary review of the 2010-2011 biennium revealed a budget gap of USD 2 million, which represents 38% of the organisation’s budget for its work on gender, women and health at headquarters during this period. Recently, this funding gap was addressed and the team now receives approximately 50% of its funding from regular resources.

MI 4.2 – Environmental health

All survey respondents were asked whether WHO sufficiently promotes environmental health in its work. The majority (64%) rated WHO as adequate, 16% as inadequate or below.

WHO does not mainstream environment, as such, but one of its 13 strategic objectives is related to the promotion of environmental health. The document review rated WHO as very strong for the promotion of environmental health as a focus area (Strategic Objective 8 of the MTSP 2008-2013: To promote a healthier environment, intensify primary prevention and influence public policies in all sectors so as to address the root causes of environmental threats to health). Six OWERs were outlined under this strategic objective and progress on performance indicators for each OWER were reported in the Mid Term Review and the Performance Assessment Report associated with the Programme Budget 2010-2011. There is evidence that WHO promotes environmental health in various sectors and initiatives, and that it has conducted reviews of the activities carried out and results achieved in the promotion of environmental health.

WHO’s Public Health and Environment Department has overall responsibility for Strategic Objective 8. While the organisation does not have a separate policy or strategy that describes how it promotes environmental health in other areas of WHO’s programming, it does provide support for policy-making, such as research, evaluation, technical advice, training and toolkits.

MI 4.3 – Good governance

All survey respondents were asked whether WHO sufficiently promotes the principles of good governance in its work. The majority (68%) rated WHO as adequate or above. MOPAN donors at headquarters and donors in-country were less positive than direct partners and the difference was statistically significant.

As WHO does not mainstream good governance, this MI and associated criteria were adapted to WHO’s work to promote the principles of good governance. WHO was rated adequate in this area based on the fact that strengthening the governance of health systems as a form of good governance is central to the mandate of WHO. As mentioned in “WHO’s Role in Global Health Governance” (EB132/5 Add.5), WHO’s role in global health governance is a practical expression of the Constitutional function to act as “the directing and co-ordinating authority on international health work”. Hence, the MTSP 2008-2013 refers to various elements of good governance (under Strategic Objective 10), such as improving national capacity for framing policy, regulating, managing, monitoring and co-ordinating health systems. However, it is worth noting that documents made available did not include an explicit commitment to develop institutional capacity or to monitor this focus area (e.g. in planning, allocation of resources, monitoring and evaluation).

There is some evidence of the vertical integration of good governance, such as WHO’s work to strengthen health systems and its framework for good governance in the pharmaceutical sector. However, these examples are exceptions. The document review noted that there is no horizontal integration of good governance in WHO’s programming.

MI 4.4 – Human rights-based approaches

All survey respondents were asked whether WHO sufficiently mainstreams human rights-based approaches in its work. The majority (71%) rated WHO as adequate or above. MOPAN donors at headquarters and donors in-country were less positive than direct partners and the difference was statistically significant.

The document review assessed WHO’s human rights-based approach (HRBA) as adequate. While WHO does not mainstream human rights-based approaches, promoting health-related human rights is one of the seven priority areas and one of the cross-cutting issues under the Global Health Agenda in the 11th General Programme of Work for 2006-2015, and one of the Strategic Objectives (7) of the Medium-Term Strategic Plan for 2008-2013.

WHO is working on clarifying its commitment to mainstreaming gender, equity and human rights. It has undertaken a situation analysis and planning related to these areas in the Action Plan for Gender, Equity and Human Rights Mainstreaming (OWAP). The OWAP includes organisation-wide performance standards, associated technical guidance for their implementation, and a GER accountability framework.

Moreover, the new Gender, Equity and Human Rights Team includes health and human rights experts whose roles are to strengthen the capacity of WHO and its member states to integrate a human rights- based approach to health, advance the right to health in international law and international development processes, and advocate for health-related human rights.[16] While the organisation is integrating institutional systems and associated capacities, as well as defining accountability mechanisms to ensure monitoring and continuous improvement of mainstreaming efforts as part of its reform process, no evidence was found of an expenditure review/costing and budgetary allocation for the implementation of human rights mainstreaming activities.

KPI 5: Country focus on results

Finding 5: Whereas survey respondents perceived WHO’s Country Co-operation Strategies and workplans as adequately reflecting a focus on results, the document review provided ratings of inadequate on most indicators in this key performance area.

Survey respondents rated WHO as adequate on four MIs in this KPI and rated it strong for expected results aligned and consistent with national strategies and the Development Assistance Framework (UNDAF). The document review rated WHO as adequate on expected results being consistent with national strategies and the UNDAF but inadequate on other MIs.

At the country level, WHO prepares Country Co-operation Strategies (CCS) that are medium-term visions of WHO’s technical co-operation with each member state. The CCS outlines a strategic agenda that considers the outcomes of an analysis undertaken in the preparation phase, the national health priorities, the contributions of the other UN agencies and development partners to the National Health Policy, Strategy or Plan (NHPSP), as well as WHO’s comparative advantage; the CCS are not designed to present the results framework. Based on both the CCS and the MTSP, a workplan comprising office and country- specific expected results (OSERs) is developed in each country.

WHO’s low ratings for its country focus on results were echoed in the Joint Inspection Unit’s “Review of Management, Administration and Decentralization” (2013) which concluded that the relationship between the CCS and WHO’s planning instruments is weak.[17] However, it is important to note that WHO’s CCS guide is being revised in line with the 12th GPW results chain to ensure clear linkages between national priorities, CCS priorities, and results.

Figure 3.9 | KPI 5: Country focus on results, ratings of micro-indicators

MI 5.1 – Frameworks link results at project, programme, sector and country levels

MOPAN donors in-country, direct partners and technical partners were asked whether WHO’s country programme documents link results from project, sector and country levels. The majority of survey respondents (70%) rated WHO as adequate or above, and 46% of direct partners and 29% of technical partners gave ratings of strong and very strong.

The document review rated WHO as inadequate on this MI based on the quality of its results frameworks in country workplans. . WHO’s workplans sampled link activities, products and services to the organisation’s expected results at country level. Similarly, the organisation’s sector strategies are linked to expected results at country level. However, while country workplans include statements of office and country- specific expected results (OSER), in most of the six countries sampled, these results statements are articulated as activities, outputs and outcomes, reflecting the same confusion between results levels as in the organisation- wide results framework (see KPI 3). Current revisions to the CCS Guide, in line with the 12th GPW results chain, may help to strengthen the formulation of results and their links with CCS and national priorities.

MI 5.2 – Frameworks include indicators at project, programme, sector and country levels

The majority of survey respondents (71%) rated WHO as adequate or above for including national, sectoral, and project/programme indicators in its results frameworks and 36% rated WHO strong or very strong on this MI.

The document review rated WHO as inadequate on this MI. To assess this MI, country workplans and OSER progress reports were assessed. Country workplans contain indicators for country-specific expected results (OSER) that are actually the same as the indicators for organisation-wide expected results (OWER) but adapted to the country level. No indicators are used at the activity, product and services level.

Although data sources and data collection methods are not identified, the majority of the indicators are clear and monitorable. However, while each OSER is measured against at least one indicator, these are a mix of output and outcome indicators that are not always relevant or appropriate to the level of results with which they are associated.

MI 5.3 – Expected results consistent with national strategies and UNDAF

On this MI, 72% of survey respondents rated WHO adequate or above for including statements of expected results that are consistent with national development strategies in its country programme documents (CCSs and workplans).

The document review rated WHO as adequate on this MI. The document review found that WHO supports countries under the broader UN umbrella through its contribution to the development of the Common Country Assessment (CCA) and the UN Development Assistance Framework (UNDAF). There are clear and explicit links between WHO’s expected results and those identified in the national health policy, strategy or plan (NHPSP) and the UNDAF.[18] Since the development of the CCS strategic agenda is jointly agreed upon and developed with national authorities in support of the NHPSP, it takes into account the national health priorities and contributions of the other UN agencies and development partners. Hence, in most cases, the links between WHO’s results (OSERs) and those identified in the NHPSP and UNDAF are implicit.

MI 5.4 – Expected results developed in consultation with direct partners/beneficiaries

Survey respondents were asked whether WHO consults with direct partners and beneficiaries to develop its expected results. The majority (63%) rated WHO as adequate or above, and 29% as inadequate or below.

MI 5.5 – Results for cross-cutting priorities included in country frameworks

MOPAN donors in-country, direct partners and technical partners were asked whether WHO’s country programme documents (CCS and workplans) include results related to cross-cutting priorities such as gender equality, environmental health, principles of good governance, and human rights-based approaches. The majority of respondents (59%) rated WHO as adequate or above. MOPAN donors in- country had the highest level of ‘don’t know’ with 37%.

The review of documents rated WHO as inadequate for the inclusion of cross-cutting priorities in its country strategies. Whereas all country programme documents (CCSs or workplans) reviewed refer to one or more cross-cutting priorities, they do not consistently identify all four thematic priorities assessed by MOPAN (see KPI 4) or articulate these in their results frameworks. More than half of the country documents lack clear evidence of strategies and approaches to address or apply a particular cross-cutting theme. This is particularly the case for gender equality and human rights; environmental health and good governance are aligned with WHO’s strategic objectives.

3.3.3 Operational management

WHO has strong performance in some areas of operational management, most notably financial accountability and delegation of authority. Results-based budgeting received the lowest ratings but the document review noted that recent changes will likely lead to improvements in this area.

Figure 3.10 below shows the overall survey and document review ratings for the KPIs in the operational management performance area.

Survey respondents and the document review consider that WHO performs adequately or above on all KPIs in operational management other than results-based budgeting. WHO’s strongest performance in operational management relates to managing human resources, financial accountability and delegating authority. It continues to face difficulties in implementing results-based budgeting.

Figure 3.10 | Performance area II: Operational management, survey and document review ratings

Figure 3.11 shows the mean scores for the KPIs for all survey respondents, and by respondent group.

Figure 3.11 | Performance area II: Operational management, mean scores by respondent group¹

Very weak = 1.00–1.49
Weak = 1.5–2.49
Inadequate = 2.5–3.49
Adequate = 3.5–4.49
Strong = 4.5–5.49
Very strong = 5.5–6.00

1. Peer organisation respondents were asked questions in KPIs related to humanitarian response and cross-cutting themes.

KPI 6: Resource allocation decisions

Finding 6: WHO was considered adequate for making its funding criteria publicly available and for allocating its resources according to established criteria, although some survey respondents note room for improvement in this area. WHO has recognised this weakness and its on-going reforms include the development and application of a more transparent resource allocation system.

Survey respondents rated WHO adequate overall for making its funding criteria publicly available and for allocating its resources according to the established criteria. However, donor respondents at headquarters provided significantly lower ratings on both of these questions.

The document review found that WHO makes criteria for allocating funding publicly available and is working on improving the transparency of its resource allocation system, but could not assess whether resources are released according to agreed schedules due to the absence of data. . The document review was designed to draw data from the 2010 Survey on Monitoring the Paris Declaration. The white diamond indicates that the data required for the assessment of WHO on this micro-indicator was unavailable.

Figure 3.12 | KPI 6: Resource allocation decisions, ratings of micro-indicators

MI 6.1 – Criteria for allocating funding publicly available

All survey respondents groups other than peer organisations were asked whether WHO makes readily available its criteria for allocating resources. Slightly less than half of the respondents (44%) rated WHO as adequate or above on this MI and 37% rated it as inadequate or below. MOPAN donors at headquarters and donors in country were less positive than direct partners and the difference was statistically significant. MOPAN donors at headquarters rated this MI as inadequate, suggesting that there could still be greater transparency in how funding is allocated within the organisation.

The document review rated WHO as adequate on this MI. While a policy for the allocation of resources to countries is available on the organisation’s website in multiple languages, the “Review of Management, Administration and Decentralization” (2013) by the Joint Inspection Unit (JIU) reports a lack of transparency in the actual allocation of resources to countries. Interviews with regional offices confirmed that resource allocations are based primarily on country needs, historical allocations, and the budget implementation rate of the previous biennium. Both the JIU and the “Independent Evaluation Report: Stage One”[19] confirm the non-application of criteria and the absence of a functioning validation mechanism to distribute funds among countries at the outset of the process.[20] As part of its reform process, WHO is moving towards a much more transparent system and plans to improve its resource allocation mechanism based on a realistic assessment of income and WHO’s implementation capacity.

MI 6.2 – Resource allocations conform to criteria

This MI was assessed by survey only. Overall, 43% of survey respondents rated WHO as adequate or above for allocating its resources according to the established criteria, 31% provided ratings of inadequate or below, and 26% answered ‘don’t know’. MOPAN donors in country and technical partners provided high levels of ‘don’t know’ answers (44 and 35% respectively). MOPAN donors at headquarters and donors in-country were less positive than direct partners, providing a rating of inadequate, and the difference was statistically significant.

MI 6.3 – Resources released according to agreed schedules

This MI was intended to be assessed by document review and is based on Indicator 7 of the Paris Declaration on Aid Effectiveness, which measures the gap between aid scheduled and aid effectively disbursed and recorded in countries’ accounting systems. The document review could not assess WHO on this MI as no data was provided by the organisation.

KPI 7: Results-based budgeting

Finding 7: Both survey and document review ratings suggest that WHO is still in the early stages of a shift towards results-oriented budgeting. This is meant to be a key output of its internal managerial reforms, which were underway at the time of this assessment.

The survey component of this KPI was assessed only by donors at headquarters, who rated WHO as inadequate overall.

The document review found WHO inadequate or weak on the two micro-indicators assessed. The assessment team looked at the results orientation of WHO budgeting and reporting on expenditures to its stakeholders based on the practices in place in the early part of 2013.

As part of its reform agenda, WHO has identified explicit outputs and deliverables designed to better align its budget to the results chain in the 12th GPW. At the time of the assessment, key reforms (for example, the design of a methodology for standardised costing of outputs, and a methodology for assessing contribution of outputs to outcomes and outcomes to impact) were underway or due to commence but were not yet evident in WHO institutional documents.

Figure 3.13 | KPI 7: Results-based budgeting, ratings of micro-indicators

MI 7.1 – Allocations linked to expected results

MOPAN donors at headquarters were asked whether WHO links budget allocations to expected results. While 53% rated WHO as adequate or above, 43% gave ratings of inadequate or below on this MI.

In the document review, WHO was rated as inadequate in linking budget allocations to expected programmatic results. The assessment team distinguished between what is presented in institutional documents (such as the Programme Budget, Programme Budget Performance Assessment) – which were the focus of the document review on this key performance indicator – and the information available in WHO’s management instruments (such as GSM and HQ and regional planning documents). While budget information is disaggregated in several ways (by strategic objective, by region and Headquarters, and by special programmes and collaborative arrangements), the institutional budget documents assessed were not linked to organisation-wide expected results nor did they present output and/or outcome costs from the DRF and MRF. Although this information can be found in the enterprise resource planning system – the General Management System – it does not appear in the organisation-wide budget. As part of the reform, WHO will implement a new results-based budgeting system (RBB) based on a revised results chain with a methodology for costing of outputs and an approach to assess contribution. These reforms, if implemented as planned in 2013-2014, represent important steps towards becoming a more performance-oriented organisation.

MI 7.2 – Expenditures linked to results

MOPAN donors at headquarters were asked if WHO reports on results include the amount disbursed to achieve those results. The majority (57%) rated WHO as inadequate or below, and 37% gave ratings of adequate or above.

The document review, which looked at whether the organisation-wide financial reports (presented to external stakeholders in the Financial Reports and Programme Budget Performance Assessments) make explicit links between disbursements and reported results, rated WHO as weak in this area. As above, it is important to note that while such information is not available in key institutional reports, some of it is available through internal management instruments, such as GSM and the Financial Management Report, which track expenditures against budget (by strategic objective) and variances, among other indicators.

Since the new results-based budgeting initiatives have not been implemented, reports based on the new system will only be available in the next strategic cycle.

KPI 8: Financial accountability
 

Finding 8: The document review found that WHO has strong policies and processes for financial accountability, but highlighted strategies for risk management as an area for improvement. Survey respondents rated WHO as adequate in providing credible information in internal financial audits and for its procurement and contract management processes.

Survey respondents, who were asked to rate the organisation on two of the seven MIs in the area of financial accountability, rated WHO as adequate.

The document review found financial accountability to be one of WHO’s strengths. The systems and practices in place for external and internal audits are well detailed and there is evidence that policies are followed. Although WHO received a rating of very weak for its practices in risk management as it does not have an organisation-wide framework on risk management that is corporately approved, it is working on an organisation-wide common framework and harmonisation of risk management practices.

Figure 3.14 | KPI 8: Financial accountability, ratings of micro-indicators

MI 8.1 – External financial audits performed across the organisation

The document review assessed whether external financial audits meeting recognised international standards are performed across the organisation, and rated WHO as very strong. According to the Financial Regulations of the World Health Organization,[21] the External Auditor(s), which “shall be the Auditor-General of a Member government and appointed by the Health Assembly”, should issue a report on the audit of the biennium financial report. Documents reviewed confirmed that external audits of WHO’s financial statements were conducted by external auditor(s) every two years. All financial reports reviewed (e.g. 2008, 2010 and 2012) included letters from an external auditor, in this case Comptroller and Auditor-General of India, confirming that they were conducted in accordance with international standards and in conformity with WHO Financial Regulations. WHO has been gradually adopting International Public Sector Accounting Standards (IPSAS) with full implementation of IPSAS expected in 2012-2013.

MI 8.2 – External financial audits performed at the regional, country, or project level

WHO received a rating of adequate in the document review on the appropriateness of its regional and country-level external audits of programmes and projects. The document review found evidence of audits conducted during the latest financial period in all the regional offices, including selected country offices and specific units. Each audit is conducted in conformity with generally accepted auditing standards and in accordance with the Additional Terms of Reference of the Financial Regulations and Financial Rules. WHO does not make internal and external audit reports available to the public.

The Office of Internal Oversight Services (OIOS) is responsible for providing internal oversight services (including financial audit). Audits are conducted in accordance with the International Standards for the Professional Practice of Internal Auditing of the Institute of Internal Auditors (IAA). Although there is no rule or procedure for audit coverage, evidence was found of clearly defined and delineated functions between internal and external audit. OIOS and the external auditors co-ordinate their work in order to avoid duplication and ensure sufficient audit coverage at country level.

MI 8.3 – Policy on anti-corruption

In the document review, WHO was rated as strong for its policy guidelines on anti-corruption. The Fraud Prevention Policy and Fraud Awareness Guideline (2005) covers both fraud prevention and the contingency measures that may be taken. The policy applies to any misappropriation, irregularities and illegal acts characterised by deceit, concealment or violation of trust involving staff members as well as consultants, contractors, outside agencies doing business with WHO, and/or other parties with a business relationship with WHO. To support the application of this policy, WHO has specific procedures that govern investigations on allegations of staff misconduct. The Office of Internal Oversight Services is empowered to initiate and conduct investigations.

New staff members are trained at the time of their appointment on: i) their duty to communicate certain matters; ii) a list of the types of matters, including actual or suspected fraud, to be communicated along with specific examples; and iii) information on how to communicate those matters. WHO also has a whistle- blower protection policy and procedures to protect staff members who report suspected violations of WHO’s regulations and rules, or who co-operate with an audit or investigation.

While the policy defines the roles, responsibilities and accountabilities of WHO staff in implementing and complying with the policy, it does not commit the organisation to review its activities on combating fraud and corruption and the organisation has not reviewed its policy and/or practice in this area.

As part of its reform agenda, WHO has committed to establish a Compliance, Risk Management and Ethics office, under the direction of the Executive Director and the Director General Office.[22]

MI 8.4 – Systems for immediate measures against irregularities

The document review found that WHO has a strong system to address financial irregularities at the country level. Together, WHO’s financial regulations and rules and the OIOS audit, evaluation and investigation process provide detailed guidance on irregularities to be investigated by internal and external audits.

An Independent Expert Oversight Advisory Committee advises the Programme, Budget and Admin- istration Committee of the Executive Board on matters including: the review of WHO’s financial statements, financial reporting and accounting policies, advice on internal control and risk management, effectiveness of the organisation’s internal and external audit functions, and monitoring of the implementation of audit findings and recommendations.

A report on the implementation of the external auditor’s recommendations and internal audits is provided by the Secretariat on a regular basis to the Programme, Budget and Administration Committee and is then reviewed by the Committee in conjunction with the reports prepared separately by the Office of Internal Oversight Services (OIOS) and the external auditor.

The document review did not find any evidence of timelines for the response to irregularities identified during an external financial audit.

MI 8.5 – Internal financial audits provide credible information

MOPAN donors at headquarters perceived WHO’s internal financial audits to be performing well in providing credible information to its governing bodies: 76% rated WHO as adequate or above.

Documents reviewed suggest that WHO is very strong in its use of internal financial/organisational audits to provide management governing bodies with credible information. The OIOS, which provides independent, objective assurance and advisory services to the Programme, Budget and Administration Committee of the Executive Board, is responsible for organising and directing a programme of internal auditing in compliance with the Standards for the Professional Practice of Internal Auditing of the Institute of Internal Auditors. The OIOS Manual provides all important policies and procedures to be followed in administering and managing internal audits (including evaluations and investigations at WHO).

As part of its reform agenda, WHO has increased its capacity for audit and oversight by recruiting additional staff for internal audits and investigations. This was confirmed by the Joint Inspection Unit, which recognises in its 2013 “Review of Management, Administration and Decentralization” in the WHO, that the audit function in WHO has been strengthened with the introduction of a new follow-up tracking system, and that the country level audit coverage has been improved.

MI 8.6 – Effective procurement and contract management processes

MOPAN donors in-country, direct partners and technical partners were asked whether WHO’s procurement and contract management processes for the provision of services or goods are effective. While 49% rated WHO as adequate or above, 32% answered ‘don’t know’.

WHO was rated strong on this MI by the document review as it appears to have solid and effective procurement and contract management processes. It has established a policy on the procurement of goods and services under the overall responsibility of the Global Procurement and Logistic Unit. While the policy does not explicitly set targets or requirements for timeliness of delivery of products and services, it states that every reasonable endeavour should be made to obtain goods and services of a quality suited to the purpose for which they are to be used, for delivery by the time required and at the lowest cost.[23]

According to the 2013 “Review of Management, Administration and Decentralization”, the Global Procure- ment Unit has a good understanding of the client perspective and seeks to further increase its client orientation.[24] In addition, in early 2013, Draft Terms of Reference for a Commercial Expertise Review of the WHO were developed to provide a detailed assessment of WHO’s commercial expertise, with a particular focus on procurement policy, practice and strategy, including a risk analysis of the procurement function. The review is intended to provide an overall assessment of WHO’s procurement and commercial capability, an analysis of areas where WHO could potentially achieve improved value for money (VfM) in its procurement strategies and processes, areas of greatest potential risk in terms of procurement, as well as key conclusions and recommendations.

MI 8.7 – Strategies for risk management

This MI was assessed by the document review only, which rated WHO as very weak as it does not have an organisation-wide, corporately-approved framework on risk management. However, WHO has been moving forward towards an organisation-wide common framework and harmonisation of risk management practices. As mentioned in MI 8.3, the establishment of a Compliance, Risk Management and Ethics Office was approved by the Director General to strengthen accountability and ensure organisational integrity by focusing on anticipating and preventing risk occurrence, thereby increasing the organisation’s capacity in compliance and risk management. Moreover, as part of its reform agenda, the organisation is planning to establish a risk management framework that would include criteria for identification and prioritisation of risks, terms of reference for risk managers, a risk management policy, as well as a corporate risk register.

KPI 9: Using performance information

Finding 9: Survey respondents rated WHO’s use of performance information on results as adequate overall. The document review provided ratings ranging from weak to strong in this area.

WHO was rated adequate on the four MIs assessed by the survey.

The document review found that WHO is doing well in using information to revise and adjust policies and being proactive in managing poorly performing initiatives, but that it needs to improve its use of information to plan new interventions and act upon evaluation recommendations.

Figure 3.15 | KPI 9: Using performance information, ratings of micro-indicators

MI 9.1 – Using information to revise and adjust policies

MOPAN donors at headquarters were asked whether WHO uses project/programme, sector and country information on performance to revise organisational policies. The majority (53%) rated WHO as adequate or strong and 33% provided ratings of inadequate or below on this MI.

The document review rated WHO as strong in its use of performance information to revise and adjust policies and programmes. The document review found evidence that WHO: analyses its performance in a systematic manner; takes into account recommendations from performance reports, organisation-wide audits, reviews and/or evaluations in order to revise and adjust its policies, systems and practices; takes steps to respond to specific performance-related problems; and adjusts its broader programming and policies in response to performance issues raised in reports. For example, based on reviews and evaluations WHO has made the following important changes in its systems and practices such as: i) the elimination of the MTSP by combining the high-level strategic vision of the General Programme of Work (GPW) with a special emphasis on WHO’s priorities, ii) the reduction of GPW’s duration from 10 years to 6 years to improve alignment with planning and budgeting cycles, and iii) the addition of high level results at outcome and impact level and associated performance indicators, baselines and targets.

MI 9.2 – Using information to plan new interventions

MOPAN donors in-country, direct partners and technical partners were asked whether WHO uses information on its projects/programmes or initiatives to plan new areas of co-operation at the country level. The majority (64%) rated WHO as adequate or above.

The document review rated WHO as inadequate on this MI. WHO has a process for formulating, implementing and monitoring its country co-operation strategies, detailed in the Country Co-operation Strategy Guide 2010, which takes into account the national health priorities, the contributions of other UN agencies and development partners to the NHPSP, the analysis of the implications of the strategic agenda for WHO Secretariat, the production of the draft CCS document, and the achievement of CCS-MTSP mapping exercise. While this process is documented in WHO: Operational Planning – Business Rules, Procedures (2007) and in the WHO Country Co-operation Strategies Guide (2010), these do not provide guidance for performance assessments of previous CCS to inform the preparation and planning of strategies at the end of each cycle. The OSERs in country office workplans have indicators that are monitored. However, there is no evidence in monitoring reports (on either the CCS or the workplans) of how performance information was used to plan new interventions. It is worth noting, however, that most interviews with staff from the six focus country offices revealed the existence of an informal process for planning new interventions. This process involves discussing the progress made on previous CCS (with all stakeholders, as well as regional and headquarter level) in order to inform the preparation of the next CCS. In addition, a network of strategic objective facilitators and technical teams peer review proposed country- and office-specific expected results and related workplans.

MI 9.3 – Proactive management of poorly performing initiatives

MOPAN donors in-country, direct partners and technical partners were asked whether WHO’s poorly performing programmes and projects are addressed proactively to improve performance. Their views were mixed: 45% rated WHO adequate or above, 22% rated it inadequate or below, and 33% answered ‘don’t know’.

The document review rated WHO adequate in addressing poorly performing programmes, projects and initiatives. WHO reports on its country workplans through progress reports (every six months), Mid-Term Reviews and End of Biennium Assessments. The document review found evidence of a process for reviewing the performance of its activities in both WHO: Operational Planning – Business Rules, Procedures (2007) and country-level reports. While this process allows for following-up on poorly performing initiatives by using a traffic light system, it does not provide sufficient guidance on the actions required to enhance implementation of the initiative, especially with regard to tasks and OSERs that have been identified“at risk”.

MI 9.4 – Evaluation recommendations are acted upon

Donors at headquarters were asked whether WHO appropriately tracks the implementation of evaluation recommendations reported to its Executive Committee/Board. The majority (57%) rated WHO as adequate or above, 31% as inadequate or below.

The document review rated WHO as weak on this MI. Prior to 2012, WHO did not have a formal organisation-wide procedure for tracking implementation of the recommendations from evaluations. In the spirit of decentralisation, the WHO Evaluation Guidelines (2006) only mentioned that the responsible units should act upon recommendations from the evaluators. The newly adopted Evaluation Policy (2012) continues to apply this practice. It is still the responsibility of the officer to “utilise the findings of the evaluation”. However, this Evaluation Policy does now specify that “an appropriate management response” should be “issued in a timely manner”. The management response must contain an assessment of the recommendations, an action plan and a timeline for the implementation of the recommendations. The newly established Global Network on Evaluation will be responsible for following up on the status of management responses to evaluation recommendations and will report to the Executive Board through the Office of Internal Oversight Services. The new policy and system may help to improve WHO’s follow up on evaluations, but it is too early to say and as yet there is little evidence.

KPI 10: Managing human resources

Finding 10: The document review noted that WHO has systems in place to conduct performance assessment and reward staff. A recent review by JIU, however, shows that these systems have been applied inconsistently. Surveyed stakeholders suggest that WHO’s country staff deployment adequately supports partnership development.

As part of its reform agenda, WHO is making efforts to review its policy and practices to better address staff performance management.

Figure 3.16 | KPI 10: Managing human resources, ratings of micro-indicators

MI 10.1 – Results-focused performance assessment systems for senior staff

This MI was assessed by document review only, which rated the organisation as adequate. WHO Staff Regulations state that: “supervisors shall periodically make a formal evaluation of the performance, conduct and development potential of all staff members under their supervision”. To this end, WHO established the Performance Management Development System (PMDS) in 2002. The PMDS is required for all staff at D2 level and below. The system contains performance objectives, competencies and a personal development plan. Apart from being a development tool for improving staff performance, it is also a legal document used by the organisation for a number of HR-related processes (reassignment, inter-agency transfer, selection, appointment, classification of post, conversion to continuing contract, and restructur- ing). WHO has developed a number of tools to manage the system: a policy, a user’s guide, and a website.

WHO reports annually on the proportion of staff in compliance with the cycle of the Performance Management Development System across the organisation in the Mid-Term Review and the Performance Assessment Report. The compliance rate for 2011 appears to be high in headquarters (91%), but less satisfactory in a number of regional offices (e.g. the Eastern Mediterranean Regional Office had a compliance rate of 38%, while the Regional Office for the Americas’ rate was 29%).[25] The quality of the PMDS has also been questioned by the JIU Report because of cultural factors and the fact that negative feedback is seldom given which sometimes leads to a tendency to overrate staff performance.

MI 10.2 – Transparent system to manage staff performance

This MI was assessed only through document review, which rated WHO as adequate.

WHO’s human resources documentation states clearly how performance relates to reassignment, inter- agency transfer, selection, appointment, classification of post, within-grade increase (WIGI), conversion to continuing contract, and restructuring. However, staff performance is not explicitly linked to promotion. Human resources management was identified as the most complex and problematic area of WHO administration in the 2013 review by the Joint Inspection Unit. The review indicates that human resources policies and rules do exist, but they are inconsistently implemented. WHO is making efforts to review its policy and practices to better address staff performance management. A new human resources strategy, a new performance development and management system, and policies on rewards and recognition and improving performance are planned as part of the WHO reform.

MI 10.3: Country staff deployment supports partnership development

This MI was assessed by survey only. MOPAN donors in-country, direct partners and technical partners were asked whether WHO keeps deployed international staff in country offices for a sufficient time to maintain effective partnerships at country level. The majority (73%) rated WHO as adequate or above. Both direct partners and technical partners were more positive than other respondent groups and the difference was statistically significant.

KPI 11: Performance-oriented programming

Finding 11: Although survey respondents viewed the performance orientation of WHO’s country programming processes to be adequate, documentary evidence indicated there is significant room for improvement.

Survey respondents rated WHO as adequate for subjecting new initiatives to benefit/impact analysis before their approval while the document review found it inadequate.

The document review provided a rating of adequate for WHO’s use of milestones and targets to monitor implementation of its activities.

Figure 3.17 | KPI 11: Performance-oriented programming, ratings of micro-indicators

MI 11.1 – New initiatives subject to benefit/impact analysis

MOPAN donors in-country were asked whether WHO subjects new programming initiatives to benefit/ impact analyses. Almost half of survey respondents (44%) answered ‘don’t know’, and 37% rated WHO adequate or strong on this MI.

Strictly adhering to MOPAN criteria, the document review rated WHO as inadequate. According to the documentation reviewed, neither programmes nor projects are subject to formal benefit/impact analyses. Although WHO does not make investments in specific projects, like other more operational development agencies and multilateral development banks, it does make significant investments in global initiatives that could benefit from an assessment of the potential impact of their implementation.

MI 11.2 – Milestones/targets set to monitor implementation

The document review rated WHO as adequate for setting milestones to track the progress of project implementation. WHO’s Operational Planning, Business Rules and Procedures, as well as General Management System Guidance state that any task that is critical for execution of a project can be flagged as milestone, however, there is no system functionality built around milestones in the system as they are used only for information and monitoring purposes.[26] Therefore, milestones (or any activity) can be set for products and services to monitor the progress in delivering the workplan. While the majority of the workplans assessed contained specific activities and dates for achievement under products and services, for the most part they did not include indicators to measure successful activity completion or associated baselines and targets. Indicators, baselines and targets were only found at OSER level.

KPI 12: Delegating authority

Finding 12: Survey respondents rated WHO as adequate overall on its delegation of decision- making authority and the document review provided ratings of strong in this area.

Figure 3.18 | KPI 12: Delegating authority, ratings of micro-indicators

MI 12.1 – Aid reallocation decisions can be made locally

MOPAN donors in-country, direct partners and technical partners were asked whether WHO country offices have sufficient delegated authority to manage activities at country level. The majority (68%) rated WHO as adequate or above on this MI.

The document review, which examined whether aid reallocation decisions can be made locally, rated WHO as strong. The WHO Accountability Framework (2006) delineates the principal lines of accountability for programmatic and managerial decisions, and decisions on operational planning and budget are illustrated in Programme Management in WHO: Operational Planning – Business Rules, Procedures (2007). Aid reallocation decisions are governed by a series of Standard Operating Procedures (2010) that provide step- by-step guidance to manage and modify workplans (budgets, activities and results). These procedures show that WHO is a decentralised, field-based organisation and that its country offices have a certain level of autonomy to make adjustments and changes to activities, such as revising budget allocations, managing planned costs and award budgets, managing workplan information and implementation, among others.

MI 12.2 – New programmes/projects can be approved locally within a budget cap

MOPAN donors in-country, direct partners and technical partners were asked whether funding for new areas of co-operation can be approved locally, within a budget cap. While 44% rated WHO as adequate or above, 40% of survey respondents answered ‘don’t know’.

WHO received a rating of strong from the document review. According to the Programme Management in WHO: Operational Planning – Business Rules, Procedures (2007), WHO’s country offices are responsible for defining activities, products and services, determining costing, and setting indicators, baselines and targets for the results planned. While the workplans are reviewed and approved at regional and headquarters level, the accountability and authority for managing planned costs rests with the project manager of the workplan. Planned costs must be kept within set limits, so as not to plan a greater share of the budget centre’s allocation than authorised or intended by the budget centre manager.

It is worth noting, however, that the JIU “Review of Management, Administration and Decentralization” (2013) found that the system of budget ceilings by strategic objective is too rigid and often causes significant problems at the country level when resources are available for a specific purpose but cannot be implemented because the spending limit in the corresponding area has been reached.[27] It also found that ceilings are often perceived as arbitrary at the country level and impede fundraising by country offices even if resources are potentially available. Hence, the consultation process for changing the ceilings is complicated and leads to significant delays in implementation.

KPI 13: Humanitarian principles

Finding 13: WHO’s adherence to humanitarian principles in its field operations was perceived by survey respondents as strong overall. The document review rated the organisation adequate for its inclusion of humanitarian principles in its strategic framework.

Survey respondents based in Ethiopia, Indonesia and Mozambique rated WHO as strong on respecting humanitarian principles in delivering assistance and adequate for maintaining on-going policy dialogue with partners on the importance of observing humanitarian principles in delivering emergency assistance.

The document review found that WHO has demonstrated its willingness to improve its institutional capacity with regard to the application of its humanitarian mandate. The organisation received a rating of adequate for including humanitarian principles in its strategic framework.

Figure 3.19 | KPI 13: Humanitarian principles, ratings of micro-indicators

MI 13.1 – Strategic framework includes humanitarian principles

This MI was assessed by document review only and was rated as adequate according to the documents reviewed. With the release of the Planning Framework – Strengthening WHO’s Institutional Capacity for Humanitarian Health Action 2009-2013, WHO demonstrated its willingness to improve its institutional capacity for humanitarian action. WHO’s application of its humanitarian mandate is articulated in its Emergency Response Framework (ERF) and the Inter-Agency Standing Committee’s Global Health Cluster Guide.[28] These two documents contain clear accountabilities and operational mechanisms for the co-ordination of WHO’s humanitarian functions and roles. The ERF, which was published in 2013, is managed by the Global Emergency Management Team at headquarters. While it does not clearly define the principles of humanity, impartiality, neutrality, and independence in its own strategic framework, it describes in detail WHO’s core commitments, steps during initial alert, grading process, performance standards, roles of the team and emergency response procedure. Through a number of consultations, reviews and/or evaluations both within the WHO and the Health Cluster, the organisation is strengthening its commitment to humanitarian response by improving organisational practices.

MI 13.2 – Humanitarian principles respected while delivering assistance

This MI was assessed by survey only. All survey respondents at country level were asked whether WHO’s commitments to humanitarian principles are respected in the delivery of emergency assistance and/or in humanitarian response. The majority (85%) rated WHO as adequate or above, and 24% provided ratings of very strong.

MI 13.3 – On-going policy dialogue with partners on observing humanitarian principles

This MI was assessed by survey only. All respondent groups other than donors at headquarters were asked whether WHO maintains on-going policy dialogue with partners on the importance of observing human- itarian principles in delivering emergency assistance, particularly in cases of protracted crises and complex emergencies. The majority (79%) rated WHO as adequate or above.

3.3.4 Relationship management

Overall, WHO was seen as adequate in the area of relationship management by both survey respondents and the document review. WHO has made consistent efforts to improve and monitor the effectiveness of its aid although the global indicators used to measure progress in this area do not always capture the nature of WHO’s mandate in setting norms and standards and providing technical co-operation in the health sector.

Figure 3.20 below shows the survey and document review ratings for the six KPIs in the relationship management performance area. In the survey, WHO was rated as adequate on all aspects of relationship management and strong in its contribution to policy dialogue. The document review considered WHO adequate in using country systems and harmonising procedures.

Several of the KPIs in this performance area are based on the indicators of the Paris Declaration on Aid Effectiveness. Although WHO performance on these indicators as a whole is adequate, some of the PD indicators reflect practices that are more typical of organisations that provide funding to partners and that only partially capture WHO’s mandate of providing technical co-operation (for example, MI 16.2 on use of national systems and MI 18.3 on the use of programme-based approaches (PBA)).

WHO has been active in monitoring its progress on aid effectiveness commitments. It is a signatory and active participant in IHP+[29] and has reported data for each of the performance reports published by the independent IHP+ Results Consortium (the most recent was published in 2012). Internally, WHO has expanded its coverage in monitoring its commitments to the Paris Declaration (from 22 country offices in 2008 to 80 in 2011 and all country offices participating in 2013) and helped to increase the validity and reliability of data in the last round of monitoring of the Paris Declaration by establishing a Helpdesk at Headquarters to provide guidance and manage the data collection process. WHO also produced an internal report, “Analysis of the 2011 Paris Declaration Monitoring Surveys: Key Highlights”, which provides overall findings and data on each of the PD indicators as well as explanations for achieving or falling short of achieving the various indicators.

Figure 3.20 | Performance area III: Relationship management, survey and document review ratings

Figure 3.21 shows the mean scores for the six KPIs for all survey respondents, and by respondent groups.

Figure 3.21 | Performance area III: Relationship management, mean scores by respondent group

Very weak = 1.00–1.49
Weak = 1.5–2.49
Inadequate = 2.5–3.49
Adequate = 3.5–4.49
Strong = 4.5–5.49
Very strong = 5.5–6.00

KPI 14: Supporting national plans

Finding 14: WHO was perceived as adequate in its support of national and partner plans.

This KPI was assessed by survey only. Respondent groups gave WHO an overall rating of adequate for its support of national and partner plans.

Figure 3.22 | KPI 14: Supporting national plans, ratings of micro-indicators

MI 14.1 – Funding proposals developed with national government or direct partners

This MI was assessed by survey only. MOPAN donors in-country, direct partners and technical partners were asked whether WHO supports funding proposals designed and developed by the national government or other direct partners. The majority of respondents (73%) rated the organisation as adequate or above.

KPI 15: Adjusting procedures

Finding 15: Overall, WHO was perceived by respondents to be adequate in taking into account local conditions and capacities.

In this KPI, which was assessed by survey only, WHO was rated adequate on all MIs.

Figure 3.23 | KPI 15: Adjusting procedures, ratings of micro-indicators

MI 15.1 – Procedures easily understood and completed by partners

MOPAN donors in-country, direct partners and technical partners were asked whether WHO uses procedures that can be easily understood and followed by partners. Overall, the majority (63%) rated WHO as adequate or above.

MI 15.2 – Time for procedures does not affect implementation

Survey respondents (MOPAN donors in-country, direct partners and technical partners) had mixed opinions on whether the length of time it takes to complete WHO procedures affects implementation. WHO was rated adequate or above by 51% of respondents and inadequate or below by 25%.

MI 15.3 – Ability to respond quickly to changing circumstances

WHO was rated as adequate for the adjustment of its work/portfolio in a country to respond to changing circumstances: 61% of respondents (MOPAN donors in-country, direct partners and technical partners) rated WHO as adequate or above and 20% as inadequate or below.

MI 15.4 – Flexibility in implementation of projects/programmes

MOPAN donors in-country, direct partners and technical partners were asked whether WHO adjusts its implementation of individual projects/programmes as learning occurs. While the majority of survey respondents (56%) rated WHO as adequate or above, 23% provided ratings of inadequate or below.

KPI 16: Using country systems

Finding 16: Survey respondents perceived WHO as adequate in its use of country systems. The review of documents considered WHO strong in ensuring that aid disbursements are recorded in national budgets and avoiding the use of PIUs and adequate in the use of country financial systems.

Figure 3.24 | KPI 16: Using country systems, ratings of micro-indicators

MI 16.1 – ODA disbursements/support recorded in annual budget

This MI assesses the percentage of an organisation’s overall Official Development Assistance (ODA) disbursements recorded in annual national budgets as revenue, grants or ODA loans (indicator 3 of the Paris Declaration on Aid Effectiveness). The document review rated WHO as strong in this area. In its “Analysis of the 2011 Paris Declaration Monitoring Surveys: Key Highlights”, an internal document that outlines the organisation’s contribution to the PD Survey 2011, WHO reported that of the total ODA disbursed in 80 countries (USD 680 333 694), more than 98% was for the government sector, which exceeds the 2010 target of 85%. The WHO scorecard published by IHP+Results noted progress towards the target in the 19 countries participating in that survey. WHO notes that sometimes there are challenges due to the “sensitivity of the indicator to the government’s policy of either including or excluding aid in the budget and to delays in WHO reporting aid due to different planning cycles.”[30] On the whole, by aligning with governments’ plans and priorities, WHO demonstrates a good degree of compliance with the Paris Declaration principle of alignment.

MI 16.2 – Use of country financial systems for operations

Survey respondents (MOPAN donors in-country, direct partners and technical partners) were asked whether WHO uses financial country systems (i.e. public financial management and procurement) as a first option for its operations where appropriate. Almost half of the respondents (47%) rated WHO as adequate or above, and 32% answered ‘don’t know’.

The document review provided an overall rating of adequate on the use of country public financial management systems (based on indicator 5a of the Paris Declaration on Aid Effectiveness) and on the use of country procurement systems (based on indicator 5b of the Paris Declaration on Aid Effectiveness).

On the use of country public financial management systems, WHO reported the following percentages in the 2011 PD Survey:[31] 21.9% of the full amount disbursed by WHO to the government sector in 56 countries used national budget execution procedures, 21.3% used national financial reporting procedures, and 20.9% used national auditing procedures.[32] The data suggest that WHO has made limited progress towards the indicative target of ODA using country systems for disbursements and operations. However, it is important to note that WHO provides a significant amount of ODA through the technical co-operation provided by its staff and these staff costs are not paid using government financial management systems.[33]

In addition, the IHP+Results scorecard notes that since 2004/2005, WHO has increased the percentage of health sector aid provided by the agency that uses country public financial management systems in the IHP+ signatory countries participating in the 2011 survey.

On the use of country procurement systems, WHO fell short of meeting the 2010 target of 80% according to its internal statistics. In its “Analysis of the 2011 Paris Declaration Monitoring Surveys: Key Highlights”, WHO reports that only 4.2% of ODA funds used national procurement systems (e.g. Ethiopia, 2%; Guatemala, 0%; Indonesia, 0%; Mozambique, 1.9%; Pakistan, 0%; and Viet Nam, 19.5%). The results of the IHP+Results survey also indicate limited progress or regression in the use of country systems, although based on input from a smaller number of countries.[34] WHO provides many reasons for this low percentage, including: centralised procurement carried out by WHO in order to achieve economies of scale, weak national capacities, WHO systems used by partners due to tax exemptions and/or to channel donations, the procurement of highly specialised/technical equipment, and the provision of technical assistance rather than goods and services.

MI 16.3 – Use of country non-financial systems for operations[35]

Survey respondents (MOPAN donors in-country, direct partners and technical partners) were asked whether WHO uses country non-financial systems (e.g. monitoring and evaluation) as a first option for its operations, where appropriate. Just above half of the respondents (51%) rated WHO as adequate or above, and 32% answered ‘don’t know’. MOPAN donors in-country were less positive than direct partners and technical partners and the difference was statistically significant.

MI 16.4 – Parallel implementation structures avoided

Survey respondents (MOPAN donors in-country, direct partners and technical partners) were asked whether WHO avoids the use of parallel project implementation units. Just over half of survey respondents (54%) rated WHO as adequate or above and 29% answered ‘don’t know’.

The document review rated WHO as strong on avoiding parallel implementation structures (indicator 6 of the Paris Declaration on Aid Effectiveness). WHO’s programmes support national health policies, strategies or plans and, as such, it does not create parallel implementation units (PIUs).

MI 16.5 – Promotion of mutual accountability in its partnerships

This MI was assessed by survey only. Survey respondents (MOPAN donors in-country, direct partners and technical partners) were asked whether WHO encourages mutual accountability assessment of Paris Declaration and subsequent Aid Effectiveness commitments (Accra Agenda for Action, Busan High Level Forum). Half of the respondents (50%) rated WHO as adequate or above, and 29% answered ‘don’t know’.

KPI 17: Contributing to policy dialogue

Finding 17: According to survey respondents, WHO makes a strong contribution to policy dialogue while respecting the views of its partners in the process.

This KPI was assessed by the survey only and WHO was rated strong. MOPAN donors at headquarters and direct partners held more favourable views than technical partners on WHO’s input to policy dialogue. This KPI received some of the highest ratings given by survey respondents, highlighting the work of WHO as a technical organisation.

Figure 3.25 | KPI 17: Contributing to policy dialogue, ratings of micro-indicators

MI 17.1 – Reputation for valuable input to policy dialogue

All survey respondent groups other than peer organisations were asked whether WHO provides valuable inputs to policy dialogue. The majority (86%) rated WHO as adequate or above. MOPAN donors at headquarters and direct partners were more positive than technical partners and the difference was statistically significant.

MI 17.2 – Policy dialogue respects partner views

All survey respondent groups other than peer organisations were asked whether WHO respects the views of partners when it undertakes policy dialogue. The majority (81%) rated WHO as adequate or above.

KPI 18: Harmonising procedures

Finding 18: WHO was recognised by survey respondents and the document review as adequate in the harmonisation of its procedures with other actors.

MOPAN donors in-country, direct partners and technical partners rated WHO as adequate on this KPI. The document review considered WHO adequate on all MIs.

Figure 3.26 | KPI 18: Harmonising procedures, ratings of micro-indicators

MI 18.1 – Participation in joint missions

MOPAN donors in-country, direct partners and technical partners were asked whether WHO engages in joint planning, programming, monitoring, and reporting with bilateral and multilateral partners. The majority (67%) rated WHO as adequate or above.

The document review rated WHO as adequate on the extent to which it engages in joint planning, programming, monitoring and reporting (indicator 10a of the Paris Declaration on Aid Effectiveness). The document review found evidence of joint missions only and therefore could not assess the organisation on other aspects of this PD indicator. WHO reports that it fell just short of the 2010 target of 40% for this indicator (36.6%).[36] Among the six countries sampled for this assessment there was considerable variance for this indicator (Ethiopia 57.14%, Guatemala 100%, Indonesia 96.55%, Mozambique 16%, Pakistan 100%, and Viet Nam 5.08%).

MI 18.2 – Technical co-operation disbursed through co-ordinated programmes

MOPAN donors in-country, direct partners and technical partners were asked whether WHO’s technical assistance is provided through co-ordinated programmes in support of capacity development. The majority (74%) rated WHO as adequate or above.

The document review rated WHO as adequate on the extent to which technical co-operation is disbursed through co-ordinated programmes (indicator 4 of the Paris Declaration on Aid Effectiveness). The target for 2010 was 50%. WHO reports that of the total ODA, 39% was disbursed in the form of technical co- operation (i.e. professional staff costs, contracts for technical work, technical missions carried out by regional and HQ technical staff, capacity building, workshops and seminars). However, according to WHO, there may be under-reporting of the amount WHO spent on technical co-operation because of the definitions used for technical co-operation in the PD survey. [37]

Although WHO did not reach the target for this indicator among all countries, the six countries sampled for this assessment reported high levels of compliance with this indicator (100% for Ethiopia, Guatemala, Indonesia, Pakistan, and Viet Nam, and 98% for Mozambique).

MI 18.3 – ODA disbursements/support for government-led PBAs

The document review rated WHO as adequate on the percentage of overall ODA disbursements / support for government-led PBAs (indicator 9 of the Paris Declaration on Aid Effectiveness).

In 2010, WHO reported that 42.2% of its ODA was disbursed to or supported PBAs (7% in direct budget support and 35% to other assistance provided in support of initiatives adopting PBAs). This fell short of the 2010 indicative target of 66%. Among the six countries sampled there were considerable variances reported on this indicator (Ethiopia 7.35%, Guatemala 11.6%, Indonesia 41.38%, Mozambique 11.25%, Pakistan 100%, and Viet Nam 7.95%).[38]

However, as a specialised health agency, WHO provides technical co-operation in the form of guidance and policy advice that are reflected in staff and activity costs rather than in direct funding to the government. As noted above, direct financial co-operation constitutes a relatively small proportion of WHO’s contributions. In addition, results from the IHP+Results survey indicate that in 2011 WHO had surpassed the 66% target, based on data from 19 participating countries.

KPI 19: Managing the cluster

Finding 19: WHO’s cluster management was perceived as adequate by survey respondents.

This KPI was assessed by survey only. MOPAN donors in-country, direct partners, technical partners and peer organisations provided ratings of adequate on all four MIs under this KPI.

Figure 3.27 | KPI 19: Managing the cluster, ratings of micro-indicators

MI 19.1 – Co-ordinated rapid assessments

All survey respondents at country level were asked whether WHO implements co-ordinated rapid assessments to identify health needs and risks. The majority (74%) rated WHO as adequate or above (85% of direct partners, 58% of technical partners, 87% of peer organisation respondents, and 50% of country office respondents).

MI 19.2 – Sufficient analytical resources

All survey respondents at country level were asked whether WHO dedicates sufficient analytical resources and policy-level engagement to strategic activities within the cluster. The majority (75%) rated WHO as adequate or above (85% of direct partners, 79% of technical partners, 69% of peer organisation respondents, and 71% of country office respondents).

MI 19.3 – Dedicated staff for co-ordination of the cluster

All survey respondents at country level were asked whether WHO provides sufficient qualified, dedicated staff for co-ordination of the cluster. The majority (81%) rated WHO as adequate or above (88% of direct partners, 86% of technical partners, 75% of peer organisations, and 79% of country office respondents).

MI 19.4 – Monitors and reports on cluster results

All survey respondents at country level were asked whether WHO monitors implementation of the cluster strategy and regularly reports on results. The majority (68%) rated WHO as adequate or above (86% of direct partners, 69% of technical partners, 60% of peer organisations, and 64% of country office respondents).

3.3.5 Knowledge management

Survey respondents found WHO’s knowledge management to be adequate overall. The document review considered WHO adequate in presenting performance information and sharing documents publicly but inadequate in evaluating results and disseminating lessons learned.

Figure 3.28 below shows the overall survey and document review ratings for the four KPIs in the knowledge management performance area.

WHO is a knowledge-based organisation and performs various critical functions in the health sector such as the gathering of the best knowledge available on health practices, the publication of global health statistics, and the publication and dissemination of health literature. WHO’s Knowledge Management Strategy has three objectives: contributing to strengthening country health systems through better knowledge management, promoting the principles and practice of knowledge management as a fundamental aspect of public health research and practice, and enabling WHO to become a better learning and knowledge sharing organisation. The Department of Knowledge Management and Sharing (KMS) is accountable for implementing the strategy as it strives to improve the understanding and application of knowledge management in the pursuit of WHO’s mandate. WHO’s Institutional Repository for Information Sharing provides access to knowledge from WHO and other sources of scientific literature.[39] In addition, WHO Press is WHO’s publisher for the dissemination of scientific, technical and medical advice that WHO wishes to disseminate.

Figure 3.28 | Performance area IV: Knowledge management, survey and document review ratings

Figure 3.29 shows the mean scores for the four KPIs for all survey respondents, and by respondent groups.

Figure 3.29 | Performance area IV: Knowledge management, mean scores by respondent group

Very weak = 1.00–1.49
Weak = 1.5–2.49
Inadequate = 2.5–3.49
Adequate = 3.5–4.49
Strong = 4.5–5.49
Very strong = 5.5–6.00

KPI 20: Evaluating results

Finding 20: WHO was perceived by survey respondents as performing adequately with regard to evaluating results. The document review highlighted areas for improvement, many of which are being addressed in the current reform process.

A review published by the Joint Inspection Unit in 2013 asserts that “Evaluation has been a weak area in WHO oversight work”, in part due to poor policies and limited resources.[40] WHO has invested considerable resources in this area and is in the process of strengthening its evaluation function. While it is making progress in systems and practices, the document review found that there is room for improvement in the coverage and quality of evaluations.

As part of its reform agenda, WHO approved an Evaluation Policy in 2012. In addition, the Office of Internal Oversight Services plans to establish a Global Network on Evaluation, disseminate an Evaluation Hand- book, present an annual workplan for evaluation, develop a web-based inventory of evaluations, and recruit additional staff to improve the capacity of the unit. While WHO’s commitment to evaluation is evident and appears to be bringing positive changes, it is too early to assess the full effects of the reform in this area.

Figure 3.30 | KPI 20: Evaluating results, ratings of micro-indicators

MI 20.1 – Independent evaluation unit

MOPAN donors at headquarters were asked whether WHO ensures the independence of its evaluation unit. Nearly two-thirds (65%) rated the organisation as adequate or above, and 27% answered ‘don’t know’. The document review rated WHO adequate on this MI. The Office of Internal Oversight Services (OIOS), which acts as co-ordinator of the decentralised evaluation function, is structurally independent from programme management. It reports directly to the Director General, and annually in a report for consideration by the Executive Board. However, on matters relating to internal audits, evaluations and investigations at WHO, the document review found that OIOS did not provide sufficient information for oversight, decision making and management purposes.[41]

According to the 2012 Evaluation Policy adopted as part of the WHO reform, OIOS is the custodian of the evaluation function.[42] The policy emphasises the impartiality and transparency of the evaluation function and clearly describes how independence is assured. The policy strengthens the evaluation function, especially regarding the publication of workplans and reports.

MI 20.2 – Sufficient evaluation coverage of programming activities

This MI was assessed by document review only, which rated WHO inadequate for the coverage of its evaluations.

The 2012 Evaluation Policy established categories for the planning and prioritisation of evaluations. According to the 2013 WHO Evaluation Practice Handbook, evaluation coverage will be established in consultation with headquarters, regional and country offices and will then be delineated in an evaluation workplan. However, the criteria on evaluation coverage and prioritisation are vague.

According to the Handbook, the Global Network on Evaluation will be responsible for co-ordinating a workplan that will be published once every two years (and updated annually). The first workplan will be published in 2014-2015. WHO published a transitional workplan in 2013 that has been presented to the Board. While it is too early to assess the effect of these new initiatives, it is likely that the Global Network on Evaluation will contribute to strengthening the evaluation culture at WHO. [43]

Of the six countries sampled for this assessment, four had conducted programmatic or thematic eval- uations during the 2008-2013 period (there was no evidence of country programme evaluations). Independent evaluation reports for the countries sampled are not available to the public,[44] but WHO is planning to develop a web-based inventory and to publish all evaluations on its website as part of its reform process. The implementation date is not yet known.

MI 20.3 – Quality of evaluations

This MI was assessed by document review only, which rated WHO as weak for its lack of a systematic approach to ensuring the quality of evaluations. Although the 2006 WHO Evaluation Guidelines foresaw that evaluations would be carried out in accordance with UNEG standards, there is no evidence that there were procedures for quality control. WHO has recently issued guidance and established procedures to enhance the quality control of evaluations. In 2013, the Evaluation Practice Handbook introduced a series of quality checklists for: evaluation Terms of Reference, evaluation reports, and compliance with WHO evaluation policy. WHO has also recently established the Global Network on Evaluation which, according to the Terms of Reference, will be responsible for the quality control and assurance system. When fully implemented, changes to policy/procedures for the quality control of evaluations along with the 2012 Evaluation Policy could help address some of the weaknesses noted by the document review.

MI 20.4 – Use of evaluation findings to inform decisions

MOPAN donors at headquarters perceived WHO’s use of evaluation findings in its decisions on programming, policy and strategy to be adequate: 57% rated it adequate or strong, 18% provided ratings of inadequate or below.

MI 20.5 – Beneficiaries and direct partners involved in evaluations

MOPAN donors in-country, direct partners and technical partners were asked whether WHO involves partners and beneficiaries in evaluations of its projects or programmes. The majority (63%) rated WHO as adequate or above, and 27% provided ratings of inadequate or below.

KPI 21: Presenting performance information

Finding 21: Survey respondents believe that WHO adequately presents performance information on its effectiveness, but evidence from the document review indicates that improvement is needed in reporting on the achievement of outcomes, programme adjustments and the use of data obtained from measuring indicators.

Donors at headquarters were the only respondent group asked about the extent to which WHO presents performance information on its effectiveness. The majority perceived WHO to perform adequately in this area.

The document review provided scores from weak to strong and found that WHO’s presentation of performance information was one of its main areas for improvement.

Figure 3.31 | KPI 21: Presenting performance information, ratings of micro-indicators

MI 21.1 – Reports on achievement of outcomes

MOPAN donors at headquarter were asked whether WHO reports to the Executive Board provide clear measures of achievement of outcomes. Their views were mixed as 61% rated WHO adequate or strong, and 37% rated it inadequate or weak.

WHO’s reporting on the achievement of outcomes was rated inadequate by the document review. There is no annual performance reporting, as such. WHO publishes a Mid-Term Review and a Performance Assessment Report for each programme budget cycle that collectively provide an overview of results achieved by WHO each year. These documents have some limitations due to the weaknesses in the structure of the results chain and the phrasing of the results statements in the MTSP (as discussed in KPI 3). The narrative of the reports describes the nature, progress and limitations of some of WHO’s interventions, but there is limited analysis of how WHO’s products and services are contributing to outcomes. These reports do not establish clear links between organisation-wide outputs and outcomes. Due to the wide-ranging nature and number of activities worldwide, it is understandable that there only is partial reporting on results (i.e. focusing on only a few results per year). However, this means that the narrative of the reports presents only a fragmented view of WHO’s contributions to results.

MI 21.2 – Reports on performance using data obtained from measuring indicators

This MI was assessed by document review only.

Based on the analysis of the most recent “Mid-Term Review” (2011) and “Performance Assessment Report” (2012) of the Programme Budget, WHO’s use of data obtained from measuring indicators to report on performance is weak. The Mid-Term Review presents a punctual assessment on all indicators (i.e. on track, at risk, in trouble), but does not present actual data against baselines or targets. In contrast, the Performance Assessment Report contains data on baselines, targets and results achieved for each organisation-wide expected result at the end of the programme budget cycle. However, these reports do not provide data on WHO’s progress over the years or of trends over the period of the MTSP. The narrative of both reports provides very little analysis of the progress towards targets or how the targets link to outcomes. Most indicators in the reports sampled do not respect SMART or CREAM criteria (see KPI 3).

MI 21.3 – Reports against corporate strategy, including expected results

MOPAN donors at headquarter were asked whether WHO reports adequately against its organisational strategy. Their views were mixed as 69% rated WHO adequate or strong, and 31% rated it inadequate or weak.

WHO was rated strong on this MI by the document review. The “Mid-Term Review” (2011) and the “Performance Assessment Report” (2012) assessed results against those defined in the development and management results frameworks in the Medium-Term Strategic Plan 2008-2013. Despite weaknesses in the results frameworks (see KPI 3), WHO received a positive rating on this indicator because it does report in a systematic way on all results (organisation-wide expected results) and on indicators identified in the results frameworks. Explanations of some variances between actual results and planned results identified in the results framework are delineated in the narrative of the report for each of the organisation-wide expected results. These brief narratives for each expected result provide some examples of contributions made and challenges faced at the regional and country levels.

MI 21.4 – Reports on aid effectiveness commitments using indicators and country targets

MOPAN donors at headquarter were asked whether WHO reports to the governing body on performance in relation to its aid effectiveness commitments (e.g. Paris Declaration/Busan). Their views were mixed as 43% rated WHO adequate or strong, 35% rated it inadequate or below.

While WHO does not independently and publicly report on Paris Declaration commitments, there is evidence that WHO is committed to assessing its organisational performance against the principles of aid effectiveness (e.g. Paris Declaration/Busan). The organisation received a rating of adequate. According to internal documentation, WHO participates actively in the OECD/DAC monitoring survey. For the 2011 report, it collected and consolidated data on Paris Declaration indicators from its country offices in a rigorous manner.[45] Even though the monitoring of the PD indicators is not conducted annually, there is evidence that WHO established robust processes to support country offices and ensured the quality of the data produced for the OECD/DAC monitoring survey in 2011. WHO is also an active member of the International Health Partnership (IHP), which leads monitoring efforts that operationalise the commitments made by development partners and countries under the Paris Declaration on Aid Effectiveness.

MI 21.5 – Reports on adjustments to policies/strategies based on performance information

This MI was assessed by document review only. Strictly adhering to MOPAN criteria, it rated WHO as adequate in reporting on policy and strategy adjustments based on performance information.

While annual performance reporting is conducted, no evidence was found at the headquarters level of a policy that defines how annual performance reporting is carried out.[46] However, various General Management System Guidance Notes provide information to WHO staff on how to input performance information in the system for both the Mid-Term Review and the End of Biennium Performance Assessment. The document review also found evidence that programme budgets (results statements, indicators, baselines and targets) are reviewed based on performance information from the previous reviews and assessments. Furthermore, interim reports on the implementation of the Programme Budget are provided to the World Health Assembly by the Secretariat following each Mid-Term Review. These reports contain information for the scaling up of specific initiatives and for undertaking corrective actions such as re-programming and allocating or reallocating resources to specific priority areas.

MI 21.6 – Reports on programming adjustments based on performance information

This MI was assessed by document review only. WHO was rated as inadequate for country-level reporting on programme adjustments based on performance information. While no evidence was found at the country level of a policy that defines how annual performance reporting is carried out, some guidance can be found in the General Management System (GSM) Guide, the GSM Guidance Notes, the WHO: Operational Planning – Business Rules, Procedures (2007), and the WHO Country Co-operation Strategy Guide (2010). A suite of country level documents (including Country and Office Specific Expected Results Progress Reports, Mid-Term Reviews, End of Biennium Progress Reports, as well as, financial implementation reports) also provides overall progress status and required actions to achieve expected results. This suite of documents is not made available to the public and is used for internal management purposes only.

KPI 22: Disseminating lessons learned

Finding 22: Survey respondents rated WHO as adequate overall in disseminating lessons learned. The document review provided a rating of inadequate.

Surveyed donors at headquarters considered WHO adequate in encouraging the identification, documentation and dissemination of lessons learned and/or best practices. The document review found that WHO continues to be committed to solidifying its role as a provider of knowledge on health (e.g. practices, statistics and research), but noted room for improvement in WHO’s reporting on how lessons learned and best practices are transforming the organisation’s programming.

Figure 3.32 | KPI 22: Disseminating lessons learned, ratings of micro-indicators

MI 22.1 – Reports on lessons learned based on performance information

MOPAN donors at headquarters were asked whether WHO identifies and disseminates lessons learned from performance information. The majority (61%) rated WHO as adequate or above.

The document review rated WHO as inadequate on this MI. While WHO prepares a range of thematic reports and guidelines, there is limited evidence that WHO systematically collects and disseminates lessons learned, including best practices, from performance reports and evaluations. WHO does not appear to have a system in place that identifies lessons learned based on performance information.

MI 22.2 – Lessons shared at all levels of the organisation

MOPAN donors at headquarters were asked whether WHO provides opportunities at all levels of the organisation to share lessons from practical experience. Their views were mixed as 37% rated WHO adequate or above, 33% rated it inadequate or below, and 31% answered ‘don’t know’.

KPI 23: Availability of documents

Finding 23: The document review rated as adequate in making its documentation available to the public.

The document review rated WHO as adequate on making documents available to the public, even though the organisation does not yet have a disclosure/access to information policy.

Figure 3.33 | KPI 23: Availability of documents, ratings of micro-indicators

MI 23.1 – Key documents available to the public

The document review rated WHO’s efforts as adequate. Most of the key documents related to governance, finance, strategy, performance are available on the organisation’s website and often in multiple languages. Resolution WHA51.30 requires that governing body documents are made available on the Internet in the six official languages (English, French, Spanish, Russian, Chinese and Arabic). WHO does not have a disclosure/access to information policy but plans to present a draft policy based on best practices in international organisations to the Executive Board in 2013. WHO does not publish audits, evaluations and operational documentation on country programmes and projects. The website has clear instructions on how to contact the organisation.

4.1 INTRODUCTION

This section presents the results of the 2013 Common Approach assessment of WHO in measuring and reporting on development results. It includes three key performance areas:

• Section 4.2: Evidence of progress towards organisation-wide outcomes (KPI A)

• Section 4.3: Evidence of contributions to country-level goals and priorities, including relevant Millenn- ium Development Goals (KPI B)

• Section 4.4: Relevance of the organisation’s objectives and programme of work to country-level stakeholders (KPI C)

The assessment of this component uses the same “traffic light” colours used in the organisational effectiveness component but applies a simplified 4-point scale. The methodology is explained in Volume II, Appendix I.

Figure 4.1 provides a summary of the assessment of the three KPIs. The detailed findings on each KPI are presented in the sections that follow.

4.2 EVIDENCE OF WHO’S PROGRESS TOWARDS ORGANISATION-WIDE OUTCOMES

4.2.1 Overview

This section presents the results of the assessment of WHO’s progress towards organisation-wide outcomes. KPI A suggests that an effective organisation should demonstrate progress towards organisation-wide, institutional outcomes.[47] These are usually related to the organisation’s strategic objectives. The assessment draws on the evidence that the organisation has available on its different result areas, primarily its reports on results.

WHO’s mandate and core functions

According to its constitution, WHO’s objective “shall be the attainment by all peoples of the highest possible level of health”.[48] To fulfil this mandate, WHO identified in its Eleventh General Programme of Work (2006-2015) a series of core functions that illustrate its role:

“providing leadership on matters critical to health and engaging in partnerships where joint action is needed; shaping the research agenda and stimulating the generation, translation and dissem- ination of valuable knowledge; setting norms and standards, and promoting and monitoring their implementation; articulating ethical and evidence-based policy options; providing technical support, catalysing change, and building sustainable institutional capacity; monitoring the health situation and assessing health trends”.[49]

WHO provides products and services to partner countries primarily to support the development of national capacities.

Figure 4.1 | Development results component – overall ratings

WHO’s results measurement and reporting at an organisation-wide level[50]

WHO’s current organisational plan, the Medium-term Strategic Plan (MTSP), is structured around a six-year cycle (2008-2013).[51] The MTSP includes 13 strategic objectives: 11 programmatic objectives (1-11) and two managerial objectives (12-13). Each strategic objective contains results statements, known in WHO’s lexicon as organisation-wide expected results[52] (OWER), through which WHO engages with partners and provides programming and other types of support. The strategic cycle is further divided into biennial programme budgets that provide an updated version of the results framework (approximately 160 indicators, baselines and targets) and budget requirements for the next biennium.

Since 2008, WHO has had a functioning enterprise resource planning (ERP) system in place known as the General Management System (GSM). The system contains information on programme management (expected results, products, services and activities) and administration (HR, finance, procurement and travel) at three levels: country offices, regional offices, and headquarters. Data are collected and entered periodically in the system based on standard procedures and then compiled and consolidated at HQ for efficiency, accountability and reporting purposes.

Based on data compiled in the GSM, WHO communicates its organisation-wide results through two documents: the Mid-Term Review (MTR) and the Performance Assessment Report (PAR). These reports, which are published every two years, provide data and analysis on each OWER. The MTR examines progress towards organisation-wide results and provides an overview of major successes and risks associated with the implementation of the Programme Budget. It does not provide a measurement against indicators, but relies instead on a self-assessment of the expected results accomplished based on a 3-point scale (on track, at risk, or in trouble). The MTR has improved since 2008-2009; in 2010-2011 it contains more narrative on each SO to describe progress and is not as narrowly focused on budget implementation as in the past. The PAR also provides an assessment of each organisation-wide expected result based on a 3-point scale (fully achieved, partly achieved, or not achieved). Unlike the MTR, the PAR provides measurements of each indicator against baselines and targets, but this is based solely on self-reported data.[53] WHO also takes stock of achievements and shortfalls in a Mid-Term Review and an end-of-cycle review of the MTSP.

For an overview of the documents used in this chapter, please see Volume II, Appendix VIII.

Data used for this assessment

The assessment of KPI A is based on survey data and a document review that considered all available organisation-wide performance information from the most recent cycle (2008-2013). The main documents consulted include the Medium-Term Strategic Plan 2008-2013 and corresponding Programme Budgets, Mid-Term Reviews, Performance Assessment Reports covering results achieved from 2008 to 2012. Evaluations were also reviewed to find complementary evidence and help validate reported achievements.

Attention was paid to the following elements: quality of results statements, including indicators, baselines and targets; the quality and consistency of evidence presented to substantiate the results achieved, including an assessment of contribution; and the evidence of progress towards organisation-wide outcomes reported by the organisation.

4.2.2 Evidence of extent of progress towards organisation-wide outcomes

This section provides an overall rating for KPI A based on documentation made available by the organisation and survey data. It also includes assessments of: the quality and consistency of reports on organisation-wide results and on the extent of progress towards WHO strategic objectives, which draws on a review of WHO’s performance information and reviewed survey data from donors at headquarters for each strategic objective.

Overall assessment

Figure 4.2 shows the overall rating for this KPI based on the review of WHO’s reports on organisation-wide results and WHO’s progress towards its strategic objectives – as expressed in WHO reports[54] and as indicated by surveyed stakeholders.

MOPAN donors at headquarters hold positive views about WHO‘s progress towards its stated results in all 11 strategic objectives of the MTSP 2008-2013. WHO’s Performance Assessment Reports for 2008-2009 and 2010-2011 provide some evidence of progress towards planned activities and outputs in WHO’s framework, but unclear and limited evidence of the results and contributions that WHO is making to outcomes (i.e. higher-level change). For example, as it is currently structured, WHO’s results framework does not provide a clear description of the contribution it is making to the reduction of communicable diseases. Results in terms of people vaccinated or vaccine national coverage are reported by WHO, but the extent to which the organisation contributed to the achievement of these results is unclear.

Figure 4.2 | KPI A: Evidence of the extent of progress toward organisation-wide outcomes, overall rating

Overall Rating: Inadequate

How WHO reports on organisation-wide results

Finding 24: Survey respondents rated WHO as adequate overall in disseminating lessons learned. The document review provided a rating of inadequate.

Finding 25: WHO provides consistent data on the same performance indicators across programme budgets, but data reliability is compromised by the absence of independent and external sources, such as evaluations.

WHO has a system to collect performance information from its Country Offices, which are required to report on the level of achievement of organisation-wide expected results as measured against a series of indicators. The precision and detail of reporting against these indicators is impressive but it says little about the extent to which WHO is making progress towards its organisation-wide results. This is due to the absence of a clear organisation-wide results chain. It is expected that an organisation will develop a results chain that clearly defines its own inputs and expected outputs as well as its expected contribution to outcomes and impact in its strategic objective areas. WHO’s strategies and reports provide neither adequate explanations of the expected changes in any given period nor of the links between outputs and expected outcomes. Consequently it is not possible to assess WHO’s contribution to outcomes.

This issue may be addressed and resolved by the current reform underway and by the introduction of a new results chain that links the work of the Secretariat (outputs) to the health and development changes to which it contributes, both in countries and globally (outcomes and impacts). The new results chain is also described in the theory of change presented in the 12th GPW. Within the new results framework, the assessment of the contribution to high-level results will be based on evidence of the accomplishment of the deliverables stated in the programme budgets.

Despite the work being carried out to improve its focus on results as part of its organisational reform, WHO’s performance measurement system relies almost exclusively on self-reported data from Country Offices. There are very few independent evaluations from which one can validate results achieved at the outcome level or higher.

Figure 4.3 presents a summary assessment of the quality of WHO’s systems and practices for measuring and reporting on its organisation-wide results. The criteria represent elements of good results reporting (see Volume II, Appendix I for full descriptions). The organisation is assessed according to whether it has met, partially met, or not met the criteria.

Figure 4.3 | WHO’s measurement and reporting on organisation-wide results

1. Theory of change’ is understood in the sense defined by Rist and Morra Imas (2009) as, “a representation of how an intervention is expected to lead to desired results” and in the sense defined by Michael Quinn Patton who has stated that a theory of change is more than the sequential order of results statements presented in a logic model; it requires key assumptions related to the results chain and context (e.g. policy and environment), and important influences and risks to be made explicit – Qualitative Research and Evaluation Methods (2002).
2. This refers to the existence of reports on outputs as defined by the OECD (i.e. lower level results). Some MOs use different terminology for the various levels of results.
3. This refers to the existence of reports on outcomes as defined by the OECD (i.e. higher level results). Some MOs use different terminology for the various levels of results.
4. Reporting according to a theory of change is understood to mean the extent to which organisations provide a narrative describing the actual implementation process and results achieved in relation to that foreseen in the initial ‘theory of change’.
5. According to Rist and Morra Imas, The Road to Results – “Reliability is the term used to describe the stability of the measurement – the degree to which it measures the same thing, in the same way, in repeated tests.” Attention is also given to the quality of the evidence– specifically, whether or not it has been derived from or validated by an external and/or independent source.

Strategic planning and theory(ies) of change

While WHO adequately describes the overall scope of each strategic objective, it does not document comprehensive or cumulative progress on results. The articulation of a clear theory of change, the establishment of an organisation-wide results chain, and the definition of robust performance indicators would allow stakeholders to understand the intended causal links and assumptions from activities and outputs identified to outcomes, including the significance of the types of results that may have been achieved. In the absence of this, it is difficult to identify WHO’s contribution towards broad strategic objectives and its progress towards organisation-wide expected results. In the context of greater demand for evidence of development results, the assessment noted some weaknesses in WHO planning and results-based management. As noted in the assessment of KPI 3 (see section 3.3.2), the quality of WHO’s results framework is poor. Result statements do not clearly differentiate between activities, outputs and outcomes. It is, therefore, difficult to understand the programme theory behind WHO’s interventions.[55] Further, the document review noted deficiencies with the defined indicators as they do not meet all SMART and CREAM criteria.[56] This renders the monitoring of progress and the measurement of results achieved very challenging.

Measuring and communicating results

Although WHO provides baselines and targets for each organisation-wide expected result to allow for adequate internal monitoring, WHO reports primarily on the “number of countries” that have received WHO’s support as the unit of analysis. While the ‘country’ unit of analysis has some validity given the role WHO plays in advising and supporting national counterparts, it does not provide sufficient inform- ation, given the confusion in the results chain, to clearly understand the significance or the depth of changes that are occurring at the country level and how they contribute to WHO’s strategic objectives at the global level.

For instance, in the area of “equitable access of all people to vaccines”, WHO’s desired achievement is a statement of activities/ interventions followed by an objective. As shown, the achievement of this organisation-wide expected result is assessed by two outcome indicators. Since there is no results chain, WHO is unable to provide a clear description and measurement of the outputs delivered in the narrative of the Performance Assessment Report. Since it is not possible to establish linkages between WHO’s products and services and outcomes achieved, the performance story therefore is incomplete.

In order to meet its accountability commitments, WHO produces a Mid-Term Review and Performance Assessment Report for each Programme Budget (two-year cycle). These reports provide a significant amount of information on the interventions that WHO is supporting. Performance information is aggregated in such a way that one can easily understand if WHO is meeting its targets. In contrast to many other international organisations, WHO deserves to be praised for the format and the performance data in its reports. However, despite the aggregated level data, and as noted above, WHO’s results framework does not allow for a clear understanding of the links between its activities and health outcomes throughout the world.

Data reliability

The performance data used by the document review to assess the extent of progress towards WHO strategic objectives is mainly self-reported. While WHO consistently provides detailed information in its Mid-Term Reviews and Performance Assessment Reports, the document review could not triangulate these findings with other sources. As noted in the assessment of KPI 20, the coverage and quality of WHO’s evaluations is unsatisfactory. As a result, this assessment could not rely on a body of reliable and credible information generated by external and/or independent evaluations. Those evaluations that do exist provide a picture of the organisation’s performance in specific areas that were not relevant to this assessment (see next section). There is therefore an over-reliance on data the quality and independence of which could be called into question. WHO’s reporting on its progress towards organisation-wide outcomes would benefit from performance information provided through independent evaluations of sectors, strategic objectives, specific themes and/or regions.

Factors that are likely to positively affect outcome achievement

Despite the absence of theories of change or robust outcome-level reporting, WHO has a number of systems and practices in place that are likely to positively influence the achievement of outcomes and impacts. However, these elements are not always adequately reflected in WHO’s results framework or reporting.

• WHO produces, uses and distributes a great amount of data, statistics and publications related to various public health subjects. For instance, the World Health Report combines an expert assessment of global health, including comprehensive international and global statistics, to provide countries, donor agencies and international organisations with the information to inform policy making and funding decisions.

• Expert staff at all levels of the organisation provide technical support and capacity building not only to member states and other health partners, but also to staff at the country level (e.g. country level staff receive technical support in order to better plan, co-ordinate and adjust WHO’s own programmes).

• WHO’s enterprise resource planning, the GSM, supplies a steady flow of data to senior management related to management and programmes. The information, which integrates all aspects of WHO’s functions, can, and most likely will, be used to improve the organisation’s management and programmes. Therefore, it is likely that the GSM can make WHO a more efficient, flexible and integrated organisation.

While the benefits of each of these factors have not been measured, taken together, they increase the likelihood that WHO’s products and services are better linked to the achievement of its strategic objectives.

Evidence of the extent of progress towards WHO’s strategic objectives

Finding 26: Surveyed stakeholders consider that WHO is making progress towards its organisation- wide strategic objectives. The document review found evidence of progress towards organisation-wide expected results in some strategic objectives; however, the data presented could not be triangulated with independent and external sources as there were very few independent evaluations.

A total of 76 MOPAN donors at headquarters were asked whether WHO is making progress towards its 11 programme-related strategic objectives. The 49 respondents rated WHO as adequate or better on all strategic objectives. The highest ratings were for communicable diseases (SO1) and HIV/AIDS, tuberculosis and malaria (SO2) – both of which were rated as strong. These two strategic objectives received approximately 45% of WHO’s total expenditures in the last two programme budgets.

As noted above, WHO does not report consistently on outcomes achieved but on a series of organisation- wide expected results (OWERs) that are a mixture of activities, outputs and outcomes. In 2010-2011 WHO’s 11 strategic objectives include 75 organisation-wide expected results (OWER). Each expected result comprises between one and five indicators for a total of around 160 indicators.

Data on progress is self-reported and the document review found very few independent evaluations (which are noted in the assessment as appropriate). It is difficult to assess the extent to which WHO has contributed to development results on the basis of WHO organisation-wide reports alone.

However, WHO organisation-wide reports do provide data on its OWERs and make reference to baselines and targets drawn from the Performance Assessment Reports (PAR). Data on progress towards WHO’s strategic objectives is self-reported and was extracted from the Performance Assessment Reports 2008-2009 and 2010-2011 (see Volume II, Appendix IX).

According to data compiled from WHO’s reports, WHO achieved most expected results in the areas of communicable diseases (SO1), chronic non-communicable conditions (SO3), and risk factors for health (SO6), but struggled to fully achieve its expected results in the areas of HIV/AIDS, tuberculosis and malaria (SO2), and a healthier environment (SO8).

Figure 4.4 presents a summary of the extent of progress towards WHO’s strategic objectives as reported by WHO. It shows the extent to which OWERs were achieved in the last two programme cycles (2008-2009 and 2010-2011).[57] WHO rates the degree of success in achieving organisation-wide expected results (OWER) according to three criteria: fully achieved, partly achieved, and not achieved.[58] The final column shows the mean scores of surveyed MOPAN donors at headquarters who were asked to rate WHO’s progress towards its stated strategic objectives. The survey used the same 6-point scale used in the assessment of practices and systems.

Figure 4.4 | Extent of progress towards WHO’s OWERs under each strategic objective (MTSP 2008-2013 cycle)

Very weak = 1.00–1.49
Weak = 1.5–2.49
Inadequate = 2.5–3.49
Adequate = 3.5–4.49
Strong = 4.5–5.49
Very strong = 5.5–6.00

1. These rates represent expenditures by SO based on the total expenditure for Programme Budget 2008-2009 and Programme Budget 2010-2011 (in percentages).
2. Although there were only 70-75 OWER in each biennium, these figures were consolidated from the Programme Budgets 2008-2009 and 2010-2011 and consequently show a total of 146 OWER.

Assessment of WHO’s strategic objectives

The following sections present the survey ratings and document review assessment for each strategic objective. They include: a description of the strategic objective; survey respondents’ rating of WHO’s progress in that area; and examples of WHO’s interventions and information from evaluation reports. [59]

WHO’s own ratings for each organisation-wide expected result are shown in Volume II, Appendix IX.

SO 1: Communicable diseases

Under this strategic objective, WHO’s work focuses on the “prevention, early detection, diagnosis, treat- ment, control, elimination and eradication measures to combat communicable diseases that dispro- portionately affect poor and marginalised popu- lations”, including vaccine-preventable, tropical, zoonotic and epidemic-prone diseases.[60] This strategic objective accounts for between 33 and 35% of WHO’s expenditures, which represents the area with the highest level of expenditure.

MOPAN donors at headquarters were asked whether WHO is making progress towards its objective of reducing the health, social and economic burden of communicable diseases. The majority of respondents (92%) rated WHO as adequate or above.

In the last two programme cycles, WHO did not reach all of its targets in OWER 1.1 or 1.2, although it reported progress against key indicators for these two expected results. WHO’s continued support facilitated co-ordination between countries on vaccination, but progress was affected by weak health infrastructures and endemic transmission of wild polio virus (e.g. Afghanistan, India, Nigeria and Pakistan).

Based on available performance data at organisation-wide level, WHO consistently achieved its expected results for 1.4, 1.5, 1.8 and 1.9 during the 2008-2009 and 2010-2011 Programme Budgets (see Volume II, Appendix IX).

In WHO’s Performance Assessment Reports, there are a few examples in the section on SO 1 of WHO’s contributions to higher-level results (see sidebar). For instance, one of WHO’s key accomplishments in this strategic objective has been the support of the research agenda on accelerating eradication and also the processing, analysis and distribution of information on the global poliomyelitis situation as part of the Global Polio Eradication Initiative.[61] In the area of surveillance and monitoring of all communicable diseases, WHO has used information produced jointly by UNICEF and WHO to refine methodology for developing immunisation coverage estimates.

However, the document review found no clear evidence of WHO’s contributions in this area in recent evaluations (e.g. “Polio Eradication Initiative: Value for Money”; “Independent Monitoring Board of theGlobal Polio Eradication Initiative”). The evaluations show that the incidence of polio has declined dramatically since 1988 (more than 99%), but only offer a global overview of the Polio Eradication Initiative and do not provide specific evidence with regard to WHO’s results.

SO 2: HIV/AIDS, tuberculosis and malaria Strategic objective 2 focuses on combating HIV/ AIDS, tuberculosis and malaria by scaling up and improving prevention, treatment, care and support interventions. It aims to achieve universal access, advancing related research; removing obstacles that block access to interventions and impediments to their use and quality; and contributing to the broader strengthening of health systems.[62] This strategic objective has six organisation-wide results and represented 12-13% of the total expenditure during the biennia 2008-2009 and 2010-2011.

Among MOPAN donors at headquarters who were asked whether WHO is making progress towards this strategic objective, the majority of respondents (90%) rated WHO as adequate or above, with 53% providing a rating of strong and 6% of very strong.

The document review found that although WHO has only partly achieved most expected results, the organisation is making some progress against most baselines in strategic objective 2 (see Volume II, Appendix IX and sidebar for some examples of WHO’s reported contribution).

WHO faced some difficulties in achieving its expected results in the Programme Budget 2008-2009 and 2010-2011 in the areas of prevention, treatment and care for HIV/AIDS, tuberculosis and malaria (2.1) and expanded gender-sensitive delivery of prevention, treatment and care interventions (2.2). Although WHO met overall targets for 2.2, the OWER was rated as partly achieved due to limited data from some regions (Americas, Europe, South East Asia and Western Pacific.[63]

Based on the evidence from the Performance Assessment Reports for 2008-2009 and 2010-2011, WHO made a contribution in the area of “advocacy and nurturing of partnerships” (2.5). WHO has nurtured partnerships at the global and regional level and built member states’ capacity in applying for Global Fund grants, resulting in an increase in financing universal access and surveillance. The organisation’s investments contributed to the achievement of all targets for this expected result in both programme budgets. The Programme Budget 2012-2013 also highlights the technical support given to member states when assessing and managing grants from the Global Fund to Fight AIDS, Tuberculosis and Malaria as a key achievement.[64]

The document review found evidence of recent external reviews of national malaria programmes, but the reviews do not provide data on progress towards WHO expected results. The “Independent Evaluation of the Stop TB Partnership” published in 2008 does not assess the current strategic cycle (2008-2013).

SO 3 Chronic non-communicable conditions Strategic objective 3 seeks to prevent and reduce disease, disability and premature death from chronic non-communicable diseases,mental disorders, violence and injuries and visual impairment. This SO, which contains five OWERs, represented 2-3% of the total expenditure during the Programme Budget 2008-2009 and 2010-2011.

Among MOPAN donors at headquarters who were asked whether WHO is making progress towards this strategic objective, the majority of respondents (84%) rated WHO as adequate or above.

WHO fully achieved OWERs 3.4 and 3.6 in both biennia (see Volume II, Appendix IX). In OWER 3.4, for instance, WHO developed evidence-based guidance on the effectiveness of interventions for the management of 12 priority conditions (depression, schizophrenia and other psychotic disorders, suicide, epilepsy, dementia, etc.) thereby meeting the established targets. Among the results achieved under 3.6, WHO developed and field tested guidelines for community-based rehabilitation and for strengthening primary health-care systems for treating tobacco dependence (training package). See sidebar for examples of WHO’s reported contributions.

WHO faced some difficulties in achieving all of its targets for 3.3 and was assessed as“partly achieved”. In 2012, WHO reported that the number of member states with a national health reporting system and annual reports that include indicators for the four major non-communicable diseases did not meet the expected target of 136, due to further refinement of the criteria and methods used for measuring the indicator.[65] A lack of basic infrastructure was also identified as a major obstacle to data collection.

SO 4 Child, adolescent, maternal, sexual and reproductive health, and ageing

The main focus of strategic objective 4 is the achievement of the Millennium Development Goals, particularly Goal 4 (reduce child mortality) and Goal 5 (improve maternal health). WHO’s efforts are directed towards the reduction of “morbidity and mortality and the improvement of health during key stages of life, including pregnancy, childbirth, the neonatal period, childhood and adolescence, and the improvement of sexual and reproductive health and promote active and healthy ageing for all individuals”.[66] WHO’s work with UNDP, UNFPA, and the World Bank on the Special Programme of Research, Development and Research Training in Human Reproduction is making a contribution to the achievement of this strategic objective.

This SO accounted for 5% of WHO’s total expenditures from 2008-2011.

Among MOPAN donors at headquarters who were asked whether WHO is making progress towards this strategic objective, the majority of respondents (90%) rated WHO as adequate or above.

According to the Performance Assessment Reports for 2008-2009 and 2010-2011, expected results on national research capacity (4.2), neonatal survival and health (4.4), and reproductive health (4.7) were fully achieved– see Volume II, Appendix IX. WHO’s investments contributed to the achievement of all targets. WHO’s expected results on adolescent health (4.6) and ageing (4.8) were not fully achieved in the last two Programme Budget cycles. While all targets were considered as met, these results were reported as partly achieved because results were uneven across regions and not all targets were achieved in all six WHO regions.

SO 5 Emergencies and disasters

WHO’s strategic objective 5 focuses on “reducing the health consequences of emergencies, disasters, crises and conflicts, and minimising their social and economic impact”. [67] It accounted for between 8 and 10% of WHO’s expenditures during the Programme Budgets 2008-2009 and 2010-2011. WHO acts as co-ordinator of the Inter-Agency Standing Committee Global Health Cluster. [68]

Among MOPAN donors at headquarters who were asked whether WHO is making progress towards this strategic objective, the majority of respondents (80%) rated WHO as adequate or above.

While WHO faced some difficulties in achieving its targets for strengthening national emergency preparedness plans (5.1) and transition and recovery phases of conflicts and disasters (5.3), the organisation consistently achieved its targets for communicable disease control in natural disaster and conflict situations (5.4).

Although WHO needs to maintain its own capacity to respond to crises, its main objective in this SO is to support of the strengthening of member state capacities. According to Programme Budget 2012-2013, one of WHO’s key accomplishments in this strategic objective has been in “preparing guidelines and forming networks for food safety, environmental health emergencies and gender mainstreaming as part of the humanitarian response to crises and disasters.”[69]

In 2011 WHO undertook a process to restructure itstwork in emergencies and disasters. Following a review of its work in humanitarian emergencies,[70] WHO decided to develop an organisation-wide emergency response framework, reorient the headquarters emergency department, and restructure this strategic objective.[71]

SO 6 Risk factors for health

Under strategic objective 8, WHO’s efforts are oriented towards “the promotion of health and development and the prevention or reduction of the occurrence of six major risk factors: use of tobacco, alcohol, drugs and other psychoactive substances; unhealthy diet; physical inactivity and unsafe sex”.[72] Operationally, the organisation supports the “development of ethical and evidence-based policies, strategies, standards, guidelines and interventions for health promotion, disease prevention and reduction of the occurrence of the major risk factors.”[73] This strategic objective constituted less than 3% of WHO’s total expenditures.

Among MOPAN donors at headquarters who were asked whether WHO is making progress towards this strategic objective, the majority of respondents (84%) rated WHO as adequate or above.

WHO has made progress and met targets in achieving most of its organisation-wide expected results for this strategic objective over the two programme budgets assessed (see Volume II, Appendix IX). It achieved its targets in four out of six expected results, namely health promotion (6.1), tobacco (6.3), alcohol, drugs and other psychoactive substance (6.4), and unhealthy diet and physical inactivity (6.5) – see sidebar for examples of WHO’s reported contributions. WHO did not fully achieve 6.6, which is related to unsafe sex, due to limited funding which resulted in limited progress in some regions, particularly the African and Eastern Mediterranean Regions.[74]

SO 7 Social and economic determinants of health WHO’s strategic objective 7 aims at “addressing the underlying social and economic determinants of health through policies and programmes that enhance health equity and integrating pro-poor, gender- responsive, and human rights-based approaches”.[75]

Expenditures related to this area represented approximately 1% of the total funds spent by the organisation.

Among MOPAN donors at headquarters who were asked whether WHO is making progress towards social and economic determinants of health, the majority (80%) rated WHO as adequate or above.

Under this strategic objective, WHO met its targets on most indicators (except 7.1) during the 2008-2009 biennium. The majority of expected results were considered as partly achieved during the 2010-2011 biennium. See sidebar for examples of WHO’s reported contributions.

 

WHO reported that 7.3, 7.4 and 7.5 were only partly achieved in 2012, although it met its targets in all three expected results (see Volume II, Appendix IX). Results were assessed as partly achieved because some regions reported partial achievements. For example, WHO reported that 7.3 was partly achieved due to results in the Eastern Mediterranean Region where many countries have not yet institutionalised the collection of disaggregated data, and in head- quarters where the monitoring of health inequities and collection and use of disaggregated data was not sufficiently mainstreamed within WHO programmes or consolidated through the Global Health Observatory.[78]

SO 8 Healthier environment

Under strategic objective 8, WHO is committed to “promote a healthier environment, intensify primary prevention and influence public policies in all sectors so as to address the root causes of environ- mental threats to health”.[79] Strategic objective 8 accounted for approximately 2% of the total expenditures for Programme Budgets 2008-2009 and 2010-2011.

Among MOPAN donors at headquarters who were asked whether WHO is making progress towards this strategic objective, the majority of respondents (78%) rated WHO as adequate or strong.

According to the “Performance Assessment Report 2008-2009”, WHO only partly achieved its five expected results (see Volume II, Appendix IX). Although the organisation met all of its targets, they were assessed as partly achieved due to limited progress in one or more regions.

During the 2010-2011 budget period, WHO fully achieved expected results: norms and standards major environmental hazards to health (8.1), national occupational and environmental health risk management systems, functions and services (8.3), health-sector leadership (8.5) and public health problems resulting from climate change (8.6). See sidebar for examples of WHO’s reported contributions.

SO 9 Nutrition and food safety

Strategic objective 9, which seeks to “improve nutrition, food safety and food security, throughout the life course, and in support of public health and sustainable development”, accounted for between 1 and 2% of all expenditures on Programme Budgets 2008-2009 and 2010-2011.

Among MOPAN donors at headquarters who were asked whether WHO is making progress towards this strategic objective, the majority of respondents (80%) rated WHO as adequate or above.

The available performance data at an organisation- wide level show that WHO consistently achieved its expected results in 9.2, 9.3 and 9.6 during the 2008-2009 and 2010-2011 Programme Budgets.

WHO faced some challenges in some areas (see Volume II, Appendix IX). For instance, expected result 9.1 was not considered fully achieved due to a need for strengthening governance mechanisms at country level, especially in the Africa and America Regions. Since 2010, WHO has shown some progress related to nutrition plans, policies and programmes (9.4). However, progress in implementing nutrition policies and plans in the Americas and South-East Asia Regions was inadequate. Although it achieved all targets for 9.5, it was rated as partly achieved due to mitigated success in some regions (e.g. Eastern Mediterranean, South-East Asia, and the Western Pacific).

SO 10 Health systems and services

Under strategic objective 10, WHO’s work focuses on “improving health services through better govern- ance, financing, staffing and management, informed by reliable and accessible evidence and research”.[82]

This strategic objective tackles the constraints faced by member states in moving towards universal health coverage. It accounted for between 8 and 9% of all expenses during the Programme Budgets for 2008-2009 and 2010-2011.

Among MOPAN donors at headquarters who were asked whether WHO is making progress towards this strategic objective, the majority of respondents (82%) rated WHO as adequate or above.

Among MOPAN donors at headquarters who were asked whether WHO is making progress towards this strategic objective, the majority of respondents (82%) rated WHO as adequate or above.

WHO consistently achieved its expected results for areas 10.1, 10.2, 10.8 and 10.9 during the Programme Budget 2008-2009 and 2010-2011 (see Volume II, Appendix IX).

The organisation struggled, however, in meeting its expected targets in both programme cycles in three areas. Although progress was observed in 10.5, the result was considered partly achieved due to insufficient capacity in the Africa and South-East Asia regions in collecting evidence for health decision making and co-ordination. For 10.6, the Regional Offices for Africa, Europe, South-East Asia and the Western Pacific, as well as headquarters, reported their contributions as partly achieved due to difficulties in obtaining reliable data on research and development resource flows. Similarly, the organisation reported that even though targets were reached for 10.12, the African and South-East Asia Regions reported their contribution as partly achieved due to a shortage of data on key components of universal coverage, especially on groups that are unable to use services for financial reasons.[83] See sidebar for examples of WHO’s reported contributions.

SO 11 Medical products and technologies

Strategic objective 11 seeks to “ensure improved access, quality and use of medical products and technologies.”[84] WHO works closely with a number of partners in this area.[85] Operationally, the organ- isation supports member states in the implementation of policies, which include promotion of sustainable financing, efficient supply management and rational use. This strategic objective represented between 3 and 4% of the total amount spent in Programme Budgets 2008-2009 and 2010-2011.

Among MOPAN donors at headquarters who were asked whether WHO is making progress towards this strategic objective, the majority of respondents (80%) rated WHO as adequate or strong.

Based on available performance data at the organisation-wide level, WHO has improved in the achievement of expected results 11.1 and 11.2 (see Volume II, Appendix IX). One key factor that contributed to the improvement was the high-priority that member states accorded to medicine policies within their agendas.

WHO partly achieved its expected results in the area of evidence-based policy guidance on promoting scientifically sound and cost-effective use (11.3). In 2011, WHO reported significant delays in the development and review of medicines lists and a shortage of funds that led to a reduction in human resources capacity, limiting achievements in normative work and country support.

4.3 EVIDENCE OF WHO’S CONTRIBUTION TO COUNTRY-LEVEL GOALS AND PRIORITIES, INCLUDING RELEVANT MDGS

4.3.1 Overview

This section presents the results of the assessment of evidence of WHO’s contributions to country-level results and relevant MDGs. By separating the KPI at the organisation-wide level from KPIs at the country level, MOPAN recognises the demand-driven nature of many of the activities of a multilateral organisation and the key role that is played by its country programming or strategy document, where expected results at the highest level (outcomes and impact) reflect a shared responsibility between the multilateral organisation and the partner country.

Section 4.3.2 examines evidence of the organisation’s contribution to country-level goals and priorities, including relevant MDGs.

(Note: Section 4.4 examines relevance in terms of the extent to which partners and donors in-country believe the organisation supports country priorities and meets changing needs.)

WHO’s work at country level

WHO’s work at country level is mainly drawn from the norms, standards and guidelines that are developed by the organisation at the global level. It also draws the support and expertise available at the regional level and headquarters to support the country and other development partners in technical matters. The nature of WHO’s work is mainly technical. At the national level, it engages in policy dialogue and provides support in the development of policies, regulations, laws, standards, guidelines and strategies, at the local level, it provides guidelines and standards for health planning, delivery of services and implementation of global and national programmes and advocacies.

WHO’s results and reporting at country level[86]

As noted in the analysis of country focus on results (see KPI 5), WHO’s work at the country level is based on a two-year programming cycle comprising Country Co-operation Strategies (CCS) that outline a strategic agenda based on the MTSP, the national health priorities, and the contributions of the other UN agencies and development partners to the National Health Policy, Strategy or Plan (NHPSP).[87]

This strategy is operationalised through a workplan developed before each programme cycle which includes office and country-specific expected results (OSERs). The country workplan lists all OSERs, as well as their connections to regional and organisation-wide expected results. Assessed and voluntary contributions allocated for the achievement of each OSER are broken down by OSER, product and services, and activity.

The achievement of each OSER is supported by a set of products and services, and activities. Country- specific expected results statements are generally formulated in terms of activity, output and the expected change (outcome) depending on the OSER (see section 4.3.2). Each OSER is directly linked to one organisation-wide expected result (OWER) of the MTSP and its achievement directly contributes to the organisation’s progress on this OWER. This describes the vertical results chain established by WHO and through which results achieved are aggregated at the country level through the regional to the HQ level for its global reporting.

In the country workplans assessed, OSERs are not listed along with performance indicators, baselines and targets. The performance indicators for OSERs are basically the same as the indicator for the associated OWER, but with slight nuances to adapt them to the country level. Activities are used as milestones for achieving a specific output; no indicators, baselines or targets were found at this level.

WHO communicates its progress on results at the country level internally through a number of key documents produced throughout the programme cycle. These include:

• Quarterly Report: This report is produced quarterly by each country office. In addition to summarising the national context and critical events related to or impacting on health, it presents progress made on the implementation of the country workplan.

• Mid-Term Review: This report assesses the implementation of the workplan for the biennium and progress towards the achievement of expected results at the end of the first year.

• End of Biennium Performance Assessment Report: This document provides a systematic assessment of WHO’s performance during the biennium according to each of the organisation’s 13 strategic objectives that are set out in the country workplan for that period. The purpose of the exercise is to evaluate the country office’s contribution to the achievement of the organisation-wide expected results. This is a self-assessment in which individual offices assess their performance in achieving the office and country-specific expected results and their indicators through the delivery of planned products and services. Each office submits an assessment of the regional and headquarters’ contributions to the achievement of organisation-wide expected results.

• WHO Country Office Annual Report: This report, published annually, presents key achievements to date. The information is presented in a more narrative way to inform WHO’s stakeholders on its work.

The implementation of the health-related Millennium Development Goals are monitored in each country (sometimes by province) and reported on at the country and the regional level by WHO.

For an overview of the documents used, please see Volume II, Appendix VIII.

Data used for this assessment

In 2013, MOPAN’s country-level assessment of WHO is based on data from the six countries sampled in the assessment (Ethiopia, Guatemala, Indonesia, Mozambique, Pakistan, and Viet Nam).[88]

In the survey, direct partners, technical partners, and MOPAN donors in-country were asked questions tailored to each of these six countries. Interviews with senior WHO country office staff also informed the analysis of the context and ensured that the assessment team had a full set of documentation with which to conduct the document review and analyse results.

All results-related information provided in documents from the most recently completed programming cycle in the six focus countries was reviewed. More specifically, WHO’s Country Co-operation Strategies, Country Workplans, Annual Reports, Mid-Term Reviews, End of Biennium Performance Assessment Reports, and any external reviews, assessments or evaluations carried out during that programming cycle.

All MDG-related documentation provided was also reviewed to assess WHO’s contribution to the MDGs. This included reports from WHO Regional Directors and from the Secretariat on the implementation of the health-related Millennium Development Goals, as well as from national and UNDP publications.

Attention was paid to the following elements throughout the document review process: quality of the results statements; the relevance of indicators, baselines and targets; the strength of the link between results statements and results achieved; the quality of evidence presented to substantiate the results achieved, including an assessment of contribution; and, the overall performance story.

The following section on country-level KPIs presents the overall results of this review of WHO’s contribution to country-level goals and priorities, with country-specific examples to illustrate the types of results achieved.

4.3.2 Evidence of contribution to country-level goals and priorities, including relevant MDGs This KPI was intended to measure the evidence of contribution to country-level (i.e. national) goals and priorities. However, the design of WHO’s results-based management systems and tools, as well as the poor quality of WHO’s performance and results-related data captured by the organisation forced the assessment to fall back to another unit of analysis: assessing the evidence of contribution to the office and country- specific expected results (OSER) associated with WHO’s 11 strategic objectives.

This section provides an overall rating for KPI B based on documentation made available by the organisation and survey data. It also includes an assessment of WHO’s measurement and reporting on country-level results, as well as an assessment of the extent of progress towards results defined by WHO in its country-level results frameworks.

Overall assessment and rating

WHO’s results statements are meant to be closely aligned to national priorities, the mandate of the organisation and to the priorities of the Medium-Term Strategic Plan. This was the case in the results matrices of the six WHO country offices reviewed. The assessment of this KPI focused on how WHO is demonstrating its contributions to the office and country-specific expected results (OSER) associated with its 11 strategic objectives. Figure 4.5 shows the overall rating for this KPI based on the review of WHO’s reports on country-level results and as indicated by surveyed stakeholders.

Figure 4.5 | KPI B: Evidence of the extent of contribution to country-level goals and priorities, rating

Overall Rating: Inadequate

How WHO reports on country results (for all countries sampled)

Finding 27: The relatively poor quality of WHO’s results-based management systems and tools and the performance and results-related data they generate limit the extent to which its contribution to country-level goals and priorities can be assessed. Despite considerable normative and technical investment in countries, WHO fails to provide strong evidence or a clear picture of the nature, magnitude or relative importance of its contributions to changes at the country level.

Figure 4.6 presents a summary assessment of the quality of the organisation’s systems and practices for measuring and reporting on country-level results. The headings show the criteria MOPAN used to assess the systems and practices.

Figure 4.6 | WHO’s measurement and reporting on country results¹

1. Internal reports from the 2008-2009 and 2010-2011 biennia were used. It is worth noting that there was considerable inconsistency in the format, content, use and quality of the results statements, frameworks and performance information among the six countries and in the two periods.
2. ‘Theory of change’ is understood in the sense defined by Rist and Morra Imas (2009) as, “a representation of how an intervention is expected to lead to desired results” and in the sense defined by Michael Quinn Patton who has stated that a theory of change is more than the sequential order of results statements presented in a logic model; it requires key assumptions related to the results chain and context (e.g. policy and environment), and important influences and risks to be made explicit - Qualitative Research and Evaluation Methods (2002).
3. This refers to the existence of reports on outputs as defined by the OECD (i.e. lower level results). Some MOs use different terminology for the various levels of results.
4. This refers to the existence of reports on outcomes as defined by the OECD (i.e. higher level results). Some MOs use different terminology for the various levels of results
5. Reporting according to a theory of change is understood to mean the extent to which organisations provide a narrative describing the actual implementation process and results achieved in relation to that foreseen in the initial ‘theory of change’.

Strategic planning and theory(ies) of change

The document review found evidence of elements of theories of change in the six Country Co-operation Strategies and workplans reviewed.

In 2010, WHO developed a Country Co-operation Strategy Guide that provides a framework for developing country strategies. Country Co-operation Strategies are made up of six sections (see sidebar) that contain some elements of a theory of change, as described in more detail below.

Situation analysis

Section 2 (health and development challenges) explicitly identifies the nature, extent and distribution of the main health-related problems in the country by providing disaggregated data on who is affected and whether the size of the problem is changing over time. This section usually addresses the reasons why the situation is problematic and worth addressing by referring to known causes or the causal pathway, as well as the consequences of the problem.

Section 4 (review of WHO co-operation over the past CCS cycle) also provides information to complement the situation analysis by presenting the level of implementation of each strategic priority by identifying key achievements and the facilitating factors and/or constraints faced.

Focus and scoping

Overall, the Country Co-operation Strategies assessed provided sufficient information to identify the focus and scope of WHO’s intervention in each country, albeit implicitly. However, this component of WHO’s theory of change should be improved to clearly set out priorities and explain what should and should not be included in the Country Co-operation Strategy (i.e. make explicit what is implicit). Nevertheless, WHO should be commended for: i) identifying the main health-related strategies, plans and policies it intends to use (Sections 2, 4, 5, and 6); ii) identifying the strategic objectives and desired focus specific to each country (Section 5); and iii) identifying those outcomes to which other actors are expected to contribute to, such as results that are beyond the direct focus of WHO (Section 3). Altogether, this information improves one’s understanding of the boundaries of WHO’s interventions in a specific country.

Results chain

Section 5 (Strategic Agenda for WHO co-operation in each country) shows hypothesised cause-and-effect relationships between WHO’s interventions in the country and the strategic priorities it wants to achieve, but does not include outcomes or a results chain. The CCS presents a mixture of activities and strategic priorities that should somehow be converted into the implied intended outcomes under each strategic priority.

Furthermore, there is considerable disconnect between the national goals and priorities included in the NHPSP, the strategic priorities and interventions in the Country Co-operation Strategy, the MTSP OWERs, and WHO’s country workplans. This shortcoming is only partially addressed by the organisation through an internal mapping exercise that seeks to align CCS strategic priorities and results from the MTSP in each country. In addition, the CCS must be read in conjunction with the country workplan which includes results statements, performance indicators, baselines and targets. For these two reasons, the CCS cannot stand alone to present a full results chain. As a result, it is difficult to fully understand how WHO’s interventions in each country will contribute to achieving the OSERs included in the workplan and address the situation described in other sections of the CCS.

It is worth noting that only in Viet Nam did the document review find evidence of theory of change diagrams to explain how an intervention could lead to a desired change. Figure 4.7 provides an example of how WHO’s normative work is intended to lead to desired outcomes and impact.

Figure 4.7 | Demonstrating the theory of change in expanded programme for immunisation in Viet Nam

Reporting according to theories of change

Some country office reports, such as the Annual Reports and the Quarterly Reports, provide some elements related to theories of change (e.g. references to national context, events related to health or impacting on health, workplan implementation status, challenges faced and required actions). While these elements could contribute to reporting according to a theory of change, the performance information provided is insufficient.

Measuring, reporting and communicating results

There is considerable room for improvement related to WHO’s practices in measuring, reporting and communicating results. In the documents provided by the six country offices for the same period (2008-9 or 2010-11), there was considerable inconsistency in format, content, use and quality of results statements, frameworks and performance information. One good example is the inconsistency in the levels of results described in the OSERs. Depending on the country office, the document, and/or the biennium, some expected results statements describe activities, while others describe outputs or outcomes. There were also variances in documentation between biennia. This is due, in part, to the fact that during the 2008-2009 biennium, the General Management System was neither stable nor fully operational and the GSM’s reporting capacity was deemed inadequate by the Joint Inspection Unit.[89]

Hence, country-level performance data from that biennium for the six countries sampled could not be provided by WHO’s Planning, Resource Co-ordination and Performance Monitoring Department to assess the organisation’s performance at country-level.

WHO’s system for measuring results does not capture country-level information. While country offices are requested to update and report on the state of achievement of each organisation-wide expected result indicator, they report and consolidate this information using an Achieved/Partially Achieved/Not Achieved system,[90] which feeds into the performance data at the organisation-wide levels (see Figure 4.8, Box 1). For each OSER in the country workplan where the country office has provided substantial support for the achievement of the related OWER, the OWER indicator will be considered as Achieved by the country office and this information will be consolidated at the regional and the global level. While this system has some advantages, such as facilitating data consolidation and demonstrating progress against OWERs, it does not provide information on the achievement of country-specific expected results or on the contribution of the country office in their achievement.

WHO’s internal reporting at country level, such as the Mid-Term Reviews, does not consistently include indicators, baselines, targets or values to date (see Figure 4.8, Box 2). The performance information is reported and consolidated using traffic lights colours (on track, at risk, in trouble), but there is no system to measure the country office’s progress on a specific result against its actual indicators (see Box 3). In addition, there are inconsistencies in the presentation of information on the progress in delivery, success factors, impediments, risks and required actions (see Box 4). These internal documents do not explain the qualitative/quantitative methodology used by WHO to determine progress in achieving its OSERs.

One of the main weaknesses of WHO’s system for communicating results at country level is its lack of transparency in the publication of detailed data on office and country-specific expected results (OSER) indicators in its performance reports. Country-level performance reports such as Quarterly Reports, Mid- Term Reviews and End of Biennium Performance Assessment reports are not made public. Only the Country Office Annual Report is available to the public.

Overall, WHO has considerable room for improvement in measuring and reporting on its activities, outputs and outcomes. This assessment was confirmed by external sources. In March 2011, the DFID “Multilateral Aid Review: Ensuring Maximum Value for Money for UK Aid through Multilateral Organisations” highlighted WHO’s shortcomings in measuring results at the country level, particularly the need to improve its strategic focus and delivery at country level, as well as results reporting, cost consciousness, financial management, and transparency. The review also identified challenges, such as the absence of systems to review organisational effectiveness, lack of a clear results chain, confusing processes with outputs, lack of a formal system to follow up on evaluations, etc. Similarly, the Joint Inspection Unit reported that the GSM is underutilised as a management tool and mainly serves as an administrative reporting and financial control instrument; it also highlighted that GSM-generated reports generally need manual manipulation in Excel to convert the data into information that is useful for decision making. These perspectives were confirmed by the document review.

Figure 4.8 | Example of country-level reporting

Measuring, reporting and communicating MDGs

Significant commitments to the MDGs have been made by countries in the past years. As a knowledge provider, WHO measures and reports specifically on three of the eight health-related Millennium Development Goals (MDGs) (Goals 4, 5 and 6). It also reports on Goals 1 and 7 which are monitored through health-related indicators.

While WHO and recipient governments provide country data on each indicator under these goals, (in regional, country or provincial reports), these do not demonstrate WHO’s contribution to progress made on the MDGs, but rather the situation in-country to which all development stakeholders are contributing.

Data Reliability

As mentioned above, WHO’s system for monitoring and measuring results at the country level is geared towards demonstrating progress against organisation-wide expected results, rather than providing information on the WHO country office’s contribution to the achievement of country-specific expected results. The evidence provided in WHO reports and documents is drawn from self-assessments. In addition, as noted in KPI 20, WHO does not have a strong practice of conducting external evaluations and reviews; therefore, the assessment could not rely on third party or independent sources to verify results achieved. The configuration, as well as the quality of WHO’s practices and reporting tools, hinder the communication of a good performance story and, for that reason, the extent of progress toward WHO results at the country level could not be adequately assessed.

Therefore, in the case of WHO, the Common Approach examined the extent to which country offices have contributed to WHO’s strategic objectives as expressed in its organisation-wide expected results (OWERs). The evidence of contribution presented in the assessment below are only examples and are not representative of the totality of WHO’s work in a specific country.

Evidence of the extent of progress toward WHO results at the country level

Finding 28: Survey respondents held positive views of WHO’s contribution to its expected results at country-level.

Finding 29: WHO provided mixed ratings for its achievement of results (including MDGs), but reported some progress on most expected results.

Finding 30: A lack of evidence related to WHO’s contributions makes it difficult to understand the linkage between WHO’s work in a given country and the progress made on national goals and priorities. The document review lacked detailed performance information from which to discern WHO’s performance story in the six focus countries.

In the following sections on all six countries sampled, the assessment examined the extent to which each WHO country office has contributed to the achievement of organisation-wide expected results. See Volume II, Appendix X for details on the extent of progress towards WHO’s country-level goals and priorities and examples of WHO’s contribution to MDGs in each country.

ETHIOPIA

Country-level respondent groups were asked whether WHO Ethiopia has effectively contributed to each strategic objective listed in Figure 4.4.[91] The majority of survey respondents rated the organisation as adequate or above on each SO. In some areas such as SO3, SO6, SO7, SO8, SO9, SO10 and SO 11, approximately one-third of survey respondents provided rating of inadequate and weak. The highest ratings were for HIV/AIDS, tuberculosis and malaria (SO2) and communicable diseases (SO1).

A summary of the extent of progress towards WHO’s strategic objectives as reported by WHO Ethiopia is presented in Volume II, Appendix X.

During the 2008-2013 programming period, the largest proportion of resources were allocated to SO1 (Communicable Diseases) and SO2 (HIV/AIDS, tuberculosis and malaria).

The assessment considered evidence from WHO Ethiopia’s 2008-2013 programming cycles, particularly the 2010-2011 biennium, for reasons mentioned above. The country workplan includes 67 country-specific expected results (mix of activities, outputs and outcomes) and includes programming activities in all 11 strategic objectives.

WHO reported that 63% of the OWERS were fully achieved (94 out of 150). The document review found that progress was made on most targets.

For the assessment of results in Ethiopia, the MOPAN team reviewed Quarterly Reports, Mid-Term Reviews, and the End of Biennium Performance Assessment for 2010-2011.

l The Quarterly Reports and MTR report on progress toward OSER indicator targets. However, some of the MTRs reviewed lacked indicators, baselines, targets and/or actual values, and the Quarterly Reports contained very little performance information.

l The End of Biennium Performance Assessment for 2010-2011 reports on OWERs and provides ratings of the extent to which these were achieved (fully achieved, partially achieved, etc.) However, it does not provide information on the country or office-specific expected results (OSERs) in Ethiopia or the actual results achieved on specific indicator targets.

The implementation of WHO’s interventions in Ethiopia has been challenging. Continued disease outbreaks (such as malaria, cholera and measles among others) and the food security situation led to a highly diversified country intervention.

During the 2010-2011 programming cycle, the WHO office supported existing partnerships and co- ordination schemes including the Joint Consultative Forum (JCF), the Joint Core Co-ordinating Committee (JCCC) and Health, Population and Nutrition (HPN) Partners Group, the Inter-Agency Planning Team (IAPT), the United Nations Development Assistance Framework (UNDAF), the Thematic Working Groups (TWGs) and the Country Co-ordinating Mechanism – Ethiopia (CCM-E).

In addition to strengthening policy dialogue, co- ordination and partnerships around health, WHO Ethiopia focused on strengthening the demand for health services, enhancing efforts to achieve the health-related Millennium Development Goals (MDGs), and improving the quality of health care in the country.

According to WHO, the Ethiopia office’s main achievements included: improved coverage of priority maternal and child health and priority disease interventions; increased and sustained vaccination coverage; increased number of mothers attending antenatal care; and increased demand for skilled assistance during pregnancy and delivery. Relatively good progress was reported in: the strengthening of national capacity and co-ordination mechanisms in the areas of HIV/AIDS, malaria, TB; emergency response to communicable diseases; and, non- communicable diseases and tropical diseases (see sidebar for an example of WHO’s work in one area).

While one independent evaluation of WHO’s contributions in the area of external disease surveil- lance was found, this evaluation was published in 2008 and therefore the progress towards expected results it measures is not based on the 2008-2013 programming. For this reason, this document was not reviewed.

Contributions to relevant MDGs

Data from “Monitoring the Implementation of the Health Millennium Development Goals Report for the African Region” (WHO, 2011) shows that Ethiopia is making limited or no progress towards the 2015 health- related MDG targets.

WHO Ethiopia’s country workplan makes an explicit link only to MDG 4 (OSER [04.05.34] [DDE] ETH). [93]

Whereas the documents reviewed indicated that the country met the OWER targets on both indicators (country is implementing strategies for increasing coverage with child health and development interventions, and country has expanded coverage of the integrated management of childhood illness to more than 75% of target districts), they do not explain the linkage between WHO’s work and the progress towards the MDG goal or target. The reports do not articulate how these actions come together to bolster the partner country’s efforts, and, for most MDG areas, there is no direct association to WHO’s role and contribution.

A summary of the self-reported data on country progress towards the achievement of MDGs is provided in Volume II, Appendix X.

GUATEMALA

MOPAN donors in-country, direct partners and technical partners were asked whether WHO Guatemala has effectively contributed to 8 of the 11 strategic objectives. The majority of survey respondents rated the organisation as adequate or above on 7 of the 8 SO, but 49% provided ratings of inadequate or below for WHO’s contribution to improving health services through better governance, financing, staffing and management, informed by reliable and accessible evidence and research in Guatemala (SO10). MOPAN recognises the difficulties in evaluating progress in health services in Guatemala, in particular during the period under review during which several changes took place in the leadership of the Ministry of Health. These changes created an unstable environment that complicated the achievement of set objectives.

A summary of the extent of progress towards WHO’s strategic objectives as reported by WHO Guatemala is provided in Volume II, Appendix X. During the 2008-2013 programming period, the largest proportions of resources were allocated to SO10 (Health systems and services) and to SO5 (Emergencies and disasters). The assessment considered evidence from WHO Guatemala’s 2008-2013 programming cycle, particularly the 2010-2011 biennium. The country workplan includes 144 country-specific expected results statement (most of them stated as outputs) and covers programming in all 11 strategic objectives.

The MOPAN team reviewed the complete self-reported “2010-2011 End of Biennium Performance Assessment” as well as Guatemala/PAHO’s own end of biennium report, “Detalles de Indicador de OSER 2010-2011”, from its internal performance management system.[94] The latter includes the same information as End of Biennium Performance Assessment Reports from other regions and, similarly, does not include indicators, baselines, targets and/or actual values related to the progress made in achieving OSER indicator targets other than specifying whether the country has met or not met the regional indicator. No Mid-Term Reviews, Quarterly Reports or Annual Reports were shared by the organisation and for this reason, the MOPAN team could not comment on or assess the work of WHO Guatemala and how it links to the progress made on office and country-specific expected results.

The MOPAN team also reviewed reports from evaluations conducted in-country during the 2008-2013 period.[95] However, none of these reports focused on WHO’s contribution to country-specific expected results. One evaluation focused on very specific areas of work, such as the use and impact of chlorine- producing equipment provided by WHO to the Minsiterio de Salud Publica y Asistencia Social (MSPAS). Another focused on a project jointly conducted with the MSPAS and the Swedish International Development Agency. While these evaluation reports may highlight efficient and effective contributions to country goals and priorities by WHO, no reference to Guatemala country office’s OSERs are found, making it impossible to assess progress on its expected results.

Contributions to relevant MDGs

Data from the 2010 report from the Guatemalan Secretariat of Planning and Programming shows that the country is making good progress towards the 2015 MDG targets overall.

WHO’s 2010-2011 country workplan makes one explicit reference to MDG 7, as one of the indicators for OSER GUT.08.01 (Se habrá contribuido al fortalecimiento de las capacidades nacionales de vigilancia y control de los riesgos ambientales para la salud) refers to the number of countries that apply WHO’s guidance with regard to drinking water. This indicator was deemed to be met by the WHO Guatemala office. Another reference is found in the management framework: OSER GUT.15.02 OPS/OMS (mantiene liderazgo en salud, con el Gobierno, dentro del SNU y con los socios en el marco de los ODMs, del CCS, CCA/UNDAF y programas conjuntos). However, no evidence was found of the linkage between WHO’s work and the progress on 2015 MDG targets for Goal 7.

WHO provided technical co-operation to support Guatemala in achieving other MDG targets, such as the development and implementation of standards and technical guidelines for the care of mothers and newborns in support of MDG 5. (See Volume II, Appendix X for a summary of the country-reported data on country progress towards the achievement of MDGs).

INDONESIA

MOPAN donors in-country, direct partners and technical partners were asked whether WHO Indonesia has effectively contributed to each strategic objective. The majority of survey respondents rated the organisation as adequate or above on each SO.

During the 2008-2013 programming period, the largest proportions of resources were allocated to SO1 (Communicable diseases) and SO2 (HIV/AIDS, tuberculosis and malaria).

Overall, WHO reported that 44% of the OSERs were achieved in Indonesia in 2008-2009, and 69% in 2010-2011 (see Volume II, Appendix X).

Challenges with the implementation of WHO’s activities in Indonesia were observed in areas such as the decentralisation, human resource distribution, access to health care for remote areas, emergency response and disaster risk reduction and emerging diseases. WHO’s self-reporting does not link its work to specific expected results, but WHO has reported having achieved a mix of activities and outputs, and contributed to some outcomes (see sidebar).

The assessment considered evidence from different sources of data from WHO Indonesia’s programming cycle.

The assessment team reviewed the “2010 Mid-Term Review”, the “2010-2011 End of Biennium Performance Assessment”, and the “World Health Organization Country Office for Indonesia Progress Report 2010”. As in other countries sampled, most of the documents provided contained very little performance information; the use and quality of results statements, indicators, baselines and targets varied considerably. They relate to the 11th GPW and 2012-2013 Programme Budget, which will soon be replaced by more robust results frameworks. The 2008-2009 End of Biennium Programme Budget Performance Assessment Reports were made available, one for each strategic objective. These reports, used to compile performance information before the implementation of the General Management System, provide WHO’s progress on all activities under each OSER.

In September 2011, an external independent review of the National Malaria Control Programme (NMCP) was published. Its objective was to conduct comprehensive in-depth review of the NMCP and recommend measures to further strengthen the programme with a view to achieving national and global targets on Millennium Development Goals (MDG) 6 related to malaria, including the review of community participation in malaria control and the partnership with various stakeholders, such as WHO colla- borative programmes, among others. While the review reported that WHO supported the finalisation of elimination guidelines and conducted operational research through local universities, it did not provide evidence of WHO contributing to the achievement of results in-country.[96] WHO did however provide technical support to the NMCP through the conduct of operational research to identify gaps in imple- mentation of NMCP’s activities. The “External Review of the National Dengue Control Program” was also assessed by the document review, but no evidence of WHO’s contributions could be retrieved from this report. The review was conducted with WHO’s technical support.

Contributions to relevant MDGs

Data from the “Republic of Indonesia Report on the Achievement of the Millennium Development Goals” (2010) shows that Indonesia has already met MDG 6.B, but none of the other targets. The 2011 “National Malaria Control Programme Review” reports that incidence has been reversed, and that the reduction of malaria has also contributed to the reduction of maternal mortality (MDG 5.A) in most provinces.[97] While such improvements have been observed, WHO’s country workplans do not make explicit reference to MDG goals.[98] Hence, the document review could not find evidence of a linkage between WHO’s work and the progress on the MDG goals or targets. A summary of Indonesia’s reported data on country progress towards the achievement of MDGs is provided in Appendix X of Volume II.

MOZAMBIQUE

MOPAN donors in-country, direct partners and technical partners were asked whether WHO Mozambique has effectively contributed to each of WHO’s strategic objectives. The majority of survey respondents rated the organisation as adequate or above on each SO. In some areas such as SO3, SO6 and SO7, approximately one-third of survey respondents provided ratings of inadequate or below. In the case of SO8, 50% of survey respondents provided ratings of adequate or above, 22% of inadequate or below, and 28% answered ‘don’t know’.

During the 2008-2013 programming period, the largest proportions of resources were allocated to SO1 (Communicable diseases) and to SO2 (HIV/AIDS, tuberculosis and malaria). WHO reported that 70% of the OWERs were achieved in 2010-2011 and 30% in 2008-2009 (see Volume II, Appendix X for a summary of the extent of progress towards WHO’s strategic objectives as reported by WHO Mozambique).

The assessment considered different sources of data from WHO Mozambique’s programming cycle. In addition to WHO’s internal planning and reporting documents, WHO’s input to the One UN Plan was also reviewed through a suite of documents.

In Mozambique, UN agencies are working together to enhance the quality and effectiveness of aid through the Delivering as One initiative. This includes the development of the One UN Plan that covers all UN agencies, serves as the UNDAF forMozambique and replaces single agency plans, such as the previous WHO Country Co-operation Strategy. The Plan is based on key national priorities and identifies the key interventions and priorities within the One UN Framework.

The assessment team reviewed the 2008-2009 Biennial Report, as well as the Mid-Term Reviews and the End of Biennium Reports for the 2008-2009 and 2010-2011 periods that were available. Most documents provided very little performance information and the use and quality of results statements, indicators, baselines and targets varied greatly from one to the other. The 2008-2009 End of Biennium Programme Budget Performance Assessment Reports were available and provided evidence of WHO’s contribution to OSERs (see sidebar).

The 2008-2009 Biennial Report also provided some evidence of contribution. For example, WHO reported that it supported Mozambique in achieving the target of eliminating leprosy by: updating and maintaining records of leprosy, delivering leprosy drugs to distribution points run by volunteers from the villages, sponsoring ‘Leprosy Days’, initiating a pilot project consisting of an electronic computer-based registration system, supporting the Campaign for the Elimination of Leprosy, and introducing a behaviour change project in Nampula and Manica (Community for improving Behaviour).[100] Another success story is the direct support to provincial governments of Maputo and Gaza in creating model of a contextualised and comprehensive response to HIV/AIDS.[101] In both provinces, situational analyses were conducted using an adapted guideline, which became the basic structure of the Reference Framework Strategy Acceleration Prevention. As a result of WHO’s support, 20 districts and 2 provinces developed and endorsed their action plans for HIV and these plans were implemented with the involvement of all stakeholders at district and provincial l levels.

Contributions to relevant MDGs

Data from the 2011 report by the Secretariat on “Monitoring the Implementation of the Health Millennium Development Goals” shows that Mozambique has made limited progress on most MDGs other than MDG 1 and 5. Evidence was found in the 2008-2009 End of Biennium Report of WHO’s contribution to the progress towards the achievement of MDG 4 and 5 targets.[102] The development and approval by the minister of health of the national integrated and costed Maternal, Newborn and Child Health plan towards MDG 4 & 5 was reported following WHO’s technical support to two provinces.

A summary of WHO’s self-reported data on country progress towards the achievement of MDGs is provided in Volume II, Appendix X.

PAKISTAN[103]

MOPAN donors in-country, direct partners and technical partners were asked whether WHO Pakistan has effectively contributed to WHO’s strategic objectives. The majority of survey respondents rated the organisation as adequate or above on each SO, with approximately one-third providing ratings of inadequate or below.

As shown in Volume II, Appendix X, WHO reported that 56% of the OWERs in the country workplan were fully achieved (78 out of 140 OWERs). During the 2008-2013 programming period, the largest proportions of resources were allocated to SO1 (Communicable diseases and to SO5 (Emergencies and disasters).

The assessment considered evidence from WHO Pakistan’s 2008-2013 programming cycle, particularly the 2008-2009 biennium, since reports from this biennium provided more complete and robust evidence of WHO’s work in-country and how this work contributed to progress towards country goals and priorities. The 2008-2009 Biennial Report, in particular, presents some evidence of linkages between WHO’s work in country and results achieved.

Contributions to relevant MDGs

Data from the “2010 Pakistan Millennium Development Goals Report” shows that Pakistan has made limited progress on most MDGs. Other reports (such as the 2011 UNDP Khyber Pakhtunkhwa, the 2011 UNDP Balochistan, and the 2011 Punjab Millennium Development Goals) were reviewed; however, it is not possible to link any progress on the MDGs to WHO based on these reports. A summary of country data on progress towards the achievement of MDGs is provided in Volume II, Appendix X.

VIET NAM

MOPAN donors in-country, direct partners and technical partners were asked whether WHO Viet Nam has effectively contributed to WHO’s strategic objectives. The majority of survey respondents rated the organisation as adequate or above on each SO. A high level of ‘don’t know’ answers were provided, ranging from 7% for SO1 to 42% for SO5.

As shown in Volume II, Appendix X, WHO reported that 56% of the OWERs in the country workplan were fully achieved (i.e. 78 out of 140 OWERs). During the 2008-2013 programming period, the largest proportions of resources were allocated to SO1 (Communicable diseases) and SO2 (HIV/AIDS, tuberculosis and malaria).

In addition to WHO’s country office planning and reporting documents, WHO’s input to the One Plan was also reviewed through a suite of documents (see sidebar for an example of WHO’s input).

A performance story can be drawn from WHO Viet Nam’s documentation about how WHO’s health diplomacy and provision of evidence-based policy options advanced the cause of public health. As reported by the organisation, WHO Viet Nam supported the Ministry of Health and the government to win a cause for public health despite pressing trade and economic objectives (a government owned and controlled tobacco corporation holds 40% share of the total market for tobacco in Viet Nam). WHO supported the government to pass the Tobacco Control Law in June 2012 by providing robust evidence of the ill effects of tobacco.

More specifically, the organisation:

• developed briefing documents and policy briefs which were presented to members of the National Assembly and to key stakeholders

• provided technical guidance on the provisions of the Framework Convention Alliance for Tobacco Control to ensure that the local law was in consonance with it

• supported the government in multi-stakeholder consultations, identified and supported champ- ions, worked with the media, co-ordinated the network of partners to support tobacco control and mobilised funding to support the develop- ment of the Tobacco Control Law.

Examples of how WHO’s normative work contri- butes to improving health in Viet Nam are also provided in Volume II, Appendix X.[104]

While there is an effort to link WHO’s work with outcomes and impact at the country level, it remains difficult to attribute any of these results to the work of WHO alone.

Contributions to relevant MDGs

Data from the 2012 report “Achieving the Health- related Millennium Development Goals in the Western Pacific Region” presents progress on most Millennium Development Goals. Although WHO’s country-level documents make explicit reference to some MDGs (OSER 08.001.WP01.VNM01 Evidence-based assessments, norms and guidance on air quality, sanitation, and drinking water developed and updated and technical support to international environmental agreements and for monitoring MDG[105]), it is difficult to find evidence of a linkage between WHO’s work and the progress on MDGs goals and targets in WHO’s external reporting. However, in an internal briefing paper presented to MOPAN, the WHO Viet Nam country office is the only organisation in Viet Nam that has a dedicated programme for essential medicines contributing to MDG 8. A summary of WHO data on country progress towards MDGs is provided in Volume II, Appendix X.

4.4 RELEVANCE OF WHO’S OBJECTIVES AND PROGRAMME OF WORK TO STAKEHOLDERS

For this KPI, MOPAN assessed relevance primarily as a measure of the extent to which a multilateral organisation is seen to support country priorities and meet the changing needs of direct partners and the target population. The assessment is based exclusively on survey data gathered from direct partners, technical partners and MOPAN donors in-country in the six countries selected for the 2013 MOPAN assessment of WHO.

Overall assessment

Across the six countries, WHO was seen to be consistently strong in responding to its partner countries’ key development priorities, in providing innovative solutions to help address these challenges, and in adapting its work to the changing needs of its partner countries.

Figure 4.9 shows the overall assessment rating and the means scores on the three survey questions on which the assessment is based.

Figure 4.9 | KPI C : Relevance of objectives and programme of work to stakeholders, overall rating and survey mean scores by country

Overall Rating: Adequate

Very weak = 1.00–1.49
Weak = 1.5–2.49
Inadequate = 2.5–3.49
Adequate = 3.5–4.49
Strong = 4.5–5.49
Very strong = 5.5–6.00

1. WHO. (2009). 2008-2009 Mid-term Review

Conclusions on organisational effectiveness

The MOPAN assessment provides a snapshot of WHO’s organisational effectiveness based on the practices and systems in place at the time of the assessment.

WHO’s commitment to organisational development and its related reform agenda are likely to improve its effectiveness and efficiency, although it is too early to assess the effects of the process.

This MOPAN assessment took place during the early stages of WHO’s implementation of an ambitious Reform Agenda. As part of this reform, the organisation is aiming to develop a set of agreed global health priorities that will guide the organisation, achieving greater coherence in global health and resolving the relative lack of clarity on the roles and functions at the country, regional and global levels. The assessment found that positive changes in systems and practices have already resulted from this process; some are well underway and others have yet to be initiated. The reform agenda is being monitored and the Board receives updates on its progress.

WHO’s mandate and comparative advantages provide a good foundation for its focus on results.

WHO is committed to revising its mandate to ensure continuing relevance. Together, the 11th General Programme of Work and the Mid-Term Strategic Plan 2008-2013 articulate the organisation’s goals and priorities and provide a clear indication of the manner in which WHO will implement the mandate during this period. WHO has also made significant improvements in defining and addressing the organisation’s priorities in developing the 12th General Programme of Work.

There is room to further strengthen WHO’s results-based management practices and tools used to manage for and report on organisation-wide results.

The MOPAN assessment found that Mid-Term Strategic Plan 2008-2013 results statements inconsistently labelled activities, outputs and/or outcomes. In addition, the results-based framework is missing levels of results between the organisation’s activities and outputs and the intermediate outcomes it aims to achieve (results chain). This discrepancy trickles down to most related performance indicators. The lack of a clear chain of plausible results from one level to the next limits the organisation’s ability to monitor and report on performance.

WHO has committed to strengthening results-based management across the organisation and is working to improve planning, monitoring, and reporting at all levels. The Programme Budget 2014-2015 includes an improved results framework.

WHO was rated as inadequate with regard to results-based budgeting, but it is introducing a new results- based budgeting system (RBB) based on a revised results chain with a methodology for costing of outputs and an approach to assess contribution. These reforms, if implemented as planned in 2013-2014, represent important steps towards becoming a more performance-oriented and accountable organisation.

WHO is commended for its technical assistance, staff expertise, normative and standard-setting work, and its convening and regulatory functions.

WHO’s technical assistance and country-level operations, staff expertise, and normative and standard setting role were seen as key organisational strengths in the 2013 MOPAN assessment. This was reflected in comments to open-ended questions, in which survey respondents highlighted WHO’s support in the development of national health strategies and plans.

Both survey respondents and the document review also commended WHO for its convening and regulatory functions, as well as its knowledge management function in the health sector. Its convening role in the negotiation of health regulations and treaties is identified as a key facet of this normative and standard-setting work. WHO performs various critical functions in the health sector, such as translating global science and evidence into products for policy-making purposes in countries, co-ordinating surveillance and response to international health threats, and gathering and disseminating the best information available on appropriate health practices.

WHO has sound policies and processes for financial accountability but does not yet have strong practices for risk management.

WHO has strong systems in place for internal and financial audits (including organisational audits), strong policies for anti-corruption, systems for immediate measures against irregularities, and effective procurement and contract management processes. The organisation is working on an organisation-wide common framework and harmonisation of risk management practices.

WHO has strengthened its evaluation function but there is still room for improvement in the coverage and quality of evaluations.

WHO has invested considerable resources in this area and is in the process of strengthening its evaluation function. While it is making progress in systems and practices, the MOPAN assessment found that there is room for improvement in the coverage and quality of evaluations. When fully implemented, the 2012 Evaluation Policy and related procedures for quality control could help to address some of the weaknesses noted by the assessment.

In contexts where it has significant humanitarian programming, WHO is fulfilling its responsibilities as a Cluster Lead and is recognised for respecting humanitarian principles.

WHO has improved its institutional capacity with regard to the application of its humanitarian mandate. WHO’s Emergency Response Framework (ERF) and the Inter-Agency Standing Committee’s Global Health Cluster Guide articulate its humanitarian mandate. Survey respondents felt that WHO adequately respects humanitarian principles and maintains on-going policy dialogue with partners on the importance of observing humanitarian principles in delivering emergency assistance. They also perceived the organ- isation performing adequately in managing the Global Health Cluster.

Conclusions on evidence of WHO’s development results and relevance to stakeholders

Limitations in WHO’s frameworks and systems to report on organisation-wide expected results make it challenging to fully understand WHO’s performance story and identify its contribution to each of its strategic objectives.

The assessment noted the work being done by WHO, under its 11 strategic objectives, to fulfil its mandate: “to achieve the attainment by all peoples of the highest possible level of health”.[106]

Surveyed stakeholders consider that WHO is making progress towards its organisation-wide strategic objectives and the document review found evidence of progress towards organisation-wide expected results in some strategic objectives. However, the data presented was largely self-reported and did not include data collected systematically and verified by a robust evaluation function.

In the absence of a clear results chain or theories of change, WHO’s organisation-wide reporting provides limited links between activities, outputs and outcomes and does not allow for an assessment of WHO’s contributions at the outcome level.

Country-level stakeholders confirm the relevance of WHO’s work and indicate that it makes contributions to its office and country-specific expected results and to partner country efforts to achieve the MDGs. However, despite considerable normative and technical investments and support to countries, WHO fails to provide strong evidence or a clear picture of the nature, magnitude or relative importance of its contributions to changes at the country level.

Although stakeholders see WHO’s work as relevant to country priorities, WHO reported limited progress towards achieving its office and country-specific expected results in the six countries sampled for the assessment.

While WHO does good work at the country level, the extent to which its contribution to country-level goals and priorities can be assessed is limited by the design of its results-based management systems and tools and by the poor quality of its performance and results-related data. The document review found limited performance information by which to understand WHO’s performance story in the six countries sampled for the assessment. It is difficult to understand how WHO’s interventions in each country contribute to achieving national goals and priorities as there is no clearly articulated chain of results. In fact, there is considerable disconnect between the national goals and priorities included in the NHPSP, the strategic priorities and interventions in the Country Co-operation Strategy, the MTSP OWERs, and WHO’s country workplans.

WHO provides consistent data on performance indicators across programme budgets, but data reliability is compromised by the absence of independent and external sources, such as evaluations.

WHO’s performance measurement system relies almost exclusively on self-reported data from Country Offices. The MOPAN assessment found very few independent evaluations that could validate the reported results achieved; the evaluations that have been conducted were in very specific, technical areas that were not relevant to this assessment. WHO’s reporting on its progress towards organisation-wide expected results would benefit from performance information provided through independent evaluations of sectors, strategic objectives, specific themes and/or regions.

List of Figures

Figure 1.1 Number of survey respondents for WHO by country and respondent group
Figure 1.2 WHO - Distribution of responses (n=394) on all questions related to micro-indicators
Figure 3.1 Overall ratings on key performance indicators of WHO’s organisational effectiveness (survey mean scores and document review ratings)
Figure 3.2 Overall ratings of WHO organisational effectiveness by respondent group (n=394)
Figure 3.3 Performance area I: Strategic management, survey and document review ratings
Figure 3.4 Performance area I: Strategic management, mean scores by respondent group
Figure 3.5 KPI 1: Providing direction for results, ratings of micro-indicators
Figure 3.6 KPI 2: Corporate strategy and mandate, ratings of micro-indicators
Figure 3.7 KPI 3: Corporate focus on results, ratings of micro-indicators
Figure 3.8 KPI 4: Focus on cross-cutting priorities, ratings of micro-indicators
Figure 3.9 KPI 5: Country focus on results, ratings of micro-indicators
Figure 3.10 Performance area II: Operational management, survey and document review ratings
Figure 3.11 Performance area II: Operational management, mean scores by respondent group
Figure 3.12 KPI 6: Resource allocation decisions, ratings of micro-indicators
Figure 3.13 KPI 7: Results-based budgeting, ratings of micro-indicators
Figure 3.14 KPI 8: Financial accountability, ratings of micro-indicators
Figure 3.15 KPI 9: Using performance information, ratings of micro-indicators
Figure 3.16 KPI 10: Managing human resources, ratings of micro-indicators
Figure 3.17 KPI 11: Performance-oriented programming, ratings of micro-indicators
Figure 3.18 KPI 12: Delegating authority, ratings of micro-indicators
Figure 3.19 KPI 13: Humanitarian principles, ratings of micro-indicators
Figure 3.20 Performance area III: Relationship management, survey and document review ratings
Figure 3.21 Performance area III: Relationship management, mean scores by respondent group
Figure 3.22 KPI 14: Supporting national plans, ratings of micro-indicators
Figure 3.23 KPI 15: Adjusting procedures, ratings of micro-indicators
Figure 3.24 KPI 16: Using country systems, ratings of micro-indicators
Figure 3.25 KPI 17: Contributing to policy dialogue, ratings of micro-indicators
Figure 3.26 KPI 18: Harmonising procedures, ratings of micro-indicators
Figure 3.27 KPI 19: Managing the cluster, ratings of micro-indicators
Figure 3.28 Performance area IV: Knowledge management, survey and document review ratings
Figure 3.29 Performance area IV: Knowledge management, mean scores by respondent group
Figure 3.30 KPI 20: Evaluating results, ratings of micro-indicators
Figure 3.31 KPI 21: Presenting performance information, ratings of micro-indicators
Figure 3.32 KPI 22: Disseminating lessons learned, ratings of micro-indicators
Figure 3.33 KPI 23: Availability of documents, ratings of micro-indicators
Figure 4.1 Development results component – overall ratings
Figure 4.2 KPI A: Evidence of the extent of progress toward organisation-wide outcomes, overall rating
Figure 4.3 WHO’s measurement and reporting on organisation-wide results
Figure 4.4 Extent of progress towards WHO’s OWERs under each strategic objective (MTSP 2008-2013 cycle)
Figure 4.5 KPI B: Evidence of the extent of contribution to country-level goals and priorities, rating
Figure 4.6 WHO’s measurement and reporting on country results
Figure 4.7 Demonstrating the theory of change in expanded programme for immunisation in Viet Nam
Figure 4.8 Example of country-level reporting
Figure 4.9 KPI C : Relevance of objectives and programme of work to stakeholders, overall rating and survey mean scores by country

 

---

[1] WHO. (2009). Basic Documents: Forty-seventh Edition. (p. 2)

[2] WHO. (2013). Financial Report and Audited Financial Statements for the year ended 31 December 2012. (pp. 69-70).

[3] See Volume II, Appendix I, section 3.4 for an explanation of the weighting formula and scheme.

[4] For instance, MOPAN recognises that the sample of six countries selected for this assignment is not representative of WHO’s offices in 150 countries, territories and areas.

[5] While most KPIs and micro-indicators were considered in the document review, not all were rated. See section 2.3.

[6] Respondents who wrote “no comment” or the like were removed from the analysis. Answers that commented on more than one element were coded in various categories.

[7] With this new structure, WHO has eliminated the need for a Medium-term Strategic Plan

[8] The normal convention for statistical significance was adopted (p<.05). See section 2.2, which describes the data analysis process.

[9] Eleventh General Programme of Work 2006–2015, Medium-term Strategic Plan 2008-2013, Programme Management in WHO: Operational Planning and Programme Budgets.

[10] WHO. (2007). Programme Management in WHO Operational Planning: Business Rules, Procedures - Including Practical Guidance.

[11] Amendments to the mandate adopted by the Twenty-sixth, Twenty-ninth, Thirty-ninth and Fifty-first World Health Assemblies (resolutions WHA26.37, WHA29.38, WHA39.6 and WHA51.23) came into force on 3 February 1977, 20 January 1984, 11 July 1994 and 15 September 2005 respectively and are incorporated in the present Basic Documents Forty-seventh Edition 2009

[12] WHO. (2007). Programme Management in WHO Operational Planning: Business Rules, Procedures - Including Practical Guidance. (pp. 65; 68).

[13] This evidence was not used in the assessment as most of the various outputs of the reform agenda are still at an early stage of implementation.

[14] Output indicators measure the quantity of products and services produced and the efficiency of production (e.g. number of member states that received support, number of tools, frameworks, guidelines produced, etc.)

[15] MOPAN has used CREAM and SMART criteria to assess the quality of indicators and results statements. CREAM criteria (clear, relevant, economic, adequate, monitorable) are usually used to evaluate indicators. SMART criteria (specific, measurable, achievable, relevant, time bound) are widely used when developing indicators.

[16] Retrieved 12 May 2013 from http://www.who.int/hhr/en/

[17] WHO. (2013). Review of Management, Administration and Decentralization in the World Health Organization, Report by the Joint Inspection Unit. (p. 78).

[18] One Plan in the case of Viet Nam.

[19] WHO. (2012). WHO Reform - Independent Evaluation Report: Stage One. (p. 15).

[20] WHO. (2013). Review of Management, Administration and Decentralization in the World Health Organization, Report by the Joint Inspection Unit. (p. 65).

[21] WHO. (2009). Basic Documents: Forty-seventh Edition. (pp. 93-94).

[22] Compliance, Risk Management, and Ethics Office Overview. (p. 1). (Internal document).

[23] WHO eManual. (p. 6). (Internal document).

[24] WHO. (2013). Review of Management, Administration and Decentralization in the World Health Organization, Report by the Joint Inspection Unit.

[25] WHO. (2013). Review of Management, Administration and Decentralization in the World Health Organization, Report by the Joint Inspection Unit. (p. 31).

[26] GSM Guidance. (Internal document).

[27] WHO. (2013). Review of Management, Administration and Decentralization in the World Health Organization, Report by the Joint Inspection Unit. (p. 35).

[28] WHO plays a leadership role within the Health Cluster.

[29] Launched in 2007, IHP+ is a group of partners, including WHO, committed to improving the health of citizens. http://www.internationalhealthpartnership.net/en/ IHP+Results Consortium is independently monitoring IHP+ signatories' commitments for mutual accountability in the international health sector and publishes scorecards on the progress being made by each agency or country in implementing the IHP+ commitments. http://ihpresults.net/

[30] IHP+Results, 2012 Partner Scorecard for WHO.

[31] Analysis of the 2011 Paris Declaration Monitoring Surveys: Key Highlights. (Internal document).

[32] 21.4% of funds used all three national procedures (with this figure varying substantially across regions)

[33] WHO also notes that in order to meet the PD target, it would have to channel 50% of its total assistance in the form of Direct Financial Contribution (DFC)

[34] 19 IHP+ country governments participated in the 2012 monitoring carried out by IHP+Results. WHO internal monitoring data covered 80 countries in 2011.

[35] The MOPAN 2013 assessment framework originally indicated that this micro-indicator would also be assessed by the document review using indicator 5b of the Paris Declaration (use of country procurement systems) but this was an error and that assessment is now presented in MI 16.2).

[36] Analysis of the 2011 Paris Declaration Monitoring Surveys: Key Highlights. (p. 9). (Internal document)

[37] Draft Report, WHO participation in the 2011 Paris Declaration Monitoring Surveys: Key Highlights (internal document). p.1. ‘’The amount "technical assistance" does not include the time and salary costs of staff travelling from the inter-country support teams as well as the regions and Headquarters, to countries; The quantification of technical co-operation excludes missions that did not interact with the Government but which are still relevant to WHO operations and capacity to support and deliver in member states; Many fixed country level expenditures that are included as part of ODA have not been reflected in the costs of technical co-operation, despite being critical to WHO's ability to provide this technical co-operation.’’

[38] Analysis of the 2011 Paris Declaration Monitoring Surveys: Key Highlights. (p. 9). (Internal document).

[39] Information retrieved 12 May 2013 from http://apps.who.int/iris/

[40] WHO. (2013). Review of Management, Administration and Decentralization in the World Health Organization, Report by the Joint Inspection Unit. (p. 46).

[41] It is worth noting, however, that within the new evaluation policy framework the OIOS will issue periodic status reports on progress in the implementation of evaluation recommendations to senior management and report annually to the Executive Board through the Programme, Budget and Administration Committee.

[42] It is also responsible for internal audit and investigation of wrongdoing/harassment.

[43] The Global Network on Evaluation will be participating in the preparation of the biennial organisation-wide evaluation workplan and its annual update; submitting relevant evaluation reports to the evaluation inventory; following up on the status of management responses to evaluation recommendations; acting as focal points for evaluation in their respective areas; and advising programmes across WHO on evaluation issues, as needed. Terms of Reference of the Global Network on Evaluation. (p. 1) (Internal document)

[44] The Assessment Team had access to an internal report on evaluations status.

[45] The Assessment Team used this data to assess KPI 16 and 18.

[46] The document review found that the Programme Management Handbook 2012 of the Western Pacific Region serves as a unified source and central repository of current policies, rules, procedures, roles and responsibilities, step-by-step guidance for planning, implementation, monitoring and assessment of WHO programmes in the Western Pacific Region within the General Management System (GSM) environment.

[47] Each organisation may use a different term to refer to this level of results.

[48] WHO. Basic Documents (2009). Constitution of the World Health Organization. (pp. 1-2).

[49] WHO. (2006). Eleventh General Programme of Work: A Global Health Agenda. (p. iii).

[50] Please refer to the sections on KPIs 3 and 21 for the analysis of WHO’s results-based systems and practices.

[51] It is important to note that WHO is changing the structure of its strategic planning in 2014. Instead of having both a General Programme of Work and a Medium-term Strategic Plan, WHO will produce only one document, the Twelfth General Programme of Work, which will contain the high-level strategic vision and WHO’s focus and priorities.

[52] There are 75 organisation-wide expected results in the 11 programmatic strategic objectives.

[53] In this case, self-reported report is understood as an appraisal of the performance based on the achievement of expected results and indicators by individual offices rather than independent or external bodies.

[54] A list of documents consulted is provided in Vol. II, Appendix VIII.

[55] . Programme theory refers to both a description of the interventions and a theory of change.

[56] CREAM (clear, relevant, economic, adequate, monitorable) and SMART (specific, measurable, achievable, relevant, time bound).

[57] As WHO had only published a Mid-Term Review at the time of the assessment, data from the Programme Budget 2012-2013 was not taken into account.

[58] As stated by WHO, fully achieved means that all indicators targets for the organisation-wide expected results were met, including across all six regions. Partly achieved indicates that one or more indicator targets for the organisation-wide expected results were not met. Not achieved means that no indicator targets were met.

[59] Although the MOPAN Common Approach examines the extent to which WHO has made progress towards all of its strategic objectives, the examples in this section are not necessarily representative of the totality of WHO’s work in a specific area. However, attention was paid to the level of expenditures.

[60] WHO. (2011). Programme Budget 2012-2013. (p. 20).

[61] WHO. (2011). Programme Budget 2012-2013. (p. 19).

[62] WHO. (2011). Programme Budget 2012-2013. (p. 39).

[63] WHO. (2012). Programme Budget 2010-2011: Performance Assessment Report. (p. 36).

[64] WHO. (2011). Programme Budget 2012-2013. (p. 27).

[65] WHO. (2012). Programme Budget 2010-2011: Performance Assessment Report. (p. 55).

[66] WHO. (2011). Programme Budget 2012-2013. (p. 39).

[67] WHO’s actions also involve a number of collaborative mechanisms such as intersectoral action for reducing risk and vulnerability within the framework of the United Nations International Strategy for Disaster Reduction and the global alert and response system for environmental and food-safety public health emergencies within the framework of the International Health Regulations

[68] In emergencies and disasters, WHO engages with numerous partners, including national authorities, civil society, United Nations agencies, Global Health Cluster partners, international non-governmental organisations (NGOs), the Red Cross movement, existing and new donors and the private sector.

[69] Although WHO needs to maintain its own capacity to respond to crises, its main objective in this SO is to support the strengthening of member state capacities. According to Programme Budget 2012-2013, one of WHO’s key accomplishments in this strategic objective has been in “preparing guidelines and forming networks for food safety, environmental health emergencies and gender mainstreaming as part of the humanitarian response to crises and disasters.”

[70] WHO. (2012). WHO’s Response, and Role as the Health Cluster Lead, in Meeting the Growing Demands of Health in Humanitarian Emergencies. (p. 1)

[71] WHO reported that it has “reduced in 2012 the number of organisation-wide expected results for strategic objective 5 from seven to two, in order to allow the work to be aligned with developments in the WHO reform process and the Inter-Agency Standing Committee Transformative Agenda”. WHO. (2013). Programme Budget 2012-2013: Mid-term Review. (p. 83)

[72] WHO. (2011). Programme Budget 2012-2013. (p. 50).

[73] WHO. (2011). Programme Budget 2012-2013. (p. 50).

[74] WHO. (2012). Programme Budget 2010-2011: Performance Assessment Report. (p. 66).

[75] WHO. (2011). Programme Budget 2012-2013. (p. 56).

[76] WHO. (2012). Programme Budget 2010-2011: Performance Assessment – Summary Report. (pp. 48; 49).

[77] WHO. (2012). Programme Budget 2010-2011: Performance Assessment – Summary Report. (pp. 52; 53).

[78] WHO. (2012). Programme Budget 2010-2011: Performance Assessment Report. (p. 105).

[79] WHO. (2011). Programme Budget 2012-2013. (p. 61).

[80] WHO. (2012). Programme Budget 2010-2011: Performance Assessment – Summary Report. (pp. 56; 49).

[81] WHO. (2012). Programme Budget 2010-2011: Performance Assessment – Summary Report. (pp. 61; 63).

[82] WHO. (2011). Programme Budget 2012-2013. (p. 72).

[83] WHO. (2012). Programme Budget 2010-2011: Performance Assessment Report. (p. 161).

[84] It is guided by the Millennium Development Goals 4, 5, 6 and target 8E (access to affordable essential medicines), as well as the third WHO Medicines Strategy 2008–2013; the Global Strategy and Plan of Action on Public Health, Innovation and Intellectual Property; the Global immunisation vision and strategy, among others.

[85] The GAVI Alliance, the Global Fund to Fight AIDS, Tuberculosis and Malaria, the International Information Network on New and Emerging Health Technologies (EuroScan International Network), OECD, UNITAID and the World Bank.

[86] Please refer to the sections in chapter 3 on KPIs 3, 5, 21 and 22 for the analysis of WHO’s results-based systems and practices.

[87] As part of the ‘’Delivering as One’’ initiative to enhance the quality and effectiveness of aid in Viet Nam, the One Country Plan replaces single agency plans such as the previous WHO Country Co-operation Strategy for Viet Nam.

[88] MOPAN recognises that this sample may not be representative of WHO work in 150 countries.

[89] WHO. (2013). Review of Management, Administration and Decentralization in the World Health Organization, Report by the Joint Inspection Unit. (p. 10).

[90] Fully achieved: All indicator targets are met or surpassed / Partially achieved: One or more indicator targets are not met / Not achieved: No indicator target is met

[91] A total of 7 MOPAN donors in-country, 14 direct partners, 25 technical partners, and 2 peer organisations

[92] WHO. (2011). Country Office Ethiopia Annual Report. (p. 24).

[93] Capacity strengthened to develop/adapt, implement and scale up high impact and cost effective child survival interventions to achieve universal coverage and for monitoring progress towards achievement of MDG-4.

[94] While the Pan American Health Organization (PAHO) has a different MTSP and management/reporting system, it reports based on WHO’s system using the End of Biennium Performance Assessment Report. PAHO uses its own Enterprise Resource Planning system.

[95] Organización Panamericana de la Salud (2010). Informe situación equipos productores de cloro in situ entregados as MSPAS por cooperación internacional hasta el 20 de agosto 2010.; Informe final cierre proyecto: Modelo Integral de Salud desarrollado e implementado sobre la base de la Rectoría, la Participación Social y la Gestión Local-ASDI III. Periodo 2005-2011.; FODM. (2011). Informe de evaluación de medio término del Programa Conjunto: Consolidando la Paz en Guatemala mediante la prevención de la violencia y gestión del conflicto.

[96] The review also reported that for 2010 the Global Fund to Fight AIDS, TB and Malaria accounted for 92% of total malaria funding and 7% of the other external support for malaria and WHO’s funding accounted for only 1% of total external funding. WHO. (2011). National Malaria Control Programme Review, Republic of Indonesia. (p. 49).

[97] WHO. (2011). National Malaria Control Programme Review, Republic of Indonesia. (p. 82).

[98] MDGs are implicitly mentioned as WHO workplans are developed in line with the MTSP’s structure and each of the 11 strategic objectives are linked to relevant MDGs goals.

[99] WHO. (2010). World Health Organization Country Office for Indonesia Progress Report 2010. (p. 7).

[100] WHO. (2010). 2008-2009 Biennial Report. (p. 17).

[101] WHO. (2010). 2008-2009 Biennial Report. (p. 21).

[102] WHO. (2010). End of Biennium Performance Assessment 2008-2009. (p. 17).

[103] Pakistan is one of eight UN member states selected in 2007 to pilot the UN reform in order to “Deliver as One” where by18 UN agencies share and synergise the diversity, knowhow and experience to maximise the benefit of the people of Pakistan.

[104] This table was shared by the WHO Viet Nam country office following interviews with Senior Management staff.

[105] WHO. (2009). 2008-2009 Mid-term Review

[106] WHO. (2009). Basic Documents: Forty-seventh Edition. (p. 2)