Preface

The Multilateral Organisation Performance Assessment Network (MOPAN) is a network of donor countries with a common interest in assessing the organisational effectiveness of multilateral organisations. MOPAN was established in 2002 in response to international fora on aid effectiveness and calls for greater donor harmonisation and coordination.

Today, MOPAN is made up of 16 donor countries: Australia, Austria, Belgium, Canada, Denmark, Finland, France, Germany, Ireland, The Netherlands, Norway, Republic of Korea, Spain, Sweden, Switzerland and the United Kingdom. For more information on MOPAN and to access previous MOPAN reports, please visit the MOPAN website (www.mopanonline.org).

Each year MOPAN carries out assessments of several multilateral organisations based on criteria agreed by MOPAN members. Its approach has evolved over the years, and since 2010 has been based on a survey of key stakeholders and a review of documents of multilateral organisations. MOPAN assessments provide a snapshot of four dimensions of organisational effectiveness (strategic management, operational management, relationship management, and knowledge management). In 2012, MOPAN is piloting a new component to examine an organisation’s development results in addition to its organisational effectiveness.

MOPAN 2012

In 2012, MOPAN assessed six multilateral organisations: the African Development Bank (AfDB), GAVI Alliance (formerly the Global Alliance for Vaccines and Immunisation), the Joint United Nations Programme on HIV/AIDS (UNAIDS), the United Nations Development Programme (UNDP), the United Nations Children’s Fund (UNICEF), and the World Bank.

MOPAN Institutional Leads liaised with the multilateral organisations throughout the assessment and reporting process. MOPAN Country Leads monitored the process in each country and ensured the success of the survey.

 

Acknowledgements

We thank all participants in the MOPAN 2012 assessment of the GAVI Alliance. The GAVI Secretariat’s senior management and staff made valuable contributions throughout the assessment and document review processes and provided lists of their global and country-level implementing partners, representatives of governments and civil society organisations they work with, and other GAVI Alliance partners and peer organisations to be surveyed. Survey respondents contributed useful insights and time to respond to the survey. The MOPAN Institutional Leads, France, Spain and Sweden, liaised with the GAVI Secretariat throughout the assessment and reporting process. The MOPAN Country Leads oversaw the process in the field and ensured the success of the survey. Consultants in each country provided vital in- country support by following up with country-level implementing partners and representatives of governments and civil society organisations that GAVI works with to enhance survey response rates.

Roles of Authors and the MOPAN Secretariat

The MOPAN Secretariat, led by Ireland in 2012 and co-chaired by Germany, worked in close cooperation with the MOPAN Technical Working Group to launch and manage the survey. MOPAN developed the Key Performance and Micro-indicators, designed the survey methodology, coordinated the development of lists of survey respondents, and approved the final survey questionnaire. MOPAN also directed the design of the approach to document review. MOPAN oversaw the design, structure, tone, and content of the reports.

Universalia and Epinion developed the survey instrument and carried out the survey and analysis. Universalia carried out the document review and wrote the reports.

Epinion is a leading consulting firm in Denmark that analyses and evaluates data to support decision making. It conducts specially designed studies for public and private organisations based on data collected among an organisation’s employees, members, customers, partners, and other sources. Epinion has 75 employees and 200 interviewers. Website: www.epinion.dk

Universalia Management Group is a Canadian consulting firm established in 1980 that specialises in evaluation and monitoring for international development. Universalia has made significant contributions to identifying best practices and developing tools in the fields of organisational assessment; planning, monitoring, and evaluation; results-based management; and capacity building. Website: www.universalia.com.

Acronyms

AAA - Accra Agenda for Action
ACPP - Advocacy, Communication, Public Policy
AfDB - African Development Bank
AMC - Advance Market Committee
ARRF - Audit Report Review Form
COMPAS - Common Performance Assessment System
CREAM - Clear, relevant, economic, adequate, monitorable
CSO - Civil society organisation
DRF - Development results framework
GFA - GAVI Fund Affiliate
GFATM - Global Fund to Fight against AIDS, Tuberculosis and Malaria
GNI - Gross national income
HPV - Human papillomavirus
HRBA - Human Rights-based Approach
HSS - Health systems strengthening
IDA - International Development Association
IFFIm - International Finance Facility for Immunisation
IRC - Independent Review Committee
ISS - Immunisation Services Support
KPI - Key Performance Indicator
M&E - Monitoring and Evaluation
MI - Micro Indicator
MOPAN - Multilateral Organisation Performance Assessment Network
MRF - Management results framework
NGO - Non-governmental organisation
NVS - New and Underused Vaccines
ODA - Overseas Development Assistance
OECD-DAC - Organisation for Economic Cooperation and Development - Development Cooperation Directorate
PAHO - Pan American Health Organisation
PBA - Programme-based approach
RBM - Results-based Management
SMART - Specific, measurable, achievable, relevant, time bound
TAP - Transparency and Accountability Policy
UNAIDS - United Nations Programme on HIV/AIDS
UNDAF - United Nations Development Assistance Framework
UNDP - United Nations Development Programme
UNEG - United Nations Evaluation Group
UNICEF - United Nations Children's Fund
WB - World Bank
WHO - World Health Organisation

Figures

Figure 2.1 Respondent Rating Scale
Figure 2.2 GAVI Alliance- Distribution of Responses (n=102) on all Questions Related to Micro-Indicators
Figure 2.3 Number of Respondents and Total Population for GAVI by Country and Respondent Group
Figure 2.4 MOPAN Ranges and Descriptions
Figure 3.1 Overall Ratings of GAVI Organisational Effectiveness by Respondent Group
Figure 3.2 Overall Ratings on Key Performance Indicators (mean scores, all respondents and document review ratings)
Figure 3.3 Quadrant I: Strategic Management, Survey and Document Review Ratings
Figure 3.4 Quadrant I: Strategic Management, Mean Scores by Respondent Group
Figure 3.5 KPI 1: Providing Direction for Results, Ratings of Micro-Indicators
Figure 3.6 KPI 2: Corporate Focus on Results, Ratings of Micro-Indicators
Figure 3.7 KPI 3: Focus on Thematic Priorities, Ratings of Micro-Indicators
Figure 3.8 KPI 4: Country Focus on Results, Ratings of Micro-Indicators
Figure 3.9 Quadrant II: Operational Management, Survey and Document Review Ratings
Figure 3.10 Quadrant II: Operational Management, Mean Scores by Respondent Group
Figure 3.11 KPI 5: Resource Allocation Decisions, Ratings of Micro-Indicators
Figure 3.12 KPI 6: Linking Support to Performance, Ratings of Micro-Indicators
Figure 3.13 KPI 7: Financial Accountability, Ratings of Micro-Indicators
Figure 3.14 KPI 8: Using Performance Information, Ratings of Micro-Indicators
Figure 3.15 KPI 9: Managing Human Resources, Ratings of Micro-Indicators
Figure 3.16 KPI 10: Performance-oriented Programming, Ratings of Micro-Indicators
Figure 3.17 Quadrant III: Relationship Management, Survey and Document Review Ratings
Figure 3.18 Quadrant III: Relationship Management, Mean Scores by Respondent Group
Figure 3.19 KPI 11: Supporting National Plans, Ratings of Micro-Indicators
Figure 3.20 KPI 12: Adjusting Procedures, Ratings of Micro-Indicators
Figure 3.21 KPI 13: Using Country Systems, Ratings of Micro-Indicators
Figure 3.22 KPI 14: Contributing to Policy Dialogue, Ratings of Micro-Indicators
Figure 3.23 KPI 15: Harmonising Procedures, Ratings of Micro-Indicators
Figure 3.24 Quadrant IV: Knowledge Management, Survey and Document Review Ratings
Figure 3.25 Quadrant IV: Knowledge Management, Mean Scores by Respondent Group
Figure 3.26 KPI 16: Evaluating External Results, Ratings of Micro-Indicators
Figure 3.27 KPI 17: Presenting Performance Information, Ratings of Micro-Indicators
Figure 3.28 KPI 18: Disseminating Lessons Learned, Ratings of Micro-Indicators

Executive Summary

This report presents the results of an assessment of the Global Alliance for Vaccines and Immunisation (GAVI) conducted by the Multilateral Organisation Performance Assessment Network (MOPAN). MOPAN assesses the organisational effectiveness of multilateral organisations based on a survey of stakeholders and a review of documents. MOPAN did not assess GAVI’s development results, although this was a pilot component included in the Common Approach for four multilateral organisations in 2012.

The year 2012 marks the first occasion for MOPAN to involve a global fund in its Common Approach assessments. Adaptations to the MOPAN methodology were required so as to reflect GAVI’s unique operating structure. GAVI was a supportive and willing partner in this process.

The GAVI Alliance is a public-private partnership, created in 2000 in response to declining immunisation rates in developing countries. It brings together multilateral organisations, civil society organisations, public health institutes, the pharmaceutical industry, donor and implementing country governments, and non-profit organisations with a common focus on increasing access to immunisation in developing countries. In 2010, the GAVI Board approved the GAVI Alliance Strategy and Business Plan 2011-2015 that articulates four main goals: 1) “the vaccine goal” of accelerating uptake and use of new and underused vaccines; 2) “the health systems goal” of strengthening the capacity of integrated health systems to deliver immunisation; 3) “the funding goal” of increasing the predictability of global financing and related “co-financing goal” aimed at improving the sustainability of national financing for immunisation; and, 4) “the market shaping goal” of minimising vaccine market prices and securing vaccine supply.

Based on a gross national income (GNI) per capita below or equal to US$ 1,520, 57 countries are eligible for GAVI support. GAVI sets conditions for each type of support. In 2012, 73 countries are receiving GAVI support (57 eligible and 16 graduating countries). The support, provided in the form of either vaccines or cash, is channelled through the regional and country offices of GAVI’s implementing partners (UNICEF, WHO, and the World Bank) or directly to implementing country governments. In 2011, GAVI raised US$4.3 billion in donor pledges. This brought GAVI’s total available resources for the period 2011-2015 to US$ 7.6 billion.

Over the past three years, GAVI’s new leadership has undertaken a number of reform efforts to improve its effectiveness, including the strengthening of its results-based, change and risk management systems.

In 2012, MOPAN assessed GAVI based on information collected at the organisation’s headquarters, from MOPAN members, and from stakeholders in fifteen GAVI-supported countries: Bangladesh, Cambodia, Democratic Republic of Congo, Georgia, Ghana, Honduras, Indonesia, Nicaragua, Niger, Nigeria, Pakistan, Rwanda, Tanzania, Yemen, and Zimbabwe.

MOPAN’s survey targeted six categories of respondents: MOPAN members based in the country and at headquarters, global and country-level implementing partners, representatives of governments and civil society organisations GAVI works with, and other GAVI Alliance partners and peer organisations. A total of 102 respondents participated in the survey. MOPAN’s document review assessed GAVI’s systems and practices through an examination of publicly available, corporate and programming documents provided by GAVI. Given the small number of responses by MOPAN donors in-country, implementing partners, government, and civil society groups, caution should be used in interpreting survey results from these respondent groups.

MOPAN assessments provide a snapshot of four dimensions of organisational effectiveness (strategic management, operational management, relationship management, and knowledge management). The main findings of the 2012 assessment of GAVI are summarised below.

Strategic Management

In strategic management, MOPAN established criteria to determine if a multilateral organisation has strategies that reflect good practices in managing for results. Overall, the 2012 assessment found that:

• Strategic management is an area where GAVI presents some strengths as well as some areas for improvement. Both survey respondents and the document review found that GAVI is strongly promoting a corporate focus on the achievement of results, with the 2011-2015 Strategy and Business Plan based on its mandate and understanding of its comparative advantage. Over the past three years, GAVI has undertaken several reforms to strengthen its results focus and improve its partners’ accountability for the delivery of the results identified in the 2011-2015 Strategy and Business Plan. This has been done by instituting quarterly reporting on the deliverables and the systematic use of indicators identified in the 2011-2015 Strategy and Business Plan to report on performance at the impact and strategic goal levels. GAVI also supports a results orientation in the programmes it funds by aligning its support with national strategies for immunisation. Survey respondents and the document review also recognised that GAVI is well- positioned in the international aid architecture and that it is addressing the cross-cutting priorities of gender and good governance.

• GAVI could improve its practices in some areas of strategic management, such as the logic of the results chain and the performance indicators in its results frameworks. While the assessment identified strong mainstreaming of governance, gender and environment, it also found GAVI’s conceptualisation of the human rights-based approach as a cross- cutting priority to be limited and that GAVI lacks specific strategies or structures to identify results in these areas and mainstream them at the country level.

Operational Management

In operational management, MOPAN established criteria to determine if a multilateral organisation manages its operations in a way that supports accountability for results and the use of information on performance. Overall, the 2012 assessment found that:

• GAVI hired an internal auditor in 2009 and its reforms since this time have led to the strengthening of its financial management and accountability systems, which the 2012 MOPAN assessment found to be its strongest operational management area. Other areas of adequate to strong performance include GAVI’s efforts to ensure transparent resource allocation based on performance information, and using performance information to revise, as well as to adjust policies and plan country-level grant windows.

• While GAVI has given some attention to results-based management practices in its budgeting process and human resource management, these areas require additional effort.

Relationship Management

In relationship management, MOPAN established criteria to determine if a multilateral organisation is engaging with its in-country partners in ways that contribute to aid effectiveness. MOPAN criteria examine the organisation’s procedures, track record on coordination and harmonisation with peers, and contributions to policy dialogue. Overall, the 2012 assessment found that:

• Relationship management is one of GAVI’s strengths. GAVI’s support of country ownership and its alignment and harmonisation of arrangements and procedures with partners were highly appreciated by survey respondents, who consider that GAVI support is aligned with national priorities and that GAVI’s procedures, such as funding application and reporting requirements, take into account local conditions and capacities. Harmonisation is also seen as a strong practice for GAVI, particularly with the launch of the Health System Funding Platform, an initiative that aims to streamline support for health systems strengthening and align it with country budgetary and programmatic cycles.

• Despite efforts to improve the reporting on its performance according to its Paris Declaration commitments, a full appreciation of GAVI’s contribution to some of these commitments can be difficult. On the one hand, GAVI has improved the predictability and delivery of its funding and vaccines, the coordination of aid development through its participation in joint missions with other donors, and the channelling of its support to countries through programme-based approaches. On the other hand, it has limited data and has reported inconsistently to the OECD-DAC on this commitment over the past decade. However, some indicators used to measure progress towards Paris Declaration commitments (e.g., the percentage of support recorded in national budget and the coordination of technical cooperation with other donors) are not applicable to GAVI due to its structure and operational approach. GAVI is working actively to address these issues.

Knowledge Management

In knowledge management, MOPAN established criteria to determine if a multilateral organisation has reporting mechanisms and learning strategies that facilitate the sharing of information inside the organisation and with the development community. Overall, the 2012 assessment found that:

• GAVI is paying increased attention to knowledge management and has recently undertaken initiatives to strengthen its evaluation function, culture and reporting systems. This is reflected in several recently implemented reforms, including the establishment of an Evaluation Advisory Committee within the Board structure and a performance management unit within the GAVI Secretariat, the recognition of quality as one of the guiding principles for evaluations, and the systematic use of performance indicators from the 2011-2015 Strategy and Business Plan in its reporting to the Board and to the public in Phase III.

• Whereas GAVI’s Evaluation Policy recognises stakeholder participation and quality as key guiding principles for evaluations, the main area for improvement in knowledge management is the coverage of GAVI’s evaluations. GAVI does not currently have clear guidelines on what it considers to be sufficient evaluation coverage.

Conclusions

The following conclusions look at messages that can contribute to dialogue between MOPAN, GAVI and its partners.

GAVI is well-positioned within the international aid architecture.

MOPAN survey respondents recognised GAVI’s mandate to save children’s lives and protect people’s health by increasing access to immunisation as one of the organisation’s main strengths. They also commended GAVI’s mandate as a public-private partnership that brings together a variety of actors from developed and developing countries and from the public and private sectors. These two elements make GAVI an organisation that is considered well- positioned within the international aid architecture.

The nature of GAVI as a public-private partnership and as an Alliance means that it faces some internal challenges in meeting the expectations of its constituent groups while operating in a complex external environment.

GAVI operates in a complex internal and external environment. Internally, GAVI works as an Alliance, which means working to meet a variety of sometimes contradictory expectations regarding its practices, structure, and scope of work; relying on the different capacities of its partners’ and supported countries to implement, while maintaining accountability for programme implementation with the whole Alliance.

Externally, GAVI works in an international aid development context characterised by partially contradictory forces. Limited availability of financial resources for aid development leads donors to insist on increased demonstrations of effectiveness and accountability. However, the Paris Declaration commitments seek increased donor coordination and country ownership.

The 2012 MOPAN assessment found that within the interplay of these forces, GAVI is holding its own and making progress to become an increasingly effective organisation. This is demonstrated by the largely positive views provided by stakeholders, the result of its last pledging conference in which pledges exceeded the target, and by the increasing country demand for GAVI’s support.

GAVI’s mandate and comparative advantage provide a good foundation for its focus on results, but there is room to further strengthen this orientation in its strategy, budget, and reporting practices.

GAVI’s 2011-2015 Strategy and Business Plan is anchored to its mandate and comparative advantage, which provide the organisation with a strong direction for results. However, the extent to which this results orientation is operationalised within the organisation does not yet meet all of GAVI’s stakeholder expectations.

Over the past three years, GAVI has undertaken reforms to strengthen its practices and become a more results-oriented organisation. These have included the development of an integrated business plan, the introduction of quarterly reporting by the GAVI Secretariat and partners on the programme deliverables identified in the 2011-2015 Strategy and Business Plan, the establishment of a performance management unit within the GAVI Secretariat, and increased mainstreaming of gender equality in GAVI’s internal and external operations.

While these reforms are important steps towards GAVI becoming a results-oriented organisation, some gaps remain: the results-based management approach, the adequacy of the results frameworks (the logic of and linkages in the results chains in the development results framework and the absence of strong results statements in the management results framework), and the results-based budgeting practices. These gaps were noted in the document review and in the survey by a small number of GAVI private sector[1] and public sector respondents. Thus, GAVI could benefit from increased efforts, and possibly increased resources, in results-based management systems and tools (especially the identification of adequate performance indicators, the strengthening of the results chain at the corporate and country levels, and RBM training). As MOPAN donors are committed to integrating cross- cutting themes, it will be important for GAVI and MOPAN to discuss the appropriate definitions and approaches to integrate human rights-based approaches in its strategy and operations and further encourage its partners to develop policies for human rights-based approaches.

GAVI is recognised for a number of its operational practices that ensure transparency and accountability in the use of resources.

Over the last two to three years, GAVI has strengthened its systems in order to manage operations in a more performance-oriented way and to promote accountability (of GAVI as a whole and of supported countries) related to results achievement and use of funds. Some of the reforms introduced to positive effect include the establishment of a performance management process and unit, and the strengthening of financial accountability systems, particularly of the risk management system for cash-based programmes (given an increase of this type of support). GAVI has also continued to promote the transparency of its decisions, including its resource allocation, through consultation mechanisms and dialogue with its partners, wide dissemination of its key documentation, and the establishment of independent mechanisms with an advisory role for decision-making, such as the Independent Review Committee and the Evaluation Advisory Committee.

In response to increasing requests to demonstrate results and accountability to different stakeholder groups, GAVI is undertaking reforms to strengthen its knowledge management practices.

GAVI has paid increasing attention to its performance and knowledge management practices since 2011 in order to respond to the demands from internal and external stakeholders (donors, partners, governments, senior managers) to demonstrate effectiveness and accountability. It has undertaken several reforms to make monitoring and evaluation integral to programme management, including: the establishment of an Evaluation Advisory Committee within the GAVI Alliance Board structure; the introduction of country grant scorecards for more regular monitoring; and, the development of a monitoring and evaluation framework and strategy. In due course, an assessment of the effectiveness of these measures may need to be done to ensure that they meet the expectations and requests of GAVI’s key internal and external stakeholders.

GAVI is successfully managing a complex set of relationships at the country level. However, adapting to the needs and expectations of partners in so many countries is a slow, delicate, and costly balancing act.

GAVI currently operates in 73 of the poorest countries through its country-level partners (both developing countries and GAVI Alliance partners). It depends heavily on the capacities of these partners in numerous areas: sustainability of vaccines, co-financing, collection of sex- disaggregated data, the integration of gender and environment-related considerations in programme implementation, and the quality of immunisation-related data.

GAVI is adapting to the contexts in these different countries by integrating some flexibility into its policies and operations by, for example, establishing different levels of co-financing and developing a specific policy for fragile and underperforming countries. At the same time, country-level stakeholders are asking GAVI to increase its country presence in order to facilitate

communications and improve support to grant recipients. These requests, however, go against the ‘light touch’ approach that has shaped GAVI’s structure and functioning until now, and that is considered by stakeholders with a global perspective to be a key factor in the Alliance’s success to date.

While the MOPAN assessment found that GAVI has managed its relationships well at the country level, this is likely to remain a delicate area that will require constant monitoring by GAVI and open dialogue with its stakeholders.

Although GAVI’s work is aligned with the principles of the Paris Declaration, progress on some specific indicators is not clear.

GAVI is one of the signatories of the 2005 Paris Declaration on Aid Effectiveness and has made efforts to align its work with the Paris Declaration principles. GAVI’s institutional architecture, characterised by a partnership between a wide range of development actors, is aligned with the principles of harmonisation and mutual accountability. Its support of countries’ national health plans and use of country systems are aligned with the principles of country ownership and alignment. Its focus on immunisation-related results and performance-based funding is coherent with the principle of managing for results. Recent initiatives, such as the Health Systems Funding Platform, the Advance Market Commitment, and the International Health Partnership, are evidence of GAVI’s adherence to PD principles.

Nonetheless, despite its commitment to the PD principles and evidence of its contribution to some of them, it is difficult to fully appreciate GAVI’s progress on its PD commitments due to several factors, including: the nature of some indicators, the lack of data on several indicators (e.g., support recorded in national budgets, the use of national systems and procedures, and the strengthening of capacity by coordinated support), as well as its inconsistent reporting to the OECD-DAC on these commitments over the past decade.

Some of the Paris Declaration indicators may not apply as readily to multilateral organisations like GAVI that provide mostly in-kind support in the form of vaccines and use the procurement and distribution systems of its partners. The OECD is currently revising these indicators. In the meantime, given that there is an expectation from donors to understand GAVI’s contribution to these principles, GAVI might consider developing some alternative indicators that measure its contribution or discuss other ways in which it can report to donors on these commitments.

Overall MOPAN Ratings of GAVI

The chart below shows the ratings on the 18 key performance indicators that MOPAN used to assess GAVI in 2012. These indicators were designed to measure organisational effectiveness (practices and systems), not development results on the ground. The indicators were adapted to GAVI’s institutional architecture as a global fund. GAVI received ratings of adequate or strong on the 18 key performance indicators assessed by survey respondents, and document review ratings ranged from inadequate to strong.

1. Introduction

1.1 MOPAN

This report presents the results of an assessment of the GAVI Alliance that was conducted in 2012 by the Multilateral Organisation Performance Assessment Network (MOPAN). In 2012 MOPAN assessed six multilateral organisations: the African Development Bank (AfDB), GAVI Alliance (formerly the Global Alliance for Vaccines and Immunisation), the Joint United Nations Programme on HIV/AIDS (UNAIDS), the United Nations Development Programme (UNDP), the United Nations Children’s Fund (UNICEF), and the World Bank.

Background

MOPAN was established in 2002 in response to international fora on aid effectiveness and calls for greater donor harmonisation and coordination. The purpose of the network is to share information and experience in assessing the performance of multilateral organisations. MOPAN supports the commitments adopted by the international community to improve the impact and effectiveness of aid as reflected in the Paris Declaration on Aid Effectiveness, the Accra Agenda for Action, and the Busan High Level Forum. MOPAN’s processes and instruments embody the principles of local ownership, alignment and harmonisation of practices, and results-based management (RBM).

MOPAN provides a joint approach (known as the Common Approach) to assess the organisational effectiveness of multilateral organisations. The approach was derived from existing bilateral assessment tools and complements and draws on other assessment processes for development organisations – such as the bi-annual Survey on Monitoring the Paris Declaration on Aid Effectiveness and annual reports of the Common Performance Assessment System (COMPAS) published by the multilateral development banks. In the long term, MOPAN hopes that this approach will replace or reduce the need for other assessment approaches by bilateral donors.

MOPAN assesses four dimensions of organisational effectiveness

MOPAN has defined organisational effectiveness as the extent to which a multilateral organisation is organised to contribute to development and/or humanitarian results in the countries or territories where it operates.

Based on a survey of stakeholders and a review of documents, MOPAN assessments provide a snapshot of a multilateral organisation’s effectiveness in four dimensions:

• Developing strategies and plans that reflect good practices in managing for development results (strategic management)

• Managing operations by results to support accountability for results and the use of information on performance (operational management)

• Engaging in relationships with direct partners and donors at the country level in ways that contribute to aid effectiveness and that are aligned with the principles of the Paris Declaration (relationship management)

• Developing reporting mechanisms and learning strategies that facilitate the sharing of knowledge and information inside the organisation and with the development community (knowledge management).

In 2012, MOPAN also piloted a new component to assess a multilateral organisation’s contributions to development results. This component was tested with four of the six organisations assessed this year (AfDB, UNDP, UNICEF, and the World Bank).

Purpose of MOPAN assessments

MOPAN assessments are intended to:

• Generate relevant, credible and robust information MOPAN members can use to meet their domestic accountability requirements and fulfil their responsibilities and obligations as bilateral donors

• Provide an evidence base for MOPAN members, multilateral organisations and direct partners to discuss organisational effectiveness and in doing so, build better understanding and improve organisational effectiveness and learning over time

• Support dialogue between MOPAN members, multilateral organisations and their partners, with a specific focus on improving organisational effectiveness over time, both at country and headquarters level.

The MOPAN methodology is evolving in response to what is being learned from year to year, and to accommodate multilateral organisations with different mandates. For example, the indicators and approach for the 2012 MOPAN review of a global fund and organisations with significant humanitarian programming were adapted to reflect the reality of these organisations.

MOPAN assessment of the GAVI Alliance

This is the first time the GAVI Alliance has been assessed by MOPAN.

1.2 Profile of the GAVI Alliance

The GAVI Alliance is a public private partnership, created in 2000 in response to declining immunisation rates in developing countries, that brings together multilateral organisations, civil society organisations, public health institutes, donor and implementing country governments, and non-profit organisations with a common focus on increasing access to immunisation within developing countries. GAVI’s mandate is “to save children’s lives and protect people’s health by increasing access to immunisation in poor countries.”

The GAVI Alliance is an incorporated Swiss foundation with offices in Geneva and Washington, DC. Originally based at UNICEF, in 2007 the organisation made the decision to centralise its governance and operations under GAVI Alliance using the legal platform of the Swiss not-for- profit GAVI Foundation. In 2009 the GAVI Foundation officially became known as the GAVI Alliance.[2]

Governance and structure

Following the merger of the previous GAVI Alliance Board and the GAVI Fund Board, GAVI officially formed a new Board of Directors that is responsible for the establishment of all policies, organisational operations, as well as monitoring of programme implementation. GAVI’s Board members include representatives from the Bill and Melinda Gates Foundation, the World Bank, UNICEF and WHO, as well as private individuals with expertise in fields critical to the successful operation of the organisation.

The GAVI Board is supported by a Secretariat that is responsible and accountable for the daily operations of GAVI such as developing policy and strategy implementation, mobilising resources, legal and financial management, coordination of programme approvals and disbursements, and Board and committee administration. The GAVI Secretariat does not implement GAVI support at the country level. As a partnership among different agencies, GAVI relies on its implementing partners (UNICEF, WHO, and the World Bank) to implement GAVI support to countries through their regional and country offices. The Secretariat is headed by a Chief Executive Officer and has a projected staff of 187 for 2012.

Strategy and Services

GAVI works with a five-year planning cycle. The most recent cycle, Phase III, began with the approval in 2010 of the GAVI Alliance Strategy and Business Plan 2011-2015. The strategy articulates four main goals: 1) “the vaccine goal” of accelerating uptake and use of new and underused vaccines; 2) “the health systems goal” of strengthening the capacity of integrated health systems to deliver immunisation; 3) “the funding goal” of increasing the predictability of global financing and “the co-financing goal” aimed at improving the sustainability of national financing for immunisation; and 4) “the market shaping goal” of minimising and stabilising vaccine market prices, and securing vaccine supply. These goals are supported by a set of cross-cutting operating principles and activities focused on monitoring and evaluation (M&E) and advocacy, as well as communications and public policy (ACPP).

Through its 2011-2015 Strategy and Business Plan, GAVI aims to successfully increase immunisation coverage and avert an additional 3.9 million deaths, provide pneumococcal and rotavirus vaccines to infants in more than 30 countries for the first time, introduce additional lifesaving antigens like human papillomavirus (HPV), and strengthen health systems in more than 40 developing countries to promote the integrated delivery of immunisation.

GAVI’s policy on country eligibility allows countries with a gross national income (GNI) per capital below or equal to US$1,500 (for 2009) to apply for vaccines and/or cash-based programme support. Applications are reviewed by the Independent Review Committee, an independent review mechanism. Countries whose applications for vaccine support have been approved by GAVI are required to contribute to the cost of the vaccines.

Finances

GAVI’s donors comprise governments, the European Commission (EC), foundations, as well as private individuals and organisations. At the beginning of its third phase (2011-2015), GAVI estimated it needed US$6.8 billion to implement the 2011-2015 Strategy and Business Plan. In 2011, GAVI raised US$4.3 billion in donor pledges. This brought GAVI’s total available resources for the period 2011-2015 to US$ 7.6 billion.

To accelerate the availability and increase the predictability of funds for the Alliance’s immunisation programme, GAVI incorporated two UK charities, the International Finance Facility for Immunisation (IFFIm) and the GAVI Fund Affiliate (GFA), that allow it to convert long-term government donor pledges into immediately available funds by issuing bonds on the global capital markets.

Organisational Development Initiatives

Since 2009, GAVI has implemented various organisational development initiatives in order to improve efficiency and accountability, such as:

• The Health Systems Funding Platform to streamline its Health Systems Strengthening (HSS) support in a single mechanism. The result has been lower transaction costs for implementing countries, increased efficiency, and a reduction in fiduciary risk.

• In order to minimise fiduciary risk in cash-based programmes, including HSS and immunisation services support (ISS), GAVI’s 2009 Transparency and Accountability Policy (TAP) requires Financial Management Assessments (FMAs) that draw on the knowledge of government and in-country development partners to reach a consensus on an appropriate financing modality for GAVI support.

• To further evaluate and strengthen risk management as well as governance processes within the Alliance, an independent internal audit function was created in 2009.

The GAVI website is: www.gavialliance.org.

2. MOPAN Methodology – 2012

2.1 Overview

Background

MOPAN continues to refine its assessment framework. In 2009, the MOPAN Common Approach replaced the Annual MOPAN Survey, which had been conducted since 2003. The Common Approach is broader and deeper than the previous surveys and includes the following components:

• Expanded survey – The MOPAN survey now brings in the views of direct partners or clients of multilateral organisations, peer organisations (or other relevant stakeholder group), and those of donors, that is, MOPAN members at both headquarters and country level.

• Document review – Since 2010, survey data are complemented by a review of documents prepared by the multilateral organisations being assessed and other sources.

• Interviews – In 2012, MOPAN complemented survey data and document review with consultations and interviews at the headquarters of multilateral organisations assessed.

In 2012 MOPAN also tested a new component to assess the results of multilateral organisations.[3]

As MOPAN’s methodology has changed significantly in the last three years, comparisons of this year’s assessments and previous assessments should take this into consideration.

The following is a summary of the MOPAN methodology in 2012.[4]

MOPAN 2012

In 2012, MOPAN assessed the effectiveness of six multilateral organisations: the African Development Bank (AfDB), GAVI Alliance (formerly the Global Alliance for Vaccines and Immunisation), the Joint United Nations Programme on HIV/AIDS (UNAIDS), the United Nations Development Programme (UNDP), the United Nations Children’s Fund (UNICEF), and the World Bank.

For all organisations, the assessment was conducted in Cambodia, Democratic Republic of Congo, Ghana, Honduras, Philippines, Morocco, Niger, Nigeria, and Zimbabwe.[5] Since GAVI works with a limited number of actors in each country, mostly one or two representatives from Ministries of Health, in order to increase the population size for the GAVI survey, eight additional GAVI-eligible countries were included: Bangladesh, Georgia, Indonesia, Nicaragua, Pakistan, Rwanda, Tanzania, and Yemen.[6]

The MOPAN Common Approach examines organisational systems, practices, and behaviours that MOPAN believes are important for aid effectiveness and that are likely to contribute to results at the country level. It groups these organisational capacities in four areas of performance: strategic management, operational management, relationship management, and knowledge management.

Key Performance Indicators and Micro-indicators – Within each performance area, organisational effectiveness is described using key performance indicators (KPIs) that are measured with a series of micro-indicators (MIs).

The micro-indicators are assessed using data from a survey and document review. The survey collects perception data from a variety of stakeholders (see Section 2.2) and the review of documents relies on a set of criteria that provide a basis for the assessment of each micro- indicator (see Section 2.3). However, not all micro-indicators are assessed by both the survey and the document review; consequently, some charts do not show survey scores and document review scores for each KPI or MI.

The GAVI Alliance was assessed using 18 KPIs and 70 MIs. The full list of MIs assessed is provided in Volume II, Appendix V (KPI and MI Data by Quadrant).

2.2 Survey

To gather diverse perspectives on the multilateral organisations being assessed, MOPAN generally seeks perceptions of diverse respondent groups. Due to the nature of GAVI, the respondent groups include:[7]

• Donor Headquarters Oversight (HQ): Professional staff, working for a MOPAN donor government, who share responsibility for overseeing / observing a multilateral organisation at the institutional level. These respondents may be based at the permanent mission of the multilateral organisation or in the donor capital.

• Donor Country Office Oversight (CO): Individuals who work for a MOPAN donor government and are in a position that shares responsibility for overseeing/observing a multilateral organisation at the country level.

• Country Government and National CSO Respondents: Government officials and representatives of civil society organisations (CSOs) from the 15 countries selected for the GAVI assessment. Government officials are representatives from health ministries or any other ministry or national department involved in applying for GAVI funding, in managing GAVI grants, or in financing immunisation programmes with GAVI support. GAVI may not support CSOs in all countries.

• In-Country Implementing Partners: Country or regional representatives of UNICEF, WHO, and the World Bank, the three members of the GAVI Alliance that support the implementation of GAVI's programmes at the country level. These respondents are knowledgeable about GAVI's programmes in the countries selected according to MOPAN criteria for 2012 (i.e., Cambodia, Democratic Republic of Congo, Ghana, Honduras, Niger, Nigeria, and Zimbabwe). Respondents from these organisations are usually based in the country or region.

• Global-Level Implementing Partners: Representatives of the three members of the GAVI Alliance: UNICEF, WHO and the World Bank. Respondents, usually based at their organisation's headquarters, include representatives who have participated in the GAVI Board, committees, initiatives, or coordination mechanisms.

• Other Global Alliance Partners and Peers: These include other members of the GAVI Alliance and peer organisations that work with GAVI at a global level. Respondents are representatives from the Bill & Melinda Gates Foundation, international non-governmental organisations, the emerging/developing country pharmaceutical industry, the industrialised country pharmaceutical industry, research and technical health institutes, and the Global Fund to Fight AIDS, Tuberculosis, and Malaria (GFATM).

MOPAN donor respondents are chosen by MOPAN member countries. The other respondents are identified by the multilateral organisation being assessed.

The survey is customised for each organisation assessed and can be completed online in English, French, or Spanish or offline (paper, email, or interview) in these same languages. See Volume II (Appendix II) for the survey. Individual responses to the survey are confidential to the independent consultants managing the online survey or collecting data offline in the field.

Respondent Ratings – Survey respondents are presented with statements describing an organisational practice, system, or behaviour and asked to rate the organisation’s performance on a scale of 1 to 6 as shown below.

Figure 2.1 Respondent Rating Scale

In some cases, not all survey questions are answered, either because: 1) the individual chose not to answer, or 2) the question was not asked of that individual. In these cases, mean scores are calculated using the actual number of people responding to the question. As noted in the methodology (Volume II, Appendix I), ‘don’t know’ survey responses are not factored into the calculation of mean scores. However, when the proportion of respondents answering ‘don’t know’ is considered notable for a micro-indicator, this is indicated in the report. The responses of various categories of respondents on the six choices, plus ‘don’t know’ are summarised across all survey questions in Figure 2.2.

Figure 2.2 GAVI Alliance- Distribution of Responses (n=102) on all Questions Related to Micro- Indicators

While there were responses in all six possible choices, relatively few responses overall were at the ‘weak’ end of the scale. The other GAP respondent group had the largest proportion of responses on the ‘inadequate’ to ‘weak’ end of the scale. Approximately one-quarter of the responses from government and CSO representatives were ‘very strong’ and almost two-thirds were ‘strong’ or ‘very strong’. Across all respondent groups, the majority of responses (63 per cent and above) were in the range of adequate to very strong.

Respondents and Response Rates

MOPAN aims to achieve a 70 per cent response rate from donors at headquarters and a 50 per cent response rate among other respondent groups (i.e., donors in-country, implementing partners, representatives of governments and civil society organisations, and other members of the GAVI Alliance). The number of respondents targeted in each category (i.e., the total population) and the actual response rates are presented in Figure 2.3 below. It is important to note that both the target and actual number of respondents in each category are small, reflecting the nature of the organisation and the way it works in each country. Response rates of all categories of respondents, except donors in-country, exceeded the 50 per cent target rate. The small numbers in the sample and the poor response rate of donors in-country are both discussed in the subsequent section on limitations. While there are wide variations in the response rates by category and location of respondents, GAVI survey results reflect the views of 102 respondents.

Figure 2.3 Number of Respondents and Total Population for GAVI by Country and Respondent Group

* This category is made up mostly of government representatives from the health sector.
** Donors in-country were surveyed only in the countries selected by MOPAN for all 2012 assessments.

Converting Individual Scores to Group Ratings

As noted above, individuals respond to survey questions on a six-point scale where a rating of “1” is considered a judgment of “very weak” up to a rating of “6” intended to represent a judgment of “very strong.” A mean score is calculated for each respondent group (e.g., donors at HQ). Since mean scores are not necessarily whole numbers (from 1 to 6) MOPAN assigns numerical ranges and descriptive ratings for each range (from very weak to very strong) as shown below.

Figure 2.4 MOPAN Ranges and Descriptions

The ranges are represented to two decimal places, which is simply the result of a mathematical calculation and should not be interpreted as representing a high degree of precision. The ratings applied to the various KPIs should be viewed as indicative judgments rather than precise measurements.

Data Analysis

First level survey data analysis includes calculations of mean scores, medians, standard deviations, frequencies, (including analysis of ‘don’t know’ and missing responses), as well as content analysis of open-ended questions. The ‘don’t know’ responses are removed from the calculation of mean scores, but the proportion of respondents choosing ‘don’t know’ is retained as potentially useful data.

A weighting scheme is applied to ensure that no single respondent group or country is under- represented in the analysis. The weighting is intended to correct for discrepancies/variation in: the number of individuals in each respondent group, the number of countries where the survey took place, the numbers of donors in-country, direct partners, and other respondent groups within each country where the survey took place. Weighted figures are carefully reviewed and analysed before inclusion in the multilateral organisation reports.

Second level analysis examines differences in the responses among categories of respondents and other variables. When significant differences are found, these are noted in the report.[8]

For a full description of survey data analysis see Volume II, Appendix I.

2.3 Document Review

The document review considers three types of documents: multilateral organisation documents, identified with the help of the organisation; internal and external reviews of the organisation’s performance, found on the organisation’s web site or provided by the organisation; external assessments such as the Survey on Monitoring the Paris Declaration, the Common Performance Assessment (COMPAS) report, and previous MOPAN surveys.

Ratings for key performance indicators (KPIs) are based on the ratings for the component micro-indicators in each KPI. For each micro-indicator, a set of criteria are established which, taken together, are thought to represent good practice in that area. The criteria are based on existing standards and guidelines (for example, UNEG or OECD-DAC guidelines), on MOPAN identification of key aspects to consider, and on the input of subject-matter specialists. The rating on any micro-indicator depends on the number of criteria met by the organisation. In cases where the micro-indicator ratings for one KPI are highly divergent, this is noted in the report.

While the document review assesses most micro-indicators, it does not assign a rating to all of them (when criteria have not been established). Consequently, some charts do not show document review scores for each KPI or MI. Documents are also used to aid in the understanding of the context in which the multilateral organisations work.

The document review and survey use the same list of micro-indicators, but some questions in the document review are worded differently from those in the survey. The document review and survey also use the same rating scale, but scores are presented separately on each chart in the report to show their degree of convergence or divergence.

2.4 Interviews

As of 2012, interviews are conducted at the headquarters of multilateral organisations with individuals who are knowledgeable in areas that relate to the MOPAN assessment.

Interviewees are asked to provide knowledge, insight, and contextual information that could assist the MOPAN Assessment Team in analysing document review data, and to identify other relevant documents for the Assessment Team to consider. This helps ensure that the Assessment Team has all the appropriate and necessary documents, enhances the Team’s ability to triangulate data from various sources, and assists the Assessment Team in the analysis of the key performance indicators by providing contextual information.

Interviews are conducted with a small number of staff who work in the primary units that relate to areas of the MOPAN assessment. Interviewees are identified by the multilateral organisation in conjunction with the Assessment Team and MOPAN. An interview guide is prepared and interviewees are advised of the content areas beforehand.

Data gathered during interviews is used to understand the context in which the agency is working, as well as how decisions are made. In the event that survey data present a picture that is very different from the document review, information from interviews can help clarify how the multilateral organisation approached a certain issue.

2.5 Basis for Judgment

From 2003 to 2009, the basis for judgment in MOPAN assessments was the perceptions of survey respondents. With the introduction of the document review in 2010 and interviews in 2012, judgments now draw on a variety of sources that can be compared and triangulated.

To the extent possible, the assessment standards and criteria are tailored to reflect the nature and operating environment of the multilateral organisations under review.

The MOPAN approach uses multiple data sources and data collection methods to validate findings. This helps eliminate bias and detect errors or anomalies.

The MOPAN reports gain trustworthiness through the multiple reviews and validation processes that are carried out by members of the network and by the multilateral organisations themselves.

2.6 Reporting

Institutional Reports

Individual institutional reports are produced for each multilateral organisation assessed. The results of the document review are presented alongside the survey results and discussed in light of the perception-based scores and interviews in order to further substantiate and contextualise the overall findings. For those agencies that were evaluated in 2009, a brief analysis of trends is included.

Country Data Summaries

When a multilateral organisation has presence at the country level, a summary of survey results may be produced for each of the countries surveyed where sufficient survey data exists.

Country Data Summaries provide feedback to those who participated in the MOPAN assessment and provide input for a dialogue process. They are not published and are shared only with individuals who attend the country workshop on the MOPAN assessment findings, which takes place in the first quarter of the year following the assessment.

2.7 Strengths and Limitations of Methodology

MOPAN continues to improve methodology based on the experience of each year of implementation. The following strengths and limitations should be considered when reading MOPAN’s report on GAVI.

Strengths

• The MOPAN Common Approach is based on the core elements of existing bilateral assessment tools. In the long term, the intent is to replace or reduce the need for other assessment approaches by bilateral donors.

• It seeks perceptual information from different perspectives: MOPAN donors (at headquarters and in-country), direct partners/clients of multilateral organisations, peer organisations, and other relevant stakeholders. This is in line with donor commitments to the Paris Declaration on Aid Effectiveness and the Accra Agenda for Action regarding harmonisation, partner voice, and mutual accountability.

• It complements perceptual data with document review and interviews, thus using multiple sources of data. This should enhance the analysis, provide a basis for discussion of agency effectiveness, and increase the validity of the assessment through triangulation of data.

• The reports undergo a validation process, including multiple reviews by MOPAN members and review by the multilateral organisation being assessed.

• MOPAN strives for consistency across its survey questions and document review for each of the multilateral organisations, while allowing for customisation to account for differences between types of multilateral organisations.

Limitations

MOPAN Framework

• The countries are generally selected based on established MOPAN criteria and comprise only a small proportion of each institution’s operations, thus limiting broader generalisations. (While additional countries were added for the GAVI assessment, the number still represents a small proportion of its work.)

• The Common Approach indicators were designed for multilateral organisations that have operations in the field. For organisations that have limited field presence or that have regional structures in addition to headquarters and country operations, there have been some modifications made in the data collection method and there will be a need for greater nuance in the analysis of the data.

Data Sources

• GAVI works with a small number of implementing partners, government, and civil society groups in each country. As a result, the target populations and the resulting samples are small and caution should be used in interpreting survey results.[9]

• In the GAVI assessment, the relatively small number of responses of MOPAN donors in- country (in contrast to the larger response of other respondent groups) underscores the need for caution in interpreting scores on charts involving the MOPAN donors in-country category.

• The MOPAN Common Approach asks MOPAN members and the organisations assessed to select the most appropriate individuals to complete the survey. While MOPAN sometimes discusses the selection with the organisation being assessed, it has no means of determining whether the most knowledgeable and qualified individuals are those that complete the survey.

• The document review component works within the confines of an organisation’s disclosure policy. In some cases, low document review ratings may be due to unavailability of organisational documents that meet the MOPAN criteria (some of which require a sample of a type of document, such as country plans, or require certain aspects to be documented explicitly). When information is insufficient to make a rating, this is noted in the charts.

Data Collection Instruments

• Three issues potentially affect survey responses. First, the survey instrument is long and a fatigue factor may affect responses and rates of response. Second, respondents may not have the knowledge to respond to all the questions (e.g., survey questions referring to internal operations of the organisation, such as financial accountability and delegation of decision-making, seem difficult for many respondents, who frequently answer ‘don’t know.’) Third, a large number of ‘don’t know’ responses may imply that respondents did not understand certain questions.

• The rating choices provided in the MOPAN survey may not be used consistently by all respondents, especially across the many cultures involved in the MOPAN assessment. One potential limitation is ‘central tendency bias’ (i.e., a tendency in respondents to avoid extremes on a scale). Cultural differences may also contribute to this bias as respondents in some cultures may be unwilling to criticise or too eager to praise.

• Because one of MOPAN’s intentions is to merge previously existing assessment tools into one, and to forestall the development of others, the survey instrument remains quite long.

Data Analysis

• While the document review can evaluate the contents of a document, it cannot assess the extent to which the spirit of that document has been implemented within the organisation (unless implementation is documented elsewhere).

• Mean scores are used in the MOPAN reports to provide central tendency values of the survey results. The mean has the advantage of being the most commonly understood measure of central tendency, however, there is a disadvantage in using the mean because of its sensitivity to extreme scores (outliers), particularly when population samples are small. The assessment team reviewed the median and standard deviations in analysing the survey results. Volume II, Appendix V provides the standard deviations for each survey question.

Basis for Judgment

• Although MOPAN uses recognised standards and criteria for what constitutes good practice for a multilateral organisation, such criteria do not exist for all MOPAN indicators. As a result, many of the criteria used in reviewing document content were developed by MOPAN in the course of the assessment process. The criteria are a work in progress and should not be considered definitive standards.

• The Common Approach assessment produces numerical scores or ratings that appear to have a high degree of precision, yet can only provide general indications of how an organisation is doing and a basis for discussion among MOPAN members, the multilateral organisation, and other stakeholders, including direct partners.

Despite some limitations, the Assessment Team believes that the MOPAN reports generally provide a reasonable picture of systems associated with the organisational effectiveness of multilateral organisations.

3. Main Findings

3.1 Introduction

This chapter presents the findings of the 2012 MOPAN assessment of GAVI. Findings are based on respondent survey data and document review.

• Section 3.2 presents overall ratings on the performance of GAVI and summarises respondent views on its primary strengths and areas for improvement;

• Section 3.3 provides findings on each of the four areas of performance (strategic, operational, relationship, and knowledge management).

3.2 Overall Ratings

This section provides a summary of overall ratings and includes: survey ratings of GAVI’s overall organisational effectiveness, survey respondent views on GAVI’s strengths and areas for improvement, and survey and document review ratings for all key performance indicators.

Survey ratings of GAVI’s organisational effectiveness

MOPAN has defined “organisational effectiveness” as the extent to which a multilateral organisation is organised to support partners in delivering expected results. Respondents were asked the question: “How would you rate the overall organisational effectiveness of GAVI?”

The responses are illustrated graphically below.

Figure 3.1 Overall Ratings of GAVI Organisational Effectiveness by Respondent Group*

*The respondent groups are the following: other GAVI Alliance partners and peers (Other GAP); global- level implementation partners (IP-Global) which include UNICEF, WHO, and the World Bank; country- level implementation partners (IP-Country) which include UNICEF, WHO, and the World Bank; supported governments and civil society organisations (GOV/CSO); MOPAN donors in-country (Donors in-Country); and MOPAN donors at headquarters (Donors at HQ).

Respondents’ Views on GAVI’s Strengths and Areas for Improvement

The survey included two open-ended questions that asked respondents to identify GAVI’s greatest strengths and areas of improvement. All 102 respondents provided responses to both questions. The comments are summarised below to illustrate the range of perceptions; they have been qualified by the approximate proportion of respondents holding such a view.

Respondents in all categories consider GAVI’s greatest overall strengths to be its mandate, its mission, and the range, modality, and scope of the support it provides to developing countries.

GAVI was commended for its mandate by 51 per cent of respondents (52). A very high percentage of global-level implementing partners (91 per cent, or 10 respondents out of 11) and of other GAVI partners and peers (92 per cent, or 11 respondents out of 12) noted its mandate as being a strength. Several aspects of GAVI’s mandate were noted by respondents. These include its partnership model, which brings together a variety of actors (from public and private sectors, and developing and developed countries) and is seen as an opportunity to coordinate interventions related to immunisation as well as to build on each partner’s comparative advantage; its business model, which provides considerable secure and predictable funding for immunisation; and its innovative financing mechanisms.

GAVI’s mission was recognised as a strength by 42 per cent of respondents (43). More than half of donors at headquarters (68 per cent, or 15 out of 22) and of global-level implementing partners (55 per cent, or 6 out of 11) commended GAVI for its mission. GAVI’s aim to save children’s lives by focusing on immunisation, its focus on the poorest countries and on underserved children, and its specific mandate on immunisation, are all elements of its mission that were mentioned by respondents.

According to 27 per cent of respondents (28), GAVI’s type of support is one of its greatest strengths. This view was expressed particularly by country-level respondents (44 per cent of donors in-country, 38 per cent of government/CSO representatives, and 36 per cent of implementing partners). A number of aspects were commended by respondents vis-à-vis GAVI’s support, including its wide range (technical support, new vaccines, health systems strengthening, in-kind and cash-based); its modality (flexible, easily accessible, un-earmarked for the reward system, responsive) and its scope (country-wide and long-term).

Respondents had diverse views on the areas in which GAVI needs to improve. Among some respondent groups, the strengthening of GAVI’s country support and partnerships emerged as two areas for improvement. No other particular trend was identified among respondent groups.

Across respondent groups, 40 per cent of respondents (41) pointed to the need for GAVI to improve its support to countries. More specifically, the majority of government/CSO respondents (77 per cent, or 20 out of 26) and half of country-level implementing partners (11) commented on this area. Respondents suggested increasing in-country/regional presence; improving interaction/communication with recipient countries (be more responsive, have more direct communication, broaden the range of interlocutors); increasing GAVI’s knowledge of countries; and improving the management of its support (such as more agile disbursement and application procedures and clarifying eligibility criteria).

The other area for improvement identified by 25 survey respondents was the strengthening of GAVI’s partnerships. This was mentioned in particular by global-level implementing partners (64 per cent, or 7 out of 11), donors in-country (44 per cent, or 4 out of 9) and other GAVI partners and peers (42 per cent, or 5 out of 11). Respondents suggested that GAVI should involve partners to a greater extent, and coordinate and communicate better with them.

Overall Ratings of Key Performance Indicators

Figure 3.2 below shows scores from the document review and the survey on key performance indicators (KPIs) in the MOPAN 2012 assessment of GAVI. The grey bar presents the survey score, while the black square presents the document review score. For example, on the first indicator, “providing direction for results”, GAVI received a score of 4.37 (adequate) in the survey and 5 (strong) in the document review.

In the overall ratings from the survey and document review, GAVI was seen to perform adequately or better on the majority of key performance indicators.

GAVI received scores of adequate or better on all 18 KPIs assessed in the survey.

GAVI received scores of adequate or better on 14 of the 16 KPIs assessed in the document review.[10]

The survey and document review ratings differed on ten KPIs – seven of which were rated higher by the document review than by survey respondents, and the opposite for the remaining two. The reasons for these differences are discussed in the following sections.

Figure 3.2 Overall Ratings on Key Performance Indicators (mean scores, all respondents and document review ratings)

3.3 GAVI’s Performance in Strategic, Operational, Relationship, and Knowledge Management

3.3.1 Overview

This section presents the results of the 2012 Common Approach assessment of GAVI in four performance areas (quadrants): Strategic, Operational, Relationship, and Knowledge Management.

The following sections (3.3.2 to 3.3.5) provide the overall survey and document review ratings for the KPIs in each quadrant, the mean scores by respondent group, and findings based on an analysis of survey and document review ratings in each quadrant.

When there were notably divergent ratings between survey respondent groups or between the survey results and document review ratings, these are noted and the information gleaned from interviews with staff is integrated when it has a bearing on the analysis. Where statistically significant differences among categories of respondents were found, these differences are noted.

The survey data for each KPI and MI by quadrant are presented in Volume II, Appendix V. The document review ratings are presented in Volume II, Appendix VI.

3.3.2 Strategic Management

Surveyed stakeholders and the document review rated GAVI adequate or strong in all areas of strategic management.

Figure 3.3 shows the overall survey and document review ratings for the four KPIs in the strategic management quadrant.

Survey respondents rated GAVI as adequate overall, and the document review rated it strong for providing direction for results and for its country focus on results.

Figure 3.3 Quadrant I: Strategic Management, Survey and Document Review Ratings

Figure 3.4 shows the mean scores for the four KPIs for all survey respondents, and by category of respondent.

Figure 3.4 Quadrant I: Strategic Management, Mean Scores by Respondent Group

KPI 1: Providing Direction for Results

Finding 1: Survey respondents rated GAVI’s executive management as adequate in providing direction for the achievement of external / beneficiary focused results. The document review rated GAVI as strong in making key documents available to the public.

Overall, survey respondents rated GAVI as adequate on the three MIs in this KPI. The document review, which only rated MI 1.3 on availability of documents, gave GAVI a score of adequate.

GAVI was seen to be providing adequate direction for results in terms of supporting a results- orientation and partner focus and in showing the required leadership on results management. The Agency’s practice of making key documents available to the public was considered strong. Since its establishment, GAVI has made important efforts to promote transparency by, among other things, publishing a broad range of key documents on its website.

Figure 3.5 KPI 1: Providing Direction for Results, Ratings of Micro-Indicators

MI 1.1 – Value system supports results-orientation and partner focus

On this MI, survey respondents rated GAVI as adequate for having a culture that reinforces a focus on results and that is focused on implementing countries (governments and supported CSOs). In their narrative comments, survey respondents recognised GAVI’s focus on results as one of its greatest strengths and commended GAVI’s mission of saving children’s lives in the poorest countries.

On both questions, almost all of the MOPAN donors at headquarters provided GAVI with ratings of adequate or better in having an institutional culture that reinforces a focus on results and that is focused on implementing countries (governments and supported civil society organisations where applicable).

Other respondent groups provided a higher percentage of ratings of inadequate or below: between 9 and 30 per cent of responses by respondent group on the first question on GAVI’s results-oriented institutional culture, and between 20 and 45 per cent on the second question on the extent to which GAVI’s institutional culture is focused on implementing countries.

MI 1.2 – Leadership on results management

MOPAN donors at headquarters, implementation partners at the global level, and other GAVI Alliance partners and peers (referred to as ‘Other GAP’ in the charts) were asked about the extent to which GAVI’s senior management shows leadership on results management. Overall, these respondent groups rated the organisation in the range from adequate to strong.

The other GAVI partners and peers group expressed mixed views on this MI, with 42 per cent rating GAVI as inadequate or lower and 58 per cent as adequate or higher. The resulting average score for this respondent group was notably lower than the average scores of the other respondent groups and the difference is statistically significant. On the other hand, the average score of MOPAN donors at headquarters was statistically significantly higher than the average scores of the other respondent groups.

MI 1.3 – Key documents available to the public

All respondent groups other than MOPAN donors at headquarters were asked whether GAVI makes key documents readily available to its stakeholders. At least half of respondents in each group rated GAVI as adequate or above on this MI.

The document review rated GAVI as strong on this MI. Since its establishment GAVI has made important efforts to promote transparency by publishing a broad range of key documents on its website (e.g., strategic plans, governance documents, financial information, evaluations and corresponding management responses) and by providing clear procedures on how to contact the organisation in case of enquiries. In addition, GAVI recently developed an access to information policy that identifies all the documents to be posted on its website and the policy is now available on its website. The availability of GAVI’s key documents in languages other than in English is subject to the Working Languages Policy, which identifies the categories of key documents requiring translation into French and states that other documents may be translated into other languages as deemed appropriate by the relevant Managing Director in the specific circumstances. However, the review of the GAVI website revealed that not all the documents identified in the policy have been translated into French. Translation of key documents has been recognised by GAVI staff as a challenge for the organisation given its relatively small size but broad scope of its work.

KPI 2: Corporate Focus on Results

Finding 2: GAVI was recognised by survey respondents for having a strong focus on the achievement of results. The document review found GAVI strong in operationalising its mandate and comparative advantage through a corporate strategy, and in promoting results-based management within the organisation. However, further efforts are required in articulating and operationalising its results-based approach.

Overall, survey respondents rated GAVI as strong on four of the seven MIs in this KPI, and adequate on three. The document review ratings varied from inadequate to strong.

Conclusions are more difficult to draw on the two MIs relating to the existence of a logical results chain and of adequate performance indicators. The document review was less positive than survey respondents on both aspects, rating GAVI inadequate on both micro-indicators.

Figure 3.6 KPI 2: Corporate Focus on Results, Ratings of Micro-Indicators

MI 2.1 – Position in international aid architecture

Survey respondents were almost unanimous in considering GAVI as appropriately positioned in the international aid architecture, with no less than 87 per cent of respondents in each group rating GAVI as adequate or above. Statistically significant differences were found between the survey responses of MOPAN donors at headquarters and MOPAN donors in-country, with the first being more positive and the second less positive than the other respondent groups.

On the following MIs 2.2 to 2.7, only global level respondents were surveyed (i.e., MOPAN donors at headquarters, implementing partners at the global level, and other GAVI partners and peers).

MI 2.2 – Organisational strategy based on clear mandate

Survey respondents were asked whether GAVI's Strategy and Business Plan is based on a clear definition of mandate and whether it rests upon a clear understanding of its comparative advantage. A large majority (82 per cent or more) agreed with these statements. Statistically significant differences were found between the responses of other GAVI Alliance partners and peers (more negative) and donors at headquarters (more positive) on both survey questions.

The document review rated GAVI very strong on this MI. The programme outlined in the GAVI Alliance Strategy and Business Plan 2011-2015 is clearly aligned with the organisation’s mandate of supporting the improvement of vaccination and immunisation in the poorest countries. The design and delivery of the plan are very strongly based on GAVI’s comparative advantage, which relies upon the organisation’s partnership and business models. In their narrative comments on the organisation’s greatest strengths, this aspect was recognised by respondents as one of GAVI’s strengths.

MI 2.3 – Structure suited to mandate and results delivery

Overall, more than 75 per cent of survey respondents rated GAVI as adequate or above on having a structure suited to its mandate and the delivery of results. However, there were notable differences across respondent groups: 58 per cent of other GAVI partners and peers view GAVI’s institutional architecture as suited to the delivery of results versus 91 per cent in other respondent groups. (This is not statistically significant.)

MI 2.4 – Organisational policy on results management

Global-level respondents were asked whether GAVI ensures the application of results management across the organisation. MOPAN donors at headquarters and implementing partners at the global level had similar views, with a very high percentage rating this MI as adequate or above (95 per cent and 100 per cent, respectively). However, other GAVI partners and peers were less positive, with 42 per cent judging GAVI’s performance as inadequate or weak and 33 per cent as adequate or strong. Statistically significant differences were found between the responses of other GAVI Alliance partners and peers (more negative) and donors at headquarters (more positive).

The review of documents found that GAVI strongly promotes a results-based management culture within the organisation as well as externally. Over the past five years, GAVI has put in place a system of funding, monitoring, and reporting based on performance. Taking into consideration the findings of the Second GAVI Evaluation (2006-2010),[11] which noted some areas for improvement in terms of implementing partners’ accountability to the whole Alliance, GAVI has strengthened partners’ accountability by institutionalising quarterly reporting on the objectives identified in the 2011-2015 Business Plan and by introducing results-based contracting and performance-based funding. GAVI has also promoted an RBM culture externally by requesting countries that apply for support to identify expected results to monitor and to report on an annual basis as a condition for subsequent funding. To coordinate these performance management processes, GAVI has recently established a performance management unit. In order to advance an RBM culture within the organisation, GAVI might consider providing opportunities for capacity building on RBM to its staff. For instance, in interviews conducted with staff it emerged that they are facing some challenges in developing and monitoring performance indicators for some programmes.

MI 2.5 – Plans and strategies contain results frameworks

Survey respondents were asked two questions about the existence of explicit development and management results in GAVI’s Strategy and Business Plan. The distribution of responses was similar to the micro-indicators above: MOPAN donors at headquarters and implementing partners at the global level had similar views, with a very high percentage rating GAVI as adequate or above (89 per cent and 95 per cent, respectively). Among other GAVI partners and peers, 71 per cent of respondents judged GAVI as adequate or above and the remaining 29 per cent as inadequate or weak.

GAVI has developed an organisation-wide results development framework to support the implementation of the 2011-2015 Strategy and Business Plan. The framework contains both outcome and output statements. GAVI has also developed a management results framework, called Corporate KPIs, which includes indicators but not results statements. As a result, the document review provided a score of adequate on this MI.

MI 2.6 – Results frameworks link outputs to final outcomes/impacts

Respondents were asked for their views on GAVI’s results chain and whether there are causal links between the different levels of development results identified in GAVI’s Strategy and Business Plan (i.e., outputs, outcomes, and impacts). GAVI was rated adequate by 38 per cent of respondents and strong or very strong by 34 per cent. Among respondents that rated GAVI’s results chain as inadequate or weak, there were differences between respondent groups: 42 per cent of other GAVI partners and peers and 18 per cent of implementing partners at the global level. No MOPAN donors at headquarters rated this MI as inadequate or weak.

The analysis of the results framework included in GAVI’s Strategy and Business Plan 2011-2015 concluded that the results chain is inadequate. The logical progression from outputs to outcomes to impacts is unclear. It was noted above that GAVI’s management results framework (called Corporate KPIs) does not have results statements, but only indicators. Thus the Assessment Team provided a rating of inadequate on this MI.

MI 2.7 – Plans and strategies contain performance indicators

Overall, survey respondents agreed that GAVI’s Strategy and Business Plan includes measurable performance indicators at the different levels of results, with no less than 73 per cent of respondents in each group providing a rating of adequate or better.

However, the review of the development results framework in GAVI’s Strategy and Business Plan revealed that performance indicators inadequately fulfil the CREAM criteria for indicators: clear, relevant, economic, adequate, monitorable.

KPI 3: Focus on Thematic Priorities

Finding 3: Overall, survey respondents indicated that GAVI maintains focus on its four cross-cutting thematic priorities, particularly in promoting good governance (anti-corruption and transparency). The document review rated GAVI strong in both gender and good governance, adequate in environment, and weaker in human rights-based approaches.

Overall, 69 per cent of survey respondents rated GAVI adequate or better on this KPI, although there were differences among the six respondent groups. Good governance was seen as the strongest area, with 81 per cent of survey respondents rating GAVI adequate or better and 47 per cent assigning a rating of strong or very strong. The document review found GAVI strongest in mainstreaming gender and promoting good governance.

Despite the document review’s lower rating on human rights-based approaches, there is evidence that GAVI is promoting the principle of equity.

Figure 3.7 KPI 3: Focus on Thematic Priorities, Ratings of Micro-Indicators

MI 3.1 – Gender equality

The majority of survey respondents (72 per cent) rated GAVI as adequate or better in mainstreaming gender equality in its work. The percentages ranged from 55 per cent among MOPAN donors in-country to 92 per cent of other GAVI partners and peers. MOPAN donors in- country and implementing partners at the global level were the least familiar with the question, with 40 per cent and 36 per cent respectively answering ‘don’t know’.

The document review confirmed that GAVI has a strong focus on mainstreaming gender across the organisation and in its work. Since the adoption of a gender policy in June 2008, GAVI has put in place a number of organisational and programmatic systems to mainstream gender as well as accountability mechanisms to ensure monitoring and continuous improvement of mainstreaming efforts. These include the establishment of an external Gender Help Desk mandated to provide support to the Secretariat (through research, training, etc.); the establishment of a gender working group within the Secretariat to monitor the implementation of the gender policy; the inclusion, in the Corporate KPIs, of a KPI related to gender; the request to applicants and implementers to provide gender-related information on access to immunisation services in country proposals and annual progress reports.

However, the accountability mechanisms for gender mainstreaming in GAVI-supported programmes do not appear to be strong enough to ensure effective monitoring or continuous improvement of mainstreaming efforts. In addition, in terms of capacities to mainstream gender into its programmatic work, it bears noting that one major challenge GAVI faces is the limited capacity of some supported countries to collect and monitor sex-disaggregated data that shows the difference in immunisation rates between boys and girls. GAVI may also need to better understand the extent of gender-related barriers to immunisation services (for example, male service providers who may affect the immunisation rate of both boys and girls).

MI 3.2 – Environment

Overall, survey respondents rated GAVI as adequate on this MI. However, respondent groups had different views: The majority of country-level respondents (MOPAN donors in-country, representatives of governments and CSOs, and implementing partners in-country) rated GAVI adequate or better in addressing environmental issues in its work (77 per cent). Approximately 53 per cent of global level respondents (implementing partners and other GAVI partners and peers) rated GAVI inadequate or below. Among MOPAN donors at headquarters, 41 per cent responded ‘don’t know’. The difference in the views of global-level respondents and country- level respondents may be due in part to different concepts around environmental issues, with global-level respondents viewing environmental issues as going beyond waste disposal management.

The document review rated GAVI adequate for its capacity to address environment-related considerations. GAVI does not have a formal policy for environment but relies on the environmental policies of WHO and UNICEF, the implementing partners on the ground. As a result of a benchmarking exercise to better understand what other organisations and partners have in place to address environmental issues, GAVI realised that developing a stand-alone environmental policy would be difficult for the organisation to administer and monitor. GAVI identifies environment as a cross-cutting priority and supports its mainstreaming by supporting the development, capacity-building and implementation of national immunisation safety and immunisation waste management frameworks within the broader health support plans to address safety. Nonetheless, environment remains an area for improvement for GAVI, as noted by two recent evaluations,[12] which noted the inadequate support provided by GAVI to countries for waste management of syringes.

MI 3.3 – Good governance

Respondent groups had generally positive views on the extent to which GAVI promotes anti- corruption and transparency among grant recipients and partners. Ratings of strong or very strong were provided by a high percentage of respondents in each group, especially by other GAVI partners and peers (50 per cent), representatives of governments and CSOs (70 per cent), and MOPAN donors at headquarters (68 per cent). The majority of implementing partners in-country (74 per cent) rated GAVI adequate or above on this MI.

The document review confirmed that GAVI promotes good governance in its activities. Since 2006 GAVI has put in place systems to prevent corruption and promote transparency (e.g., policy on anti-corruption and transparency, transparency and anti-corruption team, Internal Audit) at both the Secretariat and country level. Several results identified in its Strategy and Business Plan 2011-2015 make reference to good governance principles. In addition, GAVI- supported programmes such as the Health Strengthening Systems support, which aims to strengthen countries’ planning, budgeting, monitoring and evaluation systems, indirectly contribute to good governance. Internally, GAVI promotes transparency, a fundamental principle of good governance, within the organisation itself through the publication of a broad range of documents (e.g., Board meeting minutes, evaluations, and management responses to evaluations).

MI 3.4 – Human rights-based approaches

The majority of survey respondents in five of the six respondent groups agreed that GAVI applies human rights-based approaches in its work, with at least 67 per cent of each group rating GAVI as adequate or better on this MI. However, among implementing partners at the global level, 45 per cent rated GAVI inadequate or weak and only 36 per cent considered GAVI adequate or better. There was also a relatively high percentage of ‘don’t know’ responses among respondents of this group (18 per cent).

According to MOPAN criteria,[13] GAVI inadequately mainstreams human rights-based approaches in its work. It does not have a strategy or policy to ensure the mainstreaming of these approaches, nor has it identified roles and responsibilities or carried out expenditure review/costing and budgetary allocation for the implementation of mainstreaming activities. Nevertheless, the document review found that GAVI does apply some aspects of human rights- based approaches to its work. The concept of equity, for example, is integral to GAVI’s work. GAVI targets three dimensions of equity (wealth equity, geographic equity, and gender equity) and these are included in its organisation-wide results framework. The need for a comprehensive human rights policy that includes these aspects and others, such as participation, could be an issue for discussion between MOPAN and GAVI.

KPI 4: Country Focus on Results

Finding 4: Overall, GAVI is seen by both the survey respondents and the document review to support a results-focus in the programmes it funds, particularly in aligning its support with national strategies for immunisation. Survey respondents were more positive than the document review about the adequacy of performance indicators included in GAVI’s results frameworks.

GAVI requires applications for support to include performance indicators to track progress on immunisation-related results as well as results that are aligned with the country multi-year plan for immunisation.

GAVI was rated as strong by survey respondents on three of the five MIs in this KPI. The document review ratings ranged from adequate to very strong.

Figure 3.8 KPI 4: Country Focus on Results, Ratings of Micro-Indicators

MI 4.1 – Frameworks link results at district and national levels

Global level respondents (i.e., MOPAN donors at headquarters, implementing partners at global level, and other GAVI partners and peers) were asked to assess this MI. With the exception of other GAVI partners and peers, the majority of respondents (between 64 and 94 per cent) consider that GAVI approves applications that link results at appropriate levels. The views expressed by other GAVI partners and peers were more divided, with 42 per cent rating GAVI as adequate or above and 42 per cent as inadequate or below. A high percentage of ‘don’t know’ responses, 32 per cent, was provided by MOPAN donors at headquarters.

MI 4.2 – Frameworks include relevant indicators

Survey respondents were very positive on GAVI’s inclusion of performance indicators at applicable levels (district and national) in applications approved, with the percentage of strong and very strong responses ranging from 32 to 83 per cent on this MI. Ratings of inadequate or lower were given by 13 per cent or less of the respondent groups at the country level.

The document review of approved applications from sample countries found that the performance indicators in these applications were adequate to monitor and report on the progress made by immunisation programmes implemented with GAVI support. The indicators are, in fact, relevant and adequate to monitor progress at the outcome or impact level (e.g., improvement in immunisation coverage, reduction of child mortality rate, etc.). However, they could be improved by showing GAVI’s contribution to countries’ efforts towards increased immunisation. This is a challenging area for GAVI given that it relies upon countries to collect and report on data, but, at the same time, needs to meet donor requests to demonstrate its contribution to countries’ immunisation-related outcomes. Attribution is an eternal problem affecting all donors and programmes. Without putting excessive burden on countries for reporting, GAVI might consider adding qualitative indicators that could help describe GAVI’s contribution to the overall achievements.

MI 4.3 – Expected results consistent with MYP and/or national strategies

Two questions were asked in relation to this MI: whether approved applications include statements of expected results that appear in the multiyear plan for immunisation, and whether approved applications include statements of expected results that are consistent with the national health plan or strategy. Overall, 83 per cent of survey respondents rated GAVI as adequate or above on this MI, and 11 per cent as inadequate or below. Among MOPAN donors in-country, 35 per cent responded ‘don’t know’ on this MI.

The review of a sample of applications approved by GAVI supported the survey respondents’ assessment and provided a rating of very strong. The results included in the approved applications are explicitly linked with those in the countries’ multiyear plans.

MI 4.4 – Expected results developed in consultation with partners

Country-level respondents were asked whether approved applications include statements of expected results that are developed by national governments, CSOs (where applicable), together with other partners. Overall, 80 per cent of survey respondents rated GAVI as adequate or above on this MI.

MI 4.5 – Results for cross-cutting priorities included in results frameworks

Overall, 60 per cent of survey respondents rated GAVI as adequate or above on whether applications approved by GAVI include results for gender equality and environment (e.g., waste disposal), as appropriate, and 19 per cent as inadequate or below. A high percentage of MOPAN donors at headquarters (45 per cent) and of donors in-country (25 per cent) provided a ‘don’t know’ answer.

The document review of sampled applications approved found GAVI adequate in including results for the four cross-cutting priorities of gender equality, environment, good governance and human-rights based approaches in the proposals.

3.3.3 Operational Management

Over the past few years, GAVI has made important efforts to improve the systems guiding its operations. In the area of financial accountability, in particular, GAVI’s efforts have led to the development of strong systems and their effective implementation.

GAVI prides itself for having both a ‘light touch’ and for rapidly putting in place its systems for operations, even while they are still under development. As a result, in some areas such as results-based budgeting, there is limited documentation about the operational practices examined by the MOPAN indicators. In other areas, such as the range of topics covered under financial accountability (audit, risk, anti-corruption efforts, etc.) there is particularly strong evidence from documents that is supported by the perceptions of survey respondents.

Figure 3.9 below shows the overall survey and document review ratings for the KPIs in the operational management quadrant.

According to survey results, GAVI performs strongly on three and adequately on three KPIs in operational management. The document review rated GAVI strong on most KPIs in operational management.

Figure 3.9 Quadrant II: Operational Management, Survey and Document Review Ratings

Figure 3.10 shows the mean scores for the KPIs for all survey respondents, and by respondent groups.

Figure 3.10 Quadrant II: Operational Management, Mean Scores by Respondent Group

KPI 5: Resource Allocation Decisions

Finding 5: Survey respondents indicated that GAVI makes transparent and predictable decisions about its financial and other support. Both the survey and document review rated GAVI strong in making readily available its criteria for approving applications for support and making decisions based on independent review.

Survey respondents rated GAVI as strong in all four MIs related to this KPI, and the ratings assigned by document review ranged from adequate to very strong.

Overall, it appears GAVI has put in place a sound and transparent system for the allocation of its support. The system is based on an independent peer review of applications received from countries and guarantees that decision making is based on the results of the peer review.

Once decisions on resource allocations are made, GAVI generally delivers its support in a timely and predictable way, although document sources suggest that financial management procedures for the Health Strengthening Support are still an area for improvement.

Figure 3.11 KPI 5: Resource Allocation Decisions, Ratings of Micro-Indicators

MI 5.1 – Criteria for support publicly available

When asked if GAVI makes readily available its criteria for approving applications for support, the majority of survey respondents agreed that the criteria are accessible with 87 per cent rating GAVI’s performance in this area as adequate or higher. (MOPAN implementing partners at the global level and other GAVI partners and peers were not asked this question.)

The document review gave GAVI a rating of very strong on this MI. The system for the allocation of resources is defined in a suite of documents available on the GAVI website, including the country eligibility policy, the application guidelines for support, and the terms of reference of the Independent Review Committees (IRCs).[14] While the criteria to apply for support vary according to the type of support, one requirement common to all types of support is that the country must be eligible as per the GAVI Country Eligibility Policy. In addition to eligibility criteria, there are programme filters determining the conditions under which a country can request a specific type of support. In 2010 GAVI established a prioritisation mechanism that is applied when funding is not available for all the applications received that comply with the eligibility criteria and programme filters.

MI 5.2 – Approvals for support conform to stated criteria

MOPAN donors at headquarters, government/CSO representatives, and implementing partners at the country level were asked if GAVI’s approval for support conforms to its stated criteria.

The majority of respondents (87 per cent) rated GAVI as adequate or higher.

The review of IRC reports on applications received indicated that GAVI is adequate in this area and there is evidence that the IRC reviews country applications according to the criteria established by GAVI. However, the quality and comprehensiveness of the reports varies. In some cases, for example, there is no clear justification provided for the approval of support or there is no mention of the quality of information provided by the country on the social and gender-related barriers to health. The adequate rating is based on how information is reported by the IRC and does not necessarily mean that IRC assessments and recommendations do not comply with the established criteria. The IRC should be encouraged to provide more comments in their reports on applications that are recommended for approval and those that are not. This will be reviewed as part of the Grant Approval and Monitoring Redesign process.

MI 5.3 – Decisions on support based on independent review

MOPAN donors at headquarters, government/CSO representatives, and implementing partners at the country level also shared similar views on whether GAVI’s decision making on financial and other support is based on independent review, with 88 per cent of respondents providing a rating of adequate or higher.

The document review rated GAVI strong on this MI. The documents reviewed indicate that the IRCs review applications according to GAVI’s established criteria for the allocation of support and that their recommendations are taken into consideration by the GAVI Alliance Board. In addition, IRC members are requested by GAVI to sign a confidentiality and conflict-of-interest statement, which is also a guarantee of the independence of the review. Some concerns were raised by the 2009 Review of the IRCs[15] vis-à-vis the selection process of IRC members, the independence of the pre-review process, and the adequacy of the terms of reference of the IRC Monitoring Team. There are indications that GAVI has taken the recommendations of this review very seriously and has adopted measures to address them. However, it has not yet conducted a follow-up to ensure that these measures were sufficient.

MI 5.4 – Resources released according to agreed schedules

When asked if GAVI’s planned resources are released according to agreed schedule when preconditions are met (e.g., transparency requirements and performance targets), 89 per cent of respondents rated GAVI adequate or higher. (Implementing partners at the global level and other GAVI partners and peers were not asked about this MI.)

The document review found that GAVI has made adequate progress and has improved the predictability and delivery of funding. In terms of delivery, the International Health Partnership (IHP+) Survey rated GAVI positively, finding that 97 per cent of health sector aid disbursements provided by GAVI were released according to agreed schedules in 2009. The only area where some concerns were raised by the Second GAVI Evaluation and the Monitoring Independent Review Committee (IRC) regarding the predictability and delivery of funding was in the Health Strengthening Support (HSS) programmes, where delays were largely attributed to the financial management procedures under GAVI’s Transparency and Accountability Policy (TAP) requirements. The predictability of funding has been enhanced by GAVI’s funding arrangements with the International Finance Facility for Immunisation (IFFIm) and the Advance Market Commitment (AMC).[16]

KPI 6: Linking Support to Performance

Finding 6: Survey respondents considered GAVI strong in linking its allocations to expected results and in linking disbursements of funds and vaccine support to reported results. The document review found GAVI stronger in linking allocations to expected results than in linking disbursements to reported results.

The survey questions relating to this KPI were asked only of MOPAN donors at headquarters, who were positive on GAVI’s capacity to link allocations and disbursements to performance information.

The document review found that GAVI adequately links its budget allocations to expected results, but the link between disbursements and reported results is not shown in reports to the GAVI Alliance Board). GAVI has taken two important steps towards performance-based budgeting. First, it has revised the budget structure to include programmatic costs and Secretariat-related costs in one unified budget, meaning that there is now a closer link between programmatic costs and operational costs. Second, GAVI has introduced performance-based funding at the country level, which links GAVI cash-based support to country immunisation outcomes.

Figure 3.12 KPI 6: Linking Support to Performance, Ratings of Micro-Indicators

MI 6.1 – Allocations linked to expected results

MOPAN donors at headquarters were asked whether GAVI links funding and vaccine support to countries’ expected results. The majority (90 per cent) agreed that it does, and 64 per cent rated GAVI as strong or very strong.

The document review assessed whether GAVI’s organisation-wide budget allocations are linked to expected development results and found GAVI adequate. Since 2010, GAVI’s Business Plan Budget consists of two components: Programme Implementation and Mission Support.[17]

• Programme Implementation costs include the costs of activities to be implemented under each of the four strategic goals identified in the 2011-2015 Strategy and Business Plan.

• Mission Support costs include non-programmatic Secretariat costs (e.g., staffing, professional fees, facility and office costs/supplies and minor equipment, and capital expenditures).

Programme implementation costs are estimated by GAVI through a costing exercise with its partners on an annual basis. Since 2009, a validation process of the work plans takes place and includes an assessment of the proposed activities and budget for a two-year period by an independent committee composed of “individuals with a high level of familiarity with GAVI but with no potential or perceived conflict of interest in funding decisions.”

The document review found that GAVI’s costing exercise has improved over the years, with the introduction of the validation system and the alignment of the bi-annual business plan budget with the strategic goals and programme objectives identified in the 2011-2015 Strategy and Business Plan.

MI 6.2 – Disbursements linked to reported results

The majority of MOPAN donors surveyed shared the view that GAVI disbursements and other support link well with reported results, with 77 per cent rating GAVI as adequate or better on this MI. The remaining 23 per cent responded ‘don’t know’.

The document review looked at whether the organisation’s annual reports make explicit links between disbursements and reported results and rated GAVI as weak. The reports to the GAVI Alliance Board make a link between disbursements and progress towards results at the strategic goal level, but not for lower levels of results (i.e., programme objectives and programme deliverables). In GAVI’s annual progress reports, the link between disbursements and progress towards results is not reported at any level. Reports reviewed also do not present information on the variances in operational expenditures and results achievement.

KPI 7: Financial Accountability

Finding 7: Overall, GAVI’s financial management and accountability systems were considered an area of strength, with the majority of ratings provided by survey respondents and the document review in the range of strong to very strong.

Survey respondents rated GAVI as strong in five of the eight MIs they were asked to assess, and the document review provided ratings of strong or very strong on eight of the nine MIs in this KPI.

The document review found that GAVI has robust systems for external and internal audits, risk management (at the applicants/implementers level as well as at the programme level), and procurement and contract management. GAVI’s strong performance in the area of financial accountability seems to be linked to the establishment of an internal audit function and activities aimed at strengthening GAVI’s procedures, particularly in the area of risk management. Initiatives have included the introduction of financial management assessments and an organisation-wide risk register.

Figure 3.13 KPI 7: Financial Accountability, Ratings of Micro-Indicators

MI 7.1 – External financial audits performed across the organisation

According to MOPAN donors at headquarters, the only group asked about this MI, GAVI’s external audits are meeting the needs of the Board. In fact, 82 per cent rated the organisation as adequate or stronger; the remaining 18 per cent responded ‘don’t know’.

The document review confirmed that GAVI’s external financial audits, which have been performed in conformance with generally accepted US accounting principles, are performed across the organisation.

MI 7.2 – Support provided is subject to appropriate external financial audit

All respondent groups other than MOPAN donors in-country were asked about this MI. The majority of respondents (80 per cent) rated this MI as adequate or higher, and a notable percentage (between 43 and 60 per cent) rated GAVI strong or very strong. Twenty-seven per cent of global-level implementing partners answered ‘don’t know’.

The document review rated GAVI strong on this MI. One of GAVI’s requirements for continuation of support is the submission by recipient countries of annual audit reports of the account(s) holding GAVI funds. GAVI requires these audits to be performed using national audit systems and procedures (i.e., performed by the Auditor General of the country). However, the Access to Information Policy does not include external financial audit reports in the list of documents that shall be made available by GAVI. As explained by GAVI, the responsibility to disclose the audit reports rests with the country government. GAVI is not in a position to make public such reports.

MI 7.3 – Guidelines for immediate measures against irregularities

When asked if GAVI appropriately follows up on financial irregularities, including fraud and corruption, 80 per cent of respondents rated GAVI as adequate or above. With the exception of implementing partners at the country level, between 64 and 75 per cent of respondents rated GAVI’s performance in this area as strong or very strong. Implementing partners at the country level were divided on this question, with 49 per cent rating GAVI as adequate or better, 24 per cent as inadequate or below, and 27 per cent answered ‘don’t know’. (Global-level implementing partners and MOPAN donors in-country were not asked their views on this MI.)

The document review found that GAVI has a thorough system to address irregularities identified at the country level. It has a protocol to respond to misuse of cash-based support that details the different phases and steps (but without specifying timelines) to be taken when a possible irregularity is identified by either an external financial audit, a financial management assessment carried out by the GAVI Transparency and Accountability Policy (TAP) Team, or following allegations of misuse in other donor programmes. There is evidence that irregularities are reported to the GAVI Alliance Board. Although these procedures do not apply to in-kind support, GAVI is considering addressing this situation. As reported by GAVI, the process to formalise the management of audit reports received from countries is underway. GAVI has already developed some tools such as: the Audit Report Review Form (ARRF), a checklist for review of audit reports, a sample letter for review of audit reports, and a guidance note to clarify GAVI audit requirements. All these tools are being reviewed and finalised and are expected to be implemented before the end of the current fiscal year.

MI 7.4 – Systems for audit of sub-contractors

MOPAN donors at headquarters, country-level implementing partners, and other GAVI partners and peers were asked their views on this MI.

The majority of respondents (59 per cent) considered that GAVI has appropriate systems in place for the external audit of sub-contracted entities and rated this MI as adequate or higher. There was a high percentage of ‘don’t know’ responses (26 per cent), particularly among MOPAN donors at headquarters (41 per cent) and implementing partners at country level (21 per cent). The remaining 15 per cent of respondents rated this MI as inadequate or below.

This micro-indicator needs to be contextualised to GAVI, which operates more as a coordinating mechanism and therefore processes such as procurement and audit are mostly carried out by its partners (UNICEF, WHO, etc.) and their guidelines (such as the Single Audit Principle in the UN System). As a result, GAVI does not have very developed systems for the audit of sub-contracted entities, with the exception of a procurement policy adopted in 2006 that requires suppliers and consultants to permit applicable audit and performance verification activities as set out in the contract documents. Given these considerations, the document review provided a score of adequate on this MI.

MI 7.5 – Internal financial audit processes provide credible information

MOPAN donors at headquarters, global-level implementing partners, and other GAVI partners and peers were asked their views on this MI.

Both MOPAN donors at headquarters and other GAVI partners and peers considered that GAVI’s internal financial audits provide credible information to the GAVI Board/ Chief Executive Officer (CEO), with over 80 per cent of respondents rating GAVI as adequate or better. Overall, there was a high percentage of ‘don’t know’ responses on this MI (26 per cent), with the highest among global-level implementing partners (55 per cent).

Documents reviewed suggest that internal financial audit processes are used to provide the GAVI Secretariat and Board with credible information. GAVI established an internal audit function in November 2009 and the system appears sound. The function is independent of the Secretariat as the Internal Auditor is appointed by and reports directly to the Board.

MI 7.6 – Policy on anti-corruption

This MI was assessed only through the document review, which found that GAVI has a strong system in place to combat corruption. It has adopted a number of measures to prevent corruption, including a whistle blower policy, financial management assessments for cash- based programmes, and a conflict of interest policy. These measures allow GAVI to manage programmes and services that are compliant with preventing and combating fraud and corruption and to pro-actively support solutions to counter corruption, including at the local level.

MI 7.7 – Effective procurement and contract management processes

All respondent groups other than MOPAN donors in-country were asked about this MI.

The majority of survey respondents shared the opinion that GAVI’s procurement and contract management processes for the provision of vaccines are usually effective, with 84 per cent rating the organisation as adequate or higher. A high percentage of strong and very strong responses were provided across respondent groups (59 per cent), although 23 per cent of MOPAN donors at headquarters responded ‘don’t know’.

GAVI was rated very strong on this MI by the document review. GAVI appears to have timely, efficient and effective procurement and contract management processes for the provision of services or goods. While GAVI relies on UNICEF’s and the Pan-American Health Organisation’s (PAHO) procurement and contract management processes, it has established procurement principles and generic guidelines its partners have to comply with. The principles and guidelines give special attention to issues of efficiency and quality of vaccines. GAVI has reviewed its procurement strategy on a regular basis and tried to improve over the years, in particular the aspects related to its effectiveness. The most recent strategy for vaccine supply and procurement was approved in November 2011.

MI 7.8 – Strategies for risk management

MOPAN donors in-country and government/CSO representatives were not asked their views on this MI.

According to the majority of survey respondents, GAVI has appropriate strategies and plans for risk management, with 66 per cent rating the organisation as adequate or above. GAVI’s strategies and plans for risk management were seen as inadequate or weaker by 18 per cent of respondents, with the remaining 16 per cent of respondents providing a ‘don’t know’ response. Global-level implementing partners had a higher percentage of ‘don’t know’ responses (27 per cent).

GAVI was rated strong on this MI by the document review. GAVI has recently set up, in collaboration with the Internal Auditor, a formal and systematic risk management process, which is described in a suite of documents comprising the risk register/matrix, the Transparency and Accountability Policy, the guidance note on GAVI financial management assessment, and the procedures for handling situations that warrant increased attention and suspected and actual cases of misuse of funds. GAVI has identified risks at different levels: programmatic risks (e.g., misuse of cash grants to countries, excessive wastage of vaccines), resourcing risks (e.g., GAVI not fully funded), operating risks (i.e., risks related to GAVI internal processes such as the grant decision mechanism) and corporate risks (e.g., HR or finance related risks). Roles and responsibilities for risk management are shared among GAVI Secretariat Directors and GAVI’s partners (who monitor and report on a quarterly basis on the corresponding risks), the Monitoring Independent Review Committee (monitoring country-level risks), GAVI Chief Executive Officer (reporting to the Board), the Country Responsible Officers (who conduct financial management assessments of applicants and implementers), and the Internal Auditor (monitoring and evaluation of the risk management processes). There is evidence that major risk analyses are presented to the GAVI Alliance Board and that processes and systems in place are used.

GAVI was rated strong rather than very strong in this area since its focus has been on managing risks related to its cash-based support (which represents approximately 15 per cent of its programming) and not on its in-kind support.

MI 7.9 – Risk analysis of applicants/implementers

All respondent groups other than MOPAN donors in-country were asked about this MI.

The majority of survey respondents (68 per cent) agreed that GAVI ensures a sufficient level of risk analysis/assessment of its applicants and implementers, and 41 per cent rated the organisation as strong or very strong on this MI. Higher percentages of inadequate or weaker responses were provided by global-level implementing partners (27 per cent) and by other GAVI partners and peers (25 per cent) than other respondent groups. Global-level implementing partners provided a high number of ‘don’t know’ responses (27 per cent).

The document review found that GAVI ensures a sufficient level of risk assessment of applicants and implementers. GAVI has in fact put in place systems to ensure that risks identified at the country level and associated with its way of operating through partners are monitored and reported to the GAVI Alliance Board on a regular basis. These systems include country-level financial management assessments and quarterly reporting from global-level implementing partners on the deliverables identified in the 2011-2015 Strategy and Business Plan.

KPI 8: Using Performance Information

Finding 8: The survey and document review rated GAVI strong in allocating resources based on performance information. The document review also found GAVI strong in using performance information to revise and adjust policies and very strong in using it to adjust or plan country-level grant windows.

Survey respondents rated GAVI as adequate on four of the five MIs and strong on one of the MIs in this KPI. The document review scores ranged from adequate to very strong.

GAVI makes use of its performance information at both the corporate level, to revise policies and strategies, and to decide about the funding of new or existing programmes in countries. The organisation’s documentation shows that GAVI is assigning increasing importance to performance information for its work.

Figure 3.14 KPI 8: Using Performance Information, Ratings of Micro-Indicators

MI 8.1 – Using information to revise and adjust policies

MOPAN donors at headquarters, implementing partners at the global level, and other GAVI partners and peers were asked about this MI.

When asked if GAVI uses information on performance to revise its Strategy and Business Plan, 79 per cent of survey respondents considered GAVI adequate or strong. MOPAN donors at headquarters were particularly satisfied with GAVI performance in this area with 50 per cent rating GAVI strong. The views of other GAVI partners and peers were more divided than the other two groups, with 58 per cent rating GAVI as adequate or better and 33 per cent as inadequate or weaker.

GAVI was rated as very strong on this MI by the document review. Documents consulted show that performance information, especially from evaluations, is used by GAVI to revise and adjust organisation-wide policies. GAVI has conducted a number of evaluations and reviews of its programmes, policies and systems and has subsequently modified them on the basis of the evaluation/review recommendations. The most important was the revision of the Strategy and Business Plan based on recommendations from the Second GAVI Evaluation; these included the elevation of market-shaping to the level of a strategic goal and the new performance management system to strengthen the GAVI Secretariat’s and partners’ accountability for the implementation of the business plan.

MI 8.2 – Using information for adjusting or planning grant windows

MOPAN donors at headquarters, implementing partners at the global level, and other GAVI partners and peers were surveyed on this MI.

Respondents were asked two questions with respect to using information for adjusting current grant windows and for designing and introducing new funding windows. Across all respondent groups, the majority (between 58 and 61 per cent) considered GAVI adequate or strong on both aspects, with around 20 per cent considering it inadequate or weak.

On the use of performance information to adjust grant windows, half of GAVI partners and peers rated the organisation as inadequate or weak, and one-third rated it adequate or strong. On the use of performance information to design and introduce new funding windows, 42 per cent rated GAVI inadequate or weak, 33 per cent as adequate or strong, and 17 per cent responded ‘don’t know’.

A relatively high percentage of ‘don’t’ know’ answers were provided on the first question by MOPAN donors at headquarters (23 per cent) and global-level implementing partners (27 per cent).

There is evidence, from the documents reviewed, that GAVI uses performance information to plan new grant windows at the country level. Performance information at the country level is collected through progress reports that countries receiving GAVI support submit on an annual basis. The information is analysed by the IRC Monitoring Team, which provides recommendations to the GAVI Alliance Board on the continuation of support. The performance information is also used by the GAVI Secretariat to update the GAVI Alliance Board on an annual basis. There is evidence that performance information is used to adjust GAVI support to the country, although some concerns were raised by the 2009 Review of the IRCs in terms of follow-up on the IRC Monitoring Team’s recommendations. It also bears noting that the GAVI Secretariat has introduced a new tool (the country scorecard) to monitor performance information and take management decisions on this basis. Since the tool has just been introduced, it is too early to assess its effectiveness.

MI 8.3 – Proactive management of poorly performing initiatives

When asked about GAVI’s management of poorly performing grants, 63 per cent of respondents overall rated the organisation as adequate or stronger and 17 per cent as inadequate or weak. However, the overall percentage masks the opposite views expressed by other GAVI partners and peers, the majority of whom (67 per cent) rated GAVI as inadequate or weak on this MI and 17 per cent as adequate or strong, making the average score for this respondent group notably lower than the average scores of other respondent groups. The difference is statistically significant. A relatively high percentage of ‘don’t’ know’ answers were provided by MOPAN donors at headquarters (23 per cent).

The document review rated GAVI’s capacity to proactively manage poorly performing programmes as adequate. GAVI has a process to review all the grants provided to countries on a yearly basis. The IRC Monitoring Team reviews the annual progress reports submitted by GAVI-supported countries and makes recommendations to the GAVI Alliance Board on whether to renew support or make modifications. While our review of the IRC Monitoring Team’s reports suggests that the Board has followed the IRC’s recommendations, the 2009 Review of the IRCs raised some concerns on GAVI’s performance in responding to and managing the flags identified by the IRC.

MI 8.4 – Evaluation recommendations are acted upon

MOPAN donors at headquarters, implementing partners at the global level, and other GAVI partners and peers were surveyed on this MI.

Only global-level respondents were asked whether GAVI appropriately tracks implementation of evaluation recommendations reported to the Board. MOPAN donors at headquarters and global-level implementing partners were satisfied with GAVI’s performance in this area, with respectively 91 per cent and 64 per cent of respondents rating GAVI as adequate or stronger. Among other GAVI partners and peers, 42 per cent rated GAVI as inadequate or weaker and 33 per cent as adequate or above (33 per cent). Statistically significant differences were found between the responses of other GAVI partners and peers (more negative) and donors at headquarters (more positive). A relatively high percentage of ‘don’t’ know’ answers were provided by global-level implementing partners (27 per cent) and other GAVI partners and peers (25 per cent) on this MI.

The document review rated GAVI as adequate in acting upon evaluation recommendations reported to the GAVI Alliance Board. While there is evidence that GAVI provides management responses and does in effect follow up on many recommendations, it does not have a system or guidelines that require this or that require monitoring of the actions taken.[18]

MI 8.5 – Allocated resources reflect performance

MOPAN donors in-country were not asked their views on this MI.

The survey asked two questions about the link between performance information and allocations to countries. The first asked whether GAVI’s initial grant decisions are based on plans and targets, and the majority of all respondent groups (approximately 70 per cent) rated this MI as adequate or stronger. The second question asked whether GAVI’s subsequent grants are based on performance. More than 75 per cent of all groups, other than GAVI partners and peers, rated the organisation as adequate or stronger. Among other GAVI partners and peers, half rated GAVI as inadequate or weak and 42 per cent as adequate or stronger. A relatively high percentage of ‘don’t’ know’ answers were provided by MOPAN donors at headquarters (27 per cent).

The document review rated GAVI as strong on this MI. Performance at country level is used by GAVI to make decisions on whether to continue support or not. In addition, there is evidence that GAVI is giving increasing importance to performance as a criterion for decision making on country allocations as shown by the piloting of a prioritisation mechanism that is based on performance information, and by the introduction in 2012 of a performance-based funding approach to reward countries that achieve higher levels of immunisation coverage.

For GAVI, this indicator should be interpreted with some caution because performance cannot be the only criterion behind vaccine supply-related decision making. In some cases GAVI has to strike a balance between competing or possibly conflicting objectives. For example, strictly linking allocation of support to performance could decrease the predictability of funding. In addition, technical issues related to vaccine administration need to be considered in any decision that could interrupt vaccine administration (e.g., an interruption in the administration of some vaccines could leave the targeted population less protected than before when there was a degree of natural protection).

KPI 9: Managing Human Resources

Finding 9: GAVI’s human resource management systems are a work in progress, but very few respondents were able to answer questions on this key performance indicator.

Management of human resources is an area that GAVI is working on. It is increasing its focus on results-based management and making efforts to link its performance management processes. In a short period, GAVI has seen a rapid increase in the size of its Secretariat due to the increasing demand for more coordination at the central level, more resources being mobilised and increased demand for support. Because of its small size, all of the processes and systems for staff management are not yet in place.

Overall, survey respondents rated GAVI as adequate on the two micro-indicators of this KPI, but there was a notably high level of ‘don’t know’ survey responses (65 per cent). Readers are cautioned that the ‘adequate’ survey ratings are therefore based on a very small number of respondents and that this is not considered sufficient for a valid judgement. The document review gave GAVI mixed ratings on this KPI.

Figure 3.15 KPI 9: Managing Human Resources, Ratings of Micro-Indicators

MI 9.1 – Results-focused performance assessment systems for senior staff

Only MOPAN donors at headquarters and implementing partners at the global level were asked if the GAVI Secretariat uses results-focused performance assessment for its senior staff. A very small number of respondents provided a rating (12 out of 33), and the majority (21) answered ‘don’t know’ (59 per cent among MOPAN donors at headquarters and 73 per cent among implementing partners at the global level).

GAVI was rated very strong on this MI by the document review. GAVI has put in place a staff performance assessment system (the 360 assessment) that includes senior management and the executive leadership team.

MI 9.2 – Transparent system to manage staff performance

Only MOPAN donors at headquarters and implementing partners at the global level were asked their views on whether the GAVI Secretariat uses a transparent system to manage staff performance. The majority of respondents (61 per cent, or 20 out of 33) responded ‘don’t know’ on this question (55 per cent among MOPAN donors at headquarters and 73 per cent among implementing partners at the global level).

The document review focused on the existence of an incentive and reward system for staff performance, and provided a rating of inadequate as there is no evidence of such a system in place. The GAVI Alliance Human Resources Manual (September 2011) links promotion to the duties and responsibilities performed by staff rather than to staff performance in achieving results, and does not make reference to a reward and recognition system. While GAVI shared with the Assessment Team cases where staff members were promoted or provided with opportunities for training as a result of good performance, there is insufficient evidence to consider this an institutionalised practice. Interviews conducted with GAVI staff and the documents reviewed suggest that the organisation is considering establishing a reward and recognition system linked to staff performance.

KPI 10: Performance-oriented Programming

Finding 10: Survey respondents and document review considered that GAVI-supported programming processes at the country level are performance-oriented and subject to benefits and risks analyses.

Survey respondents and document review assessed GAVI as adequate in subjecting applications to benefits and risks analyses. Survey respondents and the document review rated GAVI as strong and adequate respectively for applications that include monitoring targets.

Figure 3.16 KPI 10: Performance-oriented Programming, Ratings of Micro-Indicators

MI 10.1 – Applications subject to benefits analysis and risk analysis

MOPAN donors at headquarters and implementing partners at the country level were asked their views on this MI.

The survey asked two questions on the degree to which applications for support are subject to benefits analysis and risk analysis (e.g., environmental, economic, social, etc.). The majority of respondents rated GAVI adequate or higher in these two areas (65 per cent and 66 per cent respectively), and only a small percentage as inadequate or below (11 per cent and 6 per cent respectively). There was a relatively high number of don’t know responses on these questions: On the question on benefits analysis, 41 per cent of MOPAN donors at headquarters responded don’t know. On the second question on risk analysis, 27 per cent of both groups responded don’t know.

The document review found that GAVI conducts benefit and risks analysis of applications received. More specifically, the IRC reviews applicants’ cold chain and logistic capacity, as well as their readiness to accommodate new vaccines. However, the document review did not find evidence that specific guidelines are provided to the IRCs to conduct such analyses.

MI 10.2 – Applications include milestones/targets to monitor implementation

MOPAN implementing partners at the global level and other GAVI partners and peers were not asked their views on this MI.

The majority of survey respondents (81 per cent) agreed that applications for support submitted to GAVI include milestones/targets to monitor implementation progress. The responses of MOPAN donors at headquarters and government/CSO representatives were more positive than those of other respondent groups and these differences were statistically significant. However, 32 per cent of MOPAN donors at headquarters responded ‘don’t know’.

The document review rated this MI as adequate. The sampled applications reviewed included baseline data and targets for monitoring and reporting on progress at the outcome/impact level. However, the applications did not include milestones or targets for the activities described. It is therefore unclear how GAVI can monitor the progress of specific activities supported through its funding windows and address potential country-level performance problems.

3.3.4 Relationship Management

Relationship management was considered an area of strength for GAVI by survey respondents due to its support of country ownership and its alignment and harmonisation of arrangements and procedures with partners. The document review found GAVI strong in applying conditionality and harmonising procedures with its partners.

Stakeholders surveyed and documents reviewed indicated that GAVI has a strong organisational commitment to working in coordination with its partners. The results also reflect GAVI’s adherence to Paris Declaration principles of country ownership, alignment, and harmonisation.

Figure 3.17 below shows the overall ratings for the five KPIs in the relationship management quadrant. According to survey respondents, GAVI performs strongly on four of the five KPIs in relationship management. The document review rated GAVI as strong on the two KPIs it examined; there was insufficient data to assess a third KPI that was originally meant to be assessed through document review.

Figure 3.17 Quadrant III: Relationship Management, Survey and Document Review Ratings

Figure 3.18 shows the mean scores for the five KPIs for all survey respondents, and by respondent groups.

Figure 3.18 Quadrant III: Relationship Management, Mean Scores by Respondent Group

KPI 11: Supporting national/regional and partner plans

Finding 11: Survey respondents considered that GAVI support is aligned with national and partner plans. The document review, which looked at whether GAVI applies conditionality in a way that is aligned with the principles of country ownership, rated the organisation strong.

The survey respondents rated GAVI as strong on the extent to which approved applications support national plans and partner plans. The document review was conducted for one of the three MIs. (Global-level implementing partners and other GAVI partners and peers were not asked their views on this KPI.)

Overall, it appears that GAVI promotes country ownership by supporting the national plans of the countries that are approved for support. GAVI requires that applications for support are aligned with the national health strategy and the national immunisation strategy. Its conditions for country support (both cash-based and in-kind support) support the principle of country ownership.

This assessment is supported by the Second GAVI Evaluation (2009-2010) and the HSS Evaluation (2009) which concluded that GAVI’s approach to programme implementation promotes country ownership and is aligned with national strategies and plans in terms of supporting priority areas and objectives.

Figure 3.19 KPI 11: Supporting National Plans, Ratings of Micro-Indicators

MI 11.1 – Funding proposals developed with national governments/partners

Overall, survey respondents agreed with the statement that proposals submitted to GAVI are developed by national governments or other applicants, with 92 per cent rating GAVI as adequate or above. MOPAN donors in-country responded more negatively than the other respondent groups who were surveyed on this micro-indicator and the difference was statistically significant.

MI 11.2 – Conditionality draws on national benchmarks/indicators

On this MI, survey respondents were asked to rate GAVI on two aspects: i) its practice of co- financing and ii) the extent to which GAVI agrees with governments on co-financing schedules based on governments’ ability and willingness to pay for vaccines. Respondents were satisfied with GAVI’s performance on both aspects, with 56 per cent rating this MI as strong or very strong and over 80 per cent as adequate or above. On the second question related to government willingness to pay for vaccines, there were high percentages of ‘don’t know’ responses among MOPAN donors at headquarters (32 per cent) and government/CSO representatives (25 per cent).

The document review found that GAVI applies conditionality in a way that is aligned with the principles of country ownership and rated GAVI strong on this MI. In the case of cash-based support, GAVI’s conditions do not bind the country on how or when support shall be used. In the case of in-kind support, GAVI requires countries to co-finance, according to established thresholds, the vaccines obtained through GAVI’s New and Underused Vaccines (NVS) programme.

This assessment was confirmed by the Second GAVI Evaluation (2009-2010) and the HSS Evaluation (2009) which found that GAVI’s approach to programme implementation promoted country ownership and was aligned with national strategies and plans in terms of supporting priority areas and objectives, though the HSS evaluation found that few proposals demonstrate a clear link between the constraints identified and the objectives specified for GAVI HSS funding.

MI 11.3 – National priorities respected

Overall, 85 per cent of survey respondents rated GAVI as adequate or above for providing support that respects national priorities.

Government/CSO representatives and country-level implementing partners were the most positive respondent groups (95 per cent rated this MI adequate or above). MOPAN donors in- country had more mixed views (65 per cent adequate or above and 35 per cent inadequate).

KPI 12: Adjusting Procedures

Finding 12: Survey results suggested that GAVI’s procedures, such as funding application and reporting requirements, take into account local conditions and capacities.

Survey respondents, who were the only source of data to assess whether GAVI’s procedures take into account local conditions and capacities, rated the organisation as adequate or better.

None of the questions relating to this key performance indicator were asked of global-level implementing partners and other GAVI partners and peers. In addition, the views of MOPAN donors at headquarters were solicited only with regard to MI 12.3.

Figure 3.20 KPI 12: Adjusting Procedures, Ratings of Micro-Indicators

MI 12.1 – Procedures easily understood and completed by partners/applicants

Within this key performance area, this MI received the highest percentage of strong and very strong responses (52 per cent) from survey respondents and a high percentage of responses of at least adequate (83 per cent). In addition, almost all government/CSO representatives (98 per cent) rated GAVI as adequate or better on its use of procedures (e.g., funding application and reporting requirements) that can be easily understood and followed by national governments and other applicants. One-quarter of the responses provided by MOPAN donors in-country were ‘don’t know’.

MI 12.2 – Length of time for procedures does not affect implementation

According to the majority of respondents (83 per cent), the length of time it takes to complete GAVI procedures does not negatively affect the implementation of programmes. Across respondent groups, in fact, 83 per cent of responses were adequate or above.

MI 12.3 – Partners can adjust implementation to changing circumstances

This MI was also an area in which survey respondents found GAVI adequate or stronger, with 75 per cent of responses across respondent groups adequate or above, 9 per cent inadequate or weak, and the remaining 16 per cent ‘don’t know’. More than one-quarter of responses from MOPAN donors at headquarters (27 per cent) and in-country (30 per cent) were ‘don’t know’.

KPI 13: Using Country Systems

Finding 13: Survey respondents considered GAVI strong in using country systems Although the document review had insufficient data to make an assessment, it noted this as an area that GAVI is paying attention to and in which it is making efforts to improve.

Survey respondents’ opinions on GAVI’s use of country systems were sought for two of three MIs for this KPI. Global-level implementing partners and other GAVI partners and peers were not asked about this KPI. Although the document review was not able to rate this area due to insufficient data available from global-level assessments such as the Survey on Monitoring the Paris Declaration and the IHP+Results,[19] GAVI’s work appears to be aligned with the Paris Declaration principles.

GAVI is trying to find a balance between the need to align its procedures with country systems, as per the Paris Declaration principles on alignment, and increasing requests for accountability from its donors. Considering its business model, which requires limited country presence and working through its partners, GAVI relies very much on systems that are already in place and that are used by its partners. In order to take into account the different levels of countries’ institutional development, GAVI is developing the Country by Country Approach, a policy that clearly defines the GAVI Alliance’s approach to fragile and under-performing countries.

Figure 3.21 KPI 13: Using Country Systems, Ratings of Micro-Indicators

MI 13.1 – ODA disbursements/ support recorded in annual budget

Although MOPAN originally intended to rate GAVI on this MI in the document review, the document review found that this MI was not applicable to GAVI. GAVI’s support is mostly in the form of vaccines that countries receive, most often through UNICEF, and is not recorded in their national budgets. As for cash-based support (HSS, ISS, CSO), GAVI’s templates for application and country progress reports request countries to provide this type of information. GAVI acknowledged that there are still some bottlenecks that hinder the alignment of its support with countries’ budget cycles. The Joint HSS platform is seen by GAVI as an opportunity to improve in this area. However, it is too early to assess whether this mechanism has brought changes in terms of alignment of GAVI’s support with countries’ budget cycles.

MI 13.2 – Use of country systems

The majority of survey respondents (86 per cent) rated GAVI adequate or better for its use of country systems (e.g., procurement, public financial management, etc.) where appropriate. These views were shared across respondent groups and no answers of ‘weak’ or ‘very weak’ were recorded.

There was insufficient data for the document review to assess GAVI’s performance on the use of national systems and procedures. This is due in part to the fact that governments almost exclusively use UNICEF and the Pan American Health Organization to procure vaccines with the support they receive from GAVI and, as a consequence, procurement is done by these two agencies.

With regard to GAVI’s cash-based support, GAVI has recently put in place a system of financial management assessments with the objective of identifying and agreeing with the country the best national financing mechanism to manage GAVI’s funds. This suggests that GAVI does not require countries to establish new processes or systems to disburse funds received, but relies on national systems. For example, GAVI does not require the establishment of Project Implementation Units to manage its support and mostly uses UNICEF systems for the procurement of vaccines. In addition, recent initiatives adopted such as the Joint HSS platform[20] and the financial management assessments are seen by GAVI as opportunities to improve its use of country systems for disbursement and operations.

MI 13.3 – Promotion of mutual accountability in its partnerships

Survey respondents were asked whether GAVI promotes a mutual assessment of progress in implementing agreed partnership commitments. The majority (83 per cent) agreed that it does and provided a score of adequate or better. Among MOPAN donors in-country, 20 per cent of responses were ‘don’t know’.

KPI 14: Contributing to Policy Dialogue

Finding 14: Survey respondents considered GAVI strong in contributing to policy dialogue.

Survey respondents rated GAVI as strong on both MIs relating to the organisation’s contributions to policy dialogue. Global-level implementing partners and other GAVI partners and peers were not asked about this KPI. This KPI was not assessed by the document review.

Figure 3.22 KPI 14: Contributing to Policy Dialogue, Ratings of Micro-Indicators

MI 14.1 – Reputation for valuable input to policy dialogue

Survey respondents were asked whether GAVI provides valuable inputs to policy dialogue on immunisation. Overall, the majority (82 per cent) rated the organisation as adequate or better, although there were some differences among respondent groups – 67 per cent of country-level implementing partners versus 81 per cent of other respondent groups.

MI 14.2 – Policy dialogue respects partner views

GAVI’s strong performance also emerged in participating in policy dialogue in a way that is respectful of the views of national governments and civil society (where relevant). Overall, 81 per cent of respondents rated GAVI as adequate or better and 62 per cent as strong or very strong.

KPI 15: Harmonising Procedures

Finding 15: Overall, GAVI was seen to harmonise arrangements and procedures with other programming partners, although the survey results and document review indicated different areas as being the strongest.

GAVI’s performance in harmonising arrangements and procedures with other programming partners (donors, UN agencies, etc.) was rated between adequate and strong by survey respondents and the document review, although these two sources of data indicated different areas as being the strongest.

Country-level respondents (i.e., MOPAN donors in-country, country-level implementing partners, and government/CSO representatives) were asked the questions related to this KPI.

The document review rated GAVI adequate on one indicator and strong on another in this KPI. GAVI is moving towards increased harmonisation with other development partners in the context of the HSS Platform. This initiative aims to streamline HSS support and align it with country budgetary and programmatic cycles by supporting: one comprehensive health plan that integrates both domestic funding and international aid; one joint assessment of the national health strategy; one budget; and one tracking system for funds.

Figure 3.23 KPI 15: Harmonising Procedures, Ratings of Micro-Indicators

MI 15.1 – Participation in joint missions

GAVI’s participation in joint missions was rated as adequate or above by 64 per cent of survey respondents, inadequate or below by 27 per cent, and 9 per cent responded ‘don’t know’. There was a high percentage of ‘don’t know’ responses from donors in-country (25 per cent).

Data available from the Paris Declaration Monitoring Survey and the World Bank research on aid quality suggest that GAVI participation in joint missions is strong. In interviews conducted with GAVI staff they provided several examples of financial assessments and monitoring missions carried out by the GAVI Secretariat jointly with other donors and development partners such as WHO, UNICEF, and the Global Fund to Fight AIDS, Tuberculosis and Malaria.

MI 15.2 – Technical cooperation coordinated with other donors/partners

Overall, the majority of survey respondents (76 per cent) rated GAVI adequate or better on coordinating technical assistance with other organisations and partners (WHO/UNICEF). In fact, GAVI was rated strong or very strong on this MI by 60 per cent of government/CSO representatives and country-level implementing countries and by 40 per cent of MOPAN donors in-country.

The indicator to be assessed by document review focused on the coordinated technical cooperation in support of capacity development (taking the wording of Paris Indicator 4). GAVI does not provide capacity development support to countries as such (this is provided by, and probably reported on, by Alliance partners). As a consequence, global-level surveys (IHP+Results and the Survey on Monitoring the Paris Declaration) on organisations’ performance vis-à-vis aid effectiveness principles (such as this one on harmonisation) do not provide data for GAVI on this performance area.

MI 15.3 – Support for government-led PBAs

GAVI’s efforts to harmonise its procedures with other programming partners is especially evident from its use of programme-based approaches to support national health plans when providing grants to countries. Almost all survey respondents (96 per cent) rated GAVI as adequate or above on this MI, with over 60 per cent rating it as strong or very strong. Similar views were shared across the three respondent groups who were consulted on this area of performance.

Data from the Survey on Monitoring the Paris Declaration, the World Bank’s study,[21] and the IHP+Results all show that GAVI has increased the percentage of support that is channelled to countries through programme-based approaches.

3.3.5 Knowledge Management

GAVI is increasingly paying attention to the area of knowledge management and has recently undertaken initiatives to strengthen its evaluation function, culture and reporting systems.

Figure 3.24 below shows the overall survey and document review ratings for the three KPIs in the knowledge management quadrant.

Survey respondents rated GAVI adequate overall in knowledge management, although there were some differences among respondent groups that may indicate their different expectations in terms of how GAVI evaluates its activities and shares lessons learned.

The document review rated GAVI strong in two of the three MIs in this KPI and highlighted the independence of its evaluation function and its reporting on adjustments to policies and strategies based on performance information. The one area noted for improvement was evaluation coverage.

Figure 3.24 Quadrant IV: Knowledge Management, Survey and Document Review Ratings

Figure 3.25 shows the mean scores for the three KPIs for all survey respondents, and by respondent groups.

Figure 3.25 Quadrant IV: Knowledge Management, Mean Scores by Respondent Group

KPI 16: Evaluating External Results

Finding 16: GAVI has made improvements in its structure for carrying out evaluations, but is still facing challenges in evaluation coverage.

Results from the survey were in the adequate band on this MI while they ranged from inadequate to strong in the document review. The document review was the main source of data on this key performance indicator, as survey respondents rated only two of the five MIs.

The GAVI evaluation function lies within two organs, the Monitoring and Evaluation Team in the Secretariat and the recently established Advisory Evaluation Committee at the Board level. While the creation of the latter has enhanced the independence and quality of evaluations, the definition and operationalisation of some evaluation principles and guidelines (e.g., evaluation coverage) need further refinement.

Figure 3.26 KPI 16: Evaluating External Results, Ratings of Micro-Indicators

MI 16.1 – Independent evaluation unit

This MI was assessed only through the document review.

The evaluation function is shared between the Monitoring and Evaluation Team within the Secretariat, and the Evaluation Advisory Committee at the Board level. The Evaluation Advisory Committee is, among other things, responsible for approving GAVI’s multiyear and annual evaluation work plans; reviewing the quality and usefulness of evaluation reports from the independent consultants; and, for evaluations costing more than $500,000, reviewing and approving the terms of reference and selection of final contractors. The M&E team reports through its Managing Director to the Evaluation Advisory Committee which in turn reports to the GAVI Alliance Board. As such, the evaluation structure appears to be independent in GAVI.

MI 16.2 – Sufficient evaluation coverage of programming activities

This MI was assessed only through the document review which found that GAVI’s Evaluation Policy does not provide for adequate coverage of GAVI’s programming activities. The policy specifies that projects, programmes, policies, and GAVI as an Alliance are part of the evaluation scope and it establishes some guidelines on coverage. For example, it states that evaluation activities should be considered: when a programme comes to an end; in the mid- term, future programme or policy development; when support for a country comes to an end; and to inform future strategic planning. While there is evidence that a number of programme evaluations have been carried out by GAVI, which have included country-level case studies, GAVI does not define in its policy (or in any other document) what it considers to be a sufficient evaluation coverage of its work.

MI 16.3 – Quality of evaluations

This MI was assessed only through the document review which focused on the systems in place to ensure quality of evaluations. The review rated GAVI strong for ensuring the quality of its evaluations through the Evaluation Advisory Committee, which is part of the GAVI Alliance Board structure, and by explicitly making quality one of the guiding principles for evaluations. The Evaluation Advisory Committee was only recently established and its activities and influence on GAVI evaluation practices have so far been limited.

MI 16.4 – Use of evaluation findings to inform decisions

This MI was assessed only through the survey and only by global-level respondents.

When asked about GAVI’s use of evaluation findings in its decisions on programming, policy and strategy, 72 per cent of survey respondents rated GAVI adequate or higher.

MI 16.5 – Grant recipients and partners involved in evaluations

Two questions were asked of respondents with respect to the involvement of grant recipients and Alliance partners in the evaluation of GAVI’s programmes. The majority of respondents rated GAVI as adequate or above on involving grant recipients (59 per cent) and Alliance partners (72 per cent). However, these overall percentages mask some differences in the views of respondent groups. MOPAN donors at headquarters provided scores of adequate or above on both questions. Among the other respondent groups, inadequate ratings ranged from 13 to

50 per cent on involving grant recipients, and from 5 to 42 per cent on involving Alliance partners. The level of ‘don’t know’ answers also varied: 32 per cent of MOPAN donors at headquarters responded ‘don’t know’ on involving grant recipients and 21 per cent of government/CSO respondents answered ‘don’t know’ on involving Alliance partners.

The document review rated GAVI as strong on this MI. The GAVI Evaluation Policy recognises stakeholder participation as one of the guiding principles for evaluations and GAVI has developed guidelines on how to involve them. The guidelines, for instance, require the involvement of partners ‘to provide input on the development of the terms of reference’ and to review the draft versions of evaluation reports. However, the evaluation reports reviewed indicate that stakeholder involvement was mostly limited to interviews for data collection purposes and do not mention whether they were involved in the other phases of evaluation such as the validation of evaluation deliverables.

KPI 17: Presenting Performance Information

Finding 17: Survey respondents and the document review suggest that GAVI adequately reports on its results and strategy, with some areas stronger than others.

GAVI was rated by survey respondents on three of the six MIs, while the document review assessed GAVI’s performance on all the MIs. Scores from the survey ranged from adequate to strong and scores from the document review from very weak to strong.

GAVI has developed systems to report to the Board and to the public on a regular basis on its performance. There seems to be an improvement in Phase III in the way performance information is reported, with closer alignment with performance indicators. The main area for improvement is its reporting against Paris Declaration commitments.

Figure 3.27 KPI 17: Presenting Performance Information, Ratings of Micro-Indicators

MI 17.1 – Reports on achievement of outcomes

Only MOPAN donors at headquarters, implementing partners at the global level, and other GAVI partners and peers were asked whether GAVI's reports to the Board provide clear measures of achievement of outcomes (objectives).

The large majority of survey respondents (92 per cent) considered GAVI to be adequate or stronger on this MI. A higher percentage of other GAVI partners and peers rated GAVI as inadequate (25 per cent) than other respondent groups, and donors at headquarters were more positive than other respondent groups (this difference was statistically significant).

GAVI’s reports on the achievement of outcomes appear to be adequate. GAVI produces annual progress reports and annual financial reports on its performance in which it presents outcomes and outputs achieved. However, it remains unclear from the reports how its support contributed to the achievement of the reported results. Results reported are in fact long-term outcomes or impacts (e.g., immunisation coverage rate, deaths averted, etc.) and in terms of outputs (e.g., grants/vaccines provided), but the reports do not explain GAVI’s contribution to outcome achievement (i.e., they do not establish a link between outputs and outcomes).

MI 17.2 – Reports on performance using data obtained from measuring indicators

This MI was assessed only through the document review which found that GAVI adequately uses data obtained from measuring indicators in its annual progress reports. Indicators generally respect SMART and CREAM criteria[22] and indicator measurements are compared, in some instances, to baselines but not to targets.

MI 17.3 – Reports against corporate strategy, including expected results

The survey asked MOPAN donors at headquarters, implementing partners at the global level, and other GAVI partners and peers whether, in their opinion, GAVI reports adequately against its 2011-2015 Strategy and Business Plan. The majority of survey respondents (86 per cent) rated GAVI adequate or better. Responses from donors at headquarters were more positive than those from other groups and this difference was statistically significant. One-third of other GAVI partners and peers rated GAVI as inadequate, making the average score for this respondent group statistically significantly lower than the average score provided by the other respondent groups.

The review of GAVI annual progress reports and quarterly updates to the Board found that GAVI reporting against its Corporate Strategy and Corporate KPIs is adequate and that annual progress reports contain performance information collected by measuring the impact and strategic goal-level indicators identified in its development results framework. In Phase III GAVI has improved the way it reports on its performance: the quarterly updates use the specific indicators included in the Corporate KPIs and the Business Plan results framework and provide an explanation for some of the variances in result achievements. In addition, unlike the previous progress reports, the 2011 GAVI Alliance Progress Report systematically uses the indicators identified in the 2011-2015 Strategy and Business Plan to report on performance at the impact and strategic goal levels.

MI 17.4 – Reports on Paris Declaration commitments use indicators and country targets

MOPAN donors at headquarters were asked whether GAVI reports to its Board on performance in relation to its commitments under the Paris Declaration Principles (i.e., country ownership, alignment, harmonisation, managing for results, mutual accountability). Overall, 82 per cent rated GAVI from adequate to very strong.

Since GAVI does not produce a report on its performance against Paris Declaration (PD) commitments, the document review provided a score of very weak on this MI. It bears noting, though, that GAVI provides some more generic information on how its work is aligned with the Paris Declaration principles, for example in its annual progress reports.

MI 17.5 – Reports on adjustments to policies/strategies based on performance information

This MI was assessed only through the document review which found that GAVI does report to its Board on adjustments made or recommended to organisation-wide policies and strategies based on performance information. Since the GAVI Board has to approve new/modified strategies and policies, the GAVI Secretariat, in submitting the proposed adjusted strategy or policy to the Board, also provides the Board with the rationale behind the adjustment of the policy or strategy, the rationale often being based on the review of the prior strategy or policy.

MI 17.6 – Reports on programming adjustments based on performance information

This MI was assessed only through the document review which found that GAVI adequately reports adjustments made or recommended based on performance information for country-level programming. The documents reviewed show that on an annual basis, the GAVI Secretariat reports to the Board on the countries’ performance, including on changes in GAVI’s support.

KPI 18: Disseminating Lessons Learned

Finding 18: Survey respondents and the document review indicate that GAVI is reporting on lessons learned based on performance information.

Survey respondents rated GAVI as adequate and strong on the two MIs related to this key performance indicator. The document review, which rated only one MI, assigned a score of strong.

Figure 3.28 KPI 18: Disseminating Lessons Learned, Ratings of Micro-Indicators

MI 18.1 – Reports on lessons learned based on performance information

Only MOPAN donors at headquarters were surveyed for this MI and 82 per cent rated GAVI’s performance in identifying and disseminating lessons learned from performance information as adequate or higher.

Recent studies available on GAVI’s website provide evidence that GAVI is committed to the identification of lessons learned and/or best practices. Lessons learned, collected by the Programme and Policy team, are in the form of studies on GAVI’s experience in key areas linked to its strategic goals, such as the shaping of the vaccine market and financial sustainability. GAVI shares lessons learned through its website, staff meetings, and GAVI Partner Fora. However, it is unclear from the documentation available to the Assessment Team how GAVI uses internal and external lessons learned to change management and programming practices (one of the criteria for this MI).

MI 18.2 – Lessons shared at all levels of the organisation

This MI was assessed only through the survey which asked respondents about opportunities for Alliance-wide sharing of lessons learned and about learning opportunities provided by GAVI to grant recipients.

On the question on Alliance-wide sharing, 67 per cent survey respondents rated GAVI as adequate or better and 27 per cent as inadequate or lower. However, there was a relatively high number of ‘don’t know’ responses: 32 per cent of MOPAN donors at headquarters, 27 per cent of global-level implementing partners, and 25 per cent of other GAVI partners and peers.

On the question on learning opportunities provided to grant recipients, respondents were more evenly split with 46 per cent rating GAVI adequate or better and 36 per cent as inadequate or lower.

4. Conclusion

The year 2012 marks the first occasion for MOPAN to involve a global fund in its common assessment. The assessment required adaptations to the MOPAN methodology to reflect the uniqueness of an alliance such as GAVI, which operates differently than other multilateral organisations. GAVI was a supportive and willing partner in this process.

The following conclusions look at messages that can contribute to dialogue between MOPAN, GAVI and its partners.

GAVI is well-positioned within the international aid architecture.

MOPAN survey respondents recognised GAVI’s mandate to save children’s lives and protect people’s health by increasing access to immunisation as one of the organisation’s main strengths. They also commended GAVI’s mandate as a public-private partnership that brings together a variety of actors from developed and developing countries and from the public and private sectors. These two elements make GAVI an organisation that is considered well- positioned within the international aid architecture.

The nature of GAVI as a public-private partnership and as an Alliance means that it faces some internal challenges in meeting the expectations of its constituent groups while operating in a complex external environment.

GAVI operates in a complex internal and external environment. Internally, GAVI works as an Alliance, which means working to meet a variety of sometimes contradictory expectations regarding its practices, structure, and scope of work; relying on the different capacities of its partners’ and supported countries to implement, while maintaining accountability for programme implementation with the whole Alliance.

Externally, GAVI works in an international aid development context characterised by partially contradictory forces. Limited availability of financial resources for aid development leads donors to insist on increased demonstrations of effectiveness and accountability. However, the Paris Declaration commitments seek increased donor coordination and country ownership.

The 2012 MOPAN assessment found that within the interplay of these forces, GAVI is holding its own and making progress to become an increasingly effective organisation. This is demonstrated by the largely positive views provided by stakeholders, the result of its last pledging conference in which pledges exceeded the target, and by the increasing country demand for GAVI’s support.

GAVI’s mandate and comparative advantage provide a good foundation for its focus on results, but there is room to further strengthen this orientation in its strategy, budget, and reporting practices.

GAVI’s 2011-2015 Strategy and Business Plan is anchored to its mandate and comparative advantage, which provide the organisation with a strong direction for results. However, the extent to which this results orientation is operationalised within the organisation does not yet meet all of GAVI’s stakeholder expectations.

Over the past three years, GAVI has undertaken reforms to strengthen its practices and become a more results-oriented organisation. These have included the development of an integrated business plan, the introduction of quarterly reporting by the GAVI Secretariat and partners on the programme deliverables identified in the 2011-2015 Strategy and Business Plan, the establishment of a performance management unit within the GAVI Secretariat, and increased mainstreaming of gender equality in GAVI’s internal and external operations.

While these reforms are important steps towards GAVI becoming a results-oriented organisation, some gaps remain: the results-based management approach, the adequacy of the results frameworks (the logic of and linkages in the results chains in the development results framework and the absence of strong results statements in the management results framework), and the results-based budgeting practices. These gaps were noted in the document review and in the survey by a small number of GAVI private sector[23] and public sector respondents. Thus, GAVI could benefit from increased efforts, and possibly increased resources, in results-based management systems and tools (especially the identification of adequate performance indicators, the strengthening of the results chain at the corporate and country levels, and RBM training). As MOPAN donors are committed to integrating cross- cutting themes, it will be important for GAVI and MOPAN to discuss the appropriate definitions and approaches to integrate human rights-based approaches in its strategy and operations and further encourage its partners to develop policies for human rights-based approaches.

GAVI is recognised for a number of its operational practices that ensure transparency and accountability in the use of resources.

Over the last two to three years, GAVI has strengthened its systems in order to manage operations in a more performance-oriented way and to promote accountability (of GAVI as a whole and of supported countries) related to results achievement and use of funds. Some of the reforms introduced to positive effect include the establishment of a performance management process and unit, and the strengthening of financial accountability systems, particularly of the risk management system for cash-based programmes (given an increase of this type of support). GAVI has also continued to promote the transparency of its decisions, including its resource allocation, through consultation mechanisms and dialogue with its partners, wide dissemination of its key documentation, and the establishment of independent mechanisms with an advisory role for decision-making, such as the Independent Review Committee and the Evaluation Advisory Committee.

In response to increasing requests to demonstrate results and accountability to different stakeholder groups, GAVI is undertaking reforms to strengthen its knowledge management practices.

GAVI has paid increasing attention to its performance and knowledge management practices since 2011 in order to respond to the demands from internal and external stakeholders (donors, partners, governments, senior managers) to demonstrate effectiveness and accountability. It has undertaken several reforms to make monitoring and evaluation integral to programme management, including: the establishment of an Evaluation Advisory Committee within the GAVI Alliance Board structure; the introduction of country grant scorecards for more regular monitoring; and, the development of a monitoring and evaluation framework and strategy. In due course, an assessment of the effectiveness of these measures may need to be done to ensure that they meet the expectations and requests of GAVI’s key internal and external stakeholders.

GAVI is successfully managing a complex set of relationships at the country level. However, adapting to the needs and expectations of partners in so many countries is a slow, delicate, and costly balancing act.

GAVI currently operates in 73 of the poorest countries through its country-level partners (both developing countries and GAVI Alliance partners). It depends heavily on the capacities of these partners in numerous areas: sustainability of vaccines, co-financing, collection of sex- disaggregated data, the integration of gender and environment-related considerations in programme implementation, and the quality of immunisation-related data.

GAVI is adapting to the contexts in these different countries by integrating some flexibility into its policies and operations by, for example, establishing different levels of co-financing and developing a specific policy for fragile and underperforming countries. At the same time, country-level stakeholders are asking GAVI to increase its country presence in order to facilitate communications and improve support to grant recipients. These requests, however, go against the ‘light touch’ approach that has shaped GAVI’s structure and functioning until now, and that is considered by stakeholders with a global perspective to be a key factor in the Alliance’s success to date.

While the MOPAN assessment found that GAVI has managed its relationships well at the country level, this is likely to remain a delicate area that will require constant monitoring by GAVI and open dialogue with its stakeholders.

Although GAVI’s work is aligned with the principles of the Paris Declaration, progress on some specific indicators is not clear.

GAVI is one of the signatories of the 2005 Paris Declaration on Aid Effectiveness and has made efforts to align its work with the Paris Declaration principles. GAVI’s institutional architecture, characterised by a partnership between a wide range of development actors, is aligned with the principles of harmonisation and mutual accountability. Its support of countries’ national health plans and use of country systems are aligned with the principles of country ownership and alignment. Its focus on immunisation-related results and performance-based funding is coherent with the principle of managing for results. Recent initiatives, such as the Health Systems Funding Platform, the Advance Market Commitment, and the International Health Partnership, are evidence of GAVI’s adherence to PD principles.

Nonetheless, despite its commitment to the PD principles and evidence of its contribution to some of them, it is difficult to fully appreciate GAVI’s progress on its PD commitments due to several factors, including: the nature of some indicators, the lack of data on several indicators (e.g., support recorded in national budgets, the use of national systems and procedures, and the strengthening of capacity by coordinated support), as well as its inconsistent reporting to the OECD-DAC on these commitments over the past decade.

Some of the Paris Declaration indicators may not apply as readily to multilateral organisations like GAVI that provide mostly in-kind support in the form of vaccines and use the procurement and distribution systems of its partners. The OECD is currently revising these indicators. In the meantime, given that there is an expectation from donors to understand GAVI’s contribution to these principles, GAVI might consider developing some alternative indicators that measure its contribution or discuss other ways in which it can report to donors on these commitments.

---

[1] Private sector respondents include representatives from the Bill & Melinda Gates Foundation, international non-governmental organisations, the emerging/developing country pharmaceutical industry, the industrialised country pharmaceutical industry, research and technical health institutes, and the Global Fund to Fight AIDS, Tuberculosis, and Malaria (GFATM)

[2] At this time, all the Campaign’s assets, rights, privileges, liabilities and obligations were transferred from what was formerly known as the GAVI Fund to the GAVI Alliance.

[3] This component was tested in 2012 with the African Development Bank, UNICEF, UNDP, and the World Bank.

[4] The full methodology is presented in Volume II, Appendix I.

[5] MOPAN criteria for country selection include: multilateral organisation presence in-country, presence and availability of MOPAN members, no recent inclusion in the survey, the need for geographical spread, and a mix of low income and middle income countries (middle income countries being subdivided into lower middle and upper middle). Based on these criteria, Philippines and Morocco were excluded from the assessment of GAVI because they are not eligible for GAVI support.

[6] The criteria applied for selecting the additional countries for GAVI were: mix of grant windows, geography, and graduation status. Government/CSO representatives and country-level implementing partners from eight additional countries were added to the survey population. No MOPAN donors in- country from these countries were included in the survey population since these countries were not part of the original MOPAN 2012 selection and, as a consequence, no local consultants were available for follow up.

[7] Throughout the report, we have also used the terms “global-level respondents” and “country-level respondents”. The first term encompasses the following respondent groups: donor headquarters oversights, global-level implementing partners, and other Global Alliance partners and peers. The second term is used to refer to donor country office oversight, country government and national CSO respondents, and in-country implementing partners.

[8] The normal convention for statistical significance was adopted (p≤.05).

[9] The numbers of respondents and response rates are similar to those in other global surveys used to assess GAVI, such as the survey applied in the Second GAVI Evaluation (2009-2010), where the survey response rate was 28 per cent.

[10] While most KPIs and micro-indicators were considered in the document review, not all were rated. See section 2.3.

[11] The evaluation, conducted by CEPA LLP in association with Applied Strategies in 2009-2010, focused on an assessment of the Alliance’s results and value added across all of its strategic goals since inception, but with a primary focus on 2006 to 2009 (Phase II).

[12] Evaluation of GAVI’s Injection Safety Support (2009) and the GAVI Second Evaluation Report (2010).

[13] The criteria, which can be found in Appendix VI of Vol. 2, specify elements from the United Nations Statement of Common Understanding among UN Agencies (2003) as necessary, specific, and unique to a human rights-based approach

[14] Since its establishment, GAVI has put in place a system of peer review by a committee of independent technical experts, the Independent Review Committee (IRC), to review applications for support (New Proposals IRC) and to monitor annual country progress reports (IRC Monitoring Team).

[15] HLSP. Review of GAVI Independent Review Committees (IRCs). Final Report. 30 March 2010. Available at: http://www.gavialliance.org/library/documents/gavi-documents/evaluations/review-of-gavi- independent-review-committees/

[16] IFFIm uses long-term pledges from donor governments to issue bonds in the capital markets, making large volumes of funds immediately available for GAVI programmes. The evaluation of the IFFIm found that this mechanism has provided GAVI with predictable funding (see page 23). GAVI is also working to increasing predictability of funding through the Advance Market Commitment, through which donors commit funds to guarantee the price of vaccines once they have been developed.

[17] The budget does not include costs for vaccines and cash grants provided to countries, which are estimated on the basis of expected applications, approval percentage, and implementation rate. Prior to 2010, the budget had three expenditure areas: Programme, Work Plan, and Administrative. Programme expenditures represented payments to either country immunisation programmes or to non-country specific programmes. Work Plan expenditures were costs related to the development and implementation of these programmes. Administrative expenditures covered the Secretariat’s daily operations.

[18] During the finalisation of this report, GAVI specified that they are working on improving how evaluation recommendations are acted upon by incorporating management action plans into their routine reporting through KPIs / Business Plan activities (where relevant). GAVI also specified that this process has already been tested for the CSO evaluation management action plan and will be followed for future evaluations in collaboration with the Performance Management Unit of GAVI.

[19] IHP+Results independently monitors the progress of IHP+ signatories in implementing their commitments on aid effectiveness in the health sector. It produces an annual Performance Review and scorecards that summarise each participating country government’s and development partner’s performance. See also: http://ihpresults.net/how/results/

[20] The aim of the HSS Platform is to streamline health system strengthening (HSS) support and align with country budgetary and programmatic cycles by supporting: one comprehensive health plan that integrates both domestic funding and international aid, one joint assessment of the national health strategy, one budget, and one tracking system for funds. Source: http://www.gavialliance.org/support/hsfp/

[21] The World Bank. (May 2010). Aid Quality and Donor Rankings. Policy Research Working Paper 5290.

[22] There are different criteria for assessing performance indicators, such as CREAM (clear, relevant, economic, adequate, monitor-able) and SMART (specific, measurable, achievable, relevant, time bound).

[23] Private sector respondents include representatives from the Bill & Melinda Gates Foundation, international non-governmental organisations, the emerging/developing country pharmaceutical industry, the industrialised country pharmaceutical industry, research and technical health institutes, and the Global Fund to Fight AIDS, Tuberculosis, and Malaria (GFATM)