Preface

The Multilateral Organisation Performance Assessment Network (MOPAN) is a network of donor countries with a common interest in assessing the organisational effectiveness of multilateral organisations. MOPAN was established in 2002 in response to international fora on aid effectiveness and calls for greater donor harmonisation and coordination.

Today, MOPAN is made up of 16 donor countries: Australia, Austria, Belgium, Canada, Denmark, Finland, France, Germany, Ireland, The Netherlands, Norway, Republic of Korea, Spain, Sweden, Switzerland and the United Kingdom. For more information on MOPAN and to access previous MOPAN reports, please visit the MOPAN website (www.mopanonline.org).

Each year MOPAN carries out assessments of several multilateral organisations based on criteria agreed by MOPAN members. Its approach has evolved over the years, and since 2010 has been based on a survey of key stakeholders and a review of documents of multilateral organisations. MOPAN assessments provide a snapshot of four dimensions of organisational effectiveness (strategic management, operational management, relationship management, and knowledge management). In 2012, MOPAN is piloting a new component to examine an organisation’s development results in addition to its organisational effectiveness.

MOPAN 2012

In 2012, MOPAN assessed six multilateral organisations: the African Development Bank (AfDB), GAVI Alliance (formerly the Global Alliance for Vaccines and Immunisation), the Joint United Nations Programme on HIV/AIDS (UNAIDS), the United Nations Development Programme (UNDP), the United Nations Children’s Fund (UNICEF), and the World Bank.

MOPAN Institutional Leads liaised with the multilateral organisations throughout the assessment and reporting process. MOPAN Country Leads monitored the process in each country and ensured the success of the survey.

Acknowledgements

We thank all participants in the MOPAN 2012 assessment of UNAIDS. UNAIDS’ senior management and staff made valuable contributions throughout the assessment, document review and headquarters interview processes and provided lists of their direct partners and co- sponsors to be surveyed. Survey respondents contributed useful insights and time to respond to the survey. The MOPAN Institutional Leads, Finland and France, liaised with UNAIDS throughout the assessment and reporting process. The MOPAN Country Leads oversaw the process in the field and ensured the success of the survey. Consultants in each country provided vital in-country support by following up with direct partners and co-sponsors to enhance survey response rates.

Roles of Authors and the MOPAN Secretariat

The MOPAN Secretariat, led by Ireland in 2012 and co-chaired by Germany, worked in close cooperation with the MOPAN Technical Working Group to launch and manage the survey. MOPAN developed the Key Performance and Micro-indicators, designed the survey methodology, coordinated the development of lists of survey respondents, and approved the final survey questionnaire. MOPAN also directed the design of the approach to document review. MOPAN oversaw the design, structure, tone, and content of the reports.

Universalia and Epinion developed the survey instrument and carried out the survey and analysis. Universalia carried out the document review and wrote the reports.

Epinion is a leading consulting firm in Denmark that analyses and evaluates data to support decision making. It conducts specially designed studies for public and private organisations based on data collected among an organisation’s employees, members, customers, partners, and other sources. Epinion has 75 employees and 200 interviewers. Website: www.epinion.dk

Universalia Management Group is a Canadian consulting firm established in 1980 that specialises in evaluation and monitoring for international development. Universalia has made significant contributions to identifying best practices and developing tools in the fields of organisational assessment; planning, monitoring, and evaluation; results-based management; and capacity building. Website: www.universalia.com.

Acronyms

AAA - Accra Agenda for Action
AfDB - African Development Bank
CCO - Co-sponsors Coordinating Committee
CEWG - Co-sponsors Evaluation Working Group
COMPAS - Common Performance Assessment System
ECOSOC - Economic and Social Council
GAVI - Global Alliance for Vaccines and Immunisation
IDA - International Development Association
KPI - Key Performance Indicator
M&E - Monitoring and Evaluation
MERG - Monitoring and Evaluation Reference Group
MI - Micro Indicator
MOPAN - Multilateral Organisation Performance Assessment Network
MoU - Memorandum of Understanding
NGO - Non-governmental organisation
NSP - National Strategic Plan
ODA - Overseas Development Assistance
OECD-DAC - Organisation for Economic Cooperation and Development – Development Cooperation Directorate
PAF - Programme Accelerated Funds
PAS - Performance Appraisal System
PBA - Programme-based approach
PCB - Programme Coordinating Board
RBM - Results-based Management
SIE - Second Independent Evaluation
UBRAF - Unified Budget, Results and Accountability Framework
UBW - Unified Budget and Workplan
UN - United Nations
UNAIDS - United Nations Programme on HIV/AIDS
UNDAF - United Nations Development Assistance Framework
UNDG - United Nations Development Group
UNDP - United Nations Development Programme
UNICEF - United Nations Children's Fund

Figures

Figure 2.1 Respondent Rating Scale
Figure 2.2 UNAIDS – Distribution of Responses (n=283) on all Questions Related to Micro- Indicators
Figure 2.3 Number of Respondents and Total Population for UNAIDS by Country and Respondent Group
Figure 2.4 MOPAN Ranges and Descriptions
Figure 3.1 Overall Ratings of UNAIDS Organisational Effectiveness by Respondent Group
Figure 3.2 Overall Ratings on Key Performance Indicators (mean scores, all respondents and document review ratings)
Figure 3.3 Quadrant I: Strategic Management, Survey and Document Review Ratings
Figure 3.4 Quadrant I: Strategic Management, Mean Scores by Respondent Group
Figure 3.5 KPI 1: Providing Direction for Results, Ratings of Micro-Indicators
Figure 3.6 KPI 2: Corporate Focus on Results, Ratings of Micro-Indicators
Figure 3.7 KPI 3: Focus on Thematic Priorities, Ratings of Micro-Indicators
Figure 3.8 KPI 4: Country Focus on Results, Ratings of Micro-Indicators
Figure 3.9 Quadrant II: Operational Management, Survey and Document Review Ratings
Figure 3.10 Quadrant II: Operational Management, Mean Scores by Respondent Group
Figure 3.11 KPI 5: Resource Allocation Decisions, Ratings of Micro-Indicators
Figure 3.12 KPI 6: Linking Aid Management to Performance, Ratings of Micro-Indicators
Figure 3.13 KPI 7: Financial Accountability, Ratings of Micro-Indicators
Figure 3.14 KPI 8: Using Performance Information, Ratings of Micro-Indicators
Figure 3.15 KPI 9: Managing Human Resources, Ratings of Micro-Indicators
Figure 3.16 KPI 10: Performance-oriented Programming, Ratings of Micro-Indicators
Figure 3.17 KPI 11: Delegating Authority, Ratings of Micro-Indicators
Figure 3.18 Quadrant III: Relationship Management, Survey and Document Review Ratings
Figure 3.19 Quadrant III: Relationship Management, Mean Scores by Respondent Group
Figure 3.20 KPI 12: Supporting National Plans, Ratings of Micro-Indicators
Figure 3.21 KPI 13: Adjusting Procedures, Ratings of Micro-Indicators
Figure 3.20 KPI 14: Using Country Systems, Ratings of Micro-Indicators
Figure 3.21 KPI 15: Contributing to Policy Dialogue, Ratings of Micro-Indicators
Figure 3.22 KPI 16: Harmonising Procedures, Ratings of Micro-Indicators
Figure 3.23 Quadrant IV: Knowledge Management, Survey and Document Review Ratings
Figure 3.24 Quadrant IV: Knowledge Management, Mean Scores by Respondent Group
Figure 3.25 KPI 17: Evaluating External Results, Ratings of Micro-Indicators
Figure 3.26 KPI 18: Presenting Performance Information, Ratings of Micro-Indicators
Figure 3.27 KPI 19: Disseminating Lessons Learned, Ratings of Micro-Indicators

Executive Summary

This report presents the results of an assessment of the Joint United Nations Programme on HIV/AIDS (UNAIDS) conducted by the Multilateral Organisation Performance Assessment Network (MOPAN). MOPAN assesses the organisational effectiveness of multilateral organisations based on a survey of stakeholders, a review of documents, and interviews with headquarter-based staff. In past years, MOPAN has not assessed an organisation’s development results, but is testing a component on this with four organisations in this year’s Common Approach assessment.[1]

UNAIDS, comprising the UNAIDS Secretariat and its co-sponsors, is an innovative partnership that has taken the lead in inspiring the world to achieve universal access to HIV treatment, prevention, care and support. UNAIDS is one of the main advocates for accelerated, comprehensive and coordinated global action on the HIV/AIDS epidemic. This implies coordinating the United Nations system’s efforts in the HIV/AIDS response, advocating and policy setting, monitoring the epidemic and providing evidence-based guidance, as well as providing technical assistance to national health and development efforts – all with the purpose of strengthening the global response to the HIV/AIDS epidemic.

In recent years, an external organisation-wide evaluation and operational reviews led UNAIDS to implement a number of reforms to improve its effectiveness, efficiency and governance. Key reforms have included the initiation of a holistic organisational change process, the improvement of results-based management systems, and the introduction of an Enterprise Resource Planning system. More concretely, the UNAIDS 2011-2015 Strategy – ‘Getting to Zero’, as well as the Unified Budget, Results and Accountability Framework (UBRAF) were developed and implemented in 2011 and a new organisational structure was implemented in early 2012.

MOPAN assessed UNAIDS based on a document review and an analysis of survey data collected at the organisation’s headquarters and in nine countries: Cambodia, Democratic Republic of Congo, Ghana, Honduras, Morocco, Niger, Nigeria, Philippines and Zimbabwe. The survey targeted UNAIDS’ co-sponsors, direct partners, and MOPAN donors based both in- country and at headquarters. A total of 283 respondents participated in the survey. MOPAN’s document review assessed UNAIDS through an examination of publicly available corporate documents and country programming documents from the nine countries selected.

MOPAN assessments provide a snapshot of four dimensions of organisational effectiveness (strategic management, operational management, relationship management, and knowledge management). The main findings of the 2012 assessment of UNAIDS are summarised below.

Strategic Management

In the area of strategic management, MOPAN established criteria to determine if a multilateral organisation has strategies and systems in place that reflect good practice in managing for results. Overall, the 2012 assessment found that:

• Strategic management is one of UNAIDS’ strengths. Survey respondents consider its focus on results and thematic priorities to be strong. The document review rated UNAIDS adequate or better on most key performance indicators in strategic management.

• UNAIDS is becoming a more results-oriented organisation. UNAIDS 2011–2015 Strategy – ‘Getting to Zero’ introduces key strategies to achieve its mandate and UNAIDS has developed a Unified Budget, Results and Accountability Framework (UBRAF), including work planning and implementation guidelines that are focused on results. Although these frameworks provide increased clarity and accountability for all stakeholders, there remains room for improvement in areas such as the clarity of results statements and selection of appropriate indicators.

• Among the cross-cutting priorities assessed by MOPAN, UNAIDS is seen to place the strongest emphasis on the promotion of human rights and gender equality. The UNAIDS 2011–2015 Strategy – ‘Getting to Zero’ includes the advancement of human rights and gender equality for the HIV response as one of its three key priorities. UNAIDS is also seen to be promoting the principles of good governance in its work, but does not have a mandate or institutional capacity to mainstream environmental issues.

Operational Management

In operational management, MOPAN established criteria to determine if a multilateral organisation manages its operations in a way that supports accountability for results and the use of information on performance. Overall, the 2012 assessment found that:

• UNAIDS’ strongest areas of performance in operational management relate to its use of performance information to revise and adjust policies and plan new interventions.

• UNAID’s financial accountability practices – in areas such as audit, anti-corruption and procurement – are viewed as appropriate and are generally supported by policies and guidelines.

• UNAIDS’ capacity to link budgets to expected development results has improved with the introduction of the UBRAF, which presents information in a result-oriented way. Indicative core resources are broken-down by strategic direction, goals, outcomes and outputs for each co-sponsor and region. Financial contributions to the sector by co-sponsors outside of the UBRAF are also noted.

• UNAIDS has revised its human resources policies to better address performance management. Its new Strategy on Human Resources 2011-2015 is fully aligned with the organisational strategy.

• The absence of an organisation-wide risk management framework is considered an area for improvement, as is the use of performance information to inform new initiatives and monitor their implementation.

Relationship Management

In relationship management, MOPAN established criteria to determine if a multilateral organisation is engaging with its partners at the country level in ways that contribute to aid effectiveness. Overall, the 2012 assessment found that:

• UNAIDS is seen to perform adequately or better across all of the key performance indicators in relationship management. It received high marks for its contribution to policy dialogue, in particular. The fact that the Joint Programme builds on and reinforces synergies with and among co-sponsors was also seen as an organisational strength.

• Feedback from MOPAN donors in-country was less positive than from other respondent groups regarding UNAIDS’ ability to adjust its procedures to local conditions and capacities.

• UNAIDS is perceived as performing adequately in the promotion of mutual accountability in its partnerships and as strong in avoiding parallel implementation structures.

Knowledge Management

In knowledge management, MOPAN established criteria to determine if a multilateral organisation has reporting mechanisms and learning strategies that facilitate the sharing of information inside the organisation and with the development community. Overall, the 2012 assessment found that:

• UNAIDS is appreciated for sharing lessons learned based on evidence.

• UNAIDS has taken steps to improve the independence of its evaluation function. The new Economics, Evaluation and Programme Effectiveness Division, as well as the Monitoring and Evaluation Reference Group (MERG), is meant to strengthen the guidance provided on monitoring and evaluation-related issues at all levels of the organisation. There is room, however, for improvement in the coverage and quality of evaluations.

• Although UNAIDS is seen to be performing adequately in most aspects of presenting performance information on its effectiveness, one area for improvement is its reporting on Paris Declaration commitments using indicators and targets.

Conclusions

The MOPAN 2012 assessment of UNAIDS was carried out at an important stage in the organisation’s evolution. UNAIDS committed to a broad organisational development process following its Second Independent Evaluation in 2009. It has shown its commitment to addressing all of the recommendations of this evaluation and has been developing systems to improve its ability to define, monitor, achieve and report on results. Many new approaches are still being tested and, as such, have yet to yield tangible effects.

The following conclusions are meant to encourage dialogue between MOPAN, UNAIDS, its direct partners and co-sponsors.

UNAIDS is highly valued by its direct partners and the co-sponsors.

In the four key performance areas examined in the MOPAN 2012 assessment, UNAIDS’s direct partners and co-sponsors were consistent in providing ratings of adequate or above, and often rated UNAIDS as strong or very strong.

UNAIDS’ commitment to organisational development has brought positive changes, although it is too early to assess the full effects of the process.

In recent years, UNAIDS has demonstrated strategic leadership and a commitment to organisational renewal while also continuing to track the epidemic and provide critical evidence- based guidance. Taking the recommendations from the Second Independent Evaluation and other operational reviews as the basis for this organisational development process, UNAIDS has reviewed its governance structure, strategies, as well as its systems and processes, all with the purpose of improving its capacity and efficiency in delivering a strengthened and more focused response in the fight against HIV/AIDS.

UNAIDS’ organisational development initiatives are likely to ensure the continuing relevance of its mandate as well as greater accountability of the UN response. However, the implementation of these initiatives has not yet caught up to the plan in areas such as results-based management and self-assessment processes (e.g., country programme assessments and accountability enhancement reviews, among others). It will take time for these initiatives to be fully implemented throughout the organisation, and UNAIDS will need provide consistent leadership to support these efforts, along with the training and financial resources required.

Although UNAIDS has made substantial progress in becoming a more performance- oriented and accountable organisation, there is room for improvement in its ability to measure its own performance.

UNAIDS has improved its focus on results and increased its efforts to measure its development effectiveness by developing and implementing the Unified Budget and Results Accountability Framework, a successor to the Unified Budget Workplan. Survey respondents were positive about UNAIDS’ new tools and processes for achieving results and the document review noted improvements in the structure and content of the organisation’s results frameworks. Nevertheless, UNAIDS has not yet maximised its use of performance information and improvements are still needed in the way results-based management is applied, notably in moving from activity-based to results-based reporting and in the use of performance indicators, baselines and targets to inform its work at the country level.

The UNAIDS Secretariat is valued for its technical expertise, evidence-based advocacy, and influence in policy setting.

UNAIDS is successfully maximising its knowledge and experience to support governments and other partners in pursuing national commitments. The technical expertise of UNAIDS’ country- based staff, as well as the use of this expertise for evidence-based advocacy, are highly valued by stakeholders on the ground. UNAIDS’ contributions to policy dialogue received the highest score of all key performance indicators in the survey. When asked to describe UNAIDS’ strengths, many country-based respondents commented on its expertise and technical support, as well as the effectiveness of UNAIDS’ influence on HIV-related policy setting.

UNAIDS’ highly consultative approach is crucial to the achievement of its mandate and its ‘Getting to Zero’ strategy.

UNAIDS is seen by direct partners and co-sponsors as highly consultative and inclusive, as it consistently seeks input from a wide range of stakeholders to inform the development and/or adjustment of its systems and tools. There is considerable evidence that UNAIDS benefited from consultative organisational development initiatives, such as the first and second multi- stakeholder consultations on the 2012-2015 Unified Budget and Results Accountability Framework and in the development of the UNAIDS 2011-2015 Strategy – ‘Getting to Zero’. This consultative and inclusive approach strengthens UNAIDS’ strategic positioning and leadership and increases its ability to achieve results. Nevertheless, UNAIDS could still improve its ability to be more strategic in its use of consultations.

UNAIDS’ effectiveness in building partnerships is highly valued and recognised by stakeholders as one of its strengths.

UNAIDS understands that partnerships and on-going relationships between and among partners are critical for the achievement of its commitments on universal access to treatment, prevention, care and support. As such, UNAIDS works to leverage existing partnerships and create new ones with various stakeholders. Examples of these collaborations include social movements, alliances, coalitions, networks, faith-based organisations, the private sector, as well as other development partners. UNAIDS convenes a wide range of stakeholders in a variety of settings to achieve common goals. The importance that UNAIDS places on meaningful, valued relationships with other organisations strengthens the UN’s capacity to meet its global targets and commitments.

UNAIDS’ unique organisational structure presents both opportunities and challenges.

UNAIDS’ structure encourages collaboration and teamwork within the UN family in response to HIV/AIDS. As such, the revised Division of Labour aims to enhance the efficiency and effectiveness of joint work among co-sponsors to improve the delivery of results. The co- sponsors outline their contributions to each area for which they are convenors or partners, and the Secretariat focuses mainly on guiding, coordinating, evidence-based policy setting, and ensuring coherence and cohesion in the overall response.

In the MOPAN assessment, some surveyed stakeholders perceived inefficiencies in the operationalisation of the Joint Programme, while others commended UNAIDS for its added value in coordination of the HIV/AIDS response. As UNAIDS unique structure has and will continue to present opportunities to the organisation, there remain significant challenges in ensuring that defined roles of the co-sponsors and the Secretariat are respected at all levels of the Joint Programme.

Overall MOPAN Ratings of UNAIDS

The chart below shows the ratings on the 19 key performance indicators that MOPAN used to assess UNAIDS in 2012. These indicators were designed to measure organisational effectiveness (practices and systems), not development results on the ground. The indicators were adapted to the work and structure of UNAIDS to encompass its joint programme dimensions. UNAIDS received ratings of strong on 11 of the 19 key performance indicators assessed by survey respondents, and document review ratings ranging from inadequate to strong.

1. Introduction

1.1 MOPAN

This report presents the results of an assessment of UNAIDS that was conducted in 2012 by the Multilateral Organisation Performance Assessment Network (MOPAN). In 2012 MOPAN assessed six multilateral organisations: the African Development Bank (AfDB), GAVI Alliance (formerly the Global Alliance for Vaccines and Immunisation), the Joint United Nations Programme on HIV/AIDS (UNAIDS), the United Nations Development Programme (UNDP), the United Nations Children’s Fund (UNICEF), and the World Bank.

Background

MOPAN was established in 2002 in response to international fora on aid effectiveness and calls for greater donor harmonisation and coordination. The purpose of the network is to share information and experience in assessing the performance of multilateral organisations. MOPAN supports the commitments adopted by the international community to improve the impact and effectiveness of aid as reflected in the Paris Declaration on Aid Effectiveness, the Accra Agenda for Action, and the Busan High Level Forum. MOPAN’s processes and instruments embody the principles of local ownership, alignment and harmonisation of practices, and results-based management (RBM).

MOPAN provides a joint approach (known as the Common Approach) to assess the organisational effectiveness of multilateral organisations. The approach was derived from existing bilateral assessment tools and complements and draws on other assessment processes for development organisations – such as the bi-annual Survey on Monitoring the Paris Declaration on Aid Effectiveness and annual reports of the Common Performance Assessment System (COMPAS) published by the multilateral development banks. In the long term, MOPAN hopes that this approach will replace or reduce the need for other assessment approaches by bilateral donors.

MOPAN Assesses Four Dimensions of Organisational Effectiveness

MOPAN has defined organisational effectiveness as the extent to which a multilateral organisation is organised to contribute to development and/or humanitarian results in the countries or territories where it operates.

Based on a survey of stakeholders and a review of documents, MOPAN assessments provide a snapshot of a multilateral organisation’s effectiveness in four dimensions:

• Developing strategies and plans that reflect good practices in managing for development results (strategic management)

• Managing operations by results to support accountability for results and the use of information on performance (operational management)

• Engaging in relationships with direct partners and donors at the country level in ways that contribute to aid effectiveness and that are aligned with the principles of the Paris Declaration (relationship management)

• Developing reporting mechanisms and learning strategies that facilitate the sharing of knowledge and information inside the organisation and with the development community (knowledge management).

In 2012, MOPAN also piloted a new component to assess a multilateral organisation’s contributions to development results. This component was tested with four of the six organisations assessed this year (AfDB, UNDP, UNICEF, and the World Bank).

Purpose of MOPAN Assessments

MOPAN assessments are intended to:

• Generate relevant, credible and robust information MOPAN members can use to meet their domestic accountability requirements and fulfil their responsibilities and obligations as bilateral donors

• Provide an evidence base for MOPAN members, multilateral organisations and direct partners to discuss organisational effectiveness and in doing so, build better understanding and improve organisational effectiveness and learning over time

• Support dialogue between MOPAN members, multilateral organisations and their partners, with a specific focus on improving organisational effectiveness over time, both at country and headquarters level.

The MOPAN methodology is evolving in response to what is being learned from year to year, and to accommodate multilateral organisations with different mandates. For example, the indicators and approach for the 2012 MOPAN review of a global fund and organisations with significant humanitarian programming were adapted to reflect the reality of these organisations.

1.2 Profile of UNAIDS

Created in 1996, in response to 1994/24 resolution of 26 July 1994 of the Economic and Social Council (ECOSOC), UNAIDS – the joint and co- sponsored United Nations programme on human immunodeficiency virus/acquired immunodeficiency syndrome (HIV/AIDS) – is a joint programme of eleven United Nations organisations[2] and the UNAIDS Secretariat.

The UNAIDS Secretariat and Headquarters are in Geneva, Switzerland and it has staff on the ground in more than 80 countries.

UNAIDS programme objectives are to:[3]

• Provide global leadership in response to the epidemic;

• Achieve and promote global consensus on policy and programmatic approaches;

• Strengthen the capacity of the United Nations system to monitor trends and ensure that appropriate and effective policies and strategies are implemented at the country level;

• Strengthen the capacity of national governments to develop comprehensive national strategies and implement effective HIV/AIDS activities at the country level;

• Promote broad-based political and social mobilisation to prevent and respond to HIV/AIDS within countries, ensuring that national responses involve a wide range of sectors and institutions;

• Advocate greater political commitment in responding to the epidemic at the global and country levels, including the mobilisation and allocation of adequate resources for HIV/AIDS-related activities.

In recent years, UNAIDS has played an active role in UN reform; it has strengthened its alignment to national processes in keeping with the reform of the United Nations and international efforts to improve aid effectiveness. Further, it has shown its commitment to the Paris Declaration on Aid Effectiveness (2003) and the Rome Declaration on Harmonisation (2005) by providing leadership and coordination within the UN family on HIV and AIDS-related responses, as defined in the UNAIDS Division of Labour and according to the Three One’s principles.

Governance and Structure

The UNAIDS Executive Director reports directly to the Programme Coordinating Board (PCB) and serves as the Secretary of the PCB. The PCB is the organisation’s governing body, composed of representatives from 22 Member States, the co-sponsors, and five NGOs (including associations of people living with HIV/AIDS).[4] The Committee of Co-sponsoring Organisations (CCO), a standing committee of the PCB, serves as a forum for the co- sponsoring organisations to meet on a regular basis and provide input into the policies and strategies of UNAIDS.[5]

At the headquarters level, UNAIDS Secretariat’s operations are carried out by a Programme branch comprised of two departments (Evidence, Policy and Innovation; and Rights, Gender and Community Mobilisation); a Political and Public Affairs branch (comprised of the Communications Department and three divisions); a Management and Governance branch (comprised of three departments: Human Resources Management; Planning, Finance and Accountability; and Technology and Innovation); and at the field level by regional and country offices, liaison offices, and Regional Support Teams.[6]

The Division of Labour document developed in 2006 defined leadership areas among the co- sponsoring agencies for various interventions to be implemented at the country level. It was updated in 2010 to include the roles and responsibilities of the Secretariat and to revise those of each co-sponsoring agency in the Joint Programme.

Strategy and Services

In 2010, the PCB adopted the UNAIDS Strategy 2011-2015 – Getting to Zero as a way of orientating the Joint Programme towards the following global commitments: achieving country set targets for universal access to HIV prevention, treatment, care and support; halting and reversing the spread of HIV; and contributing to the achievement of the Millennium Development Goals by 2015.

The strategy defines three strategic directions: to revolutionise HIV prevention; to catalyse the next phase of treatment, care and support; and to advance human rights and gender equality for the HIV response. These strategic directions as supported by 10 specific goals.[7]

As defined in the Division of Labour, the UNAIDS Secretariat is responsible for developing evidence, strategies and programmatic approaches of the response to the HIV epidemic, as well as providing evidenced-based guidance and technical support to countries in developing and implementing their national AIDS plans. The Secretariat also works to strengthen national capacities for the effective coordination, management, and monitoring of the response.

The co-sponsoring agencies are responsible for the implementation of the Joint Programme as defined in the Unified Budget and Results Accountability Framework (UBRAF). This new framework (2011) provides a link between the UNAIDS Strategy, expected results by agency, and corresponding budget allocations.

Finances

UNAIDS is able to fulfil its mandate as a result of voluntary contributions from governments, foundations, corporations, the private sector, and individuals. In 2011, total contributions reached approximately US$ 269.2 million. The organisation’s total budget for 2012-2013 is US$484.8 million, which is the same as in the previous biennium. Of this amount, US$323.4million is allocated to the Secretariat’s core budget and the remainder ($161 million) is allocated to the Joint Programme.

Organisational development initiatives

UNAIDS has commissioned two independent, organisation-wide evaluations in the past decade. The most recent was finalised in 2009 and included broad ranging recommendations about organisational effectiveness. UNAIDS responded to these recommendations by embarking on an extensive process of organisational change. An Implementation Plan was prepared to track their response to the various recommendations,[8] which included the following focus areas: mission statement and strategy development, partnerships, global programmatic mechanisms, delivery at country level, financial architecture, knowledge management, organisational issues, and governance.

In early 2012, the UNAIDS Secretariat was re-structured, thereby marking one of the final elements of the current organisational change process. Information on these changes can be found on the UNAIDS website: www.unaids.org.

1.3 Previous Assessments

Previous MOPAN Survey (2005) [9]

In 2005, MOPAN conducted a survey that gathered and analysed information on perceived behaviour of UNAIDS in its partnerships and interactions with national stakeholders and other development cooperation agencies at the country level. The survey was conducted in Albania, Bangladesh, Bolivia, Bosnia and Herzegovina, Ethiopia, Nicaragua, Tanzania, Vietnam, and Zambia.

This assessment was an opinion-based survey, which gathered the perceptions of MOPAN members about the in-country performance of UNAIDS relative to its mandate. Respondents were asked to give their views on the organisational behaviour. This assessment focused specifically on:

• UNAIDS’ national partnerships (contribution to policy dialogue, capacity development, advocacy, support to civil society, and alignment to national institutions, policies and administration); and,

• UNAIDS’ inter-agency partnerships (information sharing, inter-agency coordination, harmonisation and general local responsiveness).

The perception among MOPAN country teams was that UNAIDS was a small programme that played a supportive rather than a leading role with regard to HIV/AIDS. UNAIDS was seen as particularly effective in advocacy and policy dialogue and as performing relatively well in terms of inter-agency partnerships, especially through its promotion of the ‘Three Ones’: one agreed HIV/AIDS action framework, one national AIDS coordinating authority, and one agreed country- level monitoring and evaluation system.

2. MOPAN Methodology – 2012

2.1 Overview

Background

MOPAN continues to refine its assessment framework. In 2009, the MOPAN Common Approach replaced the Annual MOPAN Survey, which had been conducted since 2003. The Common Approach is broader and deeper than the previous surveys and includes the following components:

• Expanded survey – The MOPAN survey now brings in the views of direct partners or clients of multilateral organisations, peer organisations (or other relevant stakeholder group), and those of donors, that is, MOPAN members at both headquarters and country level.

• Document review – Since 2010, survey data are complemented by a review of documents prepared by the multilateral organisations being assessed and other sources.

• Interviews – In 2012, MOPAN complemented survey data and document review with consultations and interviews at the headquarters of multilateral organisations assessed.

In 2012 MOPAN also tested a new component to assess the results of multilateral organisations.[10]

As MOPAN’s methodology has changed significantly in the last three years, comparisons of this year’s assessments and previous assessments should take this into consideration.

The following is a summary of the MOPAN methodology in 2012.[11]

MOPAN 2012

In 2012, MOPAN assessed the effectiveness of six multilateral organisations: the African Development Bank (AfDB), GAVI Alliance (formerly the Global Alliance for Vaccines and Immunisation), the Joint United Nations Programme on HIV/AIDS (UNAIDS),[12] the United Nations Development Programme (UNDP), the United Nations Children’s Fund (UNICEF), and the World Bank. The assessment was conducted in Cambodia, Democratic Republic of Congo, Ghana, Honduras, Philippines, Morocco, Niger, Nigeria, and Zimbabwe.[13]

The MOPAN Common Approach examines organisational systems, practices, and behaviours that MOPAN believes are important for aid effectiveness and that are likely to contribute to results at the country level. It groups these organisational capacities in four areas of performance: strategic management, operational management, relationship management, and knowledge management.

Key Performance Indicators and Micro-indicators – Within each performance area, organisational effectiveness is described using key performance indicators (KPIs) that are measured with a series of micro-indicators (MIs).

The micro-indicators are assessed using data from a survey and document review. The survey collects perception data from a variety of stakeholders (see Section 2.2) and the review of documents relies on a set of criteria that provide a basis for the assessment of each micro- indicator (see Section 2.3). However, not all micro-indicators are assessed by both the survey and the document review; consequently, some charts do not show survey scores and document review scores for each KPI or MI.

UNAIDS was assessed using 19 KPIs and 67 MIs. The full list of MIs assessed is provided in Volume II, Appendix V (KPI and MI Data by Quadrant).

2.2 Survey

To gather diverse perspectives on the multilateral organisations being assessed, MOPAN generally seeks the perceptions of the following primary respondent groups:

• Donor Headquarters Oversight (HQ): Professional staff, working for a MOPAN donor government, who share responsibility for overseeing / observing a multilateral organisation at the institutional level. These respondents may be based at the permanent mission of the multilateral organisation or in the donor capital.

• Donor Country Office Oversight (CO): Individuals who work for a MOPAN donor government and are in a position that shares responsibility for overseeing/observing a multilateral organisation at the country level.

• Direct Partner (DP):[14] Typically, individuals who work for a national partner organisation (government or civil society) in a developing country. Respondents are usually professional staff from organisations that receive some sort of direct transfer from the multilateral organisation or that have direct interaction with it at country level (this could take the form of financial assistance, technical assistance, policy advice, equipment, supplies, etc.).

• Co-sponsors: This category is specific to UNAIDS and includes representatives from the ten co-sponsoring agencies based in country.

MOPAN donor respondents are chosen by MOPAN member countries. The direct partner/client, recipient government and peer organisation respondents are identified by the multilateral organisation being assessed.

The survey is customised for each organisation assessed and can be completed online in English, French, or Spanish or offline (paper, email, or interview) in these same languages. See Volume II (Appendix II) for the survey. Individual responses to the survey are confidential to the independent consultants managing the online survey or collecting data offline in the field.

Respondent Ratings – Survey respondents are presented with statements describing an organisational practice, system, or behaviour and asked to rate the organisation’s performance on a scale of 1 to 6 as shown below.

Figure 2.1 Respondent Rating Scale

In some cases, not all survey questions are answered, either because: 1) the individual chose not to answer, or 2) the question was not asked of that individual. In these cases, mean scores are calculated using the actual number of people responding to the question. As noted in the methodology (Volume II, Appendix I), ‘‘don’t know’’ survey responses are not factored into the calculation of mean scores. However, when the proportion of respondents answering ‘‘don’t know’’ is considered notable for a micro-indicator, this is indicated in the report. The responses of various categories of respondents on the six choices, plus ‘‘don’t know’’ are summarised across all survey questions in Figure 2.2.

Figure 2.2 UNAIDS – Distribution of Responses (n=283) on all Questions Related to Micro- Indicators

While there were responses in all six possible choices, relatively few responses overall were at the ‘weak’ end of the scale. Approximately 20 per cent of the responses of MOPAN donors at headquarters and in-country were ‘don’t know’, which is not unexpected given the potentially more limited familiarity of these groups with the operational specifics of UNAIDS covered in certain questions. Direct partners and co-sponsors gave fewer ‘don’t know’ responses.

Survey Response Rate

MOPAN aims to achieve a 70 per cent response rate from donors at headquarters and a 50 per cent response rate among the population of respondents in each of the survey countries (i.e., MOPAN country offices, direct partners, and co-sponsors). The number of respondents

targeted in each category (i.e., the total population) and the actual response rates are presented in Figure 2.3 below. Response rates of all categories of respondents, except MOPAN donors in-country, exceeded the 50 per cent target rate. While there are variations in the response rates by category and location of respondents, UNAIDS survey results reflect the views of 283 respondents. (See Volume II, Appendix III, Respondent Profile.)

Figure 2.3 Number of Respondents and Total Population for UNAIDS by Country and

Respondent Group

Converting Individual Scores to Group Ratings

As noted above, individuals respond to survey questions on a six-point scale where a rating of “1” is considered a judgment of “very weak” up to a rating of “6” intended to represent a judgment of “very strong.” A mean score is calculated for each respondent group (e.g., donors at HQ). Since mean scores are not necessarily whole numbers (from 1 to 6) MOPAN assigns numerical ranges and descriptive ratings for each range (from very weak to very strong) as shown below.

Figure 2.4 MOPAN Ranges and Descriptions

The ranges are represented to two decimal places, which is simply the result of a mathematical calculation and should not be interpreted as representing a high degree of precision. The ratings applied to the various KPIs should be viewed as indicative judgments rather than precise measurements.

Data Analysis

First level survey data analysis includes calculations of mean scores, medians, standard deviations, frequencies, (including analysis of ‘‘don’t know’’ and missing responses), as well as content analysis of open-ended questions. The ‘‘don’t know’’ responses are removed from the calculation of mean scores, but the proportion of respondents choosing ‘‘don’t know’’ is retained as potentially useful data.

A weighting scheme is applied to ensure that no single respondent group or country is under- represented in the analysis. The weighting is intended to correct for discrepancies/variation in: the number of individuals in each respondent group, the number of countries where the survey took place, the numbers of donors in-country, direct partners, and other respondent groups within each country where the survey took place. Weighted figures are carefully reviewed and analysed before inclusion in the multilateral organisation reports.

Second level analysis examines differences in the responses among categories of respondents and other variables. When significant differences are found, these are noted in the report.[15]

For a full description of survey data analysis see Volume II, Appendix I.

2.3 Document Review

The document review considers three types of documents: multilateral organisation documents, identified with the help of the organisation; internal and external reviews of the organisation’s performance, found on the organisation’s web site or provided by the organisation; external assessments such as the Survey on Monitoring the Paris Declaration, the Common Performance Assessment (COMPAS) report, and previous MOPAN surveys.

Ratings for key performance indicators (KPIs) are based on the ratings for the component micro-indicators in each KPI. For each micro-indicator, a set of criteria are established which, taken together, are thought to represent good practice in that area. The criteria are based on existing standards and guidelines (for example, UNEG or OECD-DAC guidelines), on MOPAN identification of key aspects to consider, and on the input of subject-matter specialists. The rating on any micro-indicator depends on the number of criteria met by the organisation. In cases where the micro-indicator ratings for one KPI are highly divergent, this is noted in the report.

While the document review assesses most micro-indicators, it does not assign a rating to all of them (when criteria have not been established). Consequently, some charts do not show document review scores for each KPI or MI. Documents are also used to aid in the understanding of the context in which the multilateral organisations work.

The document review and survey use the same list of micro-indicators, but some questions in the document review are worded differently from those in the survey. The document review and survey also use the same rating scale, but scores are presented separately on each chart in the report to show their degree of convergence or divergence.

2.4 Interviews

As of 2012, interviews are conducted at the headquarters of multilateral organisations with individuals who are knowledgeable in areas that relate to the MOPAN assessment.

Interviewees are asked to provide knowledge, insight, and contextual information that could assist the MOPAN Assessment Team in analysing document review data, and to identify other relevant documents for the Assessment Team to consider. This helps ensure that the Assessment Team has all the appropriate and necessary documents, enhances the Team’s ability to triangulate data from various sources, and assists the Assessment Team in the analysis of the key performance indicators by providing contextual information.

Interviews are conducted with a small number of staff who work in the primary units that relate to areas of the MOPAN assessment. Interviewees are identified by the multilateral organisation in conjunction with the Assessment Team and MOPAN. An interview guide is prepared and interviewees are advised of the content areas beforehand.

Data gathered during interviews is used to understand the context in which the agency is working, as well as how decisions are made. In the event that survey data present a picture that is very different from the document review, information from interviews can help clarify how the multilateral organisation approached a certain issue.

2.5 Basis for Judgment

From 2003 to 2009, the basis for judgment in MOPAN assessments was the perceptions of survey respondents. With the introduction of the document review in 2010 and interviews in 2012, judgments now draw on a variety of sources that can be compared and triangulated.

To the extent possible, the assessment standards and criteria are tailored to reflect the nature and operating environment of the multilateral organisations under review.

The MOPAN approach uses multiple data sources and data collection methods to validate findings. This helps eliminate bias and detect errors or anomalies.

The MOPAN reports gain trustworthiness through the multiple reviews and validation processes that are carried out by members of the network and by the multilateral organisations themselves.

2.6 Reporting

Institutional Reports

Individual institutional reports are produced for each multilateral organisation assessed. The results of the document review are presented alongside the survey results and discussed in light of the perception-based scores and interviews in order to further substantiate and contextualise the overall findings. For those agencies that were evaluated in 2009, a brief analysis of trends is included.

Country Data Summaries

A summary of survey results is produced for each multilateral organisation in each of the countries surveyed where sufficient survey data exists. Country Data Summaries provide feedback to those who participated in the MOPAN assessment and provide input for a dialogue process. They are not published and are shared only with individuals who attend the country workshop on the MOPAN assessment findings, which takes place in the first quarter of the year following the assessment.

2.7 Strengths and Limitations of Methodology

MOPAN continues to improve methodology based on the experience of each year of implementation. The following strengths and limitations should be considered when reading MOPAN’s report on UNAIDS.

Strengths

• The MOPAN Common Approach is based on the core elements of existing bilateral assessment tools. In the long term, the intent is to replace or reduce the need for other assessment approaches by bilateral donors.

• It seeks perceptual information from different perspectives: MOPAN donors (at headquarters and in-country), direct partners/clients of multilateral organisations, peer organisations, and other relevant stakeholders. This is in line with donor commitments to the Paris Declaration on Aid Effectiveness and the Accra Agenda for Action regarding harmonisation, partner voice, and mutual accountability.

• It complements perceptual data with document review and interviews, thus using multiple sources of data. This should enhance the analysis, provide a basis for discussion of agency effectiveness, and increase the validity of the assessment through triangulation of data.

• The reports undergo a validation process, including multiple reviews by MOPAN members and review by the multilateral organisation being assessed.

• MOPAN strives for consistency across its survey questions and document review for each of the multilateral organisations, while allowing for customisation to account for differences between types of multilateral organisations.

Limitations

MOPAN Framework

• The countries are selected based on established MOPAN criteria and comprise only a small proportion of each institution’s operations, thus limiting broader generalisations.

• The Common Approach indicators were designed for multilateral organisations that have operations in the field. For organisations that have limited field presence or that have regional structures in addition to headquarters and country operations, there have been some modifications made in the data collection method and there will be a need for greater nuance in the analysis of the data.

Data sources

• The MOPAN Common Approach asks MOPAN members and the organisations assessed to select the most appropriate individuals to complete the survey. While MOPAN sometimes discusses the selection with the organisation being assessed, it has no means of determining whether the most knowledgeable and qualified individuals are those that complete the survey. One challenge faced in the case of UNAIDS was getting sufficient respondents from MOPAN donors in-country (21 out of 43 responded).

• The document review component works within the confines of an organisation’s disclosure policy. In some cases, low document review ratings may be due to unavailability of organisational documents that meet the MOPAN criteria (some of which require a sample of a type of document, such as country plans, or require certain aspects to be documented explicitly). When information is insufficient to make a rating, this is noted in the charts.

Data Collection Instruments

• Three issues potentially affect survey responses. First, the survey instrument is long and a fatigue factor may affect responses and rates of response. Second, respondents may not have the knowledge to respond to all the questions (e.g., survey questions referring to internal operations of the organisation, such as financial accountability and delegation of decision-making, seem difficult for many respondents, who frequently answer ‘‘don’t know’.’) Third, a large number of ‘‘don’t know’’ responses may imply that respondents did not understand certain questions.

• The rating choices provided in the MOPAN survey may not be used consistently by all respondents, especially across the many cultures involved in the MOPAN assessment. One potential limitation is ‘central tendency bias’ (i.e., a tendency in respondents to avoid extremes on a scale). Cultural differences may also contribute to this bias as respondents in some cultures may be unwilling to criticise or too eager to praise.

• Because one of MOPAN’s intentions is to merge previously existing assessment tools into one, and to forestall the development of others, the survey instrument remains quite long.

Data Analysis

• While the document review can serve to evaluate the contents of a document, it cannot assess the extent to which the spirit of that document has been implemented within the organisation (unless implementation is documented elsewhere).

• Mean scores are used in the MOPAN reports to provide central tendency values of the survey results. The mean has the advantage of being the most commonly understood measure of central tendency, however, there is a disadvantage in using the mean because of its sensitivity to extreme scores (outliers), particularly when population samples are small. The assessment team reviewed the median and standard deviations in analysing the survey results. Volume II, Appendix V provides the standard deviations for each survey question.

Basis for Judgment

• Although MOPAN uses recognised standards and criteria for what constitutes good practice for a multilateral organisation, such criteria do not exist for all MOPAN indicators. As a result, many of the criteria used in reviewing document content were developed by MOPAN in the course of the assessment process. The criteria are a work in progress and should not be considered definitive standards.

• The Common Approach assessment produces numerical scores or ratings that appear to have a high degree of precision, yet can only provide general indications of how an organisation is doing and a basis for discussion among MOPAN members, the multilateral organisation, and other stakeholders, including direct partners.

Despite some limitations, the Assessment Team believes that the MOPAN reports generally provide a reasonable picture of systems associated with the organisational effectiveness of multilateral organisations.

3. Main Findings

3.1 Introduction

This chapter presents the findings of the 2012 MOPAN assessment of UNAIDS. Findings are based on respondent survey data and document review.

• Section 3.2 presents overall ratings on the performance of UNAIDS and summarises respondent views on its primary strengths and areas for improvement;

• Section 3.3 provides findings on each of the four areas of performance (strategic, operational, relationship, and knowledge management).

3.2 Overall Ratings

This section provides a summary of overall ratings. It includes: survey respondent ratings of UNAIDS’ overall organisational effectiveness, survey respondent views on UNAIDS’ strengths and areas for improvement, and survey and document review ratings for all key performance indicators.

Survey Ratings of UNAIDS Organisational Effectiveness

MOPAN has defined “organisational effectiveness” as the extent to which a multilateral organisation is organised to support direct partners in producing and delivering expected results. Respondents were asked the question: “How would you rate the overall organisational effectiveness of UNAIDS?” They were asked to provide a response that ranged from very effective to not at all effective. Their responses are presented In Figure 3.1. Of note, is the distribution of responses on the high end of the scale for co-sponsoring agencies, with 65 per cent considering UNAIDS to be effective (ratings of 5 or 6). On the low end of the scale, 21 percent of in-country donors provided the same ratings.

Figure 3.1 Overall Ratings of UNAIDS Organisational Effectiveness by Respondent Group

Respondents’ Views on UNAIDS’ Strengths and Areas for Improvement

The survey included two open-ended questions that asked respondents to identify UNAIDS’ Secretariat greatest strengths and areas of improvement. Of the 283 respondents who responded to the survey, 281 commented on the organisation’s strengths and 274 on areas for improvement.[16]

There were 392 comments on UNAIDS strengths and 329 comments on areas for improvement. The comments are summarised below with an analysis of the level of homogeneity among respondent groups as well as areas where the respondent groups felt differently about certain issues.

Respondents in all categories considered UNAIDS’ greatest strengths to be its leadership role, coordination capacity and its expertise and technical assistance.

Of the 392 comments on UNAIDS strengths, 36 per cent (143) identified UNAIDS’ leadership and coordination capacity as the organisation’s greatest strength (55 per cent among co-sponsors (52)). They noted that UNAIDS’ leadership and coordination not only support UN agency contributions in the fight against HIV and AIDS, but also encourage governments and UNAIDS partners to uphold global, regional and country commitments.

Across all respondent groups, 21 per cent of the comments (82) recognised UNAIDS’ staff expertise and technical assistance to countries as another of its strengths. A further 15 per cent (60) of comments acknowledged UNAIDS’ global advocacy on important policy issues. Two other strengths were identified by survey respondents:

• 11 per cent of comments noted partnership building with civil society, governments and others (42);

• 10 per cent of comments cited strategic guidance and programming approaches (38).

One of UNAIDS’ strengths is also identified as a weakness; UNAIDS’ cooperation and coordination function was identified by all respondent groups as its main area for improvement. There were diverse opinions among the respondent groups about other areas where UNAIDS could improve.

UNAIDS’ coordination and cooperation role and capacity (between and among the co-sponsors, the Secretariat, and beyond), was identified as both its greatest strength and main area for improvement. In fact, 18 per cent (59) of the 329 comments from survey respondents indicated a need for improvement in the promotion of synergies, the division of labour, and coordination among partners.

The apparent divergence of views between UNAIDS’ strengths and weaknesses is complex and can be explained by the organisation’s unique nature and structure within the UN family. While some stakeholders see UNAIDS’ added value in the coordination of the HIV/AIDS response, others perceive inefficiencies in the operationalisation of the Joint Programme.

Aside from comments on UNAIDS coordination, respondents had a range of opinions about other areas for improvement:

• 19 per cent of comments from donors at headquarters highlighted transparency and accountability;

• 21 per cent of comments from donors in-country identified advocacy;

• 15 per cent of comments from direct partners mentioned UNAIDS’ technical capacity;

• 27 per cent of comments from co-sponsors noted the need for more financial resources and improved capacity in resource mobilisation.

Overall Ratings of Key Performance Indicators

Figure 3.2 below presents scores from the document review and the survey on key performance indicators (KPIs) in the MOPAN 2012 assessment of UNAIDS. The grey bar presents the survey score, while the black square presents the document review score. For example, on the first indicator, “providing direction for results”, UNAIDS received a score of 4.63 (strong) in the survey and a score of 5 (strong) in the document review.

In the overall ratings from the survey and document review, UNAIDS was seen to perform adequately or better on the majority of key performance indicators.

UNAIDS received scores of strong on 11 of the 19 KPIs assessed in the survey.

UNAIDS received scores of strong on 8 of the 15 KPIs assessed in the document review.[17]

The survey and document review ratings differed on 9 KPIs – six of which were rated lower by the document review than by survey respondents, and the opposite for the remaining four. The reasons for these differences are discussed in the following sections.

Figure 3.2 Overall Ratings on Key Performance Indicators (mean scores, all respondents and document review ratings)

3.3 UNAIDS’ Performance in Strategic, Operational, Relationship, and Knowledge Management

3.3.1 Overview

This section presents the results of the 2012 Common Approach assessment of UNAIDS in four performance areas (quadrants): Strategic, Operational, Relationship, and Knowledge Management.

The following sections (3.3.2 to 3.3.5) provide the overall survey and document review ratings for the KPIs in each quadrant, the mean scores by respondent group, and findings based on an analysis of survey and document review ratings in each quadrant.

When there were notably divergent ratings between survey respondent groups or between the survey results and document review ratings, these are noted and the information gleaned from interviews with staff is integrated when it has a bearing on the analysis. Where statistically significant differences among categories of respondents were found, these differences are noted.[18]

The survey data for each KPI and MI by quadrant are presented in Volume II, Appendix V. The document review ratings are presented in Volume II, Appendix VI.

3.3.2 Strategic Management

UNAIDS is undertaking an organisational development process to become a stronger, results-based organisation. Stakeholders consistently rated UNAIDS’ performance in strategic management as strong based on MOPAN’s criteria. The document review provided several ratings of ‘strong’ and identified some areas for improvement.

Figure 3.3 shows the overall survey and document review ratings for the four KPIs in the strategic management quadrant.

The survey yielded strong ratings on all KPIs and document review yielded similar ratings, in the adequate to strong range. Stakeholder perceptions and documents reviewed suggest that there is a strong and transparent organisational commitment to results in UNAIDS’ structure, planning processes, management, and communication with stakeholders. UNAIDS is addressing cross-cutting priorities related to gender equality, good governance, and human rights, but is not yet mainstreaming the environment. Through its organisational development process, UNAIDS has made noteworthy improvements in all aspects of strategic management over the last several years.

Figure 3.3 Quadrant I: Strategic Management, Survey and Document Review Ratings

Figure 3.4 shows the mean scores for the four KPIs for all survey respondents, and by category of respondent.

Figure 3.4 Quadrant I: Strategic Management, Mean Scores by Respondent Group

KPI 1: Providing Direction for Results

Finding 1: Survey respondents rated UNAIDS’ executive management as strong in providing direction for the achievement of external/beneficiary-focused results. The document review noted that while UNAIDS makes many key documents available on its website and has a high level of transparency, it does not have a disclosure/access to information policy.

Overall, survey respondents rated UNAIDS as adequate or above on the three MIs in this KPI (see figure 3.5). The document review, which only rated the MI on availability of documents, gave UNAIDS a score of strong. The document review noted that while UNAIDS makes documents available, it does not have a disclosure / access to information policy.

Figure 3.5 KPI 1: Providing Direction for Results, Ratings of Micro-Indicators

MI 1.1 – Value system supports results-orientation and partner focus

This MI was not assessed by the document review, but was addressed in two survey questions on institutional culture. Survey respondents were asked whether UNAIDS has a value system that supports a results-orientation and a partner focus. The majority of survey respondents (82 per cent) rated UNAIDS as adequate or above for both questions. Direct partners were more positive than MOPAN donors in-country and the difference is statistically significant.[19]

MI 1.2 – Leadership on results management

MOPAN donors at headquarters were the only respondent group asked about the extent to which the Secretariat's senior management shows leadership on results management, for which the majority of the survey respondents (97 per cent) rated the organisation as adequate or above.

MI 1.3 – Key documents available to the public

Across all respondent groups, 85 per cent agreed that UNAIDS’ practice of making key documents available to key partners, including the public, is adequate or above.

The document review found UNAIDS to be strong in this area. Although the organisation does not have a disclosure / access to information policy available on its website, a ‘Privacy and Use of Data Policy Statement’, as well as most documents are available on the public website (such as Programme Coordinating Board meeting minutes and decisions, Executive Director’s statements, organisation mandate and structure, UNAIDS Division of Labour, organisation-wide strategic plan, thematic reports, and audits and evaluations, among others). It also bears noting that prior to Board meetings, UNAIDS makes documents available on its website. Most documents are available in multiple languages (English, French and Spanish). Various corporate publications are also available in English. In addition, the website has clear instructions on how to contact the organisation.

KPI 2: Corporate Focus on Results

Finding 2: UNAIDS’s organisational strategies and plans were rated as strong in terms of their focus on results. The document review recognised the recent improvements in UNAIDS’ results focus as embodied in the new Strategy 2011-2015 and the 2012-2015 Unified Budget, Results and Accountability Framework. There remains room for improvement in the quality of its results frameworks.

Over the last few years, UNAIDS has developed frameworks that operationalise its mandate and that improved its focus on results. A new organisational structure was established in the first quarter of 2012 to reflect changes to the Secretariat’s institutional architecture and strategic focus. The organisation also introduced a new 2012-2015 Unified Budget, Results and Accountability Framework (UBRAF), which is based on a four-year planning cycle, biennial budget cycles, and yearly work plans. The structure of the UBRAF, as well as the guidance developed along with it, should allow for progress towards results to be measured more accurately, improve accountability, strengthen work within the Joint Programme, and enhance the generation and use of strategic information. The latter, associated with the revised Division of Labour, represent a major advance in terms of accountability as they define responsibilities of the co-sponsors. Changes to programming, monitoring and reporting introduced along with the UBRAF could, when fully implemented, help to address certain weaknesses noted by the independent evaluations about the focus and reporting on results.

MOPAN members at headquarters were the only survey respondents questioned on this KPI.

Figure 3.6 KPI 2: Corporate Focus on Results, Ratings of Micro-Indicators

MI 2.1 – Organisational strategy based on clear mandate

All headquarters-based MOPAN donors (100 per cent) rated UNAIDS as adequate or above on the link between organisational strategies and a clear mandate, and 80 per cent perceived UNAIDS to be strong or very strong.

UNAIDS received a rating of strong from the document review. Evidence shows that successive resolutions have brought clarity and agreement on UNAIDS’ evolving role given the changing global context and needs relating to combating the HIV and AIDS pandemic. The evolving nature of UNAIDS is reflected in periodic revisions of its mandate and the reasonable linkages between the strategic direction and goals articulated in the organisational strategic plan UNAIDS 2011-2015 Strategy - Getting to Zero and the mandate. This strategy has been developed through wide consultation in order to ensure its alignment with the six components articulated in its mandate – all with the purpose of coordinating the response to the HIV/AIDS pandemic.[20]

MI 2.2 – Structure suited to mandate and results delivery

When asked whether UNAIDS’s structure was suited to its mandate and the delivery of results, the majority of MOPAN donor respondents (83 per cent) rated UNAIDS as adequate or strong.

Despite the unique structure of the organisation, which comprises a Secretariat and a Joint Programme, the document review scored UNAIDS as strong on this MI. Evidence shows that the objectives of the Economic and Social Council (ECOSOC), agreed to when UNAIDS was created, continue to be relevant and have not constrained the organisation from adapting to changing circumstances. In 2009, UNAIDS defined a new modus operandi for its interagency work, identifying five core principles and values: stronger governance and accountability, clearer focus on results, increased leadership by the convener or co-conveners, standardised interagency reporting mechanisms, and aligned coordinating and technical mechanisms. The approach is aligned with the Strategy for 2011–2015, the Partnership Strategy, the revised Division of Labour, and the respective responsibilities of co-sponsors and the Secretariat. Further, it is operationalised through numerous vertical and horizontal mechanisms and the UBRAF.

MI 2.3 – Application of results management

The majority of donors from MOPAN headquarters (91 per cent) rated UNAIDS as adequate or above in the application of results management.

The organisation received a rating of strong from the document review, based on previous Unified Budget and Work plans (UBWs), the UBRAF, and related work planning, monitoring and reporting documents. Evidence shows that UNAIDS promotes an organisation-wide policy on results management and has made efforts to improve its practice through the development and implementation of the UBRAF, which comprises a business plan, results and accountability matrix and budget, as well as resource allocation guiding principles. UNAIDS also provided training on RBM to its staff and supported increased knowledge about the framework’s use among its staff and the co-sponsors. Despite the fact that the UBRAF and the Division of Labour represent a major advance in terms of accountability as they define the responsibilities and commitments of the co-sponsors, the UNAIDS Secretariat does not have any direct authority over the implementation process carried out by the co-sponsors.

MI 2.4 – Plans and strategies contain results frameworks

MOPAN donors at headquarters were asked whether organisation-wide plans and strategies contain frameworks of expected management and development results. The majority of survey respondents (91 per cent) rated UNAIDS as adequate or above.

The document review rated UNAIDS adequate on this MI. The UBRAF, which was introduced for the 2012-2013 biennium and is a successor to the UBW, includes a results and accountability matrix detailing the strategic directions, goals and functions of the Secretariat and the co-sponsors of the Joint Programme. While the accountability matrix provides guidance on the coherence, coordination and impact of the UN’s response to AIDS by combining the efforts of the co-sponsors and the UNAIDS Secretariat, the results matrix describes outcomes, outputs and deliverables for 20+ countries, regions and global level action that the Joint Programme will focus on. However, not all statements of UBRAF results are appropriate to their results level and there is some confusion of UNAIDS outputs and country outcomes. Nonetheless, the UBRAF represents an improvement over other documents reviewed, such as previous UBWs, which did not include a management results framework or specific outcomes for the UNAIDS Secretariat.

MI 2.5 – Results frameworks link outputs to final outcomes/impacts

MOPAN donors based in their headquarters were asked whether UNAIDS' results framework (UBRAF) includes causal links from outputs through to outcomes and impact. The majority of survey respondents (70 per cent) rated UNAIDS as adequate or above.

The document review gave UNAIDS a rating of strong on this MI as both the results and accountability sections of the matrix (UBRAF) have relatively clear causal links throughout the results chain (e.g., UNAIDS Secretariat and joint programme work may lead to country outputs, outcomes, and impacts). The accountability section tracks the achievement of UNAIDS’ functional objectives (management goals) by linking outputs to outcomes. The results section has clear and plausible causal links in the results chain - from outputs to outcomes and goals under each strategic direction. The UNAIDS 2011-2015 Strategy – Getting to Zero also contributes by providing implicit and explicit descriptions of how the outputs in the results framework are linked to expected outcomes and goals. However, there is still room to improve the results chain, as well as to sharpen some of the results statements to render them SMART.[21]

MI 2.6 – Plans and strategies contain performance indicators

MOPAN donors at headquarters were asked if UNAIDS' results framework includes measurable indicators at output and outcome levels. The majority (79 per cent) rated UNAIDS as adequate or above, while 9 per cent rated it inadequate or below, and 12 per cent responded ‘don’t know’.

The document review rated UNAIDS as weak on this MI. Previous performance monitoring frameworks (related to UBWs) that were reviewed provided guidance on the required structure and key elements of the framework. They also listed all of the performance indicators related to results at the output and outcome levels. The current results and accountability matrix of the UBRAF presents joint outcome and impact performance indicators that are sometimes unclear and not fully relevant to the results with which they are associated. In addition, various performance indicators that UNAIDS uses to measure progress in the results section of the matrix lack targets with clear dates for achievement. In the accountability section of the matrix many targets have yet to be defined, thereby affecting the extent to which performance indicators can be monitored or assessed.

Despite the weak rating, UNAIDS has made progress in this area. Following the 29th Meeting of the UNAIDS PCB, the Co-sponsors Evaluation Working Group (CEWG) aligned and standardised the set of indicators to be used in the UBRAF, the co-sponsors results frameworks, and existing global indicators. A guidance note, updating the indicators as part of the monitoring and evaluation framework for the UBRAF, was published for the 30th Meeting of the PCB.

KPI 3: Focus on Thematic Priorities

Finding 3: Survey respondents rated UNAIDS as strong for mainstreaming thematic priorities that are most aligned to its core mandate. The document review rated UNAIDS as strong in mainstreaming gender equality, human rights and governance, but very weak in mainstreaming environment.

The assessment looked at four cross-cutting themes identified as priorities by MOPAN. Three of these (gender equality, good governance, and human rights) were assessed by both the document review and the survey. The environment was assessed only by the document review. These thematic issues are integrated in different ways into the strategy and programming of the organisation.

Overall, survey respondents rated UNAIDS as adequate and above on this KPI.

Ratings from both survey and document reviews suggest that UNAIDS performs relatively well in those thematic areas aligned with its core mandate (gender equality and human rights-based approach). UNAIDS has given less emphasis to good governance in its key strategic documents and this is reflected in the survey rating. Although good governance remains at the forefront of the mandate of UN agencies – the Secretariat and the co-sponsors alike – it is not identified as a priority in the latest strategic plan (2011-2015). Mainstreaming the environment is not a UNAIDS priority, and the document review rated this area very weak.

Figure 3.7 KPI 3: Focus on Thematic Priorities, Ratings of Micro-Indicators

MI 3.1 – Gender equality

Survey respondents were asked whether UNAIDS sufficiently promotes gender equality in its work. The majority of survey respondents (89 per cent) rated UNAIDS as adequate or above.

The document review rated UNAIDS as strong in its efforts to mainstream gender throughout the organisation. The UNAIDS 2011–2015 Strategy – Getting to Zero includes gender equality as one of the three strategic directions: ‘Advance human rights and gender equality for the HIV response’. Reporting from the Secretariat and the co-sponsors to the Programme Coordinating Board includes gender-specific information from activities for which the Secretariat and individual co-sponsors have defined leadership roles and programmatic responsibilities.

Recent organisational changes have also led to the formation of the new Gender Equality and Diversity unit. Further, UNAIDS developed an Agenda for Accelerated Country Action for Women, Girls, Gender Equality and HIV 2010-2014, which aims to mobilise all constituencies, starting with UNAIDS and UN Women, around strategic actions such as increased political commitment, enhanced capacity, and increased resources. This Agenda includes an accountability framework in which timelines and responsibilities have been defined for monitoring, evaluating and reporting functions. Recently, the PCB requested that all reporting on the Action Agenda and associated budgetary allocations be integrated into the UBRAF under Goal C3 - HIV-specific needs of women and girls are addressed in at least half of all national HIV responses and C4 - Zero tolerance for gender-based violence. Nevertheless, not all recommendations of the Agenda have been integrated in the UBRAF and UNAIDS has no direct authority over the budget allocations of the co-sponsors. Despite positive initiatives in certain areas, there was no evidence found of expenditure reviews for gender mainstreaming having been conducted or specific budgetary allocations for the implementation of gender mainstreaming activities.

MI 3.2 – Environment

The survey did not include a question on mainstreaming the environment.

Strictly adhering to the MOPAN criteria, UNAIDS was rated as very weak on this MI in the document review as neither the UBWs, the UBRAF, nor any other document made available to the Assessment Team identified environment as a cross-cutting theme. UNAIDS does not have a mandate or institutional capacity to mainstream environmental issues. The absence of evidence suggests that the responsibility of including social and environmental safeguards, sustainability measures, as well as undertaking situation analysis, planning, budgetary allocation and the implementation of mainstreaming activities lies with the co-sponsors. Interviews with respondents from UNAIDS headquarters highlighted that previous attempts to evaluate how HIV and AIDS are related to the environment have not yielded concrete results. This is understandable given the mandate, structure and work of UNAIDS.

MI 3.3 – Good governance

Survey respondents were asked whether UNAIDS sufficiently promotes the principles of good governance in its work. The majority of survey respondents (84 per cent) rated UNAIDS as adequate or above. However, it should be noted that despite the mean score of adequate for MOPAN donors in-country, almost one-quarter of these respondents gave a rating of inadequate or below, making the average score for this respondent group lower than the average scores for other respondent groups. The differences in mean scores are statistically significant.

The document review assessed UNAIDS as strong on this MI based on the fact that national capacity building as a form of good governance is central to the mandate of the UN agencies. The UNAIDS “Three Ones” key principles served to focus all co-sponsoring agencies on a coordinated response to HIV and AIDS by national authorities. UNAIDS attaches great importance to strengthening surveillance systems and providing countries with relevant and effective tools to track the epidemic and respond appropriately. As such, UNAIDS has provided support to strengthen the governance, institutional, financial and M&E capacity of national authorities, as well as networks and organisations of people living with HIV (PLHIV). However, according to the Second Independent Evaluation (2009), this support was not consistent across countries and UNAIDS’ support at country level has focused more on national networks and umbrella organisations and less on parliamentarians and civil servants.[22] These areas for improvement have been strengthened in the UNAIDS 2011-2015 Strategy – Getting to Zero, which addresses governance as a focus area under Strategic Direction One – ‘Accountability through ownership: people, countries and synergies’. The strategy aims to foster national ownership of the response by enabling countries to lead, manage and establish accountability systems, and by creating space for debate and policy dialogue on the governance of the response. The UBRAF also shows evidence of outputs and joint deliverables geared towards the achievement of greater impact through improved governance of the national response.

MI 3.4 – Human rights-based approaches

Survey respondents were asked whether UNAIDS sufficiently applies human rights-based approaches (HRBA) to development in its programmatic work. The majority of survey respondents (96 per cent) rated UNAIDS as adequate or above on this MI, and 66 per cent perceived UNAIDS to be strong or very strong.

UNAIDS’ support for human rights-based approaches received a rating of strong from the document review. UNAIDS 2011–2015 Strategy – Getting to Zero identifies human rights as a key priority and the UBRAF includes results and deliverables that will strengthen national responses and encourage global commitment to HIV in human rights terms (Goals C1 and C2). However, as mentioned in MI 3.1, UNAIDS has no direct authority over the budgetary allocations of the co-sponsors. In addition, a newly designed Investment Framework takes as its starting point a human rights approach to the HIV response to ensure that it is universal, equitable and inclusive. In 2011, the development of a human rights strategy was recommended by the UNAIDS Reference Group on HIV and Human Rights. This strategy intends to translate the UNAIDS Strategy and its global human rights goals and activities into country-level results. Finally, a Human Rights and Law division was recently created in the organisational structure under the Rights, Gender and Community Mobilisation Department. Although UNAIDS has not yet fully developed and integrated institutional systems and associated capacities for mainstreaming human rights, documents reviewed show evidence that the organisation is committed to improving this area.

KPI 4: Country Focus on Results

Finding 4: Although survey respondents rated UNAIDS as strong on all aspects of this key performance area, the document review found that results-based management at the country level needs improvement.

The survey questions focused on results-based frameworks at the country level. Overall, 86 per cent of respondents across all respondent groups rated UNAIDS as adequate or above. However, in each of the micro-indicators in this area, the ratings of MOPAN donors in-country were significantly less positive.

The review of documents included UNAIDS’ country office work plans and national UN HIV/AIDS Joint Support Programme 2011-2015 in eight countries (Cambodia, Democratic Republic of Congo, Ghana, Honduras, Morocco, Nigeria, Niger and Zimbabwe).

The KPI received ratings of inadequate on three MI and strong on the remaining MI by the document review. Document review acknowledged UNAIDS’ inclusion of results consistent with UNDAF and national development strategies in its country frameworks, but noted shortcomings with regard to results statements and the extent to which performance indicators provided sufficient bases upon which to assess performance.

Figure 3.8 KPI 4: Country Focus on Results, Ratings of Micro-Indicators

MI 4.1 – Frameworks link results at project, sector, and country levels

Survey respondents were asked whether UNAIDS links results at project, sector and country levels. The majority of survey respondents (86 per cent) rated UNAIDS as adequate or above, and 61 per cent of direct partners and 67 per cent of co-sponsors gave ratings of strong and very strong. MOPAN donors in-country, however, were less positive than direct partners and co-sponsors. The differences are statistically significant.

The document review rated UNAIDS’ performance as inadequate in ensuring that country programming frameworks link results from project/programme, sector and country levels. While national UN HIV/AIDS Joint Support Programme 2011-2015 contains logical frameworks and expected results that are tied to the United Nations Development Assistance Framework (UNDAF) and National Strategic Plans (NSP), UNAIDS country office work plans contain results that are linked to the UBRAF and the global strategy. One important shortcoming of the UNAIDS country office work plans sampled is that whereas activities listed reflect what the organisation does, the country offices themselves are not responsible for the outputs (i.e., results statements contained in the work plans represent national outputs and outcomes, not those for which UNAIDS is responsible or accountable).

MI 4.2 – Frameworks include indicators at project, programme, sector, and country levels

Survey respondents were asked whether UNAIDS’ results frameworks include indicators at all levels (country, sector, and project/programme). The majority of survey respondents (84 per cent) rated UNAIDS as adequate or above, and 65 per cent of direct partners and 63 per cent of co-sponsors gave ratings of strong and very strong. MOPAN donors in-country rated UNAIDS adequate on this MI, whereas direct partners and co-sponsors gave ratings of strong. These differences are statistically significant.

The document review rated UNAIDS as inadequate on this MI. More than half of the UNAIDS country work plans sampled and national UN HIV/AIDS Joint Support Programme 2011-2015 contained performance indicators that did not provide sufficient basis upon which to assess performance (i.e. not quantifiable, clear or relevant). In addition, the indicators of the UNAIDS country office work plans rarely have quantified targets or identified data sources. Associated results also lack baselines.

MI 4.3 – Expected results consistent with national strategies and UNDAF

Survey respondents were asked whether UNAIDS’ results frameworks contain statements of expected results consistent with national HIV strategies. The majority of survey respondents (92 per cent) rated UNAIDS as adequate or above, and 79 per cent of co-sponsors, 75 per cent of direct partners, and 34 per cent of MOPAN donors in-country rated UNAIDS strong and very strong on this MI. Co-sponsors were more positive than direct partners and MOPAN donors in- country, even though their ratings all fall within the range of strong. The differences are statistically significant.

The document review rated UNAIDS as strong on this MI. In most national UN HIV/AIDS Joint Support Programme strategies reviewed, the link between the Joint Programme’s expected results and those identified in United National Development Assistance Framework (UNDAF) and the National Strategic Plan was clear and credible. In almost all cases, the results framework contained a reference showing the link to expected results in the NSP and the UNDAF.

MI 4.4 – Expected results developed in consultation with co-sponsors and partners

Survey respondents were asked whether UNAIDS Secretariat consults with co-sponsors and partners to develop its expected results. The majority of survey respondents (82 per cent) rated UNAIDS as adequate or above, and 75 per cent of co-sponsors and 63 per cent of direct partners rated UNAIDS strong and very strong on this MI. MOPAN donors in-country were less positive than direct partners and co-sponsors and the differences are statistically significant.

This MI was not assessed by the document review.

MI 4.5 – Results for cross-cutting priorities included in results frameworks

Survey respondents were asked whether UNAIDS includes results related to cross-cutting priorities, such as gender and human rights, in its results frameworks. The majority of survey respondents (87 per cent) rated the UNAIDS as adequate or above on this MI.

The document review rated UNAIDS as inadequate on this MI. The document review noted that organisationally relevant cross-cutting priorities (the same as those assessed in KPI 3) were at least briefly mentioned in most UNAIDS country office work plans and in the UN HIV/AIDS Joint Support Programme country strategies reviewed. In most cases, there were specific references to gender, human rights, and good governance. However, none of the country strategies included every cross-cutting theme assessed by MOPAN for UNAIDS. Furthermore, most frameworks sampled did not provide evidence of the strategies or approaches to be used to address the cross-cutting priorities.

3.3.3 Operational Management

Despite UNAIDS’ adequate performance and recent improvements in operational management, it continues to face challenges in some areas.

Figure 3.9 below shows overall survey and document review ratings for the seven KPIs in the operational management quadrant. Figure 3.10 shows mean scores for the KPIs for all survey respondents, and by respondent groups.

In the survey, UNAIDS was rated as adequate or above on the seven KPIs assessed in operational management, and strong on performance-oriented programming and delegating authority. One-third of respondents answered ‘don’t know’ to survey questions on KPIs 5 through 9.

In the document review, UNAIDS was rated as adequate or strong on all MIs.

Figure 3.9 Quadrant II: Operational Management, Survey and Document Review Ratings

Figure 3.10 shows the mean scores for the KPIs for all survey respondents, and by respondent groups.

Figure 3.10 Quadrant II: Operational Management, Mean Scores by Respondent Group

KPI 5: Resource Allocation Decisions

Finding 5: According to survey respondents, UNAIDS performs adequately in making transparent resource allocation decisions. The document review found that UNAIDS has been able to provide co-sponsors with predictable resources and rated UNAIDS as strong in this area.

Three MIs were assessed under this KPI. One was assessed through the survey only and two by both document review and survey. Overall, survey respondents rated UNAIDS’ performance on aid allocation decisions as adequate. The document review found evidence that led to a rating of strong for both MI 5.1 and MI 5.3. The criteria for allocation of aid resources are available in the 2012-2015 UBRAF.

Figure 3.11 KPI 5: Resource Allocation Decisions, Ratings of Micro-Indicators

MI 5.1 – Criteria for allocating resources publicly available

Survey respondents were asked whether the UNAIDS Secretariat makes its criteria for allocating UBW/ UBRAF resources readily available. The majority of survey respondents (62 per cent) rated UNAIDS as adequate or above, 10 per cent as inadequate or below, and 28 per cent answered ‘don’t know’.

The document review rated UNAIDS as strong. The allocation of core resources is approved by the Programme Coordinating Board (PCB). Based on the results and deliverables identified as part of the UBRAF process, the co-sponsors and Secretariat estimate resource needs from the core budget as well as other resources that they have or expect to mobilise in the subsequent two-year period. The allocation of core resources is based on priority countries/ regions, the performance of the co-sponsors, and the funds that individual co-sponsors raise. The criteria for allocation of aid resources (including specific criteria to determine the allocation of UNAIDS Secretariat resources at country level) are described in general terms in the UBRAF, however, this publication does not provide all of the details of the allocation formula. The UBRAF is publically available.

MI 5.2 – Resource allocations conform to criteria

Survey respondents were asked whether the UNAIDS Secretariat allocates UBW/ UBRAF resources according to the criteria. The majority of survey respondents (56 per cent) rated the UNAIDS as adequate or above. MOPAN donors in-country were less positive than direct partners and donors at headquarters, even though their ratings all fall within the range of adequate.

MI 5.3 – Resources released according to agreed schedules

Survey respondents were asked whether UNAIDS releases resources according to agreed schedules. The majority of survey respondents (60 per cent) rated the UNAIDS as adequate or above and 34 per cent of all respondents answered ‘don’t know’.

The document review rated UNAIDS as strong on this MI. This indicator is based on Indicator 7 of the Paris Declaration on Aid Effectiveness, which measures the gap between aid scheduled and aid effectively disbursed and recorded in countries’ accounting systems. However, in the context of UNAIDS, this indicator has been interpreted to mean the planning and delivery of funding from the UNAIDS Secretariat to the co-sponsors as defined in the Joint Programme.

The document review found that through the UBW, the UNAIDS Secretariat provided co- sponsors with predictable resources on a biannual basis and did so early in the year, thereby allowing them to undertake and implement planned activities. One may argue that predictability in scheduling was too high within the UBWs as co-sponsors received largely the same amounts in each biennium without sufficient attention to expenditure and performance. The new policy related to the UBRAF stipulates the criteria by which UBRAF disbursements will be made and the monitoring of implementation and results achieved to be carried out. UBRAF disbursements are based on agreed 2-year programme budgets, an implementation plan, and financial implementation thresholds (e.g., an agency must spend 40 per cent of its first year indicative allocation of the biennium budget to receive the remaining amount in the second year). Further, performance will be assessed when determining the amount to be disbursed in a second tranche. The timing of this assessment did not allow for an evaluation of the timeliness of the disbursements made under the new UBRAF.

The IHP+ Results partner scorecard for UNAIDS provides an indicative result for aid disbursements made in the health sector. In 2009, 92 per cent of UNAIDS’ health sector aid disbursements were released according to agreed schedules – a decrease from 100 per cent in 2007, but still meeting the indicative target of 90 per cent.[23] This would seem to corroborate the document review findings in terms of the organisation’s predictability of disbursements.

KPI 6: Linking Aid Management to Performance

Finding 6: Surveyed donors at headquarters rated UNAIDS’ performance in linking aid management to performance as largely adequate and the document review found UNAIDS to be strong in this area.

MOPAN donors at headquarters were the only respondents asked to assess UNAIDS’s performance under this KPI. They rated the organisation as adequate in terms of linking budget allocations to expected results and in linking disbursements to expected results.

The document review found that UNAIDS’ capacity to link aid management to performance has improved with the implementation of the UBRAF and the creation of an Enterprise Resource Planning system, and rated this KPI as strong.

Figure 3.12 KPI 6: Linking Aid Management to Performance, Ratings of Micro-Indicators

MI 6.1 – Allocations linked to expected results

MOPAN Donors at headquarters were asked if they felt that UNAIDS links budget allocations to expected results. The majority (73 per cent) rated UNAIDS as adequate or above on this MI and 21 per cent answered ‘don’t know’.

The document review rated this MI as strong. It found evidence of improvements in UNAIDS’ capacity to link budgets to expected development results. The UBRAF presents information in a result-oriented way. Indicative core resources are broken-down by strategic direction, goals, outcomes and outputs for each co-sponsor and region. Further, UNAIDS uses an Enterprise Resource Planning (ERP) system to operationalise the UBRAF, thereby linking work plans and budgets to results. The ERP system tracks results from a strategic planning level (UBRAF) to operational planning and financing (country work plans) to programme implementation and monitoring and evaluation of the Joint Programme. To support budgeting and tracking of expenditures against the approved results structure, each work plan output and the costs of goods or services (including salaries) related to it, is linked to a result.

MI 6.2 – Disbursements linked to reported results

MOPAN donors at headquarters were asked if UNAIDS’ reports on results include the amount disbursed to achieve those results. The majority (52 per cent) rated UNAIDS as adequate or strong; 30 per cent of respondents answered ‘don’t know’.

The document review rated UNAIDS as adequate on this MI. Evidence of financial disbursements aligned with achieved results was found in reporting documents sampled. Under the previous UBW system, the latest Performance Monitoring Report (2010) reported the 2010-2011 UBW mid-term expenditures by principal outcome but variances in the achievement of results were neither reported nor explained. Improvement was observed under the newly implemented UBRAF system as the ERP captures information such as variances and explanations of operational expenditures and results achievement. This information is consolidated at the organisational level and provides the basis for the annual reporting to the Programme Coordinating Board. The tools being introduced should enable UNAIDS to compare actual results to expected results and to report on other performance metrics such as cost, schedule, disbursement, expenditure, implementation progress. However, there is limited evidence of their implementation as these tools are in their infancy and it is too early for reports to have been produced under the new UBRAF system.

KPI 7: Financial Accountability

Finding 7: Stakeholders surveyed considered that UNAIDS has generally adequate processes, systems and policies for financial accountability. The review of documents indicated that UNAIDS is adequate or above in all areas, with the exception of risk management.

Of the seven MIs assessed under KPI 7, five were assessed by document review and the survey, one was assessed by the survey only, and one was assessed by the document review alone. Overall, survey respondents considered UNAIDS’ policies and processes for financial accountability as adequate in all areas. The document review rated the organisation’s performance in this area as adequate or strong in all areas except risk management.

Figure 3.13 KPI 7: Financial Accountability, Ratings of Micro-Indicators

MI 7.1 – External financial audits performed across the organisation

MOPAN donors at headquarters were asked if the UNAIDS Secretariat's external financial audits are meeting the needs of donors. Almost three quarters (73 per cent) of respondents rated UNAIDS as adequate on this question.

The document review assessed whether financial audits that met recognised international standards were performed across the organisation and rated UNAIDS adequate on this MI.

At the global level, UNAIDS has clear rules and procedures for conducting audits. The External Auditor of WHO is responsible for external audit at UNAIDS. WHO provides administration in support of UNAIDS, as per ECOSOC resolution 1994/24, and Article XI of the Memorandum of Understanding among co-sponsors establishing UNAIDS. Independent of the UNAIDS Secretariat, WHO performs an external oversight function of the Joint Programme and certifies the annual financial report prepared by the Executive Director pursuant to Financial Regulation XIII (Accounts and Financial Statements) and IPSAS.

The financial reports reviewed were prepared in accordance with the United Nations System Accounting Standards (UNSAS) and WHO's Financial Regulations and Financial Rules. Further, the external financial report reviewed was accompanied by a letter from the external auditor (Controller and Auditor General of India) confirming that the audit was conducted in accordance with the International Standards on Auditing. The financial report for the biennium 2010-2011 specifies that UNAIDS along with WHO has been gradually adopting International Public Sector Accounting Standards (IPSAS) with full implementation of IPSAS expected in 2012.

MI 7.2 – External financial audits performed at the regional or country level

Survey respondents were asked whether UNAIDS Secretariat's regional or country-level operations are appropriately audited by an external body. While 36 per cent of respondents rated UNAIDS as adequate or above, the majority (56 per cent) answered ‘don’t know’.

The document review rated UNAIDS as adequate on this MI. However, it should be noted that funds are not reported on independently in the co-sponsors’ audited accounts. According to the “single audit principle”, the funds provided to co-sponsors by the UNAIDS Secretariat would be subject to internal and external audits by the respective co-sponsors. However, it is unclear whether this provides sufficient accountability to UNAIDS for funding disbursed to the co- sponsors.

MI 7.3 – Policy on anti-corruption

The document review rated UNAIDS as strong on this MI. UNAIDS’ policy on combating fraud and corruption was issued in 2005 and was reviewed in February 2012 to ensure alignment with both the Standards of Conduct for International Civil Servant and the Staff Rules and Regulations (Article I of the Staff Regulations and Section 1 of the Staff Rules) and UNAIDS’ Guide to Ethics. The safeguarding of UNAIDS’ resources is performed through an internal control system that combines fraud prevention, risk assessments, risk mitigation, segregation of duties, review and accountability mechanisms, investigation and follow-up measures, as required. A reporting mechanism is in place to ensure that any case of fraud is reported to the Programme Coordinating Board through disclosure in the annual financial report.

The UNAIDS Fraud Prevention Policy and Fraud Awareness Guidelines (issued in 2005 and reviewed in 2012) include preventive measures to counter corruption and procedures for measures to be taken against irregularities identified by an external financial audit. In addition, mitigation of inherent fraud risk by fraud prevention is also performed by external consultants as part of the accountability enhancement reviews of the country offices. UNAIDS has a Guide to Ethics and provides training to raise ethical awareness and promote a culture of integrity, responsibility and accountability.

MI 7.4 – Systems for immediate measures against irregularities

Survey respondents were asked whether the UNAIDS Secretariat follows up on financial irregularities, including fraud and corruption, in an appropriate manner. Almost half of the survey respondents (46 per cent) answered ‘don’t know’ on this MI, while 39 per cent rated it adequate or above, and 12 per cent as inadequate. Direct partners were more positive than MOPAN donors in-country and at headquarters. These differences are statistically significant.

UNAIDS was rated as adequate on this MI in the document review. The evidence reviewed included broad UNAIDS/WHO guidelines on measures to be taken against irregularities, such as the role of the World Health Organisation (WHO) Office of Internal Oversight Services (OIOS) in investigations and disciplinary actions. Within the current system, allegations of corrupt, fraudulent, collusive or coercive practices in relation to UNAIDS’ activities are promptly brought to the attention of UNAIDS internal auditor, the OIOS, and the Chief Financial Officer. Investigations are reported in the annual report of the internal auditor and any cases of fraud or corruption are disclosed in the annual financial report prepared for the UNAIDS Programme Coordinating Board.

Accountability enhancement reviews have been introduced recently to strengthen financial systems and management of country offices. However, it is not clear to what extent UNAIDS has taken appropriate action as a result of irregularities uncovered through the accountability enhancement reviews. In addition, no evidence could be found of a procedure for responding to irregularities identified during an external financial audit. Similarly, it is uncertain whether the single audit principle allows sufficient reporting on irregularities, in regards to the funds provided to co-sponsors through the Secretariat.

Despite these shortcomings, UNAIDS’ audited and non-audited financial reports usually include a list of recommendations to the PCB. The PCB generally accepts the financial reports and, at times, follows up on the recommendations.

MI 7.5 – Internal financial audit processes provide credible information

MOPAN donors at headquarters were asked if the UNAIDS Secretariat conducts internal financial audits to provide credible information to its governing bodies. The majority (70 per cent) rated UNAIDS as adequate or above on this MI and 30 per cent answered ‘don’t know’.

Although the OIOS activity relating to internal audits of UNAIDS is relatively recent and little, if any, reporting to the Board has been done on this, solid systems that meet MOPAN criteria appear to be in place.

As UNAIDS began as a WHO-sponsored programme, the internal audit services are provided by the WHO OIOS on a cost recovery basis in accordance with a Memorandum of Understanding (MoU) established on 9 January 2008. The OIOS provides UNAIDS with oversight services performed by an Audit Focal point housed in the WHO. The OIOS reports to the Director General of the WHO and internal audit reports are submitted jointly to the Director General of the WHO and the Executive Director of UNAIDS. An annual summary report of the OIOS’ activities is submitted to the Executive Director of UNAIDS and included in the Executive Director’s report to the PCB. The IOS follows up on all audit recommendations by keeping records of all open audits and by reporting on the status of recommendations to the UNAIDS Audit Focal Point. The responsibility for implementation of recommendations is assigned and agreed upon in the audit report. Moreover, accountability enhancement reviews also provide UNAIDS governing body with credible financial and management information from country offices.

MI 7.6 – Effective procurement and contract management processes

Survey respondents were asked whether the UNAIDS Secretariat's procurement and contract management processes for the provision of services or goods are effective. While 48 per cent rated UNAIDS as adequate or above, 47 per cent answered ‘don’t know’. Direct partners were more positive than donors at headquarters. The difference is statistically significant.

MI 7.7 – Strategies for risk management

MOPAN donors at headquarters were asked whether the UNAIDS Secretariat has appropriate strategies and plans for risk management. The majority (52 per cent) answered ‘don’t know’ and 36 per cent rated UNAIDS as adequate or above on this MI.

The document review rated UNAIDS as very weak on this MI given that UNAIDS does not currently have an organisation-wide policy, strategy, framework or guideline on risk management. However, UNAIDS is in the process of improving and formalising its enterprise risk management systems by developing a risk management strategy, which should be completed by the end of 2012.[24] In addition to accountability enhancement reviews, due diligence in maintaining compliance and safeguarding UNAIDS’ resources is also performed through an internal control system that combines fraud prevention, risk mitigation, detection, investigation and follow-up measures as required (see MI 7.3). Any case of fraud is reported to the Programme Coordinating Board of UNAIDS through annual financial reports.

KPI 8: Using Performance Information

Finding 8: The assessment found that UNAIDS makes good use of performance information to revise and adjust policies, plan new interventions, and manage poorly performing interventions. Follow-up on evaluation recommendations was noted as one of UNAIDS’ strengths.

Four MIs were assessed under this KPI by both the survey and the document review. Ratings provided by survey respondents were adequate and above. The document review ratings ranged from adequate to very strong.

UNAIDS provides performance information on the Joint Programme in periodic organisation- wide evaluations and performance reports.[25] Following the Second Independent Evaluation (2009-2010) and other operational reviews, UNAIDS has undergone substantive organisation- wide changes. The changes to its organisational structure, policies and interventions should enable a stronger focus on results through an improved and unified budget, results and accountability framework.

Figure 3.14 KPI 8: Using Performance Information, Ratings of Micro-Indicators

MI 8.1 – Using information for revising and adjusting policies

MOPAN donors at headquarters were asked whether UNAIDS uses project/programme, sector and country information on performance to revise its corporate policies. The majority (70 per cent) rated UNAIDS as adequate and 30 per cent answered ‘don’t know’ on this MI.

The document review rated UNAIDS very strong on its capacity to revise and adjust policies on the basis of performance information. Information on organisation-wide performance was found in a number of internal and external sources, such as independent evaluation reports, operational reviews, performance monitoring reports, country mid-term reviews, and accountability enhancement reviews. A good example to illustrate UNAIDS’ capacity in this regard would be its response to the Second Independent Evaluation, where UNAIDS established an Implementation Plan, an Implementation Support Team to address 21 of the 24 recommendations of the evaluation, and a special task to address three recommendations related to governance. As a result, changes were made in various organisational areas such as the organisation’s: mission statement and strategy development, partnerships, global programmatic mechanisms, delivery at country level, financial architecture, knowledge management, administrative issues and governance.[26] The inclusion of gender and human- rights as a strategic direction within the 2012-2015 Strategy – ‘Getting to Zero’ is also a direct result of the Second Independent Evaluation, among others.

MI 8.2 – Using information for planning new interventions

Survey respondents at the country level were asked whether UNAIDS uses evidence from projects, programmes and/or initiatives to plan new areas of cooperation at country level. The majority of survey respondents (82 per cent) rated UNAIDS as adequate or above, and 70 per cent of direct partners and 59 per cent of co-sponsors rated UNAIDS as strong and very strong in this area. MOPAN donors in-country were less positive than direct partners. These differences are statistically significant.

The document review rated UNAIDS as strong on this MI. The work planning documents sampled, such as country strategies, include sufficient information on achievements and lessons learned from previous cycles and are explicit in linking programme interventions to the results achieved, thus making it possible to deduce the rationale behind specific initiatives in most cases. In addition, mid-term reviews provide some information on programme performance at the country level, as they broadly report on outputs and achievement of results, issues faced, and key future steps. The new UNAIDS country programme assessment mechanism, which is still being tested, calls for a more explicit description of what has worked well and what could be improved in country level programming. The intention is for this tool to become an integral part of the Secretariat's results-based management approach – allowing lessons learned to be incorporated into day-to-day management decisions and programmatic choices, and promoting long-term knowledge-building and institutional learning.

MI 8.3 – Proactive management of poorly performing initiatives

Country-based survey respondents (i.e., MOPAN donors in-country and direct partners) were asked whether poorly performing projects or initiatives of the Joint Programme are managed proactively. While 51 per cent of survey respondents rated UNAIDS as adequate or above, 20 per cent gave ratings of inadequate or below, and 28 per cent answered ‘don’t know’. The difference between MOPAN donors in-country and direct partners, which rated UNAIDS as adequate and strong respectively, is statistically significant.

UNAIDS received a rating of adequate on this MI from the document review. While co-sponsors are responsible for managing the activities funded by the UBRAF, the UNAIDS Secretariat has the responsibility to monitor the use of funds that flow to co-sponsors through the Joint Programme. Given the division of labour, there are certain structural limitations to the Secretariat’s oversight and management of initiatives implemented through the Joint Programme. The Secretariat’s ability to proactively manage poorly performing programmes is much higher within their areas of responsibility than for activities in the Joint Programme.

The UBRAF is an example of how UNAIDS has improved its results framework and is working to develop more sophisticated performance monitoring mechanisms to accompany its implementation. An annual performance review process associated with the UBRAF is being developed and will identify goals where progress is not being achieved as expected. If necessary, the Committee of Co-sponsors Organisations (CCO) and the PCB will be presented with an analysis of progress on lower-level indicators, deliverables and reasons for delays and proposed mitigation plans will be developed to address poorly performing initiatives. Further, UNAIDS is in the process of implementing and testing a methodology for assessing the relevance, effectiveness and efficiency of country programmes in which poorly performing initiatives will be addressed and additional support provided through the Action Plan of the Regional Support Team.

MI 8.4 – Evaluation recommendations are acted upon

MOPAN donors at headquarters were asked whether UNAIDS appropriately tracks the implementation of evaluation recommendations reported to its governing bodies. The majority (70 per cent) rated UNAIDS as adequate or above, and 27 per cent answered ‘don’t know’ on this MI.

UNAIDS was also rated adequate on this MI in the document review. Although UNAIDS does not have an evaluation policy or guidelines for tracking the implementation of evaluation recommendations to the Board, there is evidence that the UNAIDS governing body is taking action to ensure that changes in response to evaluation recommendations are implemented. For instance, at its 25th meeting the Programme Coordinating Board (PCB) endorsed a plan to implement the recommendations of the Second Independent Evaluation (SIE) and set up a task force and an Implementation Support Team to report to the 26th meeting of the PCB on follow-up of the SIE.

KPI 9: Managing Human Resources

Finding 9: While survey responses suggested that human resource management practices are not widely known by stakeholders, UNAIDS documentation indicates that appropriate human resources management systems exist. The organisation received ratings of adequate on its system to manage staff performance and strong on its performance assessment system for senior staff.

Overall, survey respondents rated UNAIDS adequate on this KPI although there was a high percentage of ‘don’t know’ responses (43 per cent). The document review found UNAIDS strong on one of the two MIs assessed.

Following the results of the Second Independent Evaluation and internal reviews, UNAIDS developed a Strategy on Human Resources 2011-2015.[27] The strategy regularises the contracts of all UNAIDS staff and ensures that they operate under one set of Staff Regulations and Rules. It addresses key organisational elements, such as UNAIDS competency framework; workforce planning; recruitment and staffing; staff administration; staff development, career growth and mobility; performance management; staff well-being; and human resources information.

Figure 3.15 KPI 9: Managing Human Resources, Ratings of Micro-Indicators

MI 9.1 – Results-focused performance assessment systems for senior staff

MOPAN donors at headquarters were asked whether the UNAIDS Secretariat uses results- focused performance assessments for senior staff. The majority (58 per cent) answered ‘don’t know’ and 36 per cent rated UNAIDS as adequate on this MI.

UNAIDS was rated as strong on this MI in the document review. In addition to the UN Secretariat’s administrative instruction regarding the performance management and development system (2010), UNAIDS has a Performance Appraisal System (PAS) that was established in 1996 to improve and maximise the performance of individual staff members and ensure that staff performance meets expected standards. It is the cornerstone for management and personnel decisions (such as contract extensions, salary increments, standards of conduct, sanctions and separations). The PAS is required for all staff members holding appointments of at least one year, up to and including Directors of Departments. The performance year is aligned with the annual cycle. A mid-year review is held after six (6) months and appraisals are usually completed by the year-end or 12 months from the entry on duty date. Despite the procedural clarity, UNAIDS does not seem to report on the completeness and timeliness of its application of the assessment process each year.

MI 9.2 – Transparent system to manage staff performance

MOPAN donors at headquarters were asked whether the UNAIDS Secretariat uses a transparent system to manage staff performance. The majority (55 per cent) answered ‘don’t know’ and 39 per cent rated UNAIDS as adequate on this MI.

The document review rated UNAIDS as adequate on this MI. UNAIDS Performance and Appraisal System Guidelines and Reference Material broadly link the assessment of staff performance to administrative actions that may be applied, depending on particular performance-related considerations. However, little evidence was found to explain how staff performance is linked to promotions.

As a result of the recommendations from the SIE, UNAIDS is making efforts to review its policy to better address staff performance management. It has also taken other steps to improve the existing system in accordance with the competency framework and with the aim of monitoring compliance with staff performance requirements and procedures and policies outlined in the People Development and Performance Policy and Guidelines. It has provided managers with preventive tools to ensure that staff meet performance requirements, and guidance to exercise sound judgment and follow UNAIDS Performance and Appraisal System Guidelines and Reference Material, all with the purpose of rewarding strong performance and effectively managing poor performance or under-performance.

MI 9.3 – Staff rotation adequate for the development of effective partnerships

MOPAN donors in-country and direct partners were asked whether the UNAIDS Secretariat keeps deployed international staff in country for a sufficient time to maintain effective partnerships. The majority of survey respondents (68 per cent) rated UNAIDS as adequate or above.

KPI 10: Performance-oriented Programming

Finding 10: UNAIDS was rated adequate or above by survey respondents on the MIs related to performance-oriented programming processes. The document review gave UNAIDS mixed ratings in this area.

While a majority of survey respondents shared the perception that UNAIDS country/regional programming processes are performance-oriented, the document review rated UNAIDS as adequate on this KPI.

In recent years, the Secretariat has developed Workplanning, Monitoring and Reporting Guidelines to improve its delivery of results and reinforce internal accountability as outlined in the UBRAF. While the document review rated UNAIDS strong on MI 10.1, the organisation was rated inadequate on MI 10.2 as inconsistencies were found in the documents sampled regarding the inclusion of milestones, targets and baselines.

Figure 3.16 KPI 10: Performance-oriented Programming, Ratings of Micro-Indicators

MI 10.1 – New initiatives subject to evidence-based analysis

MOPAN donors at headquarters and co-sponsors were asked whether UNAIDS subjects new programming initiatives to evidence-based analyses. The majority of survey respondents (88 per cent) rated UNAIDS as adequate or above. This MI received the second strongest average ratings in the survey.

The document review rated UNAIDS’ approach to planning and quality assurance as strong. There is evidence of the implementation of the Secretariat’s work planning guidelines, which include three phases (drafting, quality assurance, and finalisation). UNAIDS has shown a commitment to improvements in this area by conducting an internal review of the quality assurance process in 2012. The review highlighted the need to: i) revise and strengthen the Guidelines and associated tools; and, ii) ensure that the process is adequately understood and conducted by peers, as well as adequately taken into account by their supervisors.

UNAIDS has also adopted a peer review as part of its quality assurance process related to Joint Programme planning. The review of Joint UN Team workplans is conducted by peers with similar functions in similar settings (e.g., UNAIDS Country Coordinators, Regional Support Teams), staff responsible for technical areas (e.g., human rights, prevention, etc.), and staff responsible for areas such as RBM, finance and budget, and information technology.[28]

In addition, a quality assurance process, funded through the Secretariat under the Programme Acceleration Funds (PAF),[29] ensures that proposals comply with the standards of quality of the Joint UN Team, the UN executing agency or implementing partner, and the UN Resident Coordinator.

MI 10.2 – Milestones/targets set to monitor implementation

Country-based survey respondents (i.e., MOPAN donors in-country, direct partners, and co- sponsors) were asked whether targets are set to enable monitoring of progress in Joint Programme implementation at the country level. The majority of survey respondents (84 per cent) rated UNAIDS as adequate or above, and 59 per cent of direct partners and 71 per cent of co-sponsors gave ratings of strong and very strong. However, 20 per cent of MOPAN donors in-country responded ‘don’t know’. MOPAN donors in-country were less positive than co- sponsors and the difference is statistically significant.

The document review rated UNAIDS inadequate in setting milestones/targets to rate the progress of implementation. Most HIV/AIDS Joint Support Programme and Operational Plans and Budgets sampled contained a description of proposed activities and associated outputs and indicators. The country work plans sampled included milestones, but did not include targets or baselines for output indicators. Other than describing in general terms what had been achieved, mid-term reviews were inconsistent in reporting on specific indicators or milestones. However, the new monitoring tool being developed for the UBRAF will include milestones and targets and is due to be ready by October 2012.

KPI 11: Delegating Authority

Finding 11: Survey respondents and the document review concurred that UNAIDS’ country offices have the authority to manage activities.

Two MIs were assessed under this KPI. Overall, country-based survey respondents rated UNAIDS as strong in delegating decision-making authority and in defining roles and responsibilities. The document review, which only assessed the extent to which operational decisions can be made locally, rated UNAIDS as strong.

Figure 3.17 KPI 11: Delegating Authority, Ratings of Micro-Indicators

MI 11.1 – Roles and responsibilities clearly defined

Direct partners and co-sponsors were asked whether the Division of Labour is respected at the country level. The majority of respondents (83 per cent) rated UNAIDS as adequate or above and 18 per cent of direct partners and 35 per cent of co-sponsors rated UNAIDS as very strong on this MI.

This MI was not assessed in the document review.

MI 11.2 – Operational decisions can be made locally

MOPAN donors in-country, direct partners, and co-sponsors were asked whether the Joint Programme is managed at the country level (i.e., whether decisions can be made locally). The majority of survey respondents (83 per cent) rated UNAIDS as adequate or above; 58 per cent of direct partners, 72 per cent of co-sponsors, and 32 per cent of MOPAN donors in-country rated UNAIDS strong and very strong on this MI. MOPAN donors in-country were less positive than co-sponsors and the difference is statistically significant.

The document review rated UNAIDS strong on this MI. There are two types of decisions made at the country level – those made by UNAIDS Country Coordinators and those made by the Joint UN Team under the authority of the Resident Coordinator.

In the first case, UNAIDS Country Coordinators are responsible for developing and implementing the work of the Secretariat and coordinating that of the Joint Programme. UNAIDS has a policy note that describes the application of UNAIDS’ existing delegation of authority, which was updated in 2008 with the introduction of the new Enterprise Resource Planning system. This policy includes the specific parameters related to the delegated authority (e.g., budget ceilings or allocations) at each level - headquarters, regional and country. The UNAIDS Secretariat’s Workplanning, Monitoring and Reporting Guidelines outline the principles that govern Secretariat-wide programmatic functions and processes, over which the UNAIDS Country Coordinator has the authority to decide on behalf the Executive Director in accordance with UNAIDS mandate and in line with national priorities.

In the second case, UN Joint Programmes of Support on AIDS are governed at the country level by guidance from the UN Development Assistance Framework (UNDAF) and joint planning, monitoring and reporting, developed by the UN Development Group. The UN Country Team is therefore responsible for making decisions related to UN Joint Programmes of Support on AIDS at the country level.

3.3.4 Relationship Management

In both the survey and document review, UNAIDS was seen to perform adequately or better across all of the KPIs in relationship management.

Figure 3.18 below shows the overall ratings for the five KPIs in the relationship management quadrant.

Survey respondents rated UNAIDS as strong on four MIs and adequate on one. UNAIDS contribution to policy dialogue received the highest total score of the survey.

The document review rated only one MI in this area and rated UNAIDS as strong on harmonising procedures with other programming partners.

Figure 3.18 Quadrant III: Relationship Management, Survey and Document Review Ratings

Figure 3.19 shows the mean scores for the five KPIs for all survey respondents, and by respondent groups.

Figure 3.19 Quadrant III: Relationship Management, Mean Scores by Respondent Group

KPI 12: Supporting National Plans

Finding 12: Survey respondents considered UNAIDS’ performance in supporting national priorities as strong.

Figure 3.20 KPI 12: Supporting National Plans, Ratings of Micro-Indicators

MI 12.1 – Funding proposals developed with national governments/partners

Donors in-country, direct partners and co-sponsors were asked whether the UNAIDS Secretariat supports funding proposals designed and developed by national governments, co- sponsors or other partners. The majority of survey respondents (81 per cent) rated UNAIDS as adequate or above. This MI was not assessed as part of the document review.

KPI 13: Adjusting Procedures

Finding 13: UNAIDS was perceived by all respondent groups as adequate in adjusting its procedures to local conditions and capacities.

This KPI was assessed by the survey only, which was administered to MOPAN donors in- country, direct partners, and co-sponsors. Overall, all respondent groups rated UNAIDS as adequate or above on this key performance area.

Figure 3.21 KPI 13: Adjusting Procedures, Ratings of Micro-Indicators

MI 13.1 – Procedures easily understood and completed by partners

Survey respondents were asked whether UNAIDS uses procedures that can be easily understood and followed by partners. Overall, the majority of survey respondents (71 per cent) rated UNAIDS as adequate or above. However, despite the mean score of adequate for MOPAN donors in- country, 40 per cent of these respondents gave a rating of inadequate or weak, making the average score for this respondent group lower than the average scores for other respondent groups. The differences are statistically significant.

MI 13.2 – Length of time for procedures does not affect implementation

Survey respondents were asked whether the length of time it takes to complete UNAIDS' procedures affects implementation or not. Overall, the majority of survey respondents (61 per cent) rated UNAIDS as adequate or above while 20 per cent gave ratings of inadequate or below on this MI.It should be noted that despite the mean score of adequate for MOPAN donors in-country, 24 per cent of these respondents gave a rating of inadequate, making the average score for this respondent group lower than the average scores for other respondent groups. The differences are statistically significant.

MI 13.3 – Ability to respond quickly to changing circumstances

Survey respondents were asked whether UNAIDS adjusts its overall portfolio in country quickly to respond to changing circumstances. Overall, the majority of survey respondents (63 per cent) rated UNAIDS as adequate or above, while 20 per cent gave ratings of inadequate or below. Co-sponsors were more positive than donors in-country and direct partners. The differences are statistically significant.

MI 13.4 – Flexibility in implementation of projects/programmes

Survey respondents were asked whether UNAIDS is flexible in adjusting its implementation of individual projects/programmes as learning occurs. The majority of survey respondents (72 per cent) rated UNAIDS as adequate or above.

KPI 14: Using Country Systems

Finding 14: Survey respondents in all groups rated UNAIDS as adequate in promoting mutual accountability in its partnerships and as strong in avoiding parallel implementation structures.

The co-sponsoring agencies rated UNAIDS as strong on avoiding parallel implementation structures and the majority of survey respondents rated UNAIDS’ promotion of mutual accountability in its partnerships as adequate or above.

The survey results on parallel implementation structures is not surprising given that UNAIDS Secretariat does not have an implementation mandate and the co-sponsors use a wide range of different administrative systems for their disbursements and operations.

Figure 3.20 KPI 14: Using Country Systems, Ratings of Micro-Indicators

MI 14.1 – Parallel implementation structures are avoided

Co-sponsors were asked whether UNAIDS relies on existing systems and partners (i.e., avoids establishing parallel implementation structures). The majority of survey respondents (90 per cent) rated UNAIDS as adequate or above, and 73 per cent as strong or very strong on this MI.

Although the document review did not assess this MI, it bears noting that UNAIDS works to strengthen national capacities in key areas of the HIV response in accordance with the Three Ones principles – one agreed HIV/AIDS Action Framework, one national AIDS coordinating authority, and one agreed country-level monitoring and evaluation system. UNAIDS has also provided leadership and coordination within the UN family according to these principles. These alignment efforts are in keeping with the reform of the United Nations and international efforts to improve aid effectiveness, as well as with the spirit of the Paris Declaration on Aid Effectiveness (2003) and the Rome Declaration on Harmonisation (2005).

MI 14.2 – Promotion of mutual accountability in its partnerships

Donors in-country, direct partners, and co-sponsors were asked whether UNAIDS encourages mutual accountability assessments of commitments to the Paris Declaration and Accra Agenda for Action. The majority of survey respondents (62 per cent) rated UNAIDS as adequate or above, 14 per cent as inadequate or below, and 23 per cent answered ‘don’t know’. It bears noting that 29 per cent of MOPAN donors in-country rated the organisation as inadequate or below. They were less positive than co-sponsors and the difference is statistically significant.

The document review did not assess this MI.

KPI 15: Contributing to Policy Dialogue

Finding 15: All respondents provided consistently strong ratings on UNAIDS’ contribution to policy dialogue.

KPI 15 was not assessed by the document review, but was addressed by two questions in the survey. The first question assessed UNAIDS’ performance in terms of providing inputs into policy dialogue and the second, whether policy dialogue conducted by UNAIDS respected partner views and perspectives. UNAIDS’ performance on both questions was rated as strong by all respondent groups.

Figure 3.21 KPI 15: Contributing to Policy Dialogue, Ratings of Micro-Indicators

MI 15.1 – Reputation for valuable input to policy dialogue

Survey respondents were asked whether UNAIDS provides valuable inputs to policy dialogue. The majority of survey respondents (87 per cent) rated UNAIDS as adequate or above.

MI 15.2 – Policy dialogue respects partner views

Survey respondents were asked whether UNAIDS respects the views of partners when it undertakes policy dialogue. The majority of survey respondents (86 per cent) rated UNAIDS as adequate or above. MOPAN donors in-country were less positive than donors at headquarters, direct partners, and co-sponsors. The differences are statistically significant.

KPI 16: Harmonising Procedures

Finding 16: Survey respondents suggested that UNAIDS is strong in the harmonisation of its procedures and the document review gave UNAIDS strong ratings in this area. The extent to which the Joint Programme builds on and reinforces synergies with and among co-sponsors was seen as an organisational strength.

The survey assessed four MIs in this KPI. Overall, survey respondents (MOPAN donors in- country, direct partners and co-sponsors) rated UNAIDS’ performance as adequate and above on all four MIs. The document review gave UNAIDS strong ratings on the three MIs assessed.

The Division of Labour, created in 2006 and revised in 2010, is clear on the roles and responsibilities of conveners, partners, and the Secretariat in the HIV response. UNAIDS has provided operational guidance on processes, such as internal communication and coordination. The Division of Labour builds on the capacities and comparative advantages of the co-sponsors and promotes integration and synergy among the different partner agencies to strengthen inter- agency work, take forward the agenda set out in the UNAIDS Strategy 2011-2015, and collectively deliver on results.

Figure 3.22 KPI 16: Harmonising Procedures, Ratings of Micro-Indicators

MI 16.1 – Participation in joint missions

Survey respondents were asked whether UNAIDS coordinates and participates in joint missions, as appropriate. The majority of survey respondents (70 per cent) rated UNAIDS as adequate or above, 16 per cent as inadequate or below, and 13 per cent answered don’t know. MOPAN donors in-country were less positive than co-sponsors and the difference is statistically significant.

This MI is based on indicator 10a of the Paris Declaration, which measures the extent to which an organisation participates in joint missions.[30] While a strict interpretation of this indicator (participation in joint missions) is difficult to measure for UNAIDS, the document review rated UNAIDS strong on this MI because joint implementation is part of the organisation’s mandate and there is clear evidence that UNAIDS harmonises its procedures with other partners. Policy development and coordination are aspects of the UNAIDS Secretariat’s core mandate. Strategic guidance to the co-sponsors in the form of clear strategies and approaches has been complemented by operational guidance. Evidence of the latter includes the creation of a clear Division of Labour among the co-sponsors, which has served to harmonise approaches as the lead/co-lead UN agencies have been clearly defined for each of the aspects of the response to the HIV epidemic. Further, the Joint UN Team on AIDS are mandated to work jointly, maximise synergies and partnerships, and improve overall coordination at the country-level. In sum, UNAIDS’ performance in terms of harmonisation and coordination is strong.

MI 16.2 – Technical cooperation disbursed through coordinated programmes

Survey respondents were asked whether UNAIDS' technical assistance is provided through coordinated programmes in support of capacity development. The majority of survey respondents overall (83 per cent) rated UNAIDS as adequate or above on this MI.

Although Indicator 4 of the Paris Declaration measures the degree of alignment of donor technical cooperation with the partner country’s development objectives and strategies,[31] the document review interpreted this indicator more broadly to include disbursements made through the Joint Programme and rated UNAIDS as strong on this MI. UNAIDS’ core resources are disbursed through a two-year, coordinated joint programme that is prepared in consultation with all co-sponsors and approved by the Programme Coordinating Board. Also, documentary evidence suggests that UNAIDS encourages national capacity building through the provision of clear guidance on the epidemic and how to maximise the effectiveness of national interventions and those of the co-sponsors. UNAIDS is strong in the provision of coordinated capacity building to national partners – particularly through support for the preparation and implementation of Global Fund proposals.

MI 16.3 – ODA disbursements/support for government-led PBAs

Survey respondents were asked whether UNAIDS participates in programme-based approaches (other than through budget support). The majority of survey respondents (77 per cent) rated UNAIDS as adequate or above. MOPAN donors in-country were less positive than direct partners and the difference is statistically significant.

There was insufficient data to analyse this MI in the document review.

MI 16.4 – Synergy with and among co-sponsors

Survey respondents were asked whether UNAIDS builds on or reinforces synergies and strengths of the co-sponsors. The majority of survey respondents (72 per cent) rated UNAIDS as adequate or above on this MI. MOPAN donors in-country were less positive than co- sponsors and the difference is statistically significant.

The document review rated UNAIDS as strong on this MI. UNAIDS has moved towards improving coordination of technical assistance by UN agencies through both its Division of Labour and the development of joint programmes of support that are aligned with national HIV/AIDS strategies. In addition, co-sponsors are now required to regularly report on achieved results against the UBRAF, including those results that can be attributed to partnerships and complementarities among the co-sponsors, thereby making the monitoring of programme performance and partnerships more transparent.

3.3.5 Knowledge Management

Survey respondents assessed UNAIDS’ overall performance in knowledge management to be adequate or better. However, the review of documents highlighted areas for improvement.

Figure 3.23 below shows the overall survey and document review ratings for the three KPIs in the knowledge management quadrant.

Being a knowledge-based organisation, UNAIDS performs acceptably in most areas with regard to performance monitoring and evaluation, performance reporting, and dissemination of lessons learned. However, there is room for improvement in some areas, including evaluation quality and evaluation coverage of programming activities, reporting on Paris Declaration commitments, and making programming adjustments based on performance information.

Figure 3.23 Quadrant IV: Knowledge Management, Survey and Document Review Ratings

Figure 3.26 shows the mean scores for the three KPIs for all survey respondents, and by respondent groups.

Figure 3.24 Quadrant IV: Knowledge Management, Mean Scores by Respondent Group

KPI 17: Evaluating External Results

Finding 17: UNAIDS has improved the independence of its evaluation structure and function, but still faces operational challenges, particularly in ensuring evaluation coverage and quality.

The survey asked respondents about two of the five MIs in evaluating results. Overall, survey respondents rated UNAIDS as strong on one and adequate on the other.

The document review, which assessed three other MIs, indicated that UNAIDS’ performance ranged from very weak to strong. UNAIDS’ new organisational structure includes the Economics, Evaluation and Programme Effectiveness Division, which will replace the Evaluation Department.

Figure 3.25 KPI 17: Evaluating External Results, Ratings of Micro-Indicators

MI 17.1 – Independent evaluation unit

The document review rated UNAIDS as adequate on this MI. UNAIDS’ new organisational structure (established in February 2012) includes an Economics, Evaluation and Programme Effectiveness Division, which will report to the Programme Branch and integrate other organisational functions. At the global level, organisation-wide evaluations have been commissioned by the Executive Director, endorsed by the Programme Coordinating Board (PCB), and managed by an independent group, committee or task force.[32] Ad hoc, independent programmatic, thematic, regional and country evaluations are also undertaken by the Cosponsors and the Secretariat in coordination by the Cosponsor Evaluation Working Group. As such, its evaluation function is considered to be independent. Further, the Monitoring and Evaluation Reference Group (MERG) provides advice on monitoring and evaluation-related issues at all levels of the programme. A recent independent assessment of the MERG recommended that it focus more strategically on a cross-agency evaluation agenda.[33]

MI 17.2 – Sufficient evaluation coverage of programming activities

Strictly adhering to MOPAN criteria, UNAIDS’ evaluation coverage was rated as very weak by the document review. There is no organisation-wide policy on evaluation. Therefore, there is no policy or framework that identifies the need for independent evaluations of projects and programmes or that defines the expected evaluation coverage for UNAIDS and the Joint Programme. The Second Independent Evaluation 2009 [34] suggested that UNAIDS promotes evaluation at the country level, but the document review found no evidence of such country- level evaluations.

UNAIDS country programme assessment approach will, however, strengthen evaluation at global and country levels. This is being done by convening a working group to develop a joint global evaluation plan structured around the priority areas of the epidemic (among other specific recommendations). It is intended to support periodic performance assessment within the context of a results-based management framework.

To complement the evaluation function, co-sponsors also provide the PCB with Broad Activity Achievement Reports for each biennium. Although these reports list key evaluations carried out by the co-sponsors for each broad activity, there is no evidence that these specific evaluation reports are shared with the PCB.

MI 17.3 – Quality of evaluations

UNAIDS was rated as inadequate in ensuring the quality of its evaluations. According to the document review criteria, the organisation is expected to have a system that ensures that evaluations and country programme assessments comply with the norms and standards of evaluation within the UN system, as well as international best practice. While major organisation-wide evaluations are thoroughly reviewed, ensuring that the methodology is robust and the majority of findings and recommendations are evidence-based and credible, little evidence was found that mechanisms are in place to ensure quality of other evaluations, such as those taking place at the country level.

MI 17.4 – Use of evaluation findings to inform decisions

MOPAN donors at headquarters were asked whether UNAIDS uses evaluation findings in its decisions on programming, policy and strategy. The majority of survey respondents (79 per cent) rated UNAIDS as adequate or above.

MI 17.5 – Beneficiaries and partners involved in evaluations

MOPAN donors in-country, direct partners, and co-sponsors were asked whether UNAIDS involves partners and beneficiaries in evaluations of its projects or programmes. The majority of survey respondents (66 per cent) rated UNAIDS as adequate or above, 19 per cent as inadequate or below, and 14 per cent answered ‘don’t know’ on this MI.

MOPAN donors in-country provided a wide range of ratings on this MI (almost half rated it adequate and above, one-third as inadequate or below, and the rest indicated ‘don’t know’). They were generally less positive than direct partners and co-sponsors and the differences are statistically significant.

KPI 18: Presenting Performance Information

Finding 18: UNAIDS was considered adequate or better in various aspects of presenting performance information on its effectiveness. One area for improvement is its reporting on Paris Declaration commitments using indicators and targets.

This KPI involved an assessment of six MIs, all of which were assessed by document review and three by survey respondents. Donors at headquarters rated UNAIDS as adequate in reporting performance information on outcomes, in reporting against its corporate strategy, and in reporting on Paris Declaration commitments. The document review rated UNAIDS as adequate or better on several aspects of reporting but as very weak in reporting on Paris Declaration commitments.

Figure 3.26 KPI 18: Presenting Performance Information, Ratings of Micro-Indicators

MI 18.1 – Reports on achievement of outcomes

MOPAN donors at headquarters were asked whether UNAIDS’ reports to governing bodies provide clear measures of achievement of outcomes. The majority of survey respondents (82 per cent) rated UNAIDS as adequate or above.

UNAIDS was rated strong on this MI by the document review which found that UNAIDS reports on the achievement of corporate outcomes and outputs. The last full performance monitoring report (2010-2011) tracked the organisation’s progress with regard to its results framework for the UBW 2010-2011, providing data on UNAIDS’ contribution to outputs and outcomes. Technical Supplement reports complement the performance monitoring report by presenting results achieved against indicators. In addition, Broad Activity Achievement Reports are provided to the PCB (one for each co-sponsor, one for the Secretariat, and one for inter-agency activities). Finally, the Report of the Committee of Co-sponsors Organisations includes co- sponsors’ progress on inter-agency work and country-level results. It also shows how they align their work with the strategic directions and goals of the UNAIDS strategy. Although no reporting has been completed under the newly implemented UBRAF, technical guidance has been finalised and an update on indicators, monitoring and evaluation of the 2012-2015 UBRAF was prepared for the June 2012 PCB.

MI 18.2 – Reports on performance using data obtained from measuring indicators

The document review rated UNAIDS as adequate on this MI. UNAIDS reports on developments in the overall AIDS response and achievements under each priority area and cross-cutting strategy of the Unified Budget and Work plan sampled. Associated performance monitoring reports such as the Unified Budget and Work plan Report Technical Supplement (2010-2011) specifically compare indicator measurements to baseline and estimated targets (including financial information) identified in related performance monitoring frameworks. However, most output and outcome indicators found in the performance monitoring frameworks sampled do not respect SMART or CREAM criteria.[35] Similarly, sampled mid-term reviews were inconsistent in reporting against selected indicators and targets.

In the new UBRAF system, an executive dashboard linked to the UBRAF will serve as a basis for reporting and data will come from AIDS Progress Reports, the co-sponsors, UN Joint Teams, and the Secretariat. Performance will be measured against UBRAF indicators, baseline, and targets at three levels (indicators for each strategic goal and outcome, and indicators linked to co-sponsor corporate results frameworks and effectiveness of the Secretariat) to measure progress on an annual or biannual basis and track progress over time.[36]

MI 18.3 – Reports against corporate strategy, including expected results

MOPAN donors at headquarters were asked whether UNAIDS reports adequately against its corporate/organisation-wide strategy). The majority of survey respondents (85 per cent) rated UNAIDS as adequate or above.

UNAIDS was rated adequate on this MI in the document review. Under the UBW, UNAIDS reported on expected results as identified in the development results framework. However, there was no management results framework or equivalent. Although the 2010-2011 report clearly aligned reporting with the 2007-2011 Strategic Framework and 2010-2011 UBW, the range of reports consulted provided insufficient explanation of the variances between achieved results and targets. Improvements are expected in this MI as the procedures established to monitor results achieved under the UBRAF are implemented.

MI 18.4 – Reports on Paris Declaration commitments using indicators and country targets

MOPAN donors at headquarters were asked whether UNAIDS reports to its governing body on the mutual accountability of the Joint Programme. The majority of survey respondents (67 per cent) rated UNAIDS as adequate or above, 12 per cent inadequate or below, and 18 per cent answered ‘don’t know’.

Strictly adhering to MOPAN criteria, the document review rated UNAIDS very weak on this MI as UNAIDS does not independently report on Paris Declaration commitments. UNAIDS has prepared one progress update and lessons learned report (2008), which outlines progress on implementing the “Three Ones” and Global Task Team recommendations within the context of international efforts towards aid effectiveness and UN Reform. In 2011, a United Nations Development Group (UNDG) report on implementation of the Paris Declaration consolidated the 2010 results reported by UN agencies, including UNAIDS, from the country level, but findings were broad and not quantitative. As mentioned in the assessment of other MIs, there is considerable evidence that UNAIDS is respecting the Paris Declaration principles and commitments in its operations.

MI 18.5 – Reports on adjustments to policies/strategies based on performance information

The document review rated UNAIDS as strong on this MI. Performance monitoring reports are produced annually and presented to the Programme Coordinating Board as a standing agenda item, as is the annual report from the UNAIDS Executive Director to the Board. There is considerable evidence of adjustments made to organisation-wide policies and strategies based on performance information – most recently as a result of the recommendations of the Second Independent Evaluation (2009). See discussion under MI 8.1 above.

MI 18.6 – Reports on programming adjustments based on performance information

The document review rated UNAIDS as inadequate on this MI. The Performance Monitoring Frameworks provide broad guidance on how performance reporting should be carried-out through mid-term reviews at country level. Sampled mid-term reviews focused on the progress achieved and proposed adjustments and follow-up actions. However, little evidence was found on how the information captured is used to adjust programming and budgets. Similarly, it is unclear how the performance information from co-sponsors’ Broad Activity Achievement Reports and the CCO Report to the PCB are used to adjust programming. There is evidence that the creation of new systems, such as the Secretariat Work planning, Monitoring and Reporting Guidelines (UBRAF), is intended to strengthen this area. As such, it is expected that the use of performance information will be improved as part of UBRAF reporting.

KPI 19: Disseminating Lessons Learned

Finding 19: Donors at headquarters and co-sponsors rated UNAIDS as strong in disseminating lessons learned and the document review rated it strong for reporting on lessons learned.

Figure 3.27 KPI 19: Disseminating Lessons Learned, Ratings of Micro-Indicators

MI 19.1 – Reports on lessons learned based on performance information

Survey respondents were asked whether UNAIDS identifies and disseminates lessons learned from performance information. The majority of survey respondents (83 per cent) rated UNAIDS as adequate or above.

The document review rated UNAIDS as strong on this MI. UNAIDS consistently builds on the experience and lessons learned from surveillance data and performance information. The previous Evaluation Unit and the new Economics, Evaluation and Programme Effectiveness unit is responsible for documenting and disseminating lessons learned and/or best practices. Lessons learned are reflected in performance monitoring reports, which usually include key lessons learned and opportunities for the Joint Programme. The co-sponsors’ Broad Activity Achievement Reports also include lessons learned for completed initiatives.

MI 19.2 – Lessons shared at all levels of the organisation

Survey respondents were asked whether UNAIDS provides opportunities throughout the organisation to share lessons from practical experience. The majority of survey respondents (73 per cent) rated UNAIDS as adequate or above, and 19 per cent answered ‘don’t know’ on this MI.

4. Conclusion

The MOPAN 2012 assessment of UNAIDS was carried out at an important stage in the organisation’s evolution. UNAIDS committed to a broad organisational development process following its Second Independent Evaluation in 2009. It has shown its commitment to addressing all of the recommendations of this evaluation and has been developing systems to improve its ability to define, monitor, achieve and report on results. Many new approaches are still being tested and, as such, have yet to yield tangible effects.

The following conclusions are meant to encourage dialogue between MOPAN, UNAIDS, its direct partners and co-sponsors.

UNAIDS is highly valued by its direct partners and the co-sponsors.

In the four key performance areas examined in the MOPAN 2012 assessment, UNAIDS’s direct partners and co-sponsors were consistent in providing ratings of adequate or above, and often rated UNAIDS as strong or very strong.

UNAIDS’ commitment to organisational development has brought positive changes, although it is too early to assess the full effects of the process.

In recent years, UNAIDS has demonstrated strategic leadership and a commitment to organisational renewal while also continuing to track the epidemic and provide critical evidence- based guidance. Taking the recommendations from the Second Independent Evaluation and other operational reviews as the basis for this organisational development process, UNAIDS has reviewed its governance structure, strategies, as well as its systems and processes, all with the purpose of improving its capacity and efficiency in delivering a strengthened and more focused response in the fight against HIV/AIDS.

UNAIDS’ organisational development initiatives are likely to ensure the continuing relevance of its mandate as well as greater accountability of the UN response. However, the implementation of these initiatives has not yet caught up to the plan in areas such as results-based management and self-assessment processes (e.g., country programme assessments and accountability enhancement reviews, among others). It will take time for these initiatives to be fully implemented throughout the organisation, and UNAIDS will need provide consistent leadership to support these efforts, along with the training and financial resources required.

Although UNAIDS has made substantial progress in becoming a more performance- oriented and accountable organisation, there is room for improvement in its ability to measure its own performance.

UNAIDS has improved its focus on results and increased its efforts to measure its development effectiveness by developing and implementing the Unified Budget and Results Accountability Framework, a successor to the Unified Budget Workplan. Survey respondents were positive about UNAIDS’ new tools and processes for achieving results and the document review noted improvements in the structure and content of the organisation’s results frameworks. Nevertheless, UNAIDS has not yet maximised its use of performance information and improvements are still needed in the way results-based management is applied, notably in moving from activity-based to results-based reporting and in the use of performance indicators, baselines and targets to inform its work at the country level.

The UNAIDS Secretariat is valued for its technical expertise, evidence-based advocacy, and influence in policy setting.

UNAIDS is successfully maximising its knowledge and experience to support governments and other partners in pursuing national commitments. The technical expertise of UNAIDS’ country- based staff, as well as the use of this expertise for evidence-based advocacy, are highly valued by stakeholders on the ground. UNAIDS’ contributions to policy dialogue received the highest score of all key performance indicators in the survey. When asked to describe UNAIDS’ strengths, many country-based respondents commented on its expertise and technical support, as well as the effectiveness of UNAIDS’ influence on HIV-related policy setting.

UNAIDS’ highly consultative approach is crucial to the achievement of its mandate and its ‘Getting to Zero’ strategy.

UNAIDS is seen by direct partners and co-sponsors as highly consultative and inclusive, as it consistently seeks input from a wide range of stakeholders to inform the development and/or adjustment of its systems and tools. There is considerable evidence that UNAIDS benefited from consultative organisational development initiatives, such as the first and second multi- stakeholder consultations on the 2012-2015 Unified Budget and Results Accountability Framework and in the development of the UNAIDS 2011-2015 Strategy – ‘Getting to Zero’. This consultative and inclusive approach strengthens UNAIDS’ strategic positioning and leadership and increases its ability to achieve results. Nevertheless, UNAIDS could still improve its ability to be more strategic in its use of consultations.

UNAIDS’ effectiveness in building partnerships is highly valued and recognised by stakeholders as one of its strengths.

UNAIDS understands that partnerships and on-going relationships between and among partners are critical for the achievement of its commitments on universal access to treatment, prevention, care and support. As such, UNAIDS works to leverage existing partnerships and create new ones with various stakeholders. Examples of these collaborations include social movements, alliances, coalitions, networks, faith-based organisations, the private sector, as well as other development partners. UNAIDS convenes a wide range of stakeholders in a variety of settings to achieve common goals. The importance that UNAIDS places on meaningful, valued relationships with other organisations strengthens the UN’s capacity to meet its global targets and commitments

UNAIDS’ unique organisational structure presents both opportunities and challenges.

UNAIDS’ structure encourages collaboration and teamwork within the UN family in response to HIV/AIDS. As such, the revised Division of Labour aims to enhance the efficiency and effectiveness of joint work among co-sponsors to improve the delivery of results. The co- sponsors outline their contributions to each area for which they are convenors or partners, and the Secretariat focuses mainly on guiding, coordinating, evidence-based policy setting, and ensuring coherence and cohesion in the overall response.

In the MOPAN assessment, some surveyed stakeholders perceived inefficiencies in the operationalisation of the Joint Programme, while others commended UNAIDS for its added value in coordination of the HIV/AIDS response. As UNAIDS unique structure has and will continue to present opportunities to the organisation, there remain significant challenges in ensuring that defined roles of the co-sponsors and the Secretariat are respected at all levels of the Joint Programme.

---

[1] AfDB, UNICEF, UNDP, and the World Bank

[2] The Office of the United Nations High Commissioner for Refugees (UNHCR), the United Nations Children's Fund (UNICEF), the World Food Programme (WFP), the United Nations Development Programme (UNDP), the United Nations Population Fund (UNFPA), the United Nations Office on Drugs and Crime (UNODC), UN Women, the International Labour Organization (ILO), the United Nations Educational, Scientific and Cultural Organization (UNESCO), the World Health Organization (WHO), and the World Bank.

[3] United Nations Economic and Social Council. (1994). Resolution 1994/24. Joint and co-sponsored United Nations programme on human immunodeficiency virus/acquired immunodeficiency syndrome (HIV/AIDS). (p 3).

[4] UNAIDS. (2012). Joint United Nations Programme on HIV/AIDS; Composition of the Programme Coordinating Board (PCB). 1 January 2012

[5] UNAIDS. (2010). The Governance Handbook. (pp. 32-34).

[6] UNAIDS. (2012). UNAIDS Organizational Structure.

[7] Reduce sexual transmission, prevent HIV among drug users, eliminate new HIV infection among children, 15 million accessing treatment, avoid TB death, close the resource gap, eliminate gender inequalities, eliminate stigma and discrimination, eliminate travel restrictions, and strengthen HIV integration.

[8] UNAIDS. (2010). UNAIDS Second Independent Evaluation Implementation Plan. (pp. 14-20).

[9] MOPAN. (2005). The MOPAN Survey 2005: Perceptions of Multilateral Partnerships at Country Level. (p. 26)

[10] This component was tested in 2012 with the African Development Bank, UNICEF, UNDP, and the World Bank.

[11] The full methodology is presented in Volume II, Appendix I.

[12] The assessment looked at the UNAIDS Secretariat and its Joint Programme dimensions (Regional Support Teams, country offices, Joint UN Teams and Programmes on AIDS, etc) but not the work of the co-sponsors, as such.

[13] MOPAN criteria for country selection include: multilateral organisation presence in-country, presence and availability of MOPAN members, no recent inclusion in the survey, the need for geographical spread, and a mix of low income and middle income countries (middle income countries being subdivided into lower middle and upper middle).

[14] In the context of IFIs, these are referred to as “clients” and some organisations refer simply to “partners.”

[15] The normal convention for statistical significance was adopted (p≤.05).

[16] Respondents who wrote “no comment” or the like were removed from the analysis. Answers comprised of various elements were coded in various categories

[17] While most KPIs and micro-indicators were considered in the document review, not all were rated. See section 2.3.

[18] The normal convention for statistical significance was adopted (p≤.05).

[19] Reference can be made to section 2.2 of the report, which describes the data analysis process, including the fact that the normal convention for statistical significance was adopted (p≤.05)

[20] Resolution 1994/24 of the Economic and Social Council identifies UNAIDS core objectives as: provide global leadership, achieve and promote global consensus on policy and programmatic approaches, strengthen the capacity of the United Nations system, strengthen the capacity of national Governments, promote political and social mobilisation, and advocate greater political commitment. (http://www.unaids.org/en/media/unaids/contentassets/dataimport/pub/externaldocument/1994/ecosoc_re solutions_establishing_unaids_en.pdf)

[21] SMART: Specific, Measurable, Achievable, Relevant, Time bound

[22] UNAIDS. (2009). Second Independent Evaluation of UNAIDS.

[23] Partner Scorecard for UNAIDS. Retrieved 25 May 2012 from http://ihpresults.net/django/media/scorecards/partner-scorecard-UNAIDS-en.pdf

[24] As part of its mandate, the Office of Organisational Performance and Ethics (OPE) is responsible for designing and implementing a comprehensive corporate-wide risk management framework for UNAIDS.

[25] Some of these are still being tested – such as the country programme assessment mechanism that assesses the relevance, effectiveness and efficiency of country programmes.

[26] UNAIDS. (2010). Second Independent Evaluation of UNAIDS; Implementation Plan. (pp. 14-20)

[27] The UNAIDS Secretariat Strategy on Human Resources will address the following key organisational issues: integrated human resources policies, staff deployment, enhanced field focus, diversity and inclusion policy, facilitated staffing decision making, and increased accountability.

[28] Co-sponsors and other stakeholders are consulted in the drafting of country and regional work plans but are not included in the review.

[29] These funds were designed to assist the UN Theme Group to play a catalytic and facilitating role in advancing the scope, scale and effectiveness of a country’s response to the HIV/AIDS epidemic

[30] OECD. (2011) Survey Guidance; 2011 Survey on Monitoring the Paris Declaration (pp.36-37)

[31] OECD. (2011) Survey Guidance; 2011 Survey on Monitoring the Paris Declaration (pp.17-18)

[32] This was the case in the Second Independent Evaluation of UNAIDS in which the Oversight Committee (OC) was mandated by the Programme Coordinating Board to ensure a credible and independent evaluation process and the preparation of a report relevant to the future of UNAIDS. Members of the OC were selected to serve in their individual capacities and the Committee was directly responsible to the Programme Coordinating Board.

[33] UNAIDS. (2009). Second Independent Evaluation of UNAIDS. (p 11-12).

[34] UNAIDS. (2009). Second Independent Evaluation of UNAIDS. (p 11-12).

[35] CREAM (clear, relevant, economic, adequate, monitor-able) and SMART (specific, measurable, achievable, relevant, time-bound)

[36] The UBRAF specifies that indicators have been assessed by the Co-sponsor Evaluation Working Group and aligned with other UN initiatives, global AIDS and MDG indicators. They are the same as and/or complement indicators used by key partners such as the Global Fund to Fight AIDS, TB and Malaria; and the United States President's Emergency Plan for AIDS Relief (PEPFAR). These indicators measure the achievements and mutual accountability of the Joint Programme. Co-sponsors will principally be using their own organisational indicators and reporting processes for in-depth reporting on achievements.