Clearinghouse Evaluation Summary

Introduction

This report presents the findings from the evaluation of the clearinghouse model used to support both the Energy and Climate Partnership of the Americas (ECPA) and Pathways to Prosperity in the Americas (Pathways) initiatives. Through Department of State Bureau of Western Hemisphere Affairs (WHA)-managed grants, the Pathways and ECPA clearinghouses are charged with deepening and expanding partnerships to support the goals of each respective initiative. The general structure of the clearinghouse model is based on one implementer serving as coordinator for identifying and engaging initiative stakeholders to create a stakeholder community that will promote initiative priorities. The clearinghouse functions include establishing the initiative’s virtual presence, managing outreach, communications, as well as technical assistance and capacity building for events held by the initiative’s stakeholder community.

Evaluation Questions

As specified in the evaluation Statement of Work (SOW), the evaluation seeks to answer the three evaluation questions and four evaluation sub-questions listed below:

1. Were the initial clearinghouse designs well defined and closely followed in implementation?

2. Are there specific characteristics in the design of each clearinghouse mechanism that were instrumental in successful or unsuccessful engagement with partner countries?

3. Have the workshops and technical assistance activities been useful to participants? How well have the clearinghouses been able to address member country needs through clearinghouse workshops and technical assistance activities? What kind of monitoring and evaluation framework, including specific components, should be built into clearinghouse-like mechanisms to best capture mechanism results?

Methodology

The evaluation employed a mixed-methods design, using a combination of qualitative and quantitative data collection methods. Primary data collection methods included: (1) a review of documents related to the ECPA and Pathways clearinghouses, (2) key informant interviews (KIIs) with 23 stakeholders, including 13 current staff members at the Department and four at the Organization of American States (OAS), associated with one or both of the two clearinghouses based in the Washington DC area and (3) two stakeholder surveys. KIIs were conducted both in-person and via telephone by members of the Dexis evaluation team. The stakeholder surveys were web-based surveys administered on-line using Survey Monkey.

Conclusions

The balance of evidence gathered during this evaluation supports the conclusion that the clearinghouse designs were not well defined and that the Department failed to provide clear guidance to the OAS and other stakeholders as to what the clearinghouses objectives or end goals were. The Department did not clearly articulate a value proposition for the clearinghouses that would incentivize member states to engage with them, nor did it provide the clearinghouses with the means to affect member states’ incentives in this regard.

There are a number of factors that were instrumental in limiting the extent to which the clearinghouses were able to engage with partner countries. Drawing conclusions related to these evaluation questions was complicated by the fact that stakeholder perceptions of the clearinghouse were inextricably intertwined with their perceptions of the Pathways and ECPA initiatives. The lack of mission and vision for the initiatives disincentivized participation in the initiatives by member states. Lacking incentives to engage with the initiative, there was in turn little incentive to engage with the clearinghouse. The lack of incentive to engage with the initiatives and the clearinghouses was reinforced by a number of other factors, including the U.S. being the sole funder, the lack of a formal governance system, and a lack of a mechanism for incentivizing member state engagement.

The evaluation concluded the decision to take the clearinghouse administration outside of the U.S. Government and house it in an independent multi-lateral institution with hemispheric credibility and heft to be the correct one.

The clearinghouses did not incorporate sufficient ongoing mechanisms to receive and incorporate feedback into their respective engagement strategies. Such mechanisms might have included, for example, an M&E system that generates more or less ongoing information on clearinghouse performance (beyond implementation outputs and milestones) or a collaborative give-and-take relationship with the funding organization, neither of which existed in the case of the ECPA and Pathways clearinghouses¹.

The activities undertaken by the two clearinghouses were, for the most part, well-implemented, well attended and perceived as informative and useful by stakeholders. Stakeholders particularly valued the meetings and events (e.g., technical workshops, public discussion forums, ministerials) hosted by the clearinghouses, which received the consistently highest scores from respondents in the stakeholder surveys and feedback during interviews. Information dissemination activities were less highly valued and do not appear to be accessed at a high rate by stakeholders.

Recommendations

1. Future multi-country initiatives like Pathways and ECPA should include some form of a clearinghouse entity. At a minimum, the clearinghouse should be responsible for collecting and housing information about the initiative and communicating this information to initiative stakeholders; a critical and essential function for any multi-country initiative, such as Pathways and ECPA.

2. Grant agreements and associated SOWs should describe carefully and in detail the clearinghouse purpose, activities and expected results, while leaving sufficient flexibility (as appropriate) for the executing agency to adapt to on-the-ground developments.

3. Clearinghouse activities and expected results defined in the grant agreement and SOW should reflect a realistic assessment of what is feasible within the political context of the initiative and should include a reasonably adequate level of funding and staff resources to undertake the planned activities and accomplish the expected results.

4. WHA should concretely and proactively communicate the purpose of the clearinghouse to all stakeholders.

5. WHA should continue to hold the clearinghouse accountable for implementation outputs and milestones through the clearinghouse M&E system, but the M&E system should also include a set of indicators that measure higher-level results indicators at the outcome and impact levels.

6. The clearinghouse should be linked/associated with a formal and legitimate initiative governance structure so as to give it the heft and legitimacy it needs to carry out its political and/or technical tasks and to influence the attitudes and behaviors of initiative stakeholders.

7. The decision to place responsibility for the clearinghouse outside of the U.S. Government and with an organization that has status and credibility among the initiative’s members states was the correct one, and should be a considered a best practice for future clearinghouses.

8. The clearinghouse should be funded and staffed at an appropriate level that allows it to carry out all of the activities and achieve all of the objectives assigned to it.

9. The clearinghouse should, if at all possible, avoid having the U.S. Government as the sole source of funding and create opportunities from its inception for other member states to contribute to it both financially and in-kind.

--------------------

Footnote:

¹ WHA note: WHA is concerned about the small sample size of respondents affecting the validity of the methodology and the results.