DRG Learning, Evaluation, and Research Activity:
Mid-Term Performance Evaluation of U.S. Government Assistance to the Mozambican Attorney General’s Office (PGR) – Summary Report

Final Report Summary
(September 11, 2015)

Prepared under Contract No.: GS-10F-0033M / AID-OAA-M-13-00013, Tasking N029

Submitted to:
USAID

Submitted by:
Linn Hammergren (Principal Investigator)
Audra Grant (Evaluation Specialist)
Renée Hendley (Project Manager and Senior Survey Director)

Contractor:
NORC at the University of Chicago
Attention: Renée Hendley
Bethesda, MD 20814
Tel: 301- 634-9489; E-mail: Hendley-Renee@norc.org

DISCLAIMER

This publication was produced for review by the United States Agency for International Development. It was prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

Mid-Term Evaluation of U.S. Government Assistance to the Mozambican Attorney General’s Office – Summary Report
 

Purpose of the Evaluation and Questions Addressed

This report presents a summary of the NORC Evaluation Team’s findings of the Mid-Term Performance Evaluation of United States Agency for International Development (USAID) and the Department of State’s Bureau for International Narcotics and Law Enforcement Affairs (INL) assistance to the Procuradoria-Geral da República (Mozambican Attorney General’s Office or PGR) and the Gabinete Central de Combate à Corrupção (Central Office to Combat Corruption or GCCC), respectively. Fieldwork for this evaluation conducted between July 27 and August 07, 2015; a draft report was submitted on September 11, 2015.

The USAID and INL projects share the general objective of strengthening the capacities of their partner agencies to prevent, investigate, and prosecute crime, including (and for the GCCC, exclusively) corruption. USAID’s project, implemented directly through a government to government (G2G) agreement with the PGR, also includes a component to strengthen its financial management systems Public Financial Management Risk Assessment Framework (PFMRAF).

NORC’s evaluation goals are two-fold: 1) to assess the extent interventions contribute to the desired results; and 2) to generate evidence-based recommendations for coordinated U.S. assistance to justice-sector partners in the future. This mid-term performance evaluation was conducted by a two-person team comprised of one international expert and one local expert.

To this end, the evaluation, at USAID and INL’s request, emphasized the following research questions:

(1) To what extent is there evidence demonstrating the degree to which USAID and INL-funded interventions have contributed to achieving their stated purposes?

(2) To what extent have USAID and INL interventions reinforced one another?

(3) What types of assistance (e.g., trainings, embedded advisors, short-term technical assistance, and studies) have been most effective in helping achieve the desired outcomes?

(4) What do Mozambican PGR/GCCC personnel, U.S. Government personnel, local partners, and other donors perceive to be the most significant challenges to achieving results, and what gaps remain?

Methodology

To address the above research questions, the evaluation employed a mixed quantitative and qualitative methodological approach that included a desk review of relevant documents; over 30 key informant interviews among PGR, INL, and USAID personnel; 6 focus groups conducted in Maputo (4) and Beira (2) among participants in trainings for public probity law, procurement, budget and asset management and interrogation techniques; and a survey with a convenience sample implemented among the PGR departments of Human Resources, Administration and Finance, and of Procurement.

Due in part to the evaluation approach and to circumstances encountered during the evaluation, the following limitations are identified:

• Possible biases in survey data collection. As this survey is based on a convenience sample, inherent biases are introduced that limit generalizability of findings. The sample, for example, is comprised of a very specific group and a readily available population. Also, the current survey was supervised by a PGR staff member, which may have influenced responses to some degree. Survey results, however, do not suggest bias, but there is some potential for external influence.
• Small survey sample size. Relatedly, the small survey sample size (n=11) precluded sophisticated statistical analysis and generalizable conclusions. The survey results, therefore, should be considered a snapshot of views rather than representative of a population.
• Low response rates. The response rates for focus group participation was somewhat low in Maputo, but this did not preclude the vibrant conversation and useful data collection.

Key Findings

This section summarizes key findings in the evaluation for each research question. Findings suggest that essentially the PGR / GCCC have made progress towards accomplishing U.S. Government (USG) program goals, though advancement is encumbered by staffing challenges and other issues. Information-based systems centered on automated processes would substantially improve performance of both institutions. Partner involvement in this process is key.

Question 1: To what extent is there evidence demonstrating the degree to which USAID and INL-funded interventions have contributed to achieving their stated purposes?

USAID and INL assistance, including the highly critical PFMRAF, was received positively by the PGR and GCCC and was considered as a means of advancing their own institutional development strategies. USAID’s G2G format has advanced without encountering any issues with use of funds, although in the beginning there was a steep learning curve for the PGR.

The USAID and INL projects are on track to complete the activities outlined in the respective organizations’ agreements and work plans. The PGR introduced its own internal working group to monitor advances. However, vague articulation of program objectives –a mixture of “capacity building” and performance improvement – and indicators that emphasize inputs and outputs rather than outcomes and impacts, impede the ability to measure progress on program goals.

In the USAID program’s institutional strengthening component, anecdotal evidence suggests that the most critical PFMRAF goals to reduce financial management risks have been met. More systematic documentation, however, would fill in the gap of empirical evidence. Furthermore, there are questions about sustainability given the tendency for talented administrative staff to seek other work with better salaries.

There are additional administrative areas not included in the PFMRAF that could use assistance. Two of the most important administrative areas requiring attention are planning (for which a long term advisor could not be located) and training (as a program not just funding of a series of courses).

While some training was provided to PGR prosecutors, much of the USAID-supported PGR training is not related to criminal investigation and prosecution. PGR interviewees explained they have requested training in the non-criminal areas where they also work, which suggests that more training in these areas may be desirable.

Except for the INL Resident Legal Advisor’s (RLA’s) recommendations to the GCCC regarding the establishment of a separate group to focus on prevention, impacts on prevention or in fact any project-funded activities to advance it could not be identified by the team at this point in the evaluation.

Question 2: To what extent have USAID and INL interventions reinforced one another?

Coordination of the USAID and INL projects was organic rather than planned, and is shaped by close relationships among USAID, the U.S. Embassy and PGR/GCCC staff at the local level. The INL’s RLA played an important role in providing advice and training to PGR as well as to GCCC prosecutors.

Question 3: What types of assistance (e.g., trainings, embedded advisors, short-term technical assistance, and studies) have been most effective in helping achieve the desired outcomes?

Respondents in interviews, focus groups, and the survey gave high marks to advisory services and training in all areas. Interviewees stressed more training was needed in the aggregate, and suggested instruction be broadened to incorporate additional themes (e.g. cybercrime). Most respondents claimed positive impacts on their performance; however, only participants in the RLA’s formal courses (the only ones to include investigative police or PIC) offered specific examples of how they were applying new insights to their work.

Two USAID activities — a series of seminars and the delivery of legal kits — were, or will soon, be completed. While impact on actual performance is less discernable, these activities have generated awareness among personnel. The seminar on combatting wildlife trafficking, in particular, was reported to have augmented external interest in the theme. It also appears to have inspired the designation of a prosecutor as advisor to the Procurador Geral on diffuse and collective rights. This step represents an incipient effort to advance the seminars’ larger goal of promoting PGR coordination with other GRM entities and CSOs.

Question 4: What do Mozambican PGR/GCCC personnel, U.S. Government personnel, local partners, and other donors perceive to be the most significant challenges to achieving results, and what gaps remain?

PGR/GCCC staff, donors and other observers differed in their identification of major performance constraints. The PGR and GCCC most often cite low budgets and staffing, and to a lesser extent, lack of training and equipment, low administrative salaries, and difficulty with recruiting and retaining qualified staff. Donors and other observers, including some advisors, emphasized structural and institutional obstacles, such as difficulty with police coordination, low quality of information management, poor investigative techniques and organization, and poorly qualified auxiliary staff. A few also were critical of what they described as poorly drafted substantive and procedural laws and the additional functions, beyond criminal investigation and prosecution, prosecutors were expected to perform.

Coordination with other donors working in Mozambique’s justice sector remains a challenge for the projects. According to one interviewee, donors tend to seek “their own territory.” This approach avoids conflicts, but it erodes the potential for capitalizing on possible synergies and may discourage coordination among Government of the Republic of Mozambique (GRM) institutions.

Recommendations

Given the findings above, the evaluation report highlights the following suggestions.

The evaluation team recommends follow-on project be undertaken. In a next round, however, efforts should be made to specify and document the intended performance improvements and to ensure they are realistic given limited time and resources. This may mean, especially for USAID, an effort to focus resources on a few issues, rather than covering a broad collection of items.

USAID’s design of a follow-up project will require making some additional decisions – how much further it will go in addressing problems identified in the PFMRAF as opposed to insisting only that existing advances be maintained; whether it will enter some administrative areas (planning and training) the PFMRAF does not address; and whether it will maintain “crime prevention as a goal” given not only lack of progress but also absence of relevant activities in the Implementation Letter (IL).

Continued, relatively informal coordination at the country level is also suggested by the evaluation team. INL/Washington and the United Nations Office on Drugs and Crime (UNODC), based in Pretoria, are too distant to help in Mozambique and when they try, can create complications.

As much was learned in these projects, it goes without saying that any future efforts should build on that knowledge. Special note should be taken of the RLA’s assessment of the GCCC’s challenges, which are also applicable to the PGR.

It is the view of the evaluation team that for both for U.S. government (USG) and Government of the Republic of Mozambique (GRM) purposes, an effort to specify measureable performance indicators would be beneficial to assess progress toward programmatic goals. Indicators that suggest aspects of performance are “improving” or merely “doing better” are not enough, and a focus on inputs and outputs is also insufficient.

To overcome obstacles faced by some PGR advisors, USAID should ensure that both higher authorities and the heads of departments will work to understand and accept their tasking orders, and that the advisors’ plans recognize the PGR’s financial and human resource constraints, including the likely limits on donor funding.

Because these are, and most likely will remain, projects with a short duration and limited resources, more coordination with other donors is highly recommended. Such coordination would not only avoid duplication and conflict of efforts — which seemed minimal — but would also encourage collaboration in providing solutions to problems neither the PGR/GCCC nor individual donors can produce on their own. Strengthening the PGR’s current role in managing this coordination is desirable, so long as the PGR is receptive to diagnostics as critical as that provided for its financial management sector. A PGR lead in coordination, however, should not preclude direct donor-to-donor discussions.

Information-based management and investigation, two separate functions, will be critical to improving PGR/GCCC performance. (Improving police-prosecutor coordination is perhaps more vital, but work with police is currently not authorized by the USG for either project.) Over the medium term, this means automation of many processes, but currently no donor can finance automation adequately. Hence, donors and the institutions are advised to collaboratively develop incremental plans for “upgrading” the PGR/GCCC systems. Part of these plans should emphasize partner input. Donors may eventually finance the software and hardware, but if the PGR and GCCC are neither organized nor have the staff to use new technologies and equipment well, the improvements will ultimately be minimal, and one may have, as has occurred in several other countries another “failed system.”