2015/North East Asia Economic Assistance Review Summary

This assessment was conducted between January 2015 and July 2015, and the report was submitted July 31, 2015.

Purpose of the Evaluation and Questions Addressed

The Bureau of Near Eastern Affairs’ Assistance Coordination Office (NEA/AC) conducted an assessment of Economic Growth (EG) programs across the Middle East and North Africa (MENA) region. The goals of this assignment were to: (i) assist the office in documenting the types of United States government (USG) economic assistance programs and activities being implemented; (ii) identify which projects were working well and why; and (iii) assist in aligning economic assistance programming with policy priorities. Evaluation questions included:

- What economic assistance programming has been conducted by USG agencies in the past three years?

- Which programs met, exceeded or failed to meet their objectives? What were the measures of performance?

- What are the reasons for meeting or not meeting objectives?

- What are the factors that have influenced/contributed to successful programming?

Methodology

For the full inventory, evaluators received an initial list of 162 projects managed by NEAPI (MEPI), NEAI (Iraq) and USAID. Through consultations with USG offices, the list was increased to 291 projects with additions from Treasury, OPIC, MCC and USAID.

NEA and dTS created a sample of 50 MEPI projects targeted for full review. To the degree possible, the sample reflects the distribution of projects in the full inventory in terms of country location, sponsoring agency, purpose and cost. Through a desk review (as well as several interviews with several project managers), the Assessment Team analyzed each project in the sample to understand its original objectives, activities, budget, Results Monitoring Plan (RMP), and evolution from initiation to completion. Each analysis contributed to the Assessment Team’s findings with respect to performance, best practices, lessons learned and, in some cases, preliminary benchmarks.

The desk review drew on all aspects of project analysis, including original design, to describe its current status and ascertain overall progress compared to objectives. The Assessment Team found it useful to devise a simple, yet inclusive, measure of project performance, i.e., whether the project exceeded (3 points), met/was meeting (2 points), or was below original expectations (1 point), or is considered a loss (0 points). This measure encompassed the analyst’s judgments derived from the operating environment, indicator status, resource utilization, stakeholder engagement, private and public sector collaboration, implementer effectiveness, and movement toward targets. The Assessment Team then carried out an inter-rater consistency exercise where each member reviewed the ratings assigned by others in the Team, noted cases in which their judgment differed from that of the initial rater, then engaged in a discussion to review the project again to confirm or change the original rating. The exercise contributed to a harmonization of criteria used for rating project performance.

Key Findings

Significant Resources for EG projects: Total commitments for Economic Growth were $2.4 billion, of which 2% was from Treasury, 7% from State/NEA, 18% from OPIC, and 73% from USAID.

Correlations between purpose (based on the F Standardized Program Structure) and funding: In the full inventory of 291 projects, “Generate Employment” is the most frequently pursued economic growth purpose (83 projects or 29%), followed by “Enhance Entrepreneurship” (60 or 21%) and “Strengthen Business” (48 projects or 16%). However, the largest amount of funding is dedicated to “Increase Investment”, followed by “Strengthen Business” and “Generate Employment”. The largest single “Increase Investment” project is in Egypt (the $300 million Egyptian-American Enterprise Fund). “Increase Investment” also leads the average budget per project category. In all categories, USAID manages the largest project for each specific purpose.

Stable environments matter: The most propitious country environments for superior results based on the assessment team’s project ratings were Bahrain and Morocco. Tunisia, Jordan and Morocco were the most consistent performers given their relative stability. The most difficult environment was Yemen. Regional projects that strive to work in several countries simultaneously faced significant implementation challenges.

A quarter of all MEPI EG projects are not performing as planned: The Assessment Team utilized a performance rating system taking into account a number of factors to determine if a project was on track to meet its objectives. A review of sample projects concluded that 13% greatly exceeded expectations, 60% were meeting original expectations, 26% were performing below expectations, and 1% was categorized as a loss.

Project Design for MEPI projects needs to improve: While many sample projects displayed an exemplary design, a very large number lacked targets, meaningful and measurable indicators, or quarterly project officer reports of performance (monitoring shortcomings), and demonstrable corrective actions. None included budgets based on expected results or outputs (to complement the line item budget), benchmarks (references of cost standards, e.g., $/trainee, from other, comparable projects), or performance ratings.

Recommendations

1. Articulate a theory of change for MENA EG projects that includes critical thinking on a project’s purpose, its context, and logical framework for success.

2. Create a “project purpose” categorization that condenses some ninety F-Project Elements related to economic growth into fewer primary purposes.

3. Establish a set of estimated benchmarks for EG investments, related to the costs of activities per outputs and outcomes, and by estimating the portion of the overall budget associated with each project activity and outcome.

4. Devote greater attention to identifying specific risks up front as well as their mitigation plans.

5. Before a project’s approval, establish a set of generic best practices for projects in general and specific best practices for projects with the same primary purpose, which would be factored into approval decisions.

6. Rate the project’s current performance on a quarterly basis. Three to five years after project completion, NEA should conduct follow-up studies of clusters of projects in the same category to gauge their long-term impact.

7. Utilize a quarterly report that summarizes the status of all current projects within the sector and country portfolios to share with senior leadership.

8. Share the technical features of the Assistance Coordination Performance Reporting System (database) with other USG agencies.

 

Table 1: Full Inventory and Sample Investment per Location

Total Funding Levels

Country

Inventory - # of Projects

Inventory - Total Budget

Sample - # of Projects

Sample - Total Budget

Algeria

5

$2,787,432

2

$1,069,180

Bahrain

2

$1,243,031

2

$1,243,031

Egypt

41

$409,457,791

6

$7,376,000

Iraq

24

$580,017,775

3

$9,517,912

Israel

23

$1,895,924

6

$498,894

Jordan

23

$574,302,846

2

$865,305

Kuwait

1

$56,889

0

$0

Lebanon

13

$109,262,500

1

$2,000,000

Libya

4

$17,364,827

0

$0

Morocco

18

$87,152,332

4

$3,609,863

Oman

2

$173,963

1

$116,590

Qatar

3

$277,469

1

$104,478

Regional

46

$36,633,988

7

$16,719,330

Saudi Arabia

3

$868,542

0

$0

Syria

1

$3,850,000

0

$0

Tunisia

35

$169,262,573

12

$20,287,904

West Bank/Gaza

28

$274,470,501

2

$2,249,918

Yemen

19

$131,914,199

1

$800,000

Total

291

$2,400,992,581

50

$66,458,405

Notes: (a) ECA programming is listed as "Regional."
(b) Unreported ECA project budgets were estimated at $120,500 per project, the average of all ECA budgets received.

 

Table 2: Projects by Primary Purpose - Full Inventory

Purpose/Category

Inventory - # of Projects

Total Budget

Average

Largest Budget

Country of Largest

Agency of Largest

Generate Employment

83

$ 282,074,912

$ 3,398,493

$ 90,000,000

Yemen

USAID

Enhance Entrepreneurship

60

$ 33,774,395

$ 562,907

$ 7,425,093

Tunisia

USAID

Strengthen Business

48

$ 401,411,510

$ 8,362,740

$ 179,800,000

Iraq

USAID

Reform Policy

30

$ 204,178,256

$ 6,805,942

$ 43,039,107

Jordan

USAID

Reinforce Public Finances

28

$ 132,519,237

$ 4,732,830

$ 46,122,834

Iraq

USAID

Increase Investment

24

$ 1,232,293,791

$ 51,345,575

$ 300,000,000

Egypt

USAID

Promote Trade

18

$ 114,740,480

$ 6,374,471

$ 58,012,718

West Bank/Gaza

USAID

Total

291

$ 2,400,992,581.23