Workshop on Measuring by Learning in Peacebuilding

U.S. Department of State Third Annual Conference on Program Evaluation - Evaluating Partnerships Track
Washington, DC
June 8, 2010

MR. SRIKANTIAH: Good afternoon, everyone.

I'd like to welcome you to our fourth workshop and final one under the theme "Evaluating Partnerships." The title of today's workshop is "Evaluating Diplomacy," and it will be presented by Aaron Chassy, Senior Technical Advisor for Catholic Relief Services.

Mr. Chassy is a specialist in good governance and civil society with more than 20 years of experience designing and managing programs in 30 countries, as well as the United States.

His areas of expertise include decentralization and democratic local governance, implementing participatory policy reform, and civil society development.

In his current position, Mr. Chassy applies best practices and lessons learned to increase program quality and governance in civil society in a half a dozen programs worldwide.

For the past nine years, he has worked with several USAID implementing partners, including both for�' and non�'for�'profit organizations, focusing primarily on policy analysis, design, municipal management, integrated rural development, public sector reform, and anti�'corruption initiatives.

And before I give the floor to Mr. Chassy, I request that the audience hold their questions for a Q&A session after the presentation.

Thank you.

Get Adobe Reader View slide presentation ]

MR. CHASSY: Thank you all very much. I hope everybody is, you know, through their post�'lunch coma.

I'm going to try to get us through the presentation itself fairly quickly, so we can have a nice robust dialogue about what's going on, because really, what I'm about to present is a work in progress.

First of all, just to give you a little bit of background of what Catholic Relief Services has done with what we call monitoring, evaluating, and learning, MEL, here is a list of some of the publications we have, and if you're interested in any of this, I really encourage you to go not to the website on my business card, for those to whom I've given it, but rather to, all one word,, and you can find these and many, many more.

So what the heck are these things that we call GAIN or GAIN templates? They were supposed to be globally accepted, but really, I mean the best we can say is they're generally appropriate. So they are very effective for a project�' level or, in some cases, higher or program level monitoring and evaluation.

The idea is, rather than having our 50�'country programs fishing around more or less for the same sets of performance measures, you know, here's something that we can have that's centralized, while not cast in stone, so this is something they can use as a template to learn from, to base their efforts on, and so it should stimulate that kind of learning process that is part and parcel of CRS's project development cycle, and then, finally, obviously most importantly, these need to be consistent with the standards or the requirements from our public donors, such as USAID.

What these are not. As I mentioned, they are not globally accepted so much as generally appropriate, and they are certainly not core or mandatory. We don't expect that all peacebuilding, governance, civil society projects worldwide use or draw from this set of indicators.

They are not representative of all the work we do in this area. With 50 country programs and program projects in another 50 countries, there is such a diversity of what we do in this particular field, one set could not possibly capture everything.

The other thing is, as I mentioned at the outset, none of this is finalized, and we are trying to go not only internally within our country programs but also with you, our peers, our colleagues in this area of practice, to get your feedback, and that's why I'm kind of rushing my presentation to get as much feedback from you as possible.

So what are some of the benefits? Well, as you'll see in front of you, I've got a couple of example templates, and we'll go through that in just a little bit, but you'll see that they emphasize qualitative analysis to add some depth, answering that why question that we've been talking about in some of the sessions to accompany the more quantitative measures.

The other thing that we tried to do is based on, again, consistency with donor requirements, is be supportive of our efforts to validate our theory of change, okay?

The other thing that we try to do is be very clear about what we mean by each of the terms that we're using in the GAIN template, and make sure that everything is not only precisely defined but specific to the context where we're asking people to apply the template.

And then the other thing that we want to make sure that people are doing �'�' this is not exactly on the paper, but we prompt the users of the template to make sure they're going out to various stakeholders, to partners, to contextualize not only the what but the how in using this template.

There are still some challenges.

I mentioned context, and that's what it's all about in peacebuilding. It's location, location, location. These are going to have to change. They should change, depending upon where you choose to apply them.

The other thing, of course, is what we've heard quite a lot about in some of the Q&A, these donor expectations. Honestly, how many people really believe that, for a peacebuilding effort, you're going to be able to achieve a statistically significant reduction in violent conflict? I mean, if you really believe that, I'm sure you have inhaled early and often, okay? So let's try to be realistic.

You know, we're trying to address and surmount some very complex development challenges. Resources are quite limited. Let's not forget that, you know, this is usually one small project within a larger donor portfolio of projects, which goes across however many donors working in the sector, and let's not believe for one moment that what donors do is somehow the be all, end all for overall efforts, state, societal, otherwise, in this particular field.

So getting into more of the nitty-gritties, what are we looking at with some of these templates? Well, in Catholic Relief Services' peacebuilding program worldwide, we have three strategic objectives. You see them up there. So that's the three.

The five are the five sub�'sectors which we identified in a nice democratic process with our community of practice. You see them up there, as well. So hence, three by five, and then we realized we still needed to have some sort of over�'arching strategic objective or pillar�'wide GAIN template, so three by five plus the additional three, and that's how many we have out there right now, and I'm sure there's more to come.

So what I would like you to do is to pick up �'�' I am not sure which is on top for most of you, but the first example I'm going to look at is the template that is looking at the indicator, the degree to which citizen participation is integrated into the government's annual budget development process. What you'll see up front, at the top there, in a sort of tabular form, is all this information, all right? So we've got a theory of change; we're trying to link that directly to the results statement.

When we look at objective, that is the Catholic Relief Services objective. Also we're looking at indicator level. The only thing I would �'�' I would point out is, for those people who have worked in democracy and governance or peacebuilding, however you choose to call it, this is always the subject of a lot of discussion between you as a technical person and maybe your mission leadership, if you're working for USAID, or between you and your fellow colleagues in other technical sectors.

And you say, well, you know, what does this have to do with achieving an outcome level or impact level result, I mean getting citizens more involved in budgetary discussions? Well, location, location, location.

If you're dealing in an environment with a
semi-authoritarian regime, like we often do, you know, getting citizens to be able to and to be able to reclaim their right to participate in these local expenditure/budget discussions, that's a pretty big deal, and it reflects a much higher level result than, let's say, a health or an education project, where that would only be the first step to maybe delivering higher quality services in that particular collectivity or municipality.

So continuing with this, I'm just going through the template, little section by section. We try to give a little background about some of the issues that might affect government capacity and willingness to be more inclusive. We talk about how you might want to map out planning for data collection.

We actually give a sample survey instrument �'�' okay? �'�' some of which is qualitative, some of which feeds into a more quantitative measure, and then we show how you might calculate based on if it's a quantitative measure to get your �'�' the little bean you put in the box �'�' right �'�' and then, you know, where you might go for further information. This is, again, trying to respect the complexity of the issues we're looking at, trying to get a better picture of the context.

We also, for internal purposes, try to show the links to some of these other GAIN templates that you might want to look at if this doesn't quite fit what you're looking for. And then, finally, there are links, both internal and external, to where you might go for more information.

So I'll look at the second example real quickly, and this is looking at the percent of targeted youth engaged in violent activities in X time period. And I hope some of you are coming here from the session next door, looking at CARSI, Central American Regional Security Initiative, because this really gets at some of the interesting work that they're doing and might offer possibly a way to measure it. Because in fact, it fits very well with an effort that we're doing in the region, in six Central American countries, called (inaudible) where we are taking an approach to youth development that includes not only vocational skills, life development skills, but also civic engagement, very much mirroring the Obama Administration's global engage for youth.

So here is our theory of change, that if youth needs are met, particularly in these areas of life skills and employment, then they are less likely to engage in violent behavior, tied to this results statement, and you'll see up here and also in the template itself, there is an emphasis on men, and I hope there are gender people who would say, hey, where's the women?

What we've seen from our experience is actually it's young men who are more likely to engage in violent activity, not to say that women don't, and we make clear later on in the template, you need to take a step back and see how, you know, young women might be involved, but this is what they were talking about in USAID 12 years ago as the muchacho factor (inaudible).

So going down again through the sort of CRS, what's the objective, what's their sub�'sector, and again, indicator level is going to depend, all right, on context and donor requirements. If you're doing an economic growth project versus a peacebuilding project, maybe this indicator would be more interesting at the IR level, but for peacebuilding, we think it might be more SO level.

Right. Moving right along. This is real important with youth in terms of defining, you know, what �'�' what's your cohort, what's your age cohort �'�' okay? �'�' and also looking at, you know, minority issues, especially where you have socio�'cultural identity issues that are creating cleavages in society, and this is also relevant when we're using what we call a people�'to�'people approach, bringing the opposing sides of the conflict together around, let's say, issues of collective interest, typically socioeconomic stuff.

Planning for data collection. Now, if you're talking about violent acts, obviously there is all kinds of sensitivities, and you'll see in the discussion there in that section, we tried to guide people a little bit about how to take care �'�' how to be mindful of some of these sensitivities. Again, walking you through a sample survey instrument and how to make the calculation, and the rest is pretty much what it looks like up there on the screen.

And we are at an end, which leaves us �'�' wow �'�' 25 minutes to talk. And that's really my point for running through this, is so that now that you have it in front of you, we can talk a little bit about, you know, where to go with this kind of an effort, in your opinion, is it useful, what could be changed to make it more useful for those parts that are not so useful, and maybe share some of your own experiences with your own respective agencies' and organizations' similar kinds of approaches.

MR. SRIKANTIAH: The floor is open to questions.

A PARTICIPANT: If everyone could use the microphones, because we are audio recording the sessions, and it just helps with feedback levels.

QUESTION: Okay. I was wondering about �'�' before working on these indicators or figuring out which indicators were appropriate, if you were building in an analysis of the levels of conflict that you're talking about, because some �'�' I think you have to adjust the indicators in accordance with the type of information you're gathering, because if you're talking about all – all out conflict, most of your indicators should deal with the �'�' with the level of reduction in conflict completely, and the underlying factors will be a part of the secondary concern. So I was just wondering how you �'�' how you work that out.

MR. CHASSY: Okay. A couple things. One is just CRS lexicon. When we say peace�'building, it isn't just the conflict transformation that you might hear about in, let's say, the USAID Office of Conflict Mitigation and Management. It would also be the same kind of stuff you would see next door in the democracy and governance office, because for us, we try to take a more holistic picture of it.

Now, in terms of the stage or the level of conflict, absolutely, that's something which some of our social cohesion indicators are going to have to take a look at, where you are dealing with a hot conflict or maybe one that is sort of in the stage of embers, or if it's actually on the opposite side as it's beginning to escalate a little bit.

I don't think that it's something that, as you'll see from both templates, that we have actually factored in as a heading, and maybe we should consider that, where, you know, people need to be mindful of, you know, how does this fit into the overall conflict analysis, and I guess the only related thing I could say is that, typically, if a CRS country program is going to engage in a peacebuilding program, part of that process is to conduct a conflict analysis, which hopefully will produce some of the information that you're talking about.

QUESTION: I was interested in the last panel, in the democracy series, where they were looking at violent crime, because most of the post�'conflict situations I've worked with is �'�' the conflict is politicized in competing factions or tribes or whatever, and that they employ unemployed young people, but in fact, it has a political objective, which is to take control of the state, if they can.

In your template here, does it make a difference whether it's the kind of stuff they were talking about in Central America, in that session, or whether you're dealing with really political conflict in the use of the template.

MR. CHASSY: If you look at the youth template, we do mention, yeah, that it is possible that what you're looking at in terms of youth violence is a result of elite manipulation for the very purposes you talk about. So you know, these templates are wide open, and they need to be contextualized.

I think it's up to, in our case, our country program staff to have done a good conflict analysis to be able to identify what are some of the drivers of conflict, and if that includes elite contestation for control of state structures, yeah, we should have that in there.

I would mention, just by way of discussion, that �'�' you're looking at Central America �'�' I've been in Honduras now, off and on, the past couple of years, and it's very clear that, you know, this sort of Jello�'like dynamic of the violence sliding from Colombia up through Central America is a result, in part, of the elites, the wealth elites, you know, using gangs to carry out their dirty work while they are able to remain legitimate and above board with their commercial enterprises.

Meanwhile, you know, they're able to launder their money through those so�'called legitimate enterprises, and it's �'�' you know, the focus is on what? Not on them and their nefarious activities but on youth gangs. So go figure that one out. That's a great point. Thank you.

MR. SRIKANTIAH: Any additional questions? Okay. Sure. Go ahead.

QUESTION: Which side of the conflict situation does this apply mostly? Is it as the conflict is escalating or in �'�' can you apply it in �'�' what a lot of USAID programs get involved in is post�'conflict, and I've worked in Liberia and Sri Lanka and Kosovo, and we were dealing essentially after the violence but still simmering, you know, potential for more violence.

MR. CHASSY: Right. I think the answer is, it depends. I would probably argue you could use it for both sides. I think, for instance, you know, if a potential trigger for conflict is upcoming elections, let's look at, let's say, Sudan, with the referendum coming up, and elites are trying to, you know, marshal their forces by provoking and manipulating youth, that would be on the side leading to potential conflict, but all the examples you gave, plus those you and I heard about last session, that's obviously post�'conflict, major conflict, let's say. So I think, with adaptation, these templates could be used on either side.


QUESTION: I may post now a stupid question. I have not been involved in this practically, I'm here to cover this conference for my superiors in Stockholm, but on the two tables here, I'm afraid I don't understand what SO and IR stand for.

MR. CHASSY: Well, God bless you. There's no stupid questions, first of all. This is the alphabet soup we all swim in every day with USAID�'funded work. "SO" is strategic objective. "IR" is intermediate result.

I knew that I had really arrived at Catholic Relief Services when I completed a full sentence, both subject and predicate, using acronyms only. The six�'pack of the Kool-Aid was in the fridge. As even people like Mary Anderson will point out, you know, “log frames ain't bad.”

John Paul Lederach, the godfather of peacebuilding �'�' there's nothing wrong with good linear thinking, as long as it's balanced with more reflective, not so linear thinking, the learning part.

MR. SRIKANTIAH: Yes. Go ahead.

QUESTION: Hi. So you mentioned that the processes on developing this, that that was very internally democratic, and I was wondering if you could share a little bit more about sort of how that worked internally.

I'm thinking about the organization where I work. I think a lot of the functional areas that we address do have really clear theories of change that could back into this, but some areas, I would wonder, maybe they are a little bit more fuzzy, and kind of �'�' could a process like that really work for us.

And then my second question is about donors and how this sort of feeds into that relationship. I think CRS receives a fair amount of private funding, is my understanding, I'm not sure, but how is that sort of tied into your obligations and responsibilities to donors?

MR. CHASSY: Great questions, both. So first is the internal process. The second is, you know, how is this truly consistent with donor requirements?

So the first question �'�' it was democratic in both how we came up with the strategy that has these three pillars and five sub�'sectors and the rest, and that took place over the last three to four years. The development of the templates �'�' we went out to what we call our peacebuilding technical commission, who are the headquarters, regional, and country level technical staff who work in the area of peacebuilding, and we said, look, do you have any really good examples of indicators that you have used in the recent times for project implementation, and we got their inputs, and then we worked with a consultant who is sort of internal.

She is the acting senior technical �'�' our regional technical advisor for monitoring and evaluation out in our Asia region, and she helped us to put together the template form itself, and then to start to populate it based on this long, arduous process of winnowing through all the various indicators that people sent in, and you know, it wasn't just internal in the sense that good people, good friends, colleagues from organizations like Search For Common Ground helped us out by giving us their own little matrix of indicators, not templates but indicators that they use in this work.

We also �'�' you know, I looked around �'�' I'm a former USAID DG officer, so I looked through some of the USAID indicators in the democracy and governance indicator handbook, 1998. It's up there, still on the USAID web page.

And then we also you know, just had a lot of internal vetting and feedback and went out to some colleagues who we knew were experts not just in peacebuilding but monitoring and evaluation.

Mark Rogers, some of you might know from your work in this particular area, who is an independent consultant but very much associated with some of the work with Search For Common Ground.

In terms of families of theories of change, we all know about the families that came out a year�'and�'a�'half or so again from CMM with their annual program �'�' CMM �'�' sorry �'�' Conflict Mitigation Management office at USAID.

They came out with their annual program statement, and they included as a requirement, if you're going to write a proposal, you have to write it from this particular family of change �'�' sorry �'�' family of theories of change and, you know, use one or two or three of them, and then describe how you're adapting it to the context, and that gets to the second question about compliance with donor requirements.

We were being pushed to do GAIN templates, and it was just serendipitous that �'�' and by the way, those �'�' it was serendipitous that this Conflict Mitigation Management procurement came out where we could really throw in the theories of change and say, look, this isn't coming out of thin air, this is what the donors want, although my understanding is the most recent annual program statement that they issued, they didn't include theories of change.

So you know, here today, gone tomorrow. What are you going to do? But we believe, as a general best practice for doing work in this particular field, you need to have a theory of change.

QUESTION: I'm with the U.S. Government Accountability Office, and one of the questions we would have if we saw something like this would be about the reliability and validity of the data about the social science principles that were used in developing the instruments by which these data are collected. And I'm surprised there isn't more detail and instructions for coordinators, because just looking through it, there is a section, tips for data collection, you know, on page 3, but some of the information that you're looking to collect is very difficult to collect, very sensitive, and I mean, it could be collected through a number of techniques, and I'm just wondering, you know, is �'�' what else have you done in that realm to assure the user and to assure, ultimately, you know, the donors of the reliability and the validity of this information to make comparisons across project and, you know, across time?

MR. CHASSY: First of all, truth in advertising. My spouse works for GAO, so I hope anything I've said here will not be used against me.

That's an excellent question. I have to say that, you know, we, in putting together these templates, did not go to do the sort of collection �'�' what's the right word? We didn't do the research to see who has been there before us, looking for similar kinds of information and what kind of instruments and methods did they use and advance?

I think that's something we really might think about doing, because not having done that, it's very hard to give you a confident answer to the better part of your question, which is what do we do to assure the validity and the reliability of these templates?

The answer is probably not a whole heck of a lot, and it's something which we should probably work on. In terms of the sensitivity involved in collecting certain data, the good news for us is we're not a contractor. We're not a contractor with a nonprofit status like a lot of our sister organizations that only go to these countries when they get donor funding.

We are there because we've been asked to come by our key partner, which is generally the Catholic Church but sometimes others, and that opens doors and really sort of softens a bit the sensitivity issues in ways that I can't even begin to describe.

It's not a panacea, but it allows us the level of access to participants and beneficiaries, and gives us a degree of credibility that we would not have otherwise, because what we do and how we do it are equally important, and as such, I think, you know, beneficiaries and participants see right away that their participation in our activities is not simply instrumental, you know.

This is of, by, and for them, a lot more than you would see in other organizations that are simply responding to the donor requirements and the program description and the request for application.

MR. SRIKANTIAH: One last question.

QUESTION: Good afternoon. Do you aggregate this information that you collect from these 50 countries?

MR. CHASSY: That was my fear when we started this whole template process. Thankfully, no.

Having said that, what we do try to do �'�' we don't pretend that everyone is going to latch onto, you know, let's say, this particular indicator if they're doing work in this particular area, and therefore, we could collect across the 25 country programs doing work in this particular area.

Instead, what we try to do every year is, when we have this workshop for members of the peacebuilding technical commission, is we try to populate a matrix that shows all the different work we're doing, where we're doing it, roughly, you know, sort of progress level reporting, nothing too, too serious. That's about as far as we go with aggregation.

I don't know if that's in the future. It sounds like a, you know, worthwhile pursuit, but again, with peacebuilding, especially, as has been mentioned in some of these other workshops on democracy and governance, it's really hard to aggregate cross�'country and cross�'temporally when, you know, you're �'�' to paraphrase the cliché, you're comparing oranges to grapefruits. You have to be careful.

MR. SRIKANTIAH: Well, I think we'll close, and thank you, Aaron, for your presentation and your questions.


MR. SRIKANTIAH: I'd just like to have you take a look at the schedule. Our next event at 4:00 will be Shelley Metzenbaum speaking at the �'�' in the auditorium.

So we'd like you all to start moving into the auditorium, if that's possible.

Thank you.