Workshop on Evaluation's Role in Informing the Design of Anti-Terrorism Assistance Programs

U.S. Department of State Third Annual Conference on Program Evaluation - Anti-Terrorism Track
Washington, DC
June 8, 2010

MODERATOR: Okay. For those who are here for session two of the anti-terrorism track, I would ask you to take your seats and we will get started.

And for this session, we have the anti-terrorism assistance program, and here to talk about that we have Dan Rosen, senior advisor for strategic plans in the Bureau of Counterterrorism.

We have Mr. Roberto Figueredo, who is the sustainment coordinator for DS/T/ATA. He oversees the institutionalization and sustainment of anti-terrorist assistance programs, has over 25 years of experience in policy formulation and strategic planning and program design. As a senior manager, he directed the development of assistance programs in economic reform, democracy, and governance, police and security sector reform in Latin America, Eastern Europe, Asia, and the Middle East.

And actually doing a presentation we have Mr. Charles Stevens, who is the chief of the assessment review and evaluation unit of DS/T/ATA, and he has over 20 years of experience in security analysis and design, including 7 years as security manager for 2 multi-billion-dollar natural gas projects in Chad and Algeria. Mr. Stevens also served as a commanding officer in the U.S. Army Special Forces. He is a graduate of Georgetown University School of Foreign Service.

So, who should I turn it over to? Dan, you want to come on up?

Get Adobe Reader View slide presentation ]

MR. ROSEN: Thanks very much. Let me just say a couple words about the ATA program overall, and then turn it over to the folks from DS to talk a little bit more about the assessment and evaluation process.

The ATA program has been around for about 27 years. It’s really the U.S. Government’s primary mechanism to provide counter-terrorism law enforcement assistance to partner nations. Its real overarching goal is to help partner nations build self-sustaining anti-terrorism capabilities. And since its inception the program has trained about 67,000 people in about 150 partner nations. So it has been really, over the years, a worldwide program.

Right now, the ATA program is active in about 63 partner nations. That large number has been scoped down a little bit and has moved around over time, as we have seen the threat shift, and capabilities needed shift.

SCT and DS/T/ATA both play important roles in the program. SCT is really responsible for policy formulation and guidance. DS/T/ATA is the implementer of the program, and provides the program administration and management. And that’s a good partnership. The two entities, the two bureaus within State have had an MOU that goes back many years, and has just been revised to reflect some of today’s realities.

The process and the policies that SCT uses for adding and removing partner nations to the program is sort of under review at the moment. We have had a process over the years for granting countries partner nation status in the first place, and that is based on a number of factors.

First, the threat, obviously, and �'�' as well as then what the U.S. interests are in that country, what our ATA �'�' what our anti-terrorism goals are in that country, and the host nation’s current capabilities and its political will to implement any additional capabilities.

Once a country is designated as a partner nation for ATA, then we really kick into gear with the assessment process and the programming process that follows from it.

That’s probably a good point at which I can turn over the briefing to C.B. to talk to you a little bit more in depth about the assessment process itself. And then I may have a few comments at the end. That’s kind of the big picture on the ATA program. C.B., you want to take it from there?

MR. STEVENS: Okay, good morning. I think it’s still morning, isn’t it? It’s late for me, anyway. Thank you, Dan.

As Dan said, it’s a partnership between SCT and ATA. And SCT handles, basically, the policy aspects and ATA handles the program implementation. And to that end, obviously, we need to be able to assess and evaluate what we’re accomplishing.

I will give you a brief �'�' very brief �'�' history of how we got to the point we are at today, in terms of evaluation, so that it might make a little more sense to you. When ATA �'�' when I came to ATA eight years ago, it was really a capacity-driven organization. In other words, we had a budget of X, and that budget bought us four of these type of course, and six of that type of course, and some of these over here, and a few of these, and a basket of apples, and whatever.

And then, all of our program managers would sit around once a year �'�' and each one of them being responsible for a portfolio of countries �'�' and he would say, “I want two of those and I want three of those and I want four of those,” and the other guy would say, “Well, I need two of those, too.” And then they would have a fist fight and decide who got which courses. And that’s how it was decided how the countries would receive training.

Obviously, that’s not a very efficient or effective way to either improve the capabilities of the partner nations, or to address U.S. strategic interests and regional interests in those partner nations.

So, the GAO and OMB and a couple of other agencies helped us realize that that wasn’t the best way to go about business, and encouraged us to find a better way to do it. And so, we went then to what we call the �'�' for want of a better term �'�' an assessment-based system, or a capability-based system, in which we developed a criteria for evaluating the capabilities within each of these partner nations.

And then, a determination would be made in a report that Country X has a lack of capability in this thing and that thing and other thing and two of these things over here, and so that was the way we could spend money.

And then the program managers could say, “Well, all right. I need $1,000 to do that, and I need $1,500 to do that. So, overall, I need $2,500 for my country.”

Well, that was better. But it still didn’t really address what U.S. objectives were in those countries, and it didn’t really address the whole thing globally, in terms of are we really helping this country develop a sustainable capability, or are we just putting Band-Aids on all the little scratches we see everywhere?

So, realizing once again that we hadn’t quite reached the point of best return, we decided that we needed to actually have objectives. Because, as I like to say to people, if we line everybody up on a start line and tell everybody that we’re going to conduct a marathon, and we’re all going to run our best, and we’re going to run 26 miles, and at the end we’re all going to say, “Yay, we accomplished it, we won,” we have to know where the finish line is. Otherwise, we’re all out there running in different directions, or we’re all running in the same direction, but we’re not heading to the right place.

So, now we operate on an objectives-based system. And the way that we do that is we refer back to the national strategic guidance documents to find out what the U.S. policy goals are for a certain region or a certain country.

And working with SCT, the strategic objectives are established. Those are then passed down to the program managers within ATA who have responsibility for certain regional and bilateral programs. And they then develop a set of objectives that fall out of those �'�' that strategic-level guidance that we get from SCT and from the national strategy documents. And they develop those objectives to address those things that ATA’s tool kit is appropriate for.

And then, those objectives come to my section, the assessment, review, and evaluation unit. And we apply to those objectives a list of what we call the 25 critical capabilities. And so, now we are getting to this point in the program. We have this list of what we call the 25 critical capabilities. And these are the 25 functional areas in which we have determined that every country must have some level of capability in order to detect, deter, deny, and defeat terrorism and any other words that start with D that you feel appropriate.

Overall, we break them down into these five categories: preventative capabilities, response capabilities, post-incident, cyber security, and then sustainment �'�' sustainment being a group of one. You can’t really break it down any further than, “Are they able to sustain the capabilities we give them?”

So, within each of these five areas �'�' I don’t think we have it listed here, not yet �'�' within each of these five areas there are a certain number that fit.

For instance, preventative capabilities are things like land border security, maritime border security, port of entry security, things like that.

Response capabilities are, for instance, SWAT or bomb squads.

Post-incident capabilities tend to be the things related to investigations, crime scene management, forensics.

Cyber security is pretty straightforward. Everything has to do with computers and the Internet and dealing with terrorism from the cyber aspect.

And then, as I said, sustainment is a group of one, which is my colleague Roberto’s area of expertise.

But it is actually, when you come down to it, the most important of the 25 because if we can’t develop a sustainable capability in the country, we’re just throwing money down the hole. We can’t go back every year and give them the same courses over and over and over again and expect anything to ever be accomplished in that respect. We also eventually would like to get out of that business with that country, so we can move on and use those resources somewhere else.

So, how do we assess these capabilities in the countries to make sure that we’re actually making progress in any one of those particular areas of the 25 that I mentioned? We develop a tool, as most people do, which we call the program measures of effectiveness.

And it’s probably remarkably similar to most other people’s assessment tools. I know it’s very similar to the one that INL uses, the CJSART �'�' I won’t even try to remember what that stands for, but it’s a tool that they use to develop the judicial �'�' their evaluation of the judicial sector and law enforcement. There are some small differences, and we will get to those a little bit later as I go through these slides.

But, essentially, when we go out to do our assessment, we look at everything �'�' at all three levels of analysis: strategic, operational, and tactical, using the military terms. Or, it might make more sense to think of it as the executive, the management, and the operator level.

So, in other words, we’re looking at the whole system globally. We’re looking at the policy stuff, we’re looking at whether the country has the legislation to support what it wants to accomplish. We’re looking at the buy-in from the highest levels of policy makers, as to whether they really support the fact that they’re fighting terrorism, or that they want to actually improve their capabilities to fight terrorism.

We go down to the next step, and we look at the unit chiefs, and we look at the implementers, the action offices, the commissioner of police, the minister of the interior, their special operations units, their special terrorism investigation units, et cetera, et cetera.

And then we go all the way down to the bottom, and we look at the people who are implementing the hard skills out on the street. We’re looking at the SWAT team members, we’re looking at the bomb squad members, we’re looking at the actual investigators who go out and do crime scene management and actually gather evidence and build a case.

And then, sometimes, in cooperation with DOJ, we will even go and look at the prosecutors, and we will look at the court system, and we will look at how they integrate those capabilities with the cops, and are they talking, and are they building cases together?

And you would be surprised �'�' or maybe you wouldn’t �'�' at how many times we find that they don’t ever talk to each other. And every time the cops build a case and turn it over to the prosecutors, the prosecutors send it back because it doesn’t meet the standard. And the police have never been told what the standard is, so how are they ever going to give them something they want?

So, in order to look at all of those things and give us a number that we can use to make an evaluation, we developed a Likert scale rating tool. I don’t know, is it Likert or is it Lickert (phonetic), does anybody know? I have heard it both ways. The gentleman whose name is spelled L-i-k-e-r-t.

So, we developed a scale which is on a scale of one to five. And, in terms of capabilities, one for us means that this capability does not exist in this country, and five means they do it so well they have no need of any instruction or assistance from us. As you might imagine, most of the countries in our portfolios are somewhere between a two and a three, and a lot of them are all the way down at a one. If they have a five, we don’t hang around very long. Four is pretty much what we expect to get them to, or what we hope to ever get them to.

Now, we also look not only at their capabilities �'�' and this is the important part �'�' we also look at the threat related to those capabilities. So, for instance, we have land border security as a capability. We have, in my section, intel analysts who, when we’re getting ready to go look at a country’s land border security capability, we will look at the threats related specifically to that land border security. And they will then rate that on a scale of one to five.

This, then, allows us, in the end, after we have done all of our analysis, to say they have got a capability of two; they have got a threat rating of three. Therefore, there is a gap between these two of, in this case, a negative one �'�' a negative being important, because that’s how we differentiate whether it’s a positive good thing, or whether it’s a bad negative thing that we need to address.

So, that’s the last bullet here. The assessment identifies the gap between capabilities and threat. The reason this is important is not only to show that the gap exists, but it also is important in helping our program managers prioritize assistance.

So, if our program manager has come up with three objectives for his country, and under each one of those objectives we in the AR unit have identified a number of critical capabilities that support that objective, and we provide to him the report that shows him the gap between the capability and the threat in each one of those capabilities, he can then look at those and say, “Okay, yes, they do need a capability in SWAT.

However, their SWAT rating is a three, and their threat rating is a three, so that’s a wash. We may or may not need to spend money on that. But if I look over here at the fact that they’ve got no capability whatsoever to do crime scene management, but they’re getting a lot of crime scenes �'�' i.e. the threat is high �'�' and they’ve got a negative rating of, say, minus two, that’s probably where I need to spend more of my money.” It’s all very intuitive, and it makes great sense, unless you don’t actually have it all written down on paper and you’re trying to just figure it all out from looking.

So, that’s the whole point of having the numerical evaluations, and being able to come up with the threat capability gap rating, and then, therefore, prioritize assistance, not to mention evaluate whether we’re getting anywhere.

So, this assessment that we do, as it says in the first bullet, provides our base line data. This would be the result of our initial capabilities assessment of the country. We go through all of those objectives that I’ve mentioned, all the critical capabilities that support those objectives, and we come up with this base line rating across the board.

As I have mentioned, the program manager will use that rating, those sets of ratings, to help him prioritize his training and create his plan of action for what currently is a five-year plan. We call it a country assistance plan. And then he implements that plan, and hopefully, somewhere in the middle of that five-year period, we will go back and we will look again, and we will call that a program review. It’s basically the same thing as an initial capabilities assessment, except that now we actually have something that we hopefully accomplished that we can now measure.

So, we go back and we look, generally, at the same things. Obviously, you want to look at the same things, but we may add some other things in if policies have changed, or if objective have changed, or if we get further guidance from SCT or someone else. But we will go back and we will look at those areas that we initially measured the first time.

And, using the same format – on a scale of one to five for both the capability and the threat �'�' we will make a determination of where the country is now. Then we will be able to take that information and compare it against the historical information, and see if we have improved. Has that gap gotten smaller? Have we built their capability above the level of threat that they have?

And if anybody has any questions, please don’t hesitate. So, yes, please. I think you have to use the microphone.

QUESTION: The numbers that you mention sound very scientific, you know: two, three, negative one, and so on. But obviously, different people might give a three or a two or �'�' you know, it could �'�' all of these things, how do you handle that?

Because, you know, you could easily imagine a situation where someone who is more parsimonious with numbers says the threat is only a two, whereas another individual would judge it to be a four, and then it would not have a capability gap, whereas you might have �'�' you know what I mean?

MR. STEVENS: Yes, exactly, and I’m glad we have you so hoodwinked that you think it’s actually scientific.

No, that’s a very good point. And we struggled with that for a while, because everyone, obviously, well aware that each individual is going to have a different frame of reference and a different subjective evaluation. We actually had a Ph.D. in our office one time who designed the system for us. So apparently, it’s kosher, as far as all of the scientific aspect goes.

But essentially, the bottom line answer to that is we used subject matter experts who are recognized in their field to go out and conduct these capabilities assessments, and to make these subjective evaluations. I will elaborate on that particular part of it in a second. But because they’re subject matter experts, their subjective evaluation has to list statistical validity.

Now, we also try to help that in the sense that we have a formalized tool. And for each one of these critical capabilities we have what, for want of a better word, is a checklist. It’s really �'�' it’s not a, you know, “Do they have X, Y, and Z, and if they don’t, they fail,” it’s more along the lines of making sure that the subject matter expert has this list at hand to help him ensure that he looks at all of the important aspects, and that he doesn’t leave anything out when he is making his evaluation.

And it formalizes it in the sense of there are some very important things that we have to make sure are there. For instance, is there legislation covering this? Is there �'�' is this unit actually provided for under law? Is this function actually financed, supported, resourced? Do they actually do training? Et cetera, et cetera.

So, while you will get a variance between individual subject matter experts, they’re all operating on the same criteria. And because they’re all talking about the same elements of the profession, they should be pretty close. And we haven’t found that anybody yet that we have used has been so far off on the left or the right that we’ve had to throw it out. Does that answer your question?

And on the threat side, we have intel analysts in our office who do that work for us. But they also vet all of that through the intel community. It’s not completely in-house. And we have �'�' the intel community knows what our gradations are, one to five, and they understand the language that’s used in that. So they know what we’re looking for, and they’re able to sort of verify whether what our intel analyst has given, in terms of a one through five, whether that rating really makes sense.

Okay. So, in terms of measuring progress, as I said, we do these periodic program reviews that allow us to measure the results and the impact. And we measure, as it says right here, on two standards. The first is skills retention and the second is institutional development.

So, in other words, have we increased their ability to accomplish this particular thing that we want them to do, whether it be SWAT tactics, or whether it be crime scene management, or whether it be cyber security, or Internet investigations?

But again, as I said earlier on, the most important thing is, are they institutionalizing? Are they putting it into their programs? And are they training each other? If they’re not doing that, then we either need to help them fix it, or we need to stop spending money there, because it’s money down a dark hole.

And, Roberto, if you have anything you want to add to any of these slides, please.

So, as I mentioned earlier, again, we try to do a program evaluation every two to three years, because our country assistance plans are a five-year program. Therefore, if we do one every two years �'�' at least every two years or three years �'�' the program manager gets at least two sets of data that he can compare and contrast before he writes his next five-year action plan.

Again, it evaluates the program results and the capabilities, both. And it will �'�' it’s tailored for each program review. In other words, it goes back to what I was saying about the objectives. We have to have specific objectives for each country, for each bilateral relationship, for each regional relationship. If we don’t have a specific objective, we can’t know what it is we’re trying to measure. We can’t possibly know where we’re trying to go, and whether we’re actually getting there or not.

And, Roberto, I’m going to let you talk about this next point. But I think �'�' are you getting ready to ask me a question, sir? Yes, please.

QUESTION: Hi, I’m Vincent Scott (phonetic), with the Department of Defense inspector general’s office. My question is you have your base line, you go in for your second or third-year review, there hasn’t been a change, although the base line was given to the program manager, they’ve applied some sort of resources to that to make some type of improvement.

I guess my question, two-part question, that second-year evaluation, who are you reporting on, the program manager or the actual function? And, secondly, are you evaluating why or why not it did or didn’t �'�' the program manager’s actions were or were not effective?

MR. STEVENS: The answer is yes. No, I �'�' we don’t evaluate the program managers from our office, per se. We are �'�' we don’t point the finger, in other words.

The evaluation looks at whether the capability has increased, but more importantly, whether the gap has decreased between the capability and the threat.

So, if the threat has fallen, but the capability hasn’t increased, that’s a positive, in the sense that the gap has gotten smaller. But it may not be a positive in the sense that we may, yes, actually be wasting money because we’re only sustaining them at that level.

So, the system recognizes all of those facets. When we write our report, it will be in there, in the narrative section of the report, to explain all of the data to the audience �'�' the primary audience being the program manager for that particular bilateral relationship.

As to what extent we can determine what it is or why it is that the capability hasn’t increased, I can’t say, you know, we’re always completely effective, because we don’t know what we don’t know, as someone famous said at some point. But we do try to figure out, when we’re there and we have determined, or when our subject matter expert has determined that progress does not appear to be made in this area, they will drill down and they will try to figure out what it is that’s causing that.

And we have another presentation that actually addresses �'�' sort of addresses that point with a particular slide, in that it’s not always the case that the training didn’t take, or that the people who received the training were incapable of understanding it or implementing it or institutionalizing it. It could be that everybody got it, and everybody loves it, and everybody really wants to do it, but there is this guy over here in charge who says, “It came from America; I have no intention of ever trying it.”

So, if we can identify those things �'�' and in the particular case I’m referring to on the other slide, we were able to identify that, and the government of that country made a change and replaced that guy, and things got a whole lot better afterwards. So there are so many different facets involved into whether you’re actually making progress or not, it’s not measurable with a checklist.

Does that answer your question? Okay.

So, Roberto, I am going to let you talk about this next section, since you are the author of that �'�' oh, I’m sorry. Yes, sir.

QUESTION: Just blending into the woodwork there.

You mentioned at the start of your portion that there was a list that you used �'�' and I forget the acronym �'�' but of the countries that you were going to work with, that that had already been created.

The question is, do you ever use this evaluation system to argue for inclusion of another country that hadn’t made the list, or for more funding for a certain country that perhaps the list had given you at a lower grade, as an example? Does that make sense?

MR. STEVENS: That’s really a two-part answer, okay?

The first part, in terms of what countries are �'�' become included on the list, and how they get there, that’s what SCT does. And so SCT figures out who should be on the list, and makes a determination as to what countries should be either now included that weren’t there before, or removed from the list. And Dan should answer the question on that, specifically.

But in terms of funding levels, and whether, yes, this format can be used to help our program managers make the argument that they need more funding for a certain program. So, Dan, you want to answer that?

MR. ROSEN: (Inaudible.)

MR. STEVENS: And, as you could imagine, pretty much everybody out there wants to get on the list, because that means they get free stuff. So I know SCT receives requests all the time from posts, from ambassadors who want to be included in the ATA program.

But, obviously, not everybody makes sense. How much of a counter-terrorism or anti-terrorism issue is there in �'�' yes, Italy, for instance? Although I would love to do the assessment to Italy, but I don’t think it’s happening any time soon.

Okay. So, Roberto, why don’t you come up here and talk about this portion?

MR. FIGUEREDO: Just to step back a little bit, we do, essentially, two kind of activities, the assessments and the evaluations. In short term, the assessment is at the beginning of a program, where we look at the capabilities to try to establish the base line data. You know, where are the weaknesses in each of the areas that we review?

C.B. mentioned about the rating tools, that we have 25 different areas, different capabilities. We, on the assessment, look at the 25 capabilities. But then, when we go to the country, because of its strategic needs, SCT and the embassy themselves, you know, they have identified the broad objectives. What are the objectives for that particular country, in regards to counter-terrorism and anti-terrorism? How this country fits in the regional strategies, and how does it fit in the overall global strategy? In that context is how we look at these things.

And then, once the program gets going, we do biannual or regular evaluations. When we go there to do the program review on evaluation, we look at the activities in country. We also take a look at whatever developments they have had, because anti-terrorism is something that, as you all know, is very dynamic. It changes continuously. So we have to look at those variables.

And then, when we look at the capabilities, we look, for example, at the unit. Let’s say in a bomb unit, we look at the skills of the people. Have they been able �'�' are they able to recognize the object? Are they able to disarm the object? Then we look at the post-blast. Were they able to gather the evidence and to report these things and to manage the things for the prosecution?

Then we look at the institutional capability. Have they been able to develop a unit? Does the unit have the resources? Do they have the personnel? Does the personnel stay complete for a specific period of time?

So, we’re looking always on �'�' in that two track, the �'�' one, the skills rotation, which is the technical, and the other one is the institutional.

Then, going back in here to this slide, for the purpose of reporting we classify the countries into five different categories: (inaudible), those countries that are at a very basic level and don’t have much capability, either in the technical or the institutional area; basic, those countries that have begun to develop some capacities, are they able to identify a threat, and if so, are they able to analyze the threat?

In answering those two basic questions, then we move to the intermediate countries. These are the countries that have the capability to identify some of the threats, to do a national plan, and to start getting their units to respond to these things.

An advanced country is the next level. Again, they are more capable of self-sustaining in that regard. They are able to identify the threat, to prepare their troops, their units. They have their own national plan and the national strategy. And the sustained countries are more or less the more advanced countries, like Turkey, Western Europe, et cetera.

So, in general terms, those are the categories that we use. We grouped these countries in here for the purpose of reporting for the State Department, because that’s essentially the base line data that we use. So we tried to group the stuff and (inaudible) facilitate for the reporting.

As I mentioned to you, we have already a skills retention and the institutional development. When we go in there �'�' let’s say a cyber unit �'�' when we provide assistance to a cyber unit, the first thing that the SME (phonetic) will do is look at the capabilities. If they have enough capability to analyze a cell phone, for example, then do they have the capability to develop the evidence? On those bases, then we provide the technical assistance and the training, and we provide some equipment.

When we come back on the program review, we look at these things. And in those ways we provide the rating that we have been talking about. We do that with every single unit that we work with in the country. And normally, in every country, at the most we have three or four different units �'�' CRT (phonetic), investigative, cyber, force security, border security.

The assessment, as we explained, you know, provides the base line data needed to design and to monitor the promise of assistance that we provide.

We have a (inaudible) that is for the strategic plan that we prepare for each country, and those places �'�' that’s how we provide the funding for the individual country, and that’s what we use when we go out in the field to do the assessments and the evaluation.

I think that that is �'�' you have any questions or anything? Essentially, that’s how we approach the process. Yes, sir?

QUESTION: Excuse me, I didn’t mean to interrupt. My name is Rob Cooper (phonetic). I’m also with the DoD inspector general. A quick question, and I don’t mean to be crossing my lane, but in the spirit of the conference, with the turnover from DoD to (inaudible) Iraq, have you done a base line for Iraq yet?

MR. FIGUEREDO: No, no. I (inaudible) planning an assessment for the end of this calendar year.

QUESTION: Okay. And my next question is, in what you all are talking about, can you give us a current example of a current country you’ve done this with?

MR. FIGUEREDO: What do you mean, an example?

QUESTION: Well, you’re talking about the program evaluation, and you’ve just been talking numbers and you’ve been talking the concepts.


QUESTION: Can you give us a concrete example?

MR. FIGUEREDO: Sure, sure.

QUESTION: Thank you.

MR. FIGUEREDO: The most recent in which I participated was the assessment of Georgia. Georgia is a strategic partner, and is one of the countries that has the largest program �'�' in fact, it’s the largest bilateral program that we have in our portfolio.

And there, what we did, we looked �'�' at the beginning, four years ago we have an interagency working group, and we got people from DoD, from the Navy, from CIA, and almost every U.S. Government Agency in the area of expertise. They went, they developed, they did the analysis, and they identified the weaknesses and the potential for activity.

Then we start working with them, and we agree, on the basis of the assessment. We determined that the best way to help them and (inaudible) all the assistance we have provided by the U.S. to develop a command and control center for the police. The reason for doing this is because DoD was working with the military, and the CIA was working with the intelligence agencies. But then we found this big weakness on the law enforcement side.

And given the strategic importance of Georgia, and their position within the region, that was key, that the police start developing these anti-terrorism capabilities. So we figured out that the best way to approach these was just in this command center.

The command center includes our 911 call center, and the strategic ability for the police to analyze and determine this �'�' trends, to analyze the trends, to determine the intelligence. So, in that sense, that’s how the program has evolved in there.

A different example would be Turkey. In Turkey, we identify cyber as one of the weaknesses. We have improvised in the training with them, but because they’re a more advanced country and they have better infrastructure, our assistance has been limited mostly just to technical assistance and the training. We are not focusing so much on institutional development.

In Afghanistan, a different thing. We are trying to just (inaudible) developing the capacity to protect the president. So that’s essentially how the program works, and how it gets evaluated.

Any other questions or comments?

QUESTION: (Inaudible) foreign office in the United Kingdom. One of the issues that we really struggle with is trying to identify �'�' because there is so much other noise going on out there �'�' what is the impact that you’ve had, and the impact that everybody else is having?

When you’re looking at the capability, the capability may have changed from a two to a three. How do you know that that’s down to you, and not other things that are going on in the country? It may be that your programs are a lot bigger than ours, so it may be easier to do, but that’s the question for you.

MR. FIGUEREDO: It’s kind of work in process, you know? On �'�' that’s one of the great difficulties that we have in our field, because we base our activities on the U.S. Government strategic objectives. So we decide, for example, let’s say border security. We look at border security with these particular objectives. Often times, other countries look at border security, but their definition of border security is somewhat different.

And what we try to do is to adhere (inaudible) �'�' what we do is adhere to our framework. And in those places we continue to measure consistently. And then, when possible, you know, we coordinate with the other countries and (inaudible) that the other countries are providing.

This is a major challenge for us, you know, with regional initiatives, because with regional initiatives �'�' let’s say border security, to stay with one topic �'�' you develop the capacity in Country A, Country B, and Country C. But how do you measure those abilities? How do you measure those things? And cause and effect is something that is very, very difficult to measure in this context. We find that very difficult, to �'�' that relationship.

MR. STEVENS: That’s a very good question, and it’s something that we �'�' I don’t say we struggle with it, because in the greater scheme of things, really it doesn’t matter who had the impact, as long as the capability increases. But in terms of is our money being spent properly, is our money making an impact, then yes, we do kind of want to know that.

And so, when we go into these countries to do our program reviews, one of the questions we ask is, “Who else are you receiving training from,” and, “What was it,” and, “Have you put it to use?” And we also often will sit down �'�' hope to sit down with graduates of our programs and get some anecdotal evidence as to whether the training that they received was put to use, and did they actually accomplish anything by using it.

Now, that doesn’t really give us a statistical evaluation, and you can’t say that they made it from 1.5 to 2 because of that class. But it at least gives us, again, some anecdotal justification, that what we have done has contributed. Does that help at all? There is no magic answer to that question, unfortunately.

QUESTION: (Inaudible.)

MR. STEVENS: Sure, sure, of course. And I also wanted to address the previous question, as well, in terms of what have we done recently, and a concrete example.

We do between 25 and 40 of these a year. So, that’s why you kind of threw us for a loop when you asked for a concrete example, because we’ve got 25 reports going through our heads at the moment.

But it’s not �'�' again, it’s not always a positive result, either. Roberto was talking about some of the successes we’ve had. But we also have, for instance, some not-so-successful programs, or some sort of sideways successful programs �'�' if one might characterize it that way.

For instance, the Palestinian authority is one of our partner nations, or one of our partners. And we help them to develop their presidential security guard. And we like to think we did a damn good job of it. Those guys got to be really professional. They went from a group of thugs who basically had everything that they owned taken away from them by the Israelis and all their cars lined up in the parking lot rolled over with a tank, and they had a couple of chairs and a desk, and that was it.

And we went in there, in consultation with the Israelis, and did things that the Israelis were okay with to help develop this group to be able to protect Abbas. And we got them to a level that was pretty darn good, and Abbas was so happy with them that he started piling everything on them, I mean, right down to, you know, putting out fires �'�' I mean literally putting out fires with fire extinguishers and trucks, and all that stuff.

So, when we went back to do our program review, we said, “Hey, you know, this is not what we’re spending our money on. We’re not trying to create one agency within the Palestinian Authority that does everything because Abbas trusts these guys.”

So, in that case, our report spelled out that we shouldn’t continue to put money into this, unless it’s very specifically for protecting Abbas, and for helping these guys develop that capability and maintain that capability to protect Abbas. All the rest of that stuff needs to go �'�' be farmed about to other agencies. “And you, Palestinian Authority, need to put people in charge you trust, if that’s what it takes, but you need to develop those other agencies, too. You can’t have everything fall under this one organization.” So, I don’t �'�' maybe that sort of addressed your question in a little more detail. I don’t know if it was helpful or not, but it’s a story I like to tell, so –

QUESTION: (Inaudible?)

MR. STEVENS: Time lines in terms of what?

QUESTION: (Inaudible.)

MR. STEVENS: Well, no, because every one of our countries is so completely different. I mean, we have a range from Chad to Turkey and the Philippines and Indonesia, you know. To be able to say, “Yes, we expect to accomplish this within one year” is not really likely.

But in terms of time lines, our program manager tries to write a plan that he expects to be able to accomplish within five years. So for some countries, it’s going to be a lot, and for some countries, it’s going to be �'�' maybe we will be able to get them to wear their uniforms by the end of the five-year period.

Are we �'�' oh, okay. Yes, sir.

QUESTION: Hi, Xavier Valverde (phonetic) from Secretary of the Navy, office of program appraisal. I would think that one of the goals of the ATA program is ultimately to improve our security here, at home.

My question is, is anyone �'�' it’s a two-part question �'�' is anyone asking whether we are making a difference with this ATA program, and improving our security at home? And, if so, how is that measured?

MR. STEVENS: Wow, that’s a good question. I think yes. I mean, our director goes up to Capitol Hill all the time to talk to �'�' to be asked questions about what it is we’re accomplishing. But I think, Dan, you can probably answer that question better than I can, because it’s more of a policy issue, as to whether we’re achieving the U.S. policy goals or not.

MR. ROSEN: Yes. I mean (inaudible).

MR. STEVENS: And I think that sort of rolls back to, really, the only sort of bottom line answer to that is that as long as we’re advancing on those objectives that are in the national strategic guidance documents, then we must be accomplishing what it is our government wants us to do. So I guess that’s the only measure of success we can have, in that regard.

Dan, did you want to make some �'�' okay. We actually had two more slides here that we didn’t get to, and I think we’re out of time. Okay.