Workshop on A Social Science Approach to Strategic Influence Evaluation: Good Evaluation Starts at the Beginning

U.S. Department of State Third Annual Conference on Program Evaluation - Anti-Terrorism Track
Washington, DC
June 8, 2010

MR. COUER: Good morning, everyone, and welcome to the evaluation conference. My name is Jim Couer (phonetic), I’m an analyst in the office of strategic and performance planning. And today it’s my pleasure to introduce a distinguished guest speaker, LTC Christopher Rate.

LTC Rate is the chief of the behavioral sciences branch at the joint military information support command. He is directly responsible for the conduct, coordination, execution, and integration of behavioral sciences into strategic psychological operations. His branch leverages best practices in research and analysis, survey development and polling, and campaign assessment.

LTC Rate holds a Ph.D. in social cognitive psychology from Yale University. Please join me in welcoming LTC Christopher Rate.


Get Adobe Reader View slide presentation ]

LTC RATE: Good morning. As Murphy’s Law would have it, we will probably take an operational pause somewhere here in the presentation, so I can actually have a couple of slides that are presently missing from my briefing. But as we get started, I will do a little dog and pony show, might maybe sing a couple of tunes, and see how you guys can evaluate my performance, and then potentially my effectiveness in persuading you never to listen to me sing again.

Who am I? As the introduction was, I am Chris Rate. I am from Tampa, Florida, part of special operations command. We are in the joint military information support command, where we basically are involved in synchronizing, planning, and executing, when necessary, strategic psychological operations.

In the JMISC, as we call it, we have PSYOP planners �'�' psychological operations planners �'�' intelligence analysts, cultural advisors, and behavioral scientists, predominantly psychologists. We have a Ph.D. social worker. And so we use all of these particular elements to fuse a certain multi-disciplinary approach to psychological operations in order to produce a better production in the end, in the hope that we can actually influence our target audiences.

Now, today, for the experienced planner and the evaluator, I’m probably not going to tell you a whole lot new. In fact, many of the questions that were brought up this morning in Senator Hagel’s brief touch on some of the things that I really want to talk about today.

Now we are going to take this operational pause.

Anyone know any good show tunes?

(Interruption to proceedings.)

LTC RATE: Of course now I’m going to have to refer back to this so I don’t know �'�' I don’t have it here on my screen, but we’ll deal with that.

All right. So, what was I saying? Let me see. For the experienced planner and the evaluator in the audience, you know, again, I’m probably not going to tell you anything that you don’t already know. But hopefully, for those people who may be new to the evaluation field or to planning, you will get a little bit of a reinforcement of things you thought you might know, or you might have known, and also maybe a little education along the lines in this idea that we can actually use perspectives such as the social science approach to strategic influence.

Now, I’m grateful for this opportunity to address an issue that is, in my opinion, often overlooked, or simply paid lip service, especially during the planning phases of strategic influence. Evaluations many times have been described or deemed too difficult, challenging, and in some instances, viewed as an obstacle to progress.

I think sometimes this is due to a lack or a fundamental misunderstanding or basic lack of knowledge regarding what evaluations can do for you in the context of campaign planning, and what they are meant to accomplish within the planning process.

So, let’s discuss �'�' put this discussion a little bit into context. We’re here to discuss effective evaluation strategy in the area of counter-radicalization and counter-terrorism. So let me suggest this. There is no short range or single-pass solution to the problem of radicalization of individuals to violent extremism. Once individuals gain membership in an extremist organization, it is difficult to influence them.

So, in the long run, the most effective counter-radicalization strategy is one that inhibits potential recruits from joining in the first place.

So, given that the agenda is set, we’re going to talk about terrorism or anti-terrorism, counter-radicalization, counter-violent extremism, whatever label you want to use for it, this is what we’re going to sort of cover today. And I say “sort of,” because I will deviate here and there. But hopefully it will cover all the main points, and get all of this done in about 30 minutes.

The nice thing about being first in the day is I get to frame the issues, and I get to tell you what I think, and then you get to judge everything else based on what I’ve said. I get to set the base line for your metrics today.

We’re going to talk a little bit about psychological operations process, the strategic communications model, why we would even talk about the social and behavioral sciences, and what they actually can bring to campaign planning. We will talk about one of my favorite theories of �'�' it’s called moral disengagement, the mechanism of moral disengagement, and then how that can be applied to strategic influence campaigns.

First here is a psychological operations process. Now, Senator Hagel mentioned this morning that we need to anchor things more than just a process. But if you see the seven-phase process, what is depicted here is the steps that a psychological operations planner would go through, and you would see that, embedded in this process, there are numerous places for people to conduct evaluations and assessments along the way.

By doctrine, Phase 1 in the psychological operations process gets the evaluators and the planners to think about measures of effectiveness. That’s what we need to make sure happens, is the evaluators and planners, together, are looking at ways to measure the effectiveness, as stated in the objectives of the campaign �'�' or stated by the objectives of the campaign �'�' of whether or not, at the end, we can show any change in attitudes or behaviors.

If it’s not done up front, then the chances are that we’re not going to have a good evaluation process. I mean, how many evaluators in the room have been asked, after a campaign is already in play, to go ahead and evaluate it. How difficult is that? Typically, it’s extremely difficult. If the evaluator isn’t there at the front to help formulate the objectives, how is the evaluator supposed to measure those objectives if the objectives are worded in such a way that they’re unmeasurable? So, you’ve got to work hand in hand with the planners.

Now, within this process, as I’ve said, there is different types of evaluations that are embedded. Now, we talk about �'�' I’m going to talk about three different types of evaluations. The first one is formative. A formative evaluation really assesses the strengths and weaknesses of campaign materials before and during the campaign. We would pretty much conduct these with interviews, focus groups. We would have some survey mechanisms in which we would actually try to obtain this particular data.

Also along the way �'�' and that typically happens at the front end of any campaign process �'�' we will also talk about process evaluations. Now, process evaluations talk about the measurement of effort, or the direct output of the campaign, the reach of the campaign, how your distribution is, the exposure, the media time that you may have purchased. Those are all �'�' that’s all data that gets at the process of a particular campaign.

And then, finally, we have the outcome measures. And these are the important measures that I think is typically overlooked, and that is the changes that result from the campaign, itself, the changes in attitudes, the changes in behaviors, the things that show that we are making progress towards the campaign objectives.

Now, in PSYOP, we tend to lump the first two in what we call measures of performance. And the latter we call the measures of effectiveness. Now, each is important, and has its own role in the campaign process. But there is no amount of the former two, no amount of measures of performance that’s going to give you a measure of effectiveness.

So, you can have things all day that people like your campaign, they like your product, you can have the best reach and the best distribution in the world. But if you don’t have a measure that is looking at, “Is there a change in the attitude and behavior as it applies to your objectives in the campaign,” then we don’t have anything that tells us whether or not our campaigns are successful.

Now, the other model that I really like �'�' this is actually one of my favorite models is �'�' this is from the state INR and BBG deep dives. Very simplistic. But the process model describes fairly simply, but you know, at least inherently within it, you know, the different types of evaluations and what should be accomplished.

If you look at the first, at the top, you have the mission. In the mission they talk about the objectives and the desired outcomes. When we move over to the market side, this is where we’re going to start looking at assessments and evaluations that help us determine the media environment, the target audience analysis, what �'�' the strategic context of our campaign, and how we’re going to work within that strategic context.

Then we have message development. We’re going to formulate, test, and refine. And message delivery will implement, assess, and revise. And then, under the metrics, we are going to assess, evaluate, and report.

What I really like about this particular model is that �'�' its explicit nature in the feedback mechanisms. Now, this one actually has �'�' it says that we are going to take those evaluations and feed it back into our campaign process, so that we can modify things as we go, make changes or sustain the status quo, whichever is necessary.

But there is an inherent limitation to process models, and this is, as Senator Hagel was mentioning, we have to have something a little bit deeper. And this is where we bring in the social and behavioral scientists �'�' sciences. Process models, by themselves, lack an underlying theoretical framework for why we are doing certain things. They can tell you how to do things.

I had spent a lot of time in the nuclear ICBM world, and we are very checklist-oriented. It’s read a step, do a step, eat a banana. Anyone can pretty much do that. And that’s what some of these process models have become. You read a step, you do a step. Banana is optional.

This isn’t a poke at planners. Most of the time their training emphasizes the process. Many planners �'�' at least with whom I am accustomed to working with �'�' have limited background beyond the training that the military gives them in the process model.

So, what we want to do is say it’s time for people to start thinking a little bit deeper, integrate multiple perspectives from the social sciences, in order for us to build a more robust campaign plan. And that’s what the social sciences does, it tells us how �'�' what they’re supposed to do, how it’s supposed to work. It’s the why, it’s the framework.

In order to influence attitudes and behaviors, it’s probably a good idea to know why those particular behaviors and attitudes occur.

So, these are a few of the social science theories that are out there. Some of these you may be very familiar with; some maybe seem a little bit more obscure. But each of these can help inform and influence campaign. They can be used separately, or they can be used in conjunction with each other. I’m not saying that you have to just use one, have to use another. Multiple can be at play at the same time.

Ones that you see a lot of times in public health campaigns, theory of reason to action. Martin Fishbein, who was one of the originators, a psychologist who was one of the originators, recently passed away. He and Ajzen formulated this theory that takes into consideration attitudes, behaviors, behavioral intentions as antecedents �'�' those things that come before �'�' changes and behavior.

So that’s my plug for the social sciences. I would recommend anyone actually go and take a look at some of these theories, get yourself a social scientist on staff if you don’t have one, and see what they can do to help out in your campaign planning.

So, let’s get back to the original context that we’re talking about. We’re talking about counter-radicalization, counter-terrorism. As I said in the beginning, one approach to counter-violent extremism is to inhibit potential recruits from joining in the first place. Therefore, if that is our goal, it is desirable to circumvent the process by which potential extremists convince themselves that violence is acceptable.

So, this is where we bring in moral disengagement. Albert Bandura, probably one of the preeminent psychologists of our time, a professor emeritus at Stanford University, defined moral disengagement as a cognitive process by which people excuse harming others. It provides one means to better understand how a campaign may be developed and assessed using sound psycho social constructs.

But the limitation of any social science theory is the bottom bullet, which is no single theory or framework, on its own, is sufficient to account for all violent extremist activities.

So, a little bit more of what is moral disengagement? Well, moral disengagement is no more and no less than disengagement from inhumane behavior. And it can take the form of eight mechanisms that focus on the behavior itself, the responsible agent of the person �'�' the person who actually is taking the action, the consequences of those effects, and the victim, themselves.

It was interesting listening again to Senator Hagel this morning talk about the Gulf spill, and how that fits in very nicely to one of these mechanisms. The mechanism in play, in this case, was diffusion of responsibility. Diffusion of responsibility is basically obscuring or minimizing the causal role played in the outcome of violent acts.

But in this case, not necessarily a violent act, but still a moral disengagement in and of itself, in that everybody �'�' BP, Halliburton, other entities involved �'�' all pointing the fingers at everyone else, not taking responsibility. It’s a way of, basically, a lack of accountability. And so, we can see this in play every day on the headlines and in the news.

Now, some of the other mechanisms such as moral justification, we will see this often times portrayed by AQ (phonetic), by other violent extremist organizations, that basically reinterpret otherwise immoral behavior in terms of a higher purpose.

Euphemistic labeling, using sanitized terms, so violent acts are deemed as mild or benign. You will hear terrorists refer to �'�' what we refer to as terrorists, you will hear them refer to themselves as freedom-fighters, a way of sanitizing it, sanitizing language that makes it seem not as bad.

We will also do this in the military. We will talk about, you know, genocide being “ethnic cleansing,” sanitizing the word, annihilating the enemy may be sanitized in another way. We do this in order for us to �'�' to help us better follow through on some of these actions that tend to be a little bit more violent, and maybe somewhat disconnected from what we might think is moral behavior in the first place.

One of my favorites ones �'�' just because my kids always use this �'�' is palliative or advantageous comparison. This one you will see all the time when someone says, “Well, you think what I did was bad, you should take a look at the guy down the street, or the guy next to me. That was bad.” So it’s a way of justifying your own behavior, and a way of comparing your behavior to something that is much more extreme.

Displacement of responsibility is placing the responsibility of harm one causes on other groups, where a soldier can say, “I am just doing something because I am just following orders.” So you can basically attribute the blame to someone else.

I talked about diffusion of responsibility. Attribution of blame, blaming the victim. You know, “They brought it upon themselves. I wouldn’t have done it, but they made me do it,” you know, “so it’s their fault.” You will see that all the time.

And misrepresenting harm. Now, this one is a little bit more like, “Well, you know, it wasn’t so bad,” you know, we kind of distort the consequences. Tends to be when people are further away from an actual act they will also be able to distort the consequences a little bit. “It wasn’t as bad as what we actually thought it was.” You are minimizing the harm caused by your particular action.

And then, dehumanization. This is one you will see a lot in propaganda. This is basically removing human qualities of people groups, and replacing them with evil or demonic qualities. Someone who is less than human, that’s kind of what we’re looking at here, with the moral disengagement and the dehumanization mechanism.

Oh, and I forgot to mention �'�' I’m going to just take another quick pause here while I’m going to catch my breath here for a second. If anyone has any questions, I would prefer this to be a little bit more interactive.

Now, in psychological operations, I am told that I have to use the truth. So my answers may be sometimes more on the MILDEC side, where I’m going to make stuff up. But if you catch me, that is fine, too. So �'�' but if anyone has a question as we go along, please just raise your hand, pop up. Sort of �'�' there we go. Or yell it out, and I will repeat your question.

QUESTION: (Inaudible.)

LTC RATE: Oh, now they tell me it’s being recorded.

QUESTION: I want to go back to something you said on the previous slide. And you don’t need to back up the slides.

I found it interesting that in the civilian world we’re starting to pick up on military terminology for things we want to do. We want to “combat” violent extremism, for example. But in the military, we’re going the other direction. Killing civilians is “collateral damage.”

What is the process that’s going on here? Why are we doing these things, when in �'�' because how you talk about things does affect what you do.

LTC RATE: Right. And that is the �'�' you know, that’s an interesting point. You would almost find that a little bit sort of backward on the civilian side, that if you want to do that mental gymnastics that is taken to sort of allow yourself to either support, defend, or commit acts of violence, that we would actually sort of go away from some of those terminologies of combating certain things.

I think the military does that in the sense that even though we justify pretty much every day what we do as almost a moral justification, there is a higher purpose that we’re serving. And it’s not necessarily always that it’s going to be necessarily a negative thing �'�' at least from our perspective, it’s not.

It does seem odd to me that we would be actually using that, the civilian world would actually be using those �'�' that kind of more or less sanitized, you know, language. I am not sure that it necessarily helps. I think it helps those people who are already ingrained. But that’s another process that’s kind of involved, which is sort of the confirmation bias. And you know, another way of looking at ways that, you know, creating cognitive dissidence, and letting people sort of kind of resolve that cognitive dissidence.

You know, so there is many things that are at play. I don’t think �'�' I’ve sort of talked around your question. And at some point you can kind of nod and say, “Yeah, you sort of got where I’m coming from,” and that way I can sort of pretend like I answered it, and then we can move on.


LTC RATE: Good, thank you.

QUESTION: And which you may be addressing, but when I’m looking at the previous slide, the chart, it seems to me that, for example, it’s very applicable, if you look at sort of what Nazis did, in terms of dehumanization, all of these things.

The question I have is �'�' in a way it’s a chicken and egg thing �'�' is there part of the theory that talks about how you get to that point? Because, obviously, not everyone gets to the point where they’re in the business of dehumanizing.

And the second part is �'�' and then I’m sure there is a feedback mechanism, where the more you dehumanize, the more it encourages these other things. What I am wondering is, is there a preconditional stage to this theory of moral disengagement?

LTC RATE: That’s an excellent question. The �'�' what tends to happen with this is whether it’s �'�' again, I think I will probably address some of this �'�' is the mechanism �'�' and again, the cognitive mental gymnastics that people go through can develop over time. And it’s a way of rationalizing behavior.

Many times a behavior will happen and you will rationalize it. Other times it’s �'�' there is a self-directed mechanism, where if something happens that �'�' if I want to believe it again �'�' and we were talking about confirmation bias. If I believe something and I want to exclude all the other information, I may start to then justify in my mind why a particular action is appropriate, or is not immoral.

There also is those folks that are taking advantage of people’s ability to be influenced through the mechanisms. And that is, as you referred to, like Nazi propaganda, AQ propaganda. And I actually have some slides that will address some of that. And that’s externally driven. And that’s getting people to start, you know, sort of playing that �'�' starting the cognitive processes of really thinking that things are potentially immoral or not immoral, that they’re appropriate behaviors.

One of the things about the moral disengagement, and in many of these theories, is the path of causality, or is �'�' is not necessarily clear. We know that there are things that are correlated, but we don’t know necessarily what did come first. Research has shown that moral disengagement is directly linked to violent behavior. But I’m not sure people have done a whole lot of research, necessarily, of precursors to moral disengagement, other than using things such as the Nazi propaganda, and exposing people to them, and showing that being exposed to these things can have people sort of move their behavior, their attitudes, a certain particular way.

And, on the flip side, they have shown that making people aware of these mechanisms have moved people away from the moral disengagement. So, hopefully that gets to some of what you’re asking.

So, let’s assume �'�' back to the moral disengagement �'�' for the next few slides, let’s assume that we are going to create or develop a multi-media campaign, you know, something that’s print, radio, TV, Internet, SMS text, whatever might be appropriate for whatever target audience we’re looking at.

So, let’s assume that we have that, and that we’re going to somehow influence �'�' in this campaign, influence the radicalization process, feeding terrorist activities �'�' that’s where we want to hit them before they become radicalized.

So, how can the moral �'�' the mechanisms of moral disengagement be applied to strategic influence campaigns? I’m going to talk about three briefly, and this is from some empirical work that we have actually �'�' that we actually did at the JMISC. I’m going to talk a little bit about target audience analysis and segmentation, prescription for content development, and measure of effectiveness: three things that can �'�' that we can use this theory for.

So, briefly, when we talk about target audience segmentation, we realize that those people who are evaluators, that when we want to determine what target audience we want to talk to, there is many ways that we can segment an audience. I mean, pick a variable. You can segment an audience on a particular variable. Pick a construct. Pick any kind of dimension. Whatever you want, you can segment the audience on. But is it appropriate? Not necessarily.

So, when the evaluators are working with the planners up front, as we are trying to define who our target audience is, then we need to make sure that we are using appropriate variables to segment this audience. In this case, we are using the moral disengagement. What we use is a two-step cluster analysis. Many ways that you can go and do segmenting. Latent squares analysis, I think, is another way. Many other techniques are out there.

But what we try to do is find relatively homogenous variables within the segments, make sure that they’re distinct from each other. And when we come up with our cluster solution, we try to find an optimal number of clusters, based on the distinctiveness of those particular clusters, and our ability to target those particular audiences.

In this case, we were able to identify two segments that were basically �'�' that we were able to discern and describe in a meaningful way. So that was the high and low moral disengagement. The blue in this case is the high moral disengagers, and the red is the low moral disengagers.

Now, our analysis revealed that the segment’s endorsement of violence was positively associated with scores on the measures of moral disengagement. We also demonstrated that there was a direct linkage between moral disengagement and opinions about the use of violence against civilians to defend Islam.

Now, let me pause one second and kind of orient you �'�' if you haven’t already seen it, up to this top, top left, my left, is that we conducted this on a multi-stage random sample of pan-Arab men, ages roughly 18 to 29 years. We had a sample size of a little over 2,000 from 9 countries. They were all self-identified as Muslim. So, some of the things I’m telling you about now is the analysis of what we found for the distinction between the two clusters.

We also found that those people that had higher moral disengagement defended the use of violence against non-combatants. Those with lower moral disengagement had increased rates of altruistic or selfless behavior. Those who also had lower moral disengagement had increased rates of tolerance of different ethnic, religious, political, and ethnic groups. Those with a higher level of moral disengagement also had increased rates of stressful life events during the last year.

And what was also fairly interesting, even though our sample size had an age of roughly 15 to 30, the younger participants reported higher levels of moral disengagement, which is very important, considering the younger age, even of this particular sample, is who is being targeted by violent extremist organizations.

So, our target audience segmentation, what this does is allow us to have greater specificity for targeting. Revelation of vulnerabilities, accessibility and susceptibility. Those are terms that we use in psychological operations. Can we reach them with the media? And even if we can reach them, are they susceptible to any influence strategy that we have, or any influence techniques that we want to employ, that we could actually see a behavior change?

Our analysis through this exercise is supported by leading scholars, whose research also indicates that those with higher moral disengagement are less pro-social, feel less guilt over inhumane conduct, and are more prone to vengeful thoughts and violent behavior.

Now, the other nice thing about having a segmentation and having the results that we have had so far, is that we now have a base line for measures of effectiveness, when we get to that point in our campaign process.

So, now that we actually know who our target is, we want to know whether or not we actually can create something, a stimulus, that will be used to influence their behavior.

So, has moral mechanism �'�' moral disengagement been used in the past? Absolutely. As I talked about, this can occur internally or externally -- Externally, the themes from propaganda that you have seen -- and it’s a key element in the radicalization of violent behavior, and of violent behavior of extremists.

The mechanisms have been applied to international terrorist organizations, violent groups as diverse as the IRA, Hezbollah, and Palestinian suicide bombers. It could easily be found in the content and appeal of extremist media, such as AQ propaganda on the Internet.

An except taken from Albert Bandura indicates language taken from AQ propaganda that shows, by using the mechanisms of moral disengagement in extremist propaganda, radical groups are able to clear themselves from blame, and reinforce continued acts of inhumane behavior.

Just by reading this example you can see many of the mechanisms in play: “religious duty,” moral justification; “lowly people,” dehumanization; “ravenous animals,” again, dehumanization; “the winds of faith have come,” moral justification; et cetera, et cetera, martyrdom, euphemistic labeling.

These are just some of the posters that you may have seen if you went on the Internet and looked at propaganda from World War II to present day. And just by looking at these particular examples, you probably will be able to see many of the mechanisms in play. “Americans will always fight for liberty,” moral justification. “Destroy this mad brute,” euphemistic labeling and dehumanization.

Unfortunately, I can’t decipher what the other one says, but you can see that one �'�' pretty much advantageous comparison might be in play on the one with the little kid.

And then, a flyer from Israel, and then another handbill that was created, I’m pretty sure, by the USG (phonetic), all showing things such as advantageous comparison, potentially blaming the victim, that they brought a lot of this upon themselves.

So, yes, it’s been used in the past. Can it be countered, can it be corrected? Yes, it can. Research indicates the knowledge of moral disengagement can help inoculate from that disengagement: peer modeling, education, emphasizing the humanity of potential victims to counter-extremist messages that vilify and demonize certain groups.

But now that we are countering the extremist propaganda in our campaign, I would at this point probably conduct some more formulative and process evaluations to see if there is anything that I need to do to improve the stimulus, improve my dissemination. All important, but what I really want to know at this point is, am I having an impact or an effect?

Assessing the cognitive processes of our target audience via surveys, interviews, is possible where direct observation of behavior is not feasible. I think this is, often times, one of the things that holds evaluations and campaign plans and planners back, is the over-emphasis on behavior. Yes, ultimately, we all want to see a behavior change, in strategic influence, especially. But many times we can never see it. If you can’t count it, you can’t change it.

People might want to argue about that. But think of this in strategic influence and persuasion. If you don’t know if the person that you’re targeting has changed their behavior or attitudes, then how do you really know that you’re persuading anybody? Again, if you can’t count it, you can’t change it.

As research has indicated, moral disengagement can be measured, which is very nice. It can help predict people’s willingness to commit or support violence. In assessing the effectiveness of a campaign, two preconditions must be met.

And this is what I would argue, that these two preconditions are necessary in assessing the effectiveness of a campaign. Do I have clearly stated, observable, and measurable objectives for the campaign? And, two, do I have reasonable assurance that my intended audience that I am evaluating is receiving the stimulus? If I can assure that those two things have been met, then I have a better chance of coming up with some measures of effectiveness.

You know, this is a measure over four different collection periods. This is actually representative of about 8,000 Muslim youths in the pan-Arab region. Just identified them on the X axis as time one, time two, time three, and time four for execution. And then, on the Y axis, is the mean score.

Now, you can see here that I have exaggerated the differences, but I did this for a purpose. If you were to take a look at this particular graph, and see that the blue is recall, the red is no recall of our particular stimulus that’s out there �'�' so let’s assume, let’s say, it’s a radio advertisement. Has our target audience �'�' are they aware of it, or are they not aware of it?

If you were to look at this, and you were to see that the higher �'�' this is actually from low moral disengagement to high moral disengagement �'�' those people with no recall have a trend line over time that’s going down. That’s in the right direction. Typically, you will hear arrow going up. That’s the right direction. In this case, no. Arrow going down, good. Arrow going up, not so good.

Well, those people who recall our particular campaign are actually going the wrong direction. So, what do you do with this as an evaluator? This goes back to one of the questions that was asked this morning, which I thought was a brilliant question, is what do you do when you have bad news that you have to tell the planners and your leadership?

So, you say, “Shoot, what do I do with this? Looks like the campaign has failed.” No, it hasn’t failed. What it means is that we have to take a position that we have to use that feedback mechanism to input this information back into the campaign process in order for us to make adjustments.

People tend to take this stuff as failure. It’s an opportunity to modify and change. That’s the important thing to take away from this. And the evaluators have to be willing to do that.

So, I know I’m coming up on my time, so let me just conclude with this particular slide. As there are limitations to this study, there is a lot of noise that’s out there. Any strategic influence campaign you run, there is a lot of noise, whether it’s partner nation, whether it’s enemy propaganda that could confound what you’re trying to collect.

Again, there is not sufficient �'�' again, the moral disengagement and other theories are not sufficient to explain all the nuances. And again, one of the caveats on this is the sample was representative of the general population, not of extremists.

But there are benefits to the theory. These are things that we can do, things we’ve talked about today.

So, remember, as evaluators, good evaluation has to start at the beginning. The social and behavioral sciences help campaigns why, the underlying framework of why people behave the way they do, and how we might employ the most effective strategies to influence them.

Working together, evaluators and planners can optimize our chances for success, and our attempts to execute and evaluate influence campaigns that meet our objectives of eradicating violent extremism. Thank you very much.