Workshop on The President's Emergency Plan for AIDS Relief: Implementation Science Program

U.S. Department of State Third Annual Conference on Program Evaluation - Health Track
Washington, DC
June 9, 2010

MS. GORDON: So good morning. I'm Alethea Gordon. I just wanted to introduce myself and welcome you to this session. This is the Health track, and I appreciate your interest in these issues. And I want to just welcome you and introduce our speaker.

We're delighted to have Dr. Dianna Edgil with us. She is a strategic information officer in the Office of the Global AIDS Coordinator. Dianna's presentation is entitled, "The President's Emergency Plan for AIDS Relief: Implementation Science Program." This presentation will give us insight into the mechanisms through which PEPFAR, the PEPFAR program, assesses impact and gains knowledge about program implementation and supportive innovation. So please join me in welcoming Dianna. We look forward to her presentation and to the opportunity to ask some questions afterward.

Thank you, Dianna.


Get Adobe Reader View slide presentation ]

DR. EDGIL: Thank you. It's my pleasure to talk to you all about the President's Emergency Plan for AIDS Relief and the newly launched Implementation Science Program. So most of this talk will be about some of the things that we have done in the past, the importance that we're now placing on implementation science into the future, and then I'm going to go back to, if I have time today, to a little bit of data from some of the evaluations that have been done or that were done in the first phase of PEPFAR.

Actually �'�' well, we'll see how this works out with the microphone. I feel like I might be going back and forth a little bit.

MS. GORDON: We can hear you. It's big enough.

DR. EDGIL: Yeah. So if it's a little bit annoying �'�' can you hear?

MS. GORDON: Yeah. It's good.

DR. EDGIL: That's all right. Because I'll be going like this and then like this probably as I look at people, so you know.

Okay. So a little bit about the President's Emergency Plan for AIDS Relief. It was launched in 2003 as the United States �'�' the USG response to the global AIDS crises. PEPFAR was then reauthorized in 2008 through 2013, and to date, approximately $32 billion have funneled through an interagency mechanism toward the global AIDS crisis. And that mechanism includes State Department, USAID, HHS, Peace Corps. I could go on. I don't actually see Treasury in there, but it includes many different agencies of the federal government.

Some of the results of �'�' that have been obtained in the first phase of PEPFAR are that we have put approximately 2.5 million people on antiretroviral treatment, that we have provided counseling and testing for nearly 29 million people, that we have prevented, through mother �'�' through prevention of mother�'to�'child transmission, or PMTCT programs, HIV transmission to nearly 340,000 babies, and that care and support has been provided for nearly 11 million people, including over 3.6 million orphans and vulnerable children.

The first phase of PEPFAR truly prioritized basic program monitoring and evaluation at the Agency level, but then also, through a centrally funded mechanism, called the Public Health Evaluation program, PEPFAR prioritized sort of a higher level kind of evaluation that was a bit more intensive and required a bit more coordination than basic program monitoring and evaluation might require.

Some of the things �'�' some of the lessons that we've learned through M&E and the PHE program are that the PEPFAR programs have benefited both the health systems of countries as well as the health status of their citizens, that for us, or for our programs, having goals and targets helped to drive our programs, that effective prevention programs required targeted, data�'driven responses, as well that our �'�' in our experience, efforts to build country capacity are key to sustainability. And for us, also, the ability to demonstrate the impact of every dollar on the community was of utmost importance.

In December of 2009, the second phase of PEPFAR's five year strategy was released with the following goals to one, promote sustainable country programs, strengthen partner government capacity, to expand our prevention care and treatment programs, to integrate and coordinate with health and development programs and as well as to invest in innovation and operations research/evaluation.

In 2010, and in accordance with those goals, PEPFAR began to focus on implementation science. Implementation science has sort of become a buzz word for many different programs. And for us, the definition that we are using is that implementation science is a scientific framework to guide health program implementation and scale up that focus on effectiveness, efficiency, and cost effectiveness in order to build the evidence base necessary to inform the best approaches to achieve sustainable prevention, care and treatment programs.

So what does implementation science mean for PEPFAR overall? Well, for our implementation science program, we would then consider the use of IS to promote collaboration and integration between program and science to improve the ways programs are designed, implemented and evaluated to accelerate an increase in health impact.

So in addition to setting out a framework wherein we would use implementation science to develop our programs and to implement our programs, there are some additional objectives that we might �'�' or that we have considered crucial to the implementation science program.

Some of those are to identify critical questions generated by the HIV/AIDS community, to address questions that PEPFAR specifically is uniquely poised to research, as well as to provide some clear answers about the efficiency, effectiveness, and impact of programs in a timely manner, which is something that PEPFAR has been criticized for in the past, to create linkages that will allow the use of evaluation results, to identify best practices and inform programmatic decision�'making, as well as to assist countries in building capacity in M&E and research, and to focus on transparency and the publication of peer reviewed articles.

So this slide is actually what we would consider to be the implementation science evaluation continuum. And that continuum includes both monitoring and evaluation, as well as higher level operations research and ultimately impact evaluation. And all of these things, all of these sorts of tools would actually be used to build up to an implementation �'�' to programmatic decision�'making that would be based on implementation science.

So based on the results that have come from �'�' that might come from those types of studies, whether they be monitoring and evaluation, up to impact evaluation, you would then ask the question how does this lead to the implementation of very well designed programs. And the questions that one might ask would then be how do we deliver interventions efficiently and effectively, how could we then transfer those interventions from one setting or population to another, how might we make informed choices between competing interventions or components of interventions or strategies for delivery. And it is our goal to use the implementation science program to actually move our programs in these directions.

So as I said before, the implementation science program is very, very new. It was only just launched in 2010. And as a matter of fact, the studies that are being funded in 2010 have not even actually gotten off the ground. So all that I can tell you about the program is sort of what we intend to happen, not necessarily what the results of implementation science have been thus far. However, the implementation science program was meant to inform programs on efficiency and effectiveness.

We are trying to get at the use of impact evaluation, where it is possible, because we consider it to be necessary. And as well, most of the research that's being done, or I could say all of the research that is being funded through our mechanisms would be done at PEPFAR funded sites.

So in 2010, there were two mechanisms that have gone out. The first one is a PEPFAR country driven mechanism, which is nearly identical to what the public health evaluation program was in the past, except now with a further focus on questions that are specifically implementation questions. And then the second mechanism was an external mechanism that was co�'funded with the NIH. And some of the specific areas of research that we focused on, because we identified them as gaps in our evaluation portfolio, were nutrition, orphans and vulnerable children, prevention of mother�'to�'child transmission. And perhaps you can read the rest of the list. I don't need to go down.

Okay. So some of the �'�' an example of an implementation science question that would be specifically related to our HIV/AIDS program are the following questions. And what we had realized at a certain point in the PHE program was that rather than focusing on something that would help to implement further a program, we really had been �'�' we had been focused on science that would inform us as to whether a program was �'�' really, we had been looking at outcomes and we hadn't been looking at either impact or moving that program forward in terms of implementation. And so using these questions, we really wanted to get at how to better implement programs with proven effectiveness.

So given that 2010 was the first year for the implementation science program, we sort of need to go back, then, and look at what �'�' how we might use the questions that were on the previous page to target all of our programmatic areas. And what I should say is that PEPFAR currently has thousands of programs in country. And so if �'�' when you think about evaluating �'�' using the implementation science program to evaluate all of those programs, it becomes an overwhelming task.

So before we get to those higher level programs that might be centrally funded and then might actually be funded through those programs, we need to focus on the fact that basic program monitoring and evaluation is going on in all of our countries through all of the agencies and that at a central level, we have not yet been collecting very good information on that, although the agencies do have that information. And so when we want to go back to them and ask them what have they done with the reports from these evaluations, have these evaluations been used to identify best practices and in programmatic decision�'making, they can tell us that.

And so some of the lessons that we've learned and some of the things that we know we need to do are based on the reports that the agencies have been fed to us. So in our �'�' in prevention care and treatment, we recognize that there are things that we need to do and that we need to include in the activities that we would fund under the implementation science program.

So in the area of prevention, the title says, "Promoting Evidence�'Based Prevention." And if you are at all familiar with prevention, HIV prevention activities, you know that there �'�' that the evidence base for those activities is limited. And so a lot of money across the board has been funded or has been funneled into prevention programs.

And now we need to go back and really show which of those programs are the best programs that should be funded and in what combinations we need to be funding those programs and in what settings those programs might differ because they would differ according to the epidemiology and the country. If you have a generalized epidemic, then you would be using prevention programs that are targeted at heterosexual couples. If you have a concentrated epidemic, you would be targeting prevention programs to the most at risk vulnerable populations like commercial sex workers and men who have sex with men.

Okay. So given that the evidence base for prevention programs is rather limited, the implementation science program is very focused on trying to get at those needs.

Strengthening systems of care and support. We have a goal of care �'�' providing care for more than 12 million people, including 5 million orphans and vulnerable children. And we have been doing a relatively good job at meeting those goals. But of course there are activities that we could continue to do and those would be to strengthen the capacity of families in communities, to provide support for OVC, expanding coverage, certainly strengthening TB HIV programs, and then ensuring equal access and equality for these programs.

We would also like to expand our treatment program. The goal is to support more than four million people on treatment. And treatment has been the one area in which we have been incredibly successful, but further activities are absolutely necessary, including further scale up. But then also ensuring retention and adherence and program quality and the monitoring of drug resistance have been things that have in the past been funded through the PHE program and will continue to be funded through that program.

And then finally, given that for �'�' that, you know, we have such an enormous amount of money flowing into many of our programs, we are putting an emphasis on finding efficiencies. So no longer just, you know, simply scaling up for scale up sake, but actually looking at where the money is going, how much things cost and whether or not we �'�' the benefits �'�' that the benefits lead to the costs in some way.

So in doing so, some of the things that we are looking at doing are costing studies in all of our program areas, coordinating with the global fund, as well as moving toward generic ARVs. So and many of these efficiency finding activities fall under the realm of evaluation, and some of them have been funded through the PHE program in the past.

So now I'm getting into some of the data that has come out of the Public Health Evaluation program in the past. So and leading from this idea of finding efficiencies and doing costing, we have used PHE data in the past to put together a costing �'�' what we call the ART costing model. So this costing model was put together in 2008 to inform the reauthorization. And the costing model was found to be so valuable that further PHEs were done to inform a better model, or the second generation of the model, that is �'�' that continues to be used to this day for planning resource needs.

Because we have this model, we suddenly realized, when we were going over the country operational plans in 2009 for many of the countries, that countries themselves are having difficulty in determining what their resource needs are. And we had done PHEs in several different countries and then never really funneled that information �'�' because we were using it at headquarters, never really funneled that information back into programmatic decision�'making for the country.

So in 2009, based on some costing data that had come out of a PHE in Mozambique, we put together a model specifically for Mozambique so that they could determine their resource needs into the future. And these are some of the results that came out of that costing model. And remember, this is specifically for ART. But what you can see is that between the first year of PEPFAR and the fifth year of PEPFAR for patients on ART, there was a dramatic decrease in the cost per year per patient.

So once Mozambique had this data, they were very excited about it and they started to talk to other countries about it at some of the annual meetings. And we have now had an increased demand for further models from other PEPFAR funded countries to aid them in estimating their resource needs. In addition to that, it has also been determined that we need to have costing models, not just for ART, but also for all of our other services and programs.

And so models that are in development are for prevention, for care, for the healthcare �'�' for determining the number and cost of healthcare workers, including voluntary counseling and testing and prevention of mother�'to�'child transmission into the ART cost model, and then patient level disease progression models.

So the second set of data that I'm going to show is actually on the other end of what a program evaluation might look like. The first is it's very high level, what does it cost and, you know, how can we predict our needs into the future. This is very on the ground looking at clinic patient data to determine whether or not programs are running optimally.

So in this case, we're looking at prevention of mother�'to�'child transmission, effectiveness in Africa, research and linkages to care and treatment. We're calling this the PEARL study. And this was a multi�'country study in Zambia, Cote D'Ivoire, South Africa, and Cameroon. And then I'm not sure how familiar everyone is with prevention with mother�'to�'child transmission. So I'll quickly go over the cascade of events that needs to happen in order for a woman to, a pregnant woman to, actually get an ARV prophylaxis that would prevent transmission of HIV to her child.

Seeking any natal care, being offered and then accepting an HIV test, getting test results, agreeing to take ARV prophylaxis, actually taking ARV prophylaxis and then also giving your newborn child a dose of ARV. Those are all of the steps that need to happen in order for transmission to actually be inhibited.

For many of these steps you can actually simply obtain a certain amount of recording, but when it comes to �'�' or reporting. When it comes to adhering to ARV prophylaxis, there is a question because often mothers will say that they took the drug, but there is no evidence that they actually did take the drug. And so in order to figure out whether the drug was actually taken, in this study, they looked at cord blood from the newborn infant. And if you can identify that the ARV is in the cord blood, that means that the woman has taken the prophylaxis.

So this was �'�' there �'�' these were some of the problems in actually collecting the cord blood, but overall, about 95 percent of the samples were obtained for cord blood, as well as all of the reporting and all of the rest of the cascade that leads up to these results.

So what you can see here is �'�' and this is once again the cascade from being zero positive to be offered a test, accepting a test all the way down to infant dosage. The black part on the bottom is the success, is the successful mediation of each of these steps, and the grey at the top is a failure in mediating these steps.

And so what you can see is that over the entire course of the cascade, the �'�' when a failure occurs, that number would then drop out of the denominator. And when you get down to the infant actually being dosed with ARVs, you are left with approximately 50 percent of what you had started with, which was zero positive �'�' HRV zero positive deliveries.

And then this is perhaps a better way of showing this. The black is the successful prophylaxis of an infant and prevention of transmission and all of the colors are the ways in which one might fail to treat the woman and the infant. And what I'm showing here is that successful prophylaxis only occurs in 54 percent of the women who were going to the clinics, to the PMTCT clinics.

And so this was really shocking data when it came out because I think that people had recognized that there were failures, but not that they were at such a scale. And then the next �'�' I know that this is a very confusing slide, but all you need to know is that these are a bunch of �'�' these are all of the clinics that were �'�' that we were obtaining samples from.

And those colors correlate with the colors on the last map. Black is successful and all of the rest of the colors are failures to dose a woman and her infant with an ARV prophylaxis. And you can see that there is not just one clinic or one failure that occurs, but the failures occur across the board in all of the clinics for the programs that were evaluated.

So basically the conclusions of this study are that, of course, PMTCT is an incredibly complex thing to deliver. There is a very complex pattern or �'�' you know, of events that need to be negotiated by a pregnant woman in order to get to the end result, which would be preventing HIV transmission. But also that simply looking at that coverage problem, simply looking at the cases that are missed in the programs that currently exist would get you probably more than new programs and new regimens that are supposed to be �'�' you know, that someone would think up in order to get populations that aren't currently seeking any natal care.

And so then in conclusion for PEPFAR in general, we feel that great progress has been made, specifically around the evaluation of our programs. But AIDS is still a crisis. And in order to sort of mitigate that crisis, we continue to need to expand our prevention, care and treatment activities. But in doing so, we also need to look at expanding country level capacity to perform these activities themselves.

Implementation science is being threaded throughout all of our programs. It needs to be integrated into our way of thinking and including in programmatic design, implementation and evaluation in order to increase our impact overall. But in reality, AIDS is a global burden that requires a true and shared global response. Thank you. And I guess I will take questions now.

MS. GORDON: Let me ask folks �'�' thank you for your presentation. That was outstanding.

Let me ask folks �'�' we are recording this �'�' if you have a question to please turn on your mike if you're sitting at the table and otherwise, we have a microphone that can go around the room. So happy to take your questions.

Maybe as you're thinking, I can ask one and this is just about the implication for the policy response. It seems as if a lot of the emphasis now is on evaluating programs for efficiencies and trying to determine what to do more of, what to do less of. How would you relate your findings to policy recommendations? It looks like there is some �'�' sort of your last slide touched on some areas, but does it show there should be greater emphasis on the prevention track versus the treatment and care side. Should it �'�' I guess what are some of your thoughts in terms of taking this to the policy next step.

DR. EDGIL: Right. I think that some of the thoughts around our programs right now are that our treatment and care programs, because many of them are so facilities and clinic�'based, are some of our best evaluated and the programs for which we have the most evidence base and the most data around implementation and what is effective.

And so obviously, we want to continue to do what is effective. When it comes to prevention, we recognize that we really need to focus on prevention. And that is where the efforts need to be, but there is also a recognition that the data is very limited. And so there is �'�' and I think that Ruth Levine may have touched on this as well. When she was talking about evaluating programs for the global health initiative, there is a balance in policy�'making that you need to play when you are thinking about implementing programs that have �'�' that don't have data on effectiveness or impact.

And so PEPFAR was an emergency plan. Many programs were rolled out with the thought that we would continue to evaluate them over time and we would figure out whether or not they were effective. The policy implications for programs that are not effective are, I think, pretty obvious. And so we need to figure out what the data says and then go back and make the hard decisions about programs that aren't doing what they should be doing.

QUESTION: Hi. Thanks. Can you talk a little bit about the ways that you ensure monitoring of treatment programs and ensure that people are continuing to come in for treatment. Because I know in one of the sessions we heard about people getting lentils if they came in for vaccination or brought their children in for vaccination. And I've heard that TB programs do that also.

DR. EDGIL: Yes. I mean, so certainly the monitoring and evaluation that goes on in our treatment programs is definitely �'�'

(Interruption to recording)

DR. EDGIL: Let me take a step back and say that because PEPFAR is an interagency program, essentially each of the smaller programs, the treatment programs that run through PEPFAR are actually run through an individual agency. So in the case of PEPFAR, many of the programs that are run on the ground are either run through CDC or through USAID. So it is up to the agency to actually perform the monitoring and the evaluation of those programs.

And in general, many of those programs are actually contracted out to other organizations, like country level organizations, so that �'�' called implementing partners for us, so that the �'�' any evaluation that goes on at that level would then be written into whatever contract the agency sets up with the program.

So we require that that happens and agencies require that that happens. The problem is that at a central level, when it comes to M&E, we have not been truly collecting the data. So we've been relying on the agencies to know their programs and to know that these things are happening. So for individual programs, you could go back to an agency and ask them exactly how they're doing the evaluation, but it's not the same for every single program that's out there

So it's kind of difficult for me to answer the question because each program has its own way of doing things. But I will say that adherence is definitely an issue and we �'�' most programs prioritize doing some amount of evaluation about how to increase adherence and how to follow up on those lost to their programs. And we have several PHE, Public Health Evaluations, that look into that for many of our programs as well.

QUESTION: Sort of a slight follow�'up to that. As you're saying, a lot of the agencies are doing their own evaluations of programs and then you also said that there might be some hard decisions to make of what �'�' where to continue funding programs that are proven successful or not. Do you have any fears that agencies and the reporting might be slightly biased because of the fear that funding will not come in, and, if so, how are you looking at mitigating that so that you are getting actual evaluations or are you looking at doing evaluations in a different way?

DR. EDGIL: Right. Yeah. I mean, thus far we have not really made the hard decisions and we haven't really sort of put out there the possibility of some of the programs being, you know, for whatever reason discontinued. So we haven't had to deal with agencies not necessarily telling us the whole truth about what their evaluations bring to bear.

But I think it is the intention of OGAC, to begin to collect some of the reports on that data and to begin to come up with, by using the costing models, some way to really figure out what the costs and what the benefits are of the programs. And they should be comparable.

But I mean, obviously, there will be ranges of cost benefit across countries and across programs, depending upon whether you're in an urban or a rural setting and what target population you're getting at, but when it comes down to it, using those models, if those models are informed by very good information, we should be able to come to some determination. That's the hope anyway.

QUESTION: In terms of the types of activities that are not facility based, like OVC, prevention community�'based care and support, I'm wondering if you have any examples where just even at a project level, if they try to tackle the effectiveness, efficiency and cost effectiveness because from what I've seen, so many of them only focused on the PEPFAR indicators and even seeing where people within them had tried to say okay let's do our costing differently so we can do cost effectiveness, but it doesn't happen.

DR. EDGIL: Right.

QUESTION: Is there any way of promoting that, too, from your �'�'

DR. EDGIL: Yeah. I mean, we �'�' so as you may have seen with the new implementation science program, in comparison to what had been going on in the past where there was a very strong emphasis on clinic�'based activities, we're now trying to get at some of the issues around OVCs and nutrition and some of those areas that have truly been left out of our evaluation portfolio in the past. But recognizing that those are �'�' the reason that they've been left out is because they're very, very difficult to do.

I do know of some USAID, which actually is sort of the agency that does most of our prevention activities at this point, does, through measure and evaluation and through other evaluation programs, they definitely do evaluate those OVC and community�'based care programs.

I'm not 100 percent sure how we get around the issue of, you know, sort of data manipulation necessarily to �'�' you know, I think that these are questions that we need to continue to have because the data for us isn't even there at a central level. So first we need to figure out what data is there and then we need to figure out whether we believe it. And then we need to figure out if we don't believe it, then how can we move forward and do evaluation that isn't biased in some way by the funding line. I don't have the answer for you, though, unfortunately right now.

QUESTION: No. I would say from my experience, that they don't have the types of baseline data to be able to do any type of things even for a small sample or, you know, particularly for cost effectiveness to being able to start from the beginning. And it's the whole issue of from the beginning whether you're building it and not all these retrospective studies.

DR. EDGIL: Right. Right. Yes. I completely agree with you and I think that we have recognized that for many of our programs, because they were not built on a platform of evaluation, that it becomes more and more difficult for us to evaluate them over time. And so one of the things that we are trying to do is to figure out alternative methods of evaluating programs. And some of them do exist. They are mathematical formulas that one can use to try to determine or to try to evaluate programs that are steady state.

And so we're moving forward in trying to figure out what those methods are and whether or not they're going to help PEPFAR evaluate it's programs. But it's early days in that conversation. So I also don't have any results yet.

QUESTION: But Dianna, I noticed in your charts when you did the color breakouts, there was a lot of data not available. It seemed like the red. So in terms of trade-offs when you look at budget allocations, is there pushback in terms of trying to improve the availability of data and the investment that you have to make in data collection to the actual implementation of the program? Do you see there is attention there or �'�'

DR. EDGIL: Well, so certainly the question of making data available is an incredibly complicated one. And it starts with data availability from the country's perspective because for all the programs that we have in country, countries consider those data to be theirs. And so in order for us to get that data, we need to ask the country and for them to give permission to us.

And then there is this sort of, you know, increasingly difficult, once again, cascade of getting those results from the country through the agency past clearances until we can finally release that data. So if you're asking about, like, transparency and making data available to the public �'�'

QUESTION: No, it was more �'�' sorry. It was more collecting the data, the investment that it takes to actually get that data, that investment versus the, you know, for example spending money on treatment.

DR. EDGIL: So right now what happens in countries is that the data is there and the data has already �'�' for most program evaluations, a report is written. And that report goes to the agency. The thing that has �'�' the link that has been lacking is OGAC headquarters getting involved and saying we also would like access to this report and we also would like to do something with this data, whether it be helping us to make decisions at the headquarters level, or whether it's simply being able to be very transparent when people come to us asking what, in fact, is going on with these evaluations.

So, in fact, I think that the amount of money that would go into that is relatively minimal, although there is certainly an emphasis on what we call knowledge management at the headquarters level.

So on many levels, trying to become organized enough and become a central repository for information, whether it be evaluation or whether it be any other programmatic data that �'�' so that the PEPFAR community, as well as potentially our other partners, might be able to access that data. But I don't think that in that case, specifically, it's going to be as much of an effort because we do, actually, collect quite a bit of data at the agency level.

QUESTION: As you mentioned, the implementation science program is quite new. And so I'm intrigued to know how you're doing a meta evaluation of the implementation science program down the line or have you already thought about what sort of milestones and goals the program as a whole is going to accomplish. And how are you looking at evaluating that?

DR. EDGIL: Yeah. Unfortunately, the program is still �'�' the only part of the implementation science program that has actually been launched is this very high end evaluation getting toward, you know, higher level operations research and impact evaluation. And so the plans for the entire program itself have not come to fruition and have not actually been released. We're really considering that to be a fiscal year 2011 activity. But it's a very good question and something that we are �'�' that we continue to discuss.

MS. GORDON: Going once. Okay. Well, let's have a warm thank you for Dianna's presentation.


MS. GORDON: We're breaking about five minutes early. So you have an opportunity for coffee and for networking, and then we'll follow the program agenda from there. So thank you very much.