Using Measurement to Drive Results

The Case for a New Strategic Measurement Framework

CSIS Commission on Smart Global Health Policy
July 2009

New Phase, New goals, New Results Focus

We are in the midst of a major transition in the core U.S. goals for global health. During the period 2003-2008, the predominant goal was to respond to a health emergency, through large-scale, dramatic, single-disease initiatives: the President's Emergency Plan for AIDS Relief (PEPFAR) and subsequently the President's Malaria Initiative (PMI). Increasingly now, the overriding goal is to help create in partner countries sustainable, long-term programs that will decrease mortality and morbidity in most at-risk populations while strengthening health systems. That will involve building on the success of the HIV/AIDS and malaria platforms, while broadening approaches to put a new emphasis on maternal and child health, family planning, and prevention of high-burden, high-risk conditions.

A New Paradigm of Accountability through Measurement

In this new phase, there is a corresponding paradigm shift with respect to measurement: from a focus on inputs and process to a higher priority on ensuring that U.S. investments are leading to the greatest health impacts possible. The ability to link new goals with concrete results is contingent on a new paradigm of measurement. There is also a need for measures that reach beyond those that quantify disease impact alone to measures that also capture whether our aid is driving enduring change in key practices. For example, are partner countries focusing their own resources more intensively on the health conditions that contribute the most to death and disease? Are U.S. inputs additional to national resources or do recipient governments respond by shifting resources out of health to other sectors? Is more money being leveraged from others as the result of U.S. investments? Do U.S. investments increase partner governments' ownership of the health agenda, evinced by higher financial commitments and strengthened management and delivery of health services?

Improved measures of both individual and broader "systems" outcomes are essential to demonstrate that U.S. investments in global health are working and need to be sustained, in the face of worsening U.S. budget conditions foreseen in the near and medium term. Achieving durable results in global health is a long-term endeavor. Improved measures of impact are fundamental to building a long-term strategic U.S. approach to global health, achieving higher efficiencies and more cost-effective returns on investments, setting priorities and making tough choices, and preserving the support of Congress and the American people. Improved measures should draw substantially from the private sector's experience in capturing impacts and building performance-based systems. A greater U.S. commitment to better define and harmonize measures could drive agreement across other donors and partner governments to focus on mutually agreed on results. It could reduce the acute burden on partner governments, high transactions costs, and confusion caused by the proliferation of divergent donor reporting requirements. Improved alignment of goals, and alignment of actions to drive toward those goals, could raise the efficiency and effectiveness of foreign aid in the health sector and beyond.

Benefits of Performance Measurement

Better impact measures will improve our understanding of comparative costs and effectiveness in three ways:

• Linking investment dollars to the improved health of individuals. This will allow looking comparatively across investments in different health areas.
• Linking investment dollars to administrative effectiveness, including host governments' own efforts to improve service delivery, recruit and retain personnel, strengthen management systems, and prioritize monitoring/evaluation.
• Linking investment dollars to improved governance by partner governments: for example, greater transparency and accountability by finance and health ministries in the use of donor funds; increased domestic funding for health; and increased high-level political leadership on health.

Need for Realism

In devising better metrics, we need to be mindful of the field's multiple challenges.

• There is limited agreement, even among "experts," on how best to measure the impact of many health interventions.
• Measures that do exist are rarely comparable with each other.
• Measurement systems can be manipulated if specific controls and independent oversight are not put in place.
• It can be difficult to directly link inputs from any single program to the improved health of an individual or a community.
• Measuring the impact of prevention programs has proven particularly challenging. New methods are needed, supported by greater research, especially in measuring HIV prevention.
• Financial data, essential to measure cost effectiveness, is often unavailable or unreliable.
• Country data and data collection capacities are often weak.
• U.S. agencies have underinvested in impact measurement.
• U.S. agencies operate side-by-side with other major bilateral and multilateral agencies with no common set of outcome measures.

Core Elements of a New U.S. Strategic Measurement Framework

Starting assumptions:

• A framework must be long term, built sequentially over a number of years.
• Important building blocks can be put in place now and need not wait for the 2011 budget process to commence.
• Significant measurement and evaluation competencies do already exist within the U.S. government and the broader global health community. These could be leveraged in a more effective and strategic manner.
• The U.S. government's stated intention to increase support for maternal and child health programs implies the organization and delivery of multiple health services; assessing results in this area can be used as an entry point to measure broader progress in strengthening health systems. Maternal mortality, it should be noted, can be very difficult to measure.
• European donors and multilateral institutions have begun a laudable, if imperfect, effort to better harmonize program measurement. A new U.S. measurement framework, backed by strong U.S. leadership, could build systematically on this dialogue. Attention to the Millennium Development Goals (MDGs) will be important.

Build from PEPFAR's foundation:

• The White House Global Health Initiative commits U.S. funding across a range of health priorities. Fully two thirds of the planned $63 billion will be directed to sustaining and accelerating PEPFAR's achievements. Given that PEPFAR will remain the foundation of U.S. engagement in global health well into the future, a first priority should be to improve the measurement of PEPFAR impacts both within HIV/AIDS, as well as on broader health systems.
• The PEPFAR experience of the last five years has already produced compelling lessons, and PEPFAR has shown a willingness to adapt and adjust its approach. Both the Institute of Medicine (IOM) and independent reviews have emphasized higher priority be given to measuring outcomes--mortality, prevalence, and incidence, as well as the capacity of health systems.
• PEPFAR-created platforms should be the basis for extending more effective measurement frameworks into other key priority areas (i.e., malaria, tuberculosis, maternal/child health, family planning, and health systems).

The approach:

• Publicly commit to a new era of accountability, transparency, and a focus on results. Senior U.S. leadership should communicate that a new effort around "outcome accountability" is a top U.S. policy priority. That commitment should include setting outcome targets and stating the intention of making them, along with other program data, publicly available. The development of sound measurement frameworks should be a top priority for inclusion in PEPFAR's Partnership Frameworks and appropriate Millennium Challenge Corporation (MCC) Compacts. In countries where the United States has a range of significant global health investments, comprehensive agreements should be reached with host governments that clearly define measures of success, as well as partner government commitments to sustainability and accountability. At the programmatic level, this new commitment should include: (1) a specific target percentage (in the range 5 to 7 percent) of program funding to support outcome evaluation; (2) a requirement in the grant/cooperative review process that stipulates inclusion of outcome evaluation measures; and (3) a commitment to accelerate research to develop improved methodologies for impact measurement, especially in prevention.
• Expand the time frame. Impacts take considerable time to achieve and measure. PEPFAR pioneered a 5-year planning framework, but a 10- to 15-year planning horizon is warranted and valuable, given what is required to change behaviors and systems. While the focus of the new framework should be on better capturing outcomes, a parallel effort should define and measure key intermediate milestones.
• Begin with a strong interagency direction. Establish a single interagency planning group, chaired at a very senior level, charged with integrating policy, budgeting, and program planning. Mandate the group to: (1) review the PEPFAR experience to date in measuring baselines, intermediate milestones, and outcomes; (2) further develop PEPFAR-supported platforms to better measure key baselines, intermediate milestones, and outcomes in other priority areas; (3) identify a core set of intermediate and outcome measures for all key priority areas (e.g., TB, malaria, maternal health); (4) map a practical plan for the phased introduction of a new measurement framework; and (5) advise on concrete steps to reduce the duplication of effort across U.S. agencies and create more streamlined, unified programs.
• Establish the core principles and collaborate with other partners to ensure broad support. The global health space is complex, with multiple stakeholders and competing agendas. The United States has a unique opportunity to change the paradigm, but this is most likely to be successful if other partners, who are willing to embrace a results-oriented approach, are brought on board early in the process. These partners could include the World Bank, World Health Organization, Global Fund, other multilaterals and selected bilateral donors. An explicit goal should be to win agreement on a common set of standards and thereby reduce the costly and unsustainable reporting burden imposed on partner governments.

The tool box:

• Create an independent evaluation team. Create a truly autonomous, external group charged with conducting evaluation of U.S. investments in global health and sharing its findings with Congress and the administration. While the group need not be a new government entity per se, it does need to be sufficiently resourced and independent to have true "teeth" and be able to carry out unbiased objective analyses. It should have access to expertise from outside the U.S. government, be staffed by a small, skilled core of professionals, have the ability to field mobile teams for firsthand in-country assessments, and the flexibility to call in outside expert assistance when needed. To be effective, this group would need to be linked directly and at an early stage to the program planning and implementation process. Such an effort has not been attempted on a sustained basis before and would likely invite resistance from agencies claiming they already perform this work. Potential conflict-of-interest issues would need to be carefully considered, but are manageable. A variety of models could be explored, including: (1) placing an independent group within the Institute of Medicine or the General Accounting Office; (2) creating a standing consortium of independent experts managed by a small secretariat; (3) relying on a competitive bidding process to contract for specific assessments; or (4) establishing a free-standing independent entity in concert with a small number of like-minded donors.
• Develop and expand core competencies. The private sector has special expertise in monitoring and measuring impacts that should be incorporated into a U.S. strategy. The private-sector mantra "if we cannot measure it, we cannot manage it" informs various industry-wide health and safety measurement systems that allow companies and industries to monitor their performance in a comprehensive and standardized way and that enable meaningful trend and cross-company comparisons. The starting point for these "scorecards" is identifying indicators that accurately reflect desired goals and may include process, outcomes, and impact measures. The pharmaceutical and globalized food industries have depth in measuring procurement and supply-chain performance in developing countries. Other industries have made particular strides in measuring quality of services/products delivered, a particularly important consideration for health programs.
  
  The Epidemiologic Intelligence Service (EIS) of the Centers for Disease Control (CDC) provides an important government model for how to structure an initiative to train U.S. and partner country experts in monitoring and measuring program impacts. EIS has demonstrated over several decades how to recruit, train, and retain experts with the competencies and skill sets to conduct epidemiologic investigations and surveillance. That experience could be translated into an initiative to train a cadre of monitoring and evaluation expertise. This effort could be extended to train overseas partners much like CDC's Field Epidemiology Training Program (FETP) does. Focused initiatives in selected high-investment African countries could create greater data collection and analysis capacity.

• Create an external advisory group. A number of U.S. government health agencies (CDC, the National Institutes of Health, and the Food and Drug Administration) make regular use of external advisory boards to help guide their efforts and grapple with difficult issues. Measurement and evaluation of U.S. government health programs could also benefit from the insights of a similar "wise persons group" drawn from the private sector, finance, foundations, academia, and elsewhere to: (1) advise on key, difficult issues (i.e., measuring the impact of HIV-prevention programs); (2) advise on building external alliances; and (3) help explore and apply promising new approaches, such as performance-based funding, to U.S. programs. There are a number of groups and initiatives that have emerged over the last several years focused on measurement and impact. There is a diversity of views among them on how to approach the issues and some significant differences of opinion; however, there would almost certainly be strong interest in engaging with the U.S. government in a more regular dialogue on how to better assess impacts.¹
• Invest diplomatically at a sustained high level to encourage a streamlined, standardized approach to measurement, a focus on results, and a reduction of excessive donor reporting burdens. Decide whether the United States should take an immediate and more active role with emerging initiatives such as the H8² and the International Health Partnership--both of which are in part focused on efforts to develop common data architecture, coordinate monitoring and evaluation, and lower transactions costs. Convene a high-level meeting of leaders from the major multilateral institutions (the Global Fund, World Bank, UNAIDS, Global Alliance for Vaccines and Immunization, and World Health Organization) to solicit input on the new U.S. approach and table forward-looking reform proposals for discussion with partner countries.

---

¹Some of the organizations with a strong measurement focus are the World Bank's Independent Evaluation Group, the University of Washington's Institute for Health Metrics and Evaluation, the World Health Organization's Health Metrics Network, and the International Initiative for Impact Evaluation.

²The "Health Eight" (H8) includes the World Health Organization (WHO); World Bank; Global Alliance for Vaccines and Immunization (GAVI); the UN Population Fund (UNFPA); the Joint UN Program on AIDS (UNAIDS); UN Children's Fund (UNICEF); the Global Fund to Fight AIDS, TB and Malaria; and the Bill & Melinda Gates Foundation.