Evidence-Based Policymaking Resources

Want to learn more about evidence-based policymaking? The Pew-MacArthur Results First Initiative's "Evidence-Based Policymaking Resource Center" provides publications about evidence-based policymaking. It includes:

• Short reference guides describing the components of evidence-based policymaking with descriptions of each component and a checklist of actions to implement them; and
• Short stories (analyses) about how states and counties have approached evidence-based policymaking in their communities.

Additionally, the Pew-MacArthur Results First Initiative reports listed below provide additional information and examples of how jurisdictions (including Colorado) have successfully implemented the framework:

• Evidence-based Policymaking: A Guide for Effective Government: A comprehensive "road map" that provides clear guidance on the evidence-based policymaking approach. Based on an extensive review of research and in-depth interviews with government officials, practitioners, and academic experts, the document identifies five steps that both the executive and legislative branches can take to drive the development, funding, implementation, and monitoring of policies and programs.
• Legislating Evidence-Based Policymaking: The Pew-MacArthur Results First Initiative reviewed more than 100 state statutes passed between 2004 and 2014 and identified five different approaches to promoting data-driven program choices. This brief examines several of these laws and looks at how state governments have used them to expand the use of evidence-based policymaking.
• How Policymakers Prioritize Evidence-Based Programs Through Law: This brief highlights laws in Washington, Oregon, and Tennessee that mandate the use of evidence-based programs and practices and documents each state’s experiences, results, and lessons learned.
• Colorado Dives Into Evidence-Based Policymaking: This brief highlights Colorado's data-driven initiatives that aim to improve outcomes for Coloradans.

The report "Reframing Evidence-Based Policy to Align with the Evidence," published by the William T. Grant Foundation, highlights the importance of supporting the social side of evidence use. Specifically, the report emphasizes honoring different types of evidence, investing in capacity building and infrastructure, and prioritizing relationships and stakeholder engagement.

Resources for High-Quality Evidence

There are several highly reputable sources for quality evidence. Clearinghouses assess and vet potential sources of evidence and present summaries of findings from those studies. The summaries include an aggregated assessment of impact.

National Results First Clearinghouse Database

The Results First Clearinghouse Database compiles research from ten national clearinghouses in one, central location. These clearinghouses conduct systematic reviews to understand impact. While each of the contributing clearinghouses focuses on different policy areas and uses slightly different inclusion criteria, each delivers quality evidence.

Policy areas include:

• Child welfare
• Criminal justice
• Education
• Juvenile justice
• Mental health
• Substance abuse
• Other social policies and programs

The Cochrane Review and Campbell Collaboration

These two organizations produce systematic reviews of assorted interventions, programs, and policies. Cochrane is primarily focused on public and personal health topics and Campbell Collaboration focuses on education, social welfare, and nutrition.

• Cochrane Review
• Campbell Collaboration

Washington State Institute for Public Policy

The Washington State Institute for Public Policy (WSIPP) summarizes a wide assortment of quality evidence to support evidence-based policymaking, with an emphasis on meta-analysis. Their website provides an overview of evidence for several policy areas and estimates benefit-cost ratios specific to Washington state. The Colorado-specific Results First Model is built off of the Washington State model.

• Adult criminal justice
• Adult mental health
• Children's mental health
• Child welfare
• Health care
• Higher education
• Juvenile justice
• Pre-K to 12 education
• Public health & prevention
• Workforce development

On February 16, 2017, the Office of State Planning and Budgeting partnered with the Center for the Study and Prevention of Violence, Invest in Kids, the Evidence-Based Practices Implementation for Capacity (EPIC), the Laura and John Arnold Foundation, the National Conference of State Legislatures, and Toby Barker from the Mississippi House of Representatives to present an evidence-based policymaking seminar to members of Colorado's legislature. The presentation can be found below. Please reach out to Luke Martel at luke.martel@ncsl.org or Erica MacKellar at erica.mackellar@ncsl.org of the National Conference of State Legislatures for more information.

OSPB is always working to integrate evidence-based policymaking into the State's budget process. OSPB developed budget questions for state departments to answer when completing a budget request. The questions, which are listed within the state's budget instructions and below for reference, aim to ensure that funding decisions for certain types of state programs use existing research to inform on programs' potential efficacy. Additionally, it is requested that state departments include a plan for program implementation and evaluation, which is of special importance for new and/or innovative programs that have not had rigorous research completed on them. The goal of these budget questions is to ensure that the state prioritizes funding to efficient and effective programs.

From page 43 of the FY 2019-20 Executive Branch Budget Instructions:

FUNDING REQUESTS FOR NEW PROGRAMS OR SERVICES

As stated above, any request for a new program or service, or expansion of an existing program or service, will be prioritized by OSPB based on the evidence and body of research supporting the program’s effect on desired outcomes and proposed implementation plan. For new programs without a body of research supporting the program’s effect on desired outcomes, departments are also required to submit a plan, timeline, and budget for evaluating the program.

Please be sure to address the following questions in your funding request:

a) Is this a new program/service, or an expansion of an existing program/service? If yes, proceed to question b.

b) Summarize and cite the research that exists on the program or service and the expected effects on outcomes.

c) What is the expected return on investment for the program/service? If it is included in the Colorado Results First analysis , please cite that in the request.

d) What is the plan, timeline, and budget for implementation? Please include information on:

• Training and coaching schedules for staff;
• Technical assistance, such as who will be providing technical assistance and for how long;
• Capacity building, such as meetings and resources that will span the department to support successful implementation; and
• Fidelity monitoring, such as the methods that will be used to ensure that the program is operating in accordance to its design.

e) If you are proposing a new program without a body of research, please include a plan, timeline, and budget to evaluate it. If additional resources are necessary to evaluate the program/service, note that in the request. Evaluations should include (when possible):

• A process evaluation;
• An outcome evaluation component; and
• A benefit-cost component.

Program evaluation is necessary to understanding the impact social programs or policies have on participants and society. The best way to learn whether a social program or policy works as intended is through a randomized controlled trial, a program evaluation in which people are randomly assigned to receive a program’s services or to be part of a control group. Randomized controlled trials, sometimes referred to as the "gold standard" in program evaluation, produce the highest form of evidence because they make it possible to isolate the effect of a program from complicating factors, even those that are unseen. This free edX course reviews how randomized controlled trials can be used to evaluate social and development programs.

If you are interested in completing a rigorous program or policy evaluation, there are several opportunities to obtain funding:

1. The Laura and John Arnold Foundation (LJAF) is offering funding opportunities for randomized controlled trials:
   • The "Moving the Needle" opportunity is designed for select education, employment/training, health, and poverty reduction programs to expand their implementation and research. LJAF anticipates awarding grants that provide between $1 million and $5 million in total funding per project to support expanded program delivery and evaluation over multiple years. Letters of interest are being accepted on an ongoing basis and there is no submission deadline.
   • The "Building Rigorous Evidence about How to Improve Postsecondary Success" opportunity seeks ideas to improve outcomes for students in higher education. The opportunity includes (i) an Innovation Tier, seeking early-stage pilot projects to test innovative ideas; and (ii) a Validation Tier, seeking rigorous—preferably randomized— evaluations of programs that either have highly promising prior evidence or are widely implemented. Letters of Interest are being accepted through January 31, 2019.
   • The "Randomized Controlled Trials to Evaluate Social Programs Whose Delivery Will Be Funded by Government or Other Entities" opportunity seeks to help governments and/or philanthropic foundation (or other funders) build the body of programs rigorously shown to produce sizable, sustained benefits to participants or society. Letters of interest are being accepted on an ongoing basis and there is no submission deadline.
2. The J-PAL State and Local Innovation Initiative is supporting state and local governments in using rigorous evaluation to answer critical questions about what works to reduce poverty. State and local governments can apply for support in using randomized evaluations to better understand the impact of policies and programs in their jurisdictions. Selected partners receive pro bono technical support from J-PAL North America staff, flexible project funding, and connections with J-PAL’s network of leading academic researchers. Application materials must be submitted by 5:00 pm EST on Monday, April 1, 2019.
3. The Patient-Centered Outcomes Research Institute (PCORI) is offering funding for high-quality clinical studies that compare the effectiveness of evidence-based clinical strategies to treat anxiety disorder in children, adolescents, and young adults. Up to $20 million in total costs is available through this funding announcement. The maximum budget per project is $5 million in direct costs, with a maximum project duration of 3.5 years. Applications may be submitted by any private-sector research organization, including any nonprofit or for-profit organization, and any public-sector research organization, including any university or college hospital or healthcare system; laboratory or manufacturer; or unit of local, state, or federal government. Letters of Interest are being accepted through January 31, 2019.
4. The Robert Wood Johnson Foundation funds a wide array of research and initiatives to help address some of America’s most pressing health challenges.
5. The Hearst Foundations provide nonprofits working in the fields of Education, Health, Culture and Social Service with funding to ensure that people of all backgrounds in the United States have the opportunity to build healthy, productive and inspiring lives. The foundations fund a variety of activities, including evaluation.

The following resources can help you develop your evaluation plans:

1. Rigorous Evaluations, by the Laura and John Arnold Foundation
2. Research Resources, by the Abdul Latif Jameel Poverty Action Lab
3. The J-PAL State and Local Innovation Initiative released a guide on how to identify good opportunities for randomized evaluations, how randomized evaluations can be feasibly embedded into the implementation of a program or policy, and how to overcome some of the common challenges in designing and carrying out randomized evaluations. The guide, which contains case studies, draws on lessons learned from the State and Local Innovation Initiative and partnerships with governments using randomized evaluations.

Funding opportunities for evaluation will be shared on this page as they become available. Resources shared may not be exhaustive and are not specifically endorsed by the Office of State Planning and Budgeting.

The following doument can help you plan for the cost of evaluation:

The Evidence Based Practices in Implementation for Capacity (EPIC), within the Colorado Department of Public Safety, Division of Criminal Justice, provides Colorado agencies that serve justice-involved individuals with implementation supports. EPIC assists agencies by:

• Assessing program readiness to implement a practice or policy;
• Collaboratively identifying barriers and devising solutions;
• Establishing methods for monitoring implementation progress and fidelity to innovation;
• Establishing ways to incorporate improvements identified through ongoing review; and
• Continuing to test the innovation and adjusting as necessary until intended outcomes are reached.

The National Implementation Research Network (known as NIRN) provides information and resources to help service providers deliver evidence-based programs with fidelity. Fidelity, or closely following the model of an evidence-based program, is extremely important if an evidence-based program is going to produce results similar to those that have been demonstrated in program evaluation. The National Implementation Research Network continually produces new research and publications related to Active Implementation Science, which is the study of factors that influence the full and effective use of innovations in practice.