ESSA | Ohio Department of Education

Empowered by Evidence: Using Level 4 Evidence-Based Strategies

Evidence-based strategies, practices or activities have been evaluated and proven to improve student outcomes. Programs may include evidence-based strategies. Districts can have confidence that evidence-based strategies are likely to produce positive results when implemented.

The Every Student Succeeds Act (ESSA) (Section 8002) and the U.S. Department of Education's NonRegulatory Guidance: Using Evidence to Strengthen Education Investments outline four levels of evidence. When you search for evidence-based strategies in Ohio’s Evidence-Based Clearinghouse, the evidence-based strategies you see meet ESSA’s definitions for Level 1 (“Strong”), Level 2 (“Moderate”) or Level 3 (“Promising”). More information about how these levels of evidence are defined is available on the Defining ESSA Levels of Evidence page.

Level 4 strategies have not yet been evaluated with the same level of rigor as Levels 1, 2 or 3. This does not mean that Level 4 strategies are not valuable, nor does it mean that they will not help you meet your needs.

Rather, using Level 4 evidence-based strategies – when those strategies are aligned to your needs and you are prepared to evaluate their impact – is an opportunity to support innovative thinking, encourage the use of local data to understand what is working in districts and contribute to the national evidence base.

What are Level 4 Evidence-Based Strategies?

ESSA defines Level 4 evidence-based strategies as:

Demonstrating a rationale based on high-quality research findings or positive evaluation that such activity, strategy or intervention is likely to improve student outcomes or other relevant outcomes.

What does this really mean?

Level 4 evidence-based strategies have demonstrated promise in early studies, but the statistical rigor and outcomes of existing research does not yet meet that required by Levels 1 (“Strong”), 2 (“Moderate”) and 3 (“Promising”). There are three related reasons why this might be the case:

When Can Level 4 Evidence-Based Strategies Be Used?

While there are cases where districts and schools will be required under ESSA to use Level 1 (“Strong”), Level 2 (“Moderate”) or Level 3 (“Promising”) evidence-based strategies, there will also be opportunities for districts to use Level 4 strategies. Options for using Level 4 strategies to address school improvement requirements or grant opportunities will be identified and detailed on a case-by-case basis.

Understanding the reasoning behind the Level 4 category of evidence-based strategies, as well as the three steps to using Level 4 evidence-based strategies, can help districts make informed decisions about what strategies will best meet their needs.

What are the Three Steps to Using Level 4 Evidence-Based Strategies?

There are three basic steps a district should plan for when considering using a Level 4 evidence-based strategy:

Step 1 – Explore Existing Research

The first step in determining whether a strategy “demonstrates a rationale” is exploring existing research to determine whether there is good cause to believe the strategy will work.

There are many resources districts can use to learn more about existing research:

Sponsored by the Institute for Education Sciences (IES), Education Resources Information Center (ERIC) is a searchable, online library of journal and non-journal articles related to education research. You can get started using ERIC here.
If a strategy is not a Level 1 through Level 3 strategy simply because it has not yet been evaluated enough – or with sufficient statistical rigor – existing clearinghouses of evidence-based strategies (What Works Clearinghouse; Evidence for ESSA, Blueprints for a Healthy Youth) may still include a wealth of information about an activity, strategy or intervention. The information included within those clearinghouses may be enough to give you good cause to believe the strategy will work for your students — and the work that you carry out to evaluate that strategy in your own local context may contribute to the evidence base for that strategy.
Regional Educational Laboratories (REL) Ask-A-REL can result in a basic bibliography, with article abstracts included, related to any question you may pose related to education research. You can learn more about the REL program on the IES website. Ohio falls within the REL’s Midwest region.
External research partners can be a great resource for learning more about existing research on an activity, strategy or intervention. If you have existing partnerships or are considering developing new research partnerships, ask them to share their insights and knowledge.

Tip: In the past, program evaluations were often designed as long-term projects that provided final insights into the success (or lack thereof) of a program many years into its implementation.

Today, advancements in data and technology mean that districts can often benefit from rapidcycle evaluation. These types of evaluations can tell you if something is working relatively quickly, allowing you to “course-correct” during program implementation.

Tip: As you explore existing research, keep in mind it is easy to focus on research and findings that support our existing assumptions about whether something will work — and critically important not to disregard existing research

Understanding why something did not work in other settings can help you decide if that strategy is the best choice for your district and can help you plan for a successful implementation.

Step 2 – Develop a Logic Model

A logic model is a visual way of showing how you anticipatethat implementing the evidence-based strategy will ultimately lead to improved results.

Going through the process of creating a logic model is more than just an exercise. The process can clarify objectives, generate important questions about assumptions and expectations and provide a roadmap for developing a robust program evaluation.

There are many existing variations on logic models. Additional resources about using logic models include the following:

Step 3 – Plan to Evaluate

When you use a Level 4 evidence-based strategy, you are committing to evaluating how that strategy works within your context. If designed appropriately, the evaluations you carry out will provide insight into:

Show you whether the strategy is improving outcomes as intended;
Shed light on reasons why a strategy is not working as intended; and Contribute to the existing, national evidence base associated with the strategy.

Some school districts have research and data analysis units that can design and carry out program evaluations. In other cases, districts may want to use external research partners to help design and conduct evaluations.

Whether you are using internal resources to carry out an evaluation or using an external partner, thoughtful planning is critical for ensuring that research evaluations yield actionable information that is — first and foremost — helpful for your district in meeting your students’ needs.

As you begin planning an evaluation, some key questions to consider include:

How will you know whether the strategy your district is using is successful? As you review your logic model, what are the key outcomes the evaluation should measure? Are there short-term and long-term outcoms to consider?
How will your district and schools use the results of the evaluation?
What data is available to use for the evaluation? Will you need to collect new data?
Who will need to know about the results of the evaluation? When will they need to know the results? How should the results of the evaluation be communicated to be most useful?
What kind of research partnership will you want to engage in for the program evaluation? How engaged will you want to be in co-designing the program evaluation? How frequently will you want to hear from the researchers and how would you like them to communicate results?
Who are possible research partners? (For example, researchers from universities or colleges, organizations such as regional education laboratories (RELs), local or national research firms.)
Are there costs associated with the evaluation?

Additional Resources: Developing Research Partnerships

There are many ways that districts can work with research partners to learn more about what is working. Resources to help you learn more about developing research partnerships include the following:

While targeted toward state education agencies, the Data Quality Campaign’s Roadmap for Effective Data Use and Research Partnerships between State Education Agencies and Education Researchers includes many points relevant to districts as well.
The National Center for Education Statistics (NCES) Forum Guide to Supporting Data Access for Researchers: A Local Education Agency Perspective provides best practices and templates for data sharing with researchers.
The National Network of Education Research-Practice Partnerships (RPP) includes an RPP Knowledge Clearinghouse.
Research-Practice Partnerships: A Strategy for Leveraging Research for Educational Improvement in School Districts outlines three major types of research practice partnerships and provides guidance on developing these partnerships.
The SEA of the Future: Building Agency Capacity for Evidence-Based Policymaking is written with state agencies in mind but offers many insights into the importance of carrying out education research and evaluation.
The William T. Grant Foundation published a wide range of resources designed to educate districts, state education agencies and researchers on how to leverage research-practice partnerships.

Engaging in research partnerships sometimes involves data sharing with external partners. When planning to share data with research partners for evaluation purposes, districts should always start by talking with their legal and IT departments. As you work with staff across your district to develop a data sharing plan, resources than can help you develop that plan include:

The Ohio Department of Education published guidance on data sharing for program evaluation. This guidance should not replace consultation with legal staff at a district but may provide helpful insights or generate important questions to ask your legal, data or program staff or research partners.
The Ohio Department of Education Data Privacy Report (2014) provides information about how federal and state law effect data collection, sharing and reporting at the state level. The report includes information about how the state approaches data privacy, data security and data sharing.
The DATA DRIVES School-Community Collaboration: Seven Principles of Effective Data Sharing is a resource developed by StriveTogether and Data Quality Campaign that, in addition to outlining seven principles of data sharing, also provides links to case studies, sample documentation and additional data sharing resources.
The National Center for Education Statistics (NCES) Forum Guide to Education Data Privacy (2016) is a resource designed to help states and districts better understand the steps they can take to protect student privacy.
The U.S. Department of Education’s Protecting Student Privacy website provides a wide range of resources for education agencies interested in understanding FERPA regulations and the impact of those regulations on student data collection, sharing and reporting.

Building Up the Evidence Base

Empowered by Evidence: Using Level 4 Evidence-Based Strategies