Building Up the Evidence Base

Empowered by Evidence: Using Level 4 Evidence-Based Strategies

Using Level 4 evidence-based strategies that are aligned to your context and needs supports innovative thinking and encourages the use of local data to understand what is working in classrooms, schools and districts.

Level 4 strategies have not yet been evaluated with the same level of rigor as Levels 1, 2 or 3. This does not mean that Level 4 strategies are not valuable, nor does it mean that they will not help you meet your students’ needs.

Tools and Resources

Explore Existing Research

The first step in determining whether a strategy "demonstrates a rationale" is exploring existing research to determine whether there is good cause to believe the strategy will work. Check out these helpful tools and resources to learn more.

Ohio Evidence-based Clearinghouse - Explore the strategies available through Ohio’s Evidence-based Clearinghouse.
Education Resources Information Center - This internet-based digital library of education research and information is sponsored by the Institute of Education Sciences (IES) of the U.S. Department of Education and provides a comprehensive, easy-to-search database of education research and information.
What Works Clearinghouse - This clearinghouse provides access to scientific evidence on education programs, products, practices, and policies - identifying high-quality research to answer the question "what works in education?".
Evidence For ESSA - This website provides a free, authoritative, user-centered database to help schools, districts, state leaders, teachers, parents, or concerned citizens easily find programs and practices that align to the ESSA evidence standards and meet their local needs.
Blueprints For Healthy Youth Development - This website provides a comprehensive registry of scientifically proven and scalable interventions that prevent or reduce the likelihood of antisocial behavior and promote a healthy course of youth development and adult maturity.
Pennsylvania Evidence Resource Center - This collection of evidence-based strategies was curated to help educators make evidence-based decisions and implement strategies, interventions, and activities that are best fit for their needs and local context.
Ask-A-REL - Midwest Region - Connect with Regional Educational Lab (REL) Midwest to learn from their work with key stakeholders in Illinois, Indiana, Iowa, Michigan, Minnesota, Ohio, and Wisconsin to develop evidence to inform consequential decisions about policy, programs, and practice.

Develop A Logic Model

A logic model is a visual way of showing how you anticipate that implementing the evidence-based strategy will ultimately lead to improved results. Find toolkits and guides for developing yours here.

EBITE Logic Model Guide - This practical guide is great for learning what Logic Models are and how important they are to designing and implementing school or classroom practices/programs. Examples used to demonstrate how to design your own Logic Model.
W. Kellogg Logic Model Guide - This guide provides an orientation to the underlying principles and language of the program logic model so it can be effectively used in program planning, implementation, and dissemination of results.
Pell Institute Evaluation Toolkit: Using a Logic Model - This interactive toolkit provides an overview of the purpose of a logic model and its core elements.

Plan To Evaluate

Whether you're using internal resources to carry out an evaluation, or workng with an external research partner, thoughtful planning is critical for ensuring that evaluations yield actionable information that's helpful for meeting students’ needs. Here are some tools to help you with this important planning process.

Forum Guide to Supporting Data Access for Researchers: A Local Education Agency Perspective - This publication recommends a set of core practices, operations, and templates that can be adopted and adapted by LEAs as they consider how to respond to requests for both new and existing data about the education enterprise.
National Network of Education Research-Practice Partnerships Clearinghouse - The Knowledge Clearinghouse provides relevant, up-to-date resources curated specifically to facilitate learning of education research-practice partnerships.
Research-Practice Partnerships: A Strategy for Leveraging Research for Educational Improvement in School Districts - This white paper assesses the predominant types of research-practice partnerships in education—and the benefits and challenges of each—so those seeking to form or fund such a group can make informed decisions about how best to do so.

Additional Resources

There are many ways that districts can work with research partners to learn more about what is working. Check out these resources to learn more about developing research partnerships and data sharing.

Turning Data into Information: The Vital Role of Research in Improving Education - This document provides school districts guidance on sharing student-level data while protecting student privacy.
Data Drives School-Community Collaboration - This resource is designed to help leaders implement complex data partnerships, with seven key lessons about how to begin and grow a data-driven initiative with schools.
Ohio Department of Education: Data Sharing for Program Evaluation - This document provides school districts guidance on sharing student-level data while protecting student privacy.
National Center for Education Statistics Forum Guide to Education Data Privacy - This is resource for state and local education agencies to use in assisting school staff in protecting the confidentiality of student data in instructional and administrative practices.

Frequently Asked Questions

Defining the Levels of Evidence

The Every Student Succeeds Act (ESSA) (Section 8002) and the U.S. Department of Education's NonRegulatory Guidance: Using Evidence to Strengthen Education Investments outline four levels of evidence: Level 1 ("Strong"), Level 2 ("Moderate"), Level 3 ("Promising") and Level 4 ("Demonstrates a Rationale").

The table below highlights the key attributes of the ESSA evidence levels. More information about the definitions of levels of evidence is available on the Defining ESSA Levels of Evidence page.

	Level 1 STRONG	Level 2 MODERATE	Level 3 PROMISING	Level 4 DEMONSTRATES RATIONALE
STUDY DESIGN	Experimental (RCT)	Quasi-experimental (propensity matching, regression discontinuity, natural experiment)	Correlational with controls for selection bias	Logic model based on solid research
RESULTS	Statistically significant positive effect	Statistically significant positive effect	Statistically significant positive relationship	Study planned or underway
SAMPLE	350+ students, multiple sites	350+ students, multiple sites	-	-
MATCH	Similar population and similar setting	Similar population or similar setting	-	-

What are Level 4 Evidence-Based Strategies?

ESSA defines Level 4 evidence-based strategies as:

Demonstrating a rationale based on high-quality research or positive evaluation that such activity, strategy or intervention is likely to improve student outcomes or other relevant outcomes.

What does this really mean?

Level 4 evidence-based strategies have demonstrated positive outcomes in early studies, but the statistical rigor and outcomes of existing research does not yet meet that required by Levels 1 ("Strong"), 2 ("Moderate") and 3 ("Promising"). There are three related reasons why this might be the case:

The strategy is new and has not yet been evaluated.
Existing studies of the strategies to not meet the requirements for levels 1, 2 or 3. (See Defining ESSA Levels of Evidence: An Overview for more information about the evidence requirements.)
Existing studies do not address your district or school’s specific needs or local characteristics.

When Can Level 4 Evidence-Based Strategies Be Used?

There are cases where districts and schools will be required under ESSA to use Level 1 ("Strong"), Level 2 ("Moderate") or Level 3 ("Promising") evidence-based strategies, such as when interventions are being applied under Title I, Section 1003 (School Improvement). Schools and districts completing the One Needs Assessment and One Plan should refer to the guidance provided for their specific context.

A lack of formal evidence does not mean that a Level 4 strategy will not be successful; it means that the evidence is not there yet. There are many opportunities for districts and schools to use Level 4 strategies. Level 4 encourages innovation and provides schools and districts with opportunities to try new approaches to meeting the needs of their students. A Level 4 strategy may be the best fit for a particular context. Importantly, to implement a Level 4 strategy, you must demonstrate a rationale for why you think the strategy will be successful by developing a logic model and committing to measuring your results.

What are the Three Steps to Using Level 4 Evidence-Based Strategies?

Step 1 – Explore Existing Research

The first step in determining whether a strategy “demonstrates a rationale” is exploring existing research to determine whether there is good cause to believe the strategy will address your needs.
Here are some tips to help you get started:

Tip: Focus on fit. A higher evidence level does not necessarily mean that a strategy is right for you. Implementing a Level 4 strategy with fidelity and committing to measuring results might get you the outcome you are seeking.
Tip: Pay attention to the research setting and the characteristics of the students and schools that were studied. Ask yourself:
- Was this research conducted in a setting like mine?
- Do the results lead me to believe that it could be effective with my students?
Tip: Visit the Tools and Resources section to get started on your exploration.

Step 2 – Develop a logic model

A logic model is a visual way of showing how you anticipate that implementing the evidence-based strategy will lead to improved results.
Going through the process of creating a logic model is more than just an exercise. The process can clarify objectives, generate important questions about assumptions and expectations and provide a roadmap for developing a robust program evaluation.
There are many existing variations on logic models. Visit the Tools and Resources section for detailed guidance and templates that you can use.

Step 3 – Plan to Evaluate

When you use a Level 4 evidence-based strategy, you are committed to evaluating how that strategy works within your context. If designed appropriately, the evaluations you carry out will:
- Show you whether the strategy is improving outcomes as intended;
- Shed light on reasons why a strategy is not working as intended; and
- Contribute to the existing evidence base associated with the strategy.
Some school districts have research and data analysis units that can design and carry out program evaluations. In other cases, districts may want to use external research partners to help design and conduct evaluations.
The Tools and Resources section has information that will help you as you develop and implement your evaluation plan.

Knowledge Exchange

Connect

Learn more about Level 4 projects and related best practices in districts around Ohio.

We want to hear about what's happening with Level 4 work - successes and challenges - in your building or district.

Explore

Check out Level 4 strategies in Ohio's Evidence Based Clearinghouse.