Level 4 Evidence-Based Strategies

Frequently Asked Questions

---

Defining the Levels of Evidence

---

The Every Student Succeeds Act (ESSA) (Section 8002) and the U.S. Department of Education's NonRegulatory Guidance: Using Evidence to Strengthen Education Investments outline four levels of evidence: Level 1 ("Strong"), Level 2 ("Moderate"), Level 3 ("Promising") and Level 4 ("Demonstrates a Rationale").

The table below highlights the key attributes of the ESSA evidence levels. More information about the definitions of levels of evidence is available on the Defining ESSA Levels of Evidence page.

	Level 1 STRONG	Level 2 MODERATE	Level 3 PROMISING	Level 4 DEMONSTRATES RATIONALE
STUDY DESIGN	Experimental (RCT)	Quasi-experimental (propensity matching, regression discontinuity, natural experiment)	Correlational with controls for selection bias	Logic model based on solid research
RESULTS	Statistically significant positive effect	Statistically significant positive effect	Statistically significant positive relationship	Study planned or underway
SAMPLE	350+ students, multiple sites	350+ students, multiple sites	-	-
MATCH	Similar population and similar setting	Similar population or similar setting	-	-

What are Level 4 Evidence-Based Strategies?

---

ESSA defines Level 4 evidence-based strategies as:

• Demonstrating a rationale based on high-quality research or positive evaluation that such activity, strategy or intervention is likely to improve student outcomes or other relevant outcomes.

What does this really mean?

• Level 4 evidence-based strategies have demonstrated positive outcomes in early studies, but the statistical rigor and outcomes of existing research does not yet meet that required by Levels 1 ("Strong"), 2 ("Moderate") and 3 ("Promising"). There are three related reasons why this might be the case:
1. The strategy is new and has not yet been evaluated.
2. Existing studies of the strategies to not meet the requirements for levels 1, 2 or 3. (See Defining ESSA Levels of Evidence for more information about the evidence requirements.)
3. Existing studies do not address your district or school’s specific needs or local characteristics.

When Can Level 4 Evidence-Based Strategies Be Used?

---

There are cases where districts and schools will be required under ESSA to use Level 1 ("Strong"), Level 2 ("Moderate") or Level 3 ("Promising") evidence-based strategies, such as when interventions are being applied under Title I, Section 1003 (School Improvement). Schools and districts completing the One Needs Assessment and One Plan should refer to the guidance provided for their specific context.

A lack of formal evidence does not mean that a Level 4 strategy will not be successful; it means that the evidence is not there yet. There are many opportunities for districts and schools to use Level 4 strategies. Level 4 encourages innovation and provides schools and districts with opportunities to try new approaches to meeting the needs of their students. A Level 4 strategy may be the best fit for a particular context. Importantly, to implement a Level 4 strategy, you must demonstrate a rationale for why you think the strategy will be successful by developing a logic model and committing to measuring your results.

What are the Three Steps to Using Level 4 Evidence-Based Strategies?

---

Step 1 – Explore Existing Research

• The first step in determining whether a strategy “demonstrates a rationale” is exploring existing research to determine whether there is good cause to believe the strategy will address your needs.
• Here are some tips to help you get started:
• Tip: Focus on fit. A higher evidence level does not necessarily mean that a strategy is right for you. Implementing a Level 4 strategy with fidelity and committing to measuring results might get you the outcome you are seeking.
• Tip: Pay attention to the research setting and the characteristics of the students and schools that were studied. Ask yourself:
  • Was this research conducted in a setting like mine?
  • Do the results lead me to believe that it could be effective with my students?
• Tip: Visit the Tools and Resources section to get started on your exploration.

Step 2 – Develop a logic model

• A logic model is a visual way of showing how you anticipate that implementing the evidence-based strategy will lead to improved results.
• Going through the process of creating a logic model is more than just an exercise. The process can clarify objectives, generate important questions about assumptions and expectations and provide a roadmap for developing a robust program evaluation.
• There are many existing variations on logic models. Visit the Tools and Resources section for detailed guidance and templates that you can use.

Step 3 – Plan to Evaluate

• When you use a Level 4 evidence-based strategy, you are committed to evaluating how that strategy works within your context. If designed appropriately, the evaluations you carry out will:
  • Show you whether the strategy is improving outcomes as intended;
  • Shed light on reasons why a strategy is not working as intended; and
  • Contribute to the existing evidence base associated with the strategy.
• Some school districts have research and data analysis units that can design and carry out program evaluations. In other cases, districts may want to use external research partners to help design and conduct evaluations.
• The Tools and Resources section has information that will help you as you develop and implement your evaluation plan.