Step 5 - Examine, Reflect, Adjust

Step 1
Step 2
Step 3Step 4Step 5

Click on Graphic above to Navigate to other Steps

First, you and your team carried out a needs assessment that helped you identify and prioritize critical needs. Then, you identified an evidence-based strategy that would help you address those needs.

What happens next?

A deliberative, systematic and data-driven needs assessment process equipped you with objective information to identify and prioritize critical needs. Similarly, a deliberative, systematic and data-driven evaluation plan will give you objective insights on whether the strategy is working and is truly helping you address your critical needs.

Just as there is no one-size-fits-all approach to meeting local needs, there is no one-size-fits-all approach to evaluating whether an evidence-based strategy is working for you. Where do you start and what are key considerations to keep in mind?

First and foremost, plan ahead. It is never too soon to think about how you will know whether your evidence-based strategy is working.

You already started laying the groundwork for evaluation when you carried out your needs assessment. As part of your needs assessment, you and your team talked about where you currently are on an outcome and compared it to where you want to be.

This means that you have already thought about baseline data and end goals; what you need now is a plan to understand how the evidence-based strategy is affecting progress towards your goals.

In planning ahead, the questions you ask yourself should address the following:

Much like carrying out a needs assessment, evaluating whether your evidence-based strategy is working is a team effort. As you think about the team you want to assemble to carry out this work, consider the following:

Stakeholders
  • Who are your key stakeholders in this work?
  • How will you communicate preliminary, interim, and final results?
  • Do your stakeholders have a clear understanding of what you are trying to accomplish and how you will get there?
Internal Team
  • Who are the key internal staff who will be able to help you understand how the evidence-based strategy was implemented?
  • Who will help you communicate key findings about the evidence-based strategy to the stakeholders?
  • Do your internal team members have a clear understanding of what you are trying to accomplish and how you will get there?
  • How will you use your existing District Leadership Teams (DLTs), Building Leadership Teams (BLTs) and Teacher-Based Teams (TBTs) in the Examine, Reflect and Adjust process?
Data Analysis & Evaluation
  • Can your internal team carry out all or part of the data collection, analysis and evaluation work?
  • Are there external partners who can help you analyze your data and evaluate whether your evidence-based strategy is working?

Having the answers that you need when you need them will put you in a better position to make programmatic, staffing, scheduling, budgetary and other operational decisions. Consider the following:

Short- and Long-Term Targets
  • How long would you expect it to take before your evidence-based strategy should improve your critical needs?
  • Are there points along the way where you will need or want to understand progress towards your goals?
Data Collection & Analysis
  • How often will you collect and analyze data?
  • How will your cycle of data collection affect your ability to answer your short- and long-term evaluation questions?
  • How often will you share results with your stakeholders?
  • How often will you share results with your internal team?
Opportunities for Course Correction
  • Will the timing of your data analysis allow you to reflect and adjust regularly throughout implementation?

The data that you used to identify your critical needs can serve as a great starting point for your program evaluation. You may find that you will need to supplement this data with additional information – especially when it comes to understanding implementation. Consider the following:

Types of Data
  • What quantitative data (e.g., enrollment counts; attendance rates; test scores) will you use in your evaluation?
  • What qualitative data (e.g., focus groups; interviews) will you use in your evaluation?
Progress Towards Goals
  • If it will take time for your evidence-based strategy to impact your long-term goals, what data will help you understand if you are on the right track?
Data Access
  • Will the people carrying out your evaluation – whether internal staff or external partners – have access to the right data?
  • If not, what steps will you need to take with your legal, information technology, or other staff to ensure access?

Data analysis gives you an objective lens through which to understand whether your evidence-based strategy is working. Data alone rarely gives us a clear-cut answer on next steps in implementing strategies to improve outcomes. Once you start to see results from data analysis and program evaluation, consider the following questions:

Review and Understand Results
  • Where the results of your evaluation surprising to the team? Why or why not?
  • Are there important contextual pieces of information that can help shed light on the findings?
Address Implementation
  • As you review the results of your evaluation, what do you know about whether implementation was fully, partially or not successful? Why was implementation fully, partially or not successful?
  • Are there steps district, building or classroom leaders can take to improve fidelity of implementation?
Adjustment vs. Changing Strategies
  • Do your evaluation results - and your team’s discussion about results, context, and implementation – imply that you should continue, improve the implementation of, or stop doing the evidence-based strategy you selected?
  • If you opt to change implementation, how will you communicate those changes?
  • How will you test whether the changes to implementation improved outcomes?

There are many ways that districts can work with research partners to learn more about what is working. Resources to help you learn more about developing research partnerships include the following:

Engaging in research partnerships sometimes involves data sharing with external partners. When planning to share data with research partners for evaluation purposes, districts should always start by talking with their legal and IT departments. As you work with staff across your district to develop a data sharing plan, resources than can help you develop that plan include:

Data Sharing for Program Evaluation: Guidance on Sharing Student Level Data While Protecting Student Privacy

National Network of Education Research-Practice Partnerships

Ohio Improvement Process Step 5: Examine, Reflect, Adjust