The Research reflection guide helps education practitioners and policymakers reflect on a piece of research that provides evidence about the effectiveness of a particular policy, program or practice (that is, an approach), which they may be considering implementing.
Using the guide
First identify a piece of research evidence on a particular approach that you are considering implementing. Then, answer the series of guiding questions below that will prompt you to consider: what the research says; how relevant the research is to your context; whether you should implement the approach; and what you can do to ensure successful implementation.
The guide can be used individually or in a group as part of a community of practice.
- What does the research say?
What policy, program or practice (i.e. approach) is being evaluated? Where and when was this evaluation conducted? How many participants were involved?
- How was the approach evaluated?
What outcomes were looked at, and how were these outcomes measured? Was there a comparison between a group of individuals who experienced the approach and a group of individuals who did not experience the approach?
- What standard of evidence does this research meet?
Is the evidence generated by the research causal or correlational? Causal evidence shows that the approach caused a change in outcomes. Correlational evidence shows that use of the approach is associated with a change in outcomes, but doesn’t rule out the possibility that the change was caused by something else, or by chance.
- What connects with my experience?
What about the research is similar to my context and our current priorities? What aspects of the research are different to my context?
- What excites me about the research?
What might be possible in my context?
- What makes the approach work?
According to the research, what are the key features of the approach that led to improved outcomes? What resources and organisational conditions (financial, human, logistical, curricular etc.) enabled success?
- Would there be a benefit if I changed to this approach?
What am I currently doing? What would I have to change in order to adopt this approach? Given what the research says, would any of the changes I make lead to improved outcomes? By how much do I think outcomes would improve? Alternatively, am I already doing something very similar to the approach, such that any changes might not improve outcomes further?
- What adaptations would I need to make?
How aligned is this approach with existing system approaches? What about the approach will I need to change? Will any changes affect the key features? Will any adaptations make the approach less effective? Will any adaptations make the approach more effective?
- What is the cost, in time, effort, and/or other resources, of changing? What will it cost me and/or my students to change what I’m doing? Where will this time, effort or other resources come from? If I implement this approach, what would it replace? What would be the consequences on my students of replacing my existing approach?
Should I implement the approach?
Are the potential benefits worth the costs of implementation?
- How can I rally resources to support implementation?
How do I make implementation as smooth as possible? What resources and/or organisational supports do I need? How do I access these resources and/or supports?
- How will I be sure that implementation is effective?
What data do I need to collect to track the effects of implementation? How will I know that any changes will be due to implementation of this new approach and not anything else?
context (or contextual factors)
Context is the social, cultural and environmental factors found in research settings. Taking context into account in research studies is important because context can affect the outcomes of research (i.e. evidence generated in one context may not necessarily apply to a different context). Evidence is most relevant when it has been generated in a context similar to the context in which it will be applied. Examples of ‘context’ may include location, demographics of research participants, or the level of organisational support for the particular approach being researched.
Relevant evidence is evidence produced in contexts that are similar to one’s own context. Evidence can also be considered relevant when it is derived from a large number of studies conducted over a wide range of contexts.
Data is information that is collected and analysed in order to produce findings and/or to inform decision-making. Data can be qualitative (for example, teacher observations or quotes from students) or quantitative (for example, student test scores or attendance data).
An educational approach is effective if it causes (see causation above) a desired change in a particular outcome. This desired change can be an increase in an outcome (for example, increases in student achievement) or it can be a decrease in an outcome (for example, reduction in student absenteeism).
association (or correlation)
An association is when there is a relationship between two elements, factors or events, but the association cannot be proved or explained. Associations can be positive (for example, higher socioeconomic status is associated with higher student achievement) or negative (for example, higher student absenteeism is associated with lower student achievement).
Evaluation is the systematic and objective assessment of an approach. Evaluation provides evidence of what has been done well, what could be done better, the extent to which objectives have been achieved and/or the impact of the approach. This evidence can then be used to inform ongoing decision-making regarding the approach.
An outcome measure is an observation that can be used to measure the effect of a particular approach. Outcome measures can be qualitative (such as quotes or observations) or quantitative (such as test scores). For example, when examining whether a particular approach helps students understand a concept, a teacher could set an assessment. The student assessment score could then be used as an outcome measure of student understanding.
causation (or cause(s)/causal/causal evidence)
Causation is when one element, factor or event is known to cause another (for example, a particular teaching practice is known to lead to improvements in student test scores). To prove causation between two things (let’s call them A and B), researchers need to show: 1. that there is an association between A and B; 2. that A happens before B; and 3. that B is not caused by a third thing (that is, C or D). In education settings, proving causation is often challenging because of the many influences on teacher and student outcomes.