The art of critical engagement when using academic research and evidence
Dr James Rogers FCCT
Director of Jim Rogers Training. Research and CPD Lead for Teaching School Council SW.
In this article I will begin by stating that I believe we should be a research and evidence-informed profession, rather than a research or evidence-led profession. The latter implies a passive engagement, while the former implies an active engagement and this, I argue, comes from critical engagement, asking the right questions of research and understanding what it can and cannot offer.
This process, I suggest, is one of the fundamental skills required for both effective leadership and teaching. Expertise developed in the process of teaching, and in the school and local community, is a crucial element of this and I will touch on ‘craft knowledge’ (Wilson et al., 2013). However, effective leadership and teaching also requires a level of reflection and critical engagement of this craft knowledge and one’s own practice. It requires an open and challenging mind, not accepting the status-quo, and being precise about the identification of needs and issues, whilst also being acutely aware of the cultural context to ensure effective implementation and impact.
I believe the opportunities to develop practice informed by research and evidence are currently limited. We have a traditionally very short period of training before becoming a teacher and no further mandatory qualifications. Once in schools, workload provides a huge barrier to accessing educational research and there is often a gap between ‘research knowledge’ and ‘everyday practice knowledge’ (MacIntyre, 2005 in Wilson, 2013 p.2). However, I cannot remember an era in education when there was so much on offer to the profession in terms of research and ‘expertise’, and much of this is now more accessible to the busy professional. Indeed, I believe we are in a golden age for a research-rich profession. While this may be the case, the system is ‘noisy’ and there are a wide range of organisations and individuals making bold claims about strategies and ‘interventions’ that ‘work’ and some of these present conflicting messages. Schools are very complex social organisations and therefore educational research is very complex, and while headlines referring to the next best thing catch the eye, the applicability and universality of the findings, I argue, needs to be context specific.
It concerns me that, while we appear to be operating in a ‘research-rich’ era, some schools struggle to develop practice and can focus on initiatives and approaches that are not appropriate for their needs or context. And while schools use data to measure performance, I believe there has been an over-emphasis on measuring ‘how much’ rather than exploring ‘how’, the process of understanding exactly how something works in order to develop practice. This, I believe, is partly down to a lack of critical engagement and challenge from the profession.
Defining (academic) ‘research’
I believe it is important to understand what research can, and cannot, bring to the profession, and this begins with understanding what research is. There is a discipline and rigour involved in academic research, and that is because research is the process of creating new knowledge and insights, in order that we can better understand something and potentially improve or control something. It can only be achieved when a specific, rigorous (and ethical) approach is applied. Findings tend to be published through peer-reviewed academic journals, peer review being an important part of the process where experts in the same field check the research validity and suitability for publication. Research design needs to account for the research question and how best to explore and answer this (Rickinson, 2005), and is informed by the theoretical perspective of the researcher. I will not go in to ontological or epistemological perspectives other than to say that what we define as ‘truth’ (is truth interpreted or is it absolute?) is important in how we determine research design, and that how we seek to gather and interpret evidence plays a huge part in how confidently we can draw conclusions. In other words, is the new knowledge the research has illuminated based on a reliable and valid research approach, and has the author been transparent about how and why the research was conducted in the way that it was?
In defining research, we also ought to acknowledge there are different research approaches, determined by the nature of the research question and the theoretical perspective of the researcher. In the CCT’s Compact Guides, Jones and Netolicky (2019) present a hierarchy of research. In this, different approaches to research are ranked from meta-analysis to cross-sectional surveys. This is a useful starting point, although I would contend that ranking is perhaps not the best way to look at research, rather one should understand further the opportunities and limitations of each approach. For example, a case-study can allow a researcher to really drill down into a particular research area, and while findings may be very context specific, they allow for a much deeper, nuanced understanding and insights that cannot be easily seen when researching on a large scale. Conversely, meta-analysis (see for example John Hattie’s Visible Learning) can provide a useful overview at scale and indicate what might work with greater confidence. Similarly, randomised control trials adopt a traditionally medical approach to research (often used in clinical trials with placebos), exploring whether an ‘intervention’ works, at scale, next to a ‘control’ (see, for example, Education Endowment Foundation’s work). Each approach is suited to the research question or focus and tells part of a story. Each is also context specific according to the research focus, scale, age groups, whether it is longitudinal (over time) and so on.
Defining ‘craft knowledge’
The other form of knowledge that I have mentioned is ‘craft knowledge’ or ‘everyday practice knowledge’ (Wilson, 2013). This is the knowledge we, as schoolteachers and leaders, develop through the experience of our roles. This is knowledge of our practice, our children, our school community, and the local community. Wilson (2013) describes this as:
‘…hard to verbalize (sic) because it is expressed through action-based skills, is difficult to make explicit or to represent in textual form because it is largely acquired informally through participation in teaching situations, and it is often so ‘taken for granted’ that teachers are often unaware of its influence on their behaviour’ (Wilson, 2013, p2).
This ‘craft knowledge’ is important in the day-to-day operational activity within the school. But, as Weinstein and Sumeracki (2019) suggest, teaching is often more informed by intuition than research, and this intuition can lead us towards practices that have little impact on efficacy or outcomes. An additional problem with relying on intuition is how we can positively reinforce intuition by ‘our tendency to seek out information that support rather than disproves our beliefs’ (Weinstein and Sumeracki 2019 p12). Furthermore, I believe it has allowed approaches and techniques, many of which ‘feel right’ or ‘appear logical’ but have had no impact on learning. If decisions are only based on experience and intuition, we run the risk of not challenging our thinking and perspectives, and recycling practice without considering alternatives. We can promote poor practice that positively reinforces poor practice.
Weinstein and Sumeracki (2019) argue that we should put intuition to one side and be led by the research. Their research approach has been to reduce teacher effectiveness to a series of laboratory experiments in order to seek causal effects between teaching approach and learning outcomes, then upscale these approaches within school settings. However, while some might argue that we should ignore teachers’ intuition entirely and rely solely on what the research tells us, I argue that just as with ignoring the limitations and context of research studies and evidence, ignoring the experience and knowledge of one’s school contexts will also limit how successful implementation and impact will be.
How to critically engage
I believe there is a crucial space that exists between the profession and the research, and this space is best utilised when the profession critically engages with both their own context, and that of research. What follows are, some suggested considerations when critically engaging with research and evidence, and some sources of useful guidance to support this and planning and implementing change.
- Understand the research context: who funded and published the research? What was the approach, scale, country, demographic, timescale and research approach?
- Where is the research published? Academic journals are peer-reviewed adding a layer of integrity. Popular media, blogs etc have little external verification other than the ‘experts’ that use the comments section.
- Are the claims too good to be true? As with anything, there can be an incentive for headlines.
- Is there more than one source of evidence and/or research saying the same thing? Researchers use triangulation to cross-reference evidence before conclusions are drawn. It is a useful approach when critically engaging with research.
In considering application and implementation, the next stage is to consider your context:
- How secure are you that ‘x’ is the issue that needs to be addressed – are more than three sources of evidence indicating the same issue?
- Are you and your staff open to new ways of working and thinking?
- How likely is this approach going to work in your context / setting?
- Are there cultural, contextual, financial, political factors that need to be considered?
A good example of how the research context is explored can be seen in Coe’s article about the extent to which retrieval practice can be translated into classroom practice (Coe, 2020). Coe carefully explores the context and limitations of the research that underpin retrieval practice and while not challenging the approach, he does highlight that translating it to the classroom is more nuanced than perhaps we might assume.
Image from ‘That’s a Claim’ (permission for use granted 1st May 2020)
Informed Health Choices is an international multidisciplinary group that has published a framework for ‘thinking critically about claims, evidence and choices’ called ‘That’s a Claim’. The framework enables the careful questioning of research and evidence in a systematic way offering three stages to the questioning. They provide a useful poster for educationalists to use when considering research and evidence and interestingly there are other fields covered from agriculture to health.
In considering effective implementation and increasing the chances of positive impact on practice and hopefully learning, the Education Endowment’s Guidance on Implementation (2019) and Jones and Netolicky’s Research-informed Practice (2019) are two really useful sources of guidance to help provide a framework for planning and implementing change.
In this article I have made the case for the teaching profession to be research and evidence-informed rather than led. For this to be effective, and for research to meaningfully impact on practice, the process of critical engagement, I have argued, is crucial. As a profession we need to remain open-minded and prepared to continually learn and reflect, as research evolves and improves our understanding of what works and what does not. A healthy pinch of scepticism should be used, and an acceptance that ‘best practice is ‘best, not in an absolute sense, but in a comparative sense. It is the ‘best so far’’ (Syed, 2020 p.230).
Coe (2020) ‘Does research into retrieval practice translate into classroom practice?’ Impact (8): 12-13.
Education Endowment Foundation. Putting Evidence to Work – A School’s Guide to Implementation. Available at https://educationendowmentfoundation.org.uk/tools/guidance-reports/a-schools-guide-to-implementation/ (accessed 1 May 2020)
Informed Health Choices: That’s A Claim available at www.thatsaclaim.org/educational (accessed 1 May 2020)
Jones and Netolicky (2019) ‘Hierarchy of Evidence’. Chartered College of Teaching Compact Guide available at https://my.chartered.college/2019/05/research-informed-practice-the-hierarchy-of-evidence/ (accessed 1 May 2020)
Jones and Netolicky (2019) ‘Research-informed Practice. Chartered College of Teaching Compact Guide available at https://my.chartered.college/resources/compact-guides/research-informed-practice-by-gary-jones-and-deborah-netolicky/ (accessed 1 May 2020)
Rickinson, M (2005) ‘How to Plan Your Research’ NFER
Syed, M. (2020) Rebel Ideas, The Power of Diverse Thinking. Murray 01 edition.
Weinstein et al. (2019) Understanding How We Learn. A David Fulton Book
Wilson et al. (2013) School-based research. Second edition. London: Sage