Features of Good Educational Research
In this blog, Professor Emeritus of Education Psychology at Massey University, James Chapman, sets out the features of good educational research to help you sift the wheat from the chaff when reviewing the findings and claims made in different studies.
Download the blog: here.
What is Educational Research?
Educational research involves a wide range of purposes and methodologies. The purpose and methodology depend on the research “problem” and research question(s). Although educational researchers mostly have a preferred methodology (e.g., quantitative/statistical; qualitative; case study; ethnographic), the methodology for any particular study should be appropriate for attempting to answer the research question(s). For example, if we wish to be able to draw generally applicable conclusions from a study then a quantitative randomised control trial is the gold standard approach.
Furthermore, it’s important that we are careful with the inferences or conclusions that can be drawn from a particular study. For example, claims that a particular approach or intervention are ‘successful’ need to be seen in the context of: firstly, what do we mean by ‘success’, secondly, successful in comparison to what other alternative; and finally, successful for how long. The main point being that not all evidence is of equal strength or value and the following guidelines will help you determine the strength and value of those claims of ‘success’.
How do I know what is high quality research?
High quality educational research should be publishable in peer reviewed journals. Blogs and opinions are generally not based on research (although they may refer to research elsewhere) and are not peer-reviewed by scholars and researchers in the field.
While the peer review process isn’t perfect, it does provide some safeguard that a particular study has been carried out appropriately, in terms of the research problem and research question(s), and that the conclusions are based on the actual findings.
Following are some points to consider when reading, or reading about educational research:
Is the research published in a reputable journal or in a book from a reputable publishing company that uses peer review?
This is not easy for non-researchers to navigate! Knowing what constitutes a reputable journal or publishing company takes years of experience. In general, reputable educational journals tend to be ones that belong to membership organisations, or run by highly regarded publishing companies which don’t charge authors for the cost of publishing their research. Examples include but are by no means limited to the following, especially regarding literacy:
American Psychological Association (e.g., Journal of Educational Psychology);
Society for the Scientific Study of Reading (Scientific Studies in Reading);
American Educational Research Association (e.g., Review of Educational Research);
Learning Difficulties Australia (e.g., Australian Journal of Learning Difficulties);
Springer Publishing (e.g., Reading and Writing);
British Psychological Society (e.g., British Journal of Educational Psychology);
International Literacy Association (e.g., Reading Research Quarterly).
Does the journal have a solid peer review process that involves an editorial board of recognizable experts in the field?
Most journals publish online the names of their editors and editorial board members as well as reviewers.
Where do experts in the field publish their results?
Researchers will publish in different journals or by invitation in book chapters. Where to publish depends on the desired audience and purpose of the study. Looking at where top researchers publish their results can provide a good guide as to the quality of their research. But top researchers will also publish in lesser journals, often to reach a wider audience.
Is the methodology used determined by the research question(s) and research purpose?
If the purpose of a particular study is to be able to generalise beyond the sample in the study, then a good study will involve a reasonably large sample with results analysed by means of various statistical procedures. There is seldom a magic number for a “good” sample size; it depends on many factors and most statistical procedures take sample size into account. This usually means that the smaller the sample, the bigger the effect needs to be in order for it to be statistically significant. And statistical significance isn’t the be all and end all of research. Some results might be statistically significant but educationally non-significant!
Are the results able to be generalised beyond the research participants?
Qualitative and case study research have an important place in educational research but cannot provide the basis for generalising the results beyond the research participants.
A full description of the research sample will help readers to decide whether the results might apply to another similar group with which they are familiar.
If the study is designed to test whether a particular intervention works better than “business as usual”, has a “randomised control trial” been used?
A randomised control trial is the ideal research design for studies with this purpose. Participants are randomly assigned to the intervention/trial group and to the control/comparison group. Although desirable for being able to control for all sorts of possible confounding or extraneous factors that could bias or nullify results, this is difficult to achieve in education, for ethical and sometimes practical reasons. Where approval (from ethics committees) is given to conduct a randomised trial, there’s usually a requirement that if the outcome of the intervention/new method is successful, the same approach will be provided to those in the control group, if this is desired or requested. It’s also important to realise that even if the results suggest a positive contribution from the intervention, this does not necessarily mean that intervention is the best intervention there is out there.
Does the research provide a good theoretical or conceptual framework?
Theory helps us to interpret and understand research findings. As a good mentor and colleague of mine said years ago, “there is nothing more practical than a good theory”!! A good study may find support for the results from a particular theory (e.g., the Simple View of Reading), or may question the theoretical framework of a particular teaching approach (e.g., whole language, Reading Recovery). Be wary of articles that claim results “prove” a particular theory or point of view. At best, we find results in support of our theoretically derived hypotheses; not proving them.
Do the researchers state what the limitations of their research are?
This is a requirement in many journals. Journal reviewers pay attention to whether or not the authors have honestly drawn attention to any shortcomings or limitations in the research. Such limitations will not necessarily undercut the research findings. There’s little “perfect” educational research around. But knowing what limitations may have been associated with a particular study is important.
A lot of good quality educational research involves "quasi-experimental" designs.
Such research usually studies existing groups, such as students placed in Reading Recovery, compared to similar students who have been wait-listed. The groups are naturally formed rather than experimentally formed. As long as the groups are reasonably similar at the start of an intervention (e.g., Reading Recovery) or a new teaching approach (e.g., children taught with decodable texts compared to meaning-based texts), the findings can be very informative.
Correlational research also forms a large and legitimate part of educational research.
To try to tease out causes of outcomes, correlational studies should ideally be conducted longitudinally over time. For example, such research has shown in many studies that children's alphabet knowledge on entry to school is significantly predictive of reading comprehension ability years later. Similarly, there is ample evidence to show that academic performance is significantly correlated with academic self-concept, with performance generally leading to the development of self-concept.
In Summary:
Educational research is complex. False findings (fake news) can be abundant. While it’s not necessary for consumers of research to fully understand the research process, the above guidelines might help to sift the wheat from the chaff.
Declaration of Viewpoint
My Masters thesis research at Victoria University of Wellington, and PhD research at the University of Alberta were conducted within a quantitative framework. My research training was largely based on quantitative approaches involving various statistical procedures. This has guided my research over 40+ years. Nonetheless, I have supervised PhD research that is predominantly qualitative in design, and as a reviewer for many journals over the years I have reviewed both qualitative and quantitative studies, and increasingly in recent years, studies that use a mix of both these approaches.