Assessing ‘evidence-based’ claims
Read or print pdf here.
Assessing the quality of PLD offerings, literacy programmes, and teaching resources
How do we know what is good? And what to avoid?
The availability and promotion of various professional learning and development offerings, curriculum programmes, and supporting resources in New Zealand—all of which claim to be evidence-based—have increased rapidly in tandem with the recent surge in interest in the science of reading and structured literacy teaching approaches.
We are fortunate in New Zealand to have many high-quality structured literacy PLD providers, programmes and teaching resources. But it would be naïve (and negligent) of us to accept the evidence-based claims of providers or sellers of resources without doing some due diligence to assess how credible any claims are. The same goes for any school claiming to be following evidence-based teaching approaches.
So what steps can a school leader take when deciding what investments to make? And how can parents question schools about their literacy instruction?
Evaluating programmes - for school leaders
Currently, there are two problems that make assessing PLD offerings and literacy programmes extremely challenging:
In New Zealand, there is not a single, reliable source where school administrators can go to obtain unbiased, high-quality information about all of these options, including what they cover and how rigorously they engage with the body of research in reading science.
The only thing that is available right now is our website's list of PLD providers that, in our opinion, are broadly in line with the science of reading and a structured literacy teaching approach. But there is no depth of information on any of the offerings or evaluation of structured literacy curricula materials, or a way to keep tabs of changes in the offerings.
That is why, one of the main suggestions in our Draft Literacy Policy White Paper we put out in December is to establish a trusted source, like an independent Science of Reading Accreditation process, and an information portal where schools can readily access this information. An example of this for programmes is the Reading Leagues’ Curriculum Navigation Reports currently in development or the Mississippi Department of Education Instructional Materials Evaluation Rubrics.
Research on the efficacy of the different PLD offerings, programmes, and instructional materials used in New Zealand schools is sadly minimal.
The Better Start Literacy Approach (BSLA), created and administered by the University of Canterbury, is the sole subject of research to date. BSLA is fortunate to have received almost $20 million in government funding, and as a university, it possesses the necessary resources and knowledge to carry out this kind of research. The expenses associated with conducting research of this calibre are simply too high for non-University based PLD providers.
The quality of the implementation and embedding of any programme is paramount. Feedback from teachers and principals on implementation post-training needs to be part of ongoing research. There must also be a focus on tracking and objectively measuring changes in teacher knowledge and teaching efficacy, along with tracking student achievement. ERO could potentially have a role here.
We therefore recommend that the government provide funding for independent studies that examine the efficacy of all PLD providers who wish to obtain accreditation for their services or for a particular structured literacy programme.
What can we use to assess quality in the meantime?
Until such accreditation and research mechanisms proposed above are established, we advise school leaders to make every effort to assess the various offerings using the Reading League’s Curriculum Evaluation Guidelines (updated in 2023).
These are more useful for assessing curriculum materials or programmes. However, they could be adapted to assess the content of PLD offerings as well.
They focus on identifying ‘red flags’ - practices that are less effective and not aligned with the science of reading.
They come with a handy Reviewer’s Workbook.
The Reading League recommends that any review team include people who have been trained in the science of reading.
They point out that there is no perfect curriculum or programme. There will always be a need to adapt and/or supplement.
They make it clear that while evidence-aligned materials are an essential component of successful literacy instruction, it is equally important to build programme-agnostic educator and leader knowledge in the science of reading and to develop a coaching system to support implementation. And then, for the greatest impact, it is essential to ensure practices are aligned across all systems (i.e., across year levels and tiers of support).
Finally, they offer this rule of thumb:
“If 85% of students are not achieving proficiency in foundational skills with a Tier 1 curriculum as measured with a reliable and valid curriculum based measure, the first assumption should be that something vital is lacking in the program or its implementation, not the students”.
Evaluating PLD offerings
As for assessing PLD offerings, we recommend that you ask yourself the following questions in addition to applying the programme guidelines mentioned above to assess the content of any PLD offerings:
What are the qualifications of the people delivering the PLD? What is the depth and breadth of their knowledge of the science of learning, and the principles and components of structured literacy approaches? Do they have a depth of experience in supporting students with the most challenging needs, those requiring tier 3 instruction?
What research evidence do they have behind the content of their training? Do they offer a whole-school approach that will align practice across year levels and all tiers of intervention?
What components of structured literacy instruction do they cover? Will you need to supplement with training from another provider?
Does their training focus on all tiers of instruction, tier 1 whole class through to tier 3, intensive one-on-one intervention support? How are teachers supported to differentiate instruction based on needs of class and individual students?
How do they stay up-to-date with advances in the science and evidence-base? And how do they promulgate any changes this might mean to teaching practice for any programmes they may also provide/support?
How much of the training is focused on building teaching knowledge (of the science of reading and a structured literacy approach) versus training on a specific structured literacy programme or set of curriculum resources?
What coaching support do they offer during training, as well as following training for embedding practice in the school?
We also recommend school leaders speak to other schools about their experiences with different offerings. There are now many schools in New Zealand that are well down the track of implementing and embedding structured literacy school wide. We suggest taking a look at our Map of Structured Literacy schools to see if there is one near you. Most are more than happy to share their experience and allow you to come and observe teaching in classrooms.
Sustaining and embedding change
In terms of sustaining and embedding change across the school, it is critical that school leaders do the training so that they are knowledgeable about the science and evidence and can lead from a place of knowledge. They also need to come up with a carefully considered implementation roadmap where they plan the focus for each term across a 2-3 year implementation time horizon. And then they need to regularly review and evaluate the effectiveness of the implementation and make adjustments as required.
The change can feel daunting or even overwhelming at times. This blog and self-reflection tool from Jocelyn Seamer provides tips on creating a positive mindset and making small changes so that the change journey does not seem so daunting. This can be used as an individual teacher, as a team, or as a whole school. This self-reflection can help in the formulation of a whole of school change implementation roadmap.
For parents worried about reading instruction in your child’s class
If you are a parent and are worried about the reading instruction in your child’s class but don’t know what questions to ask, then this guide from the Right to Read Project has an incredible set of 5 checklists to help you assess the quality of their reading instruction.
The checklists provide you with questions to ask the school and then what might be encouraging signs or signs indicating there might be a problem. These checklists cover the big 5 areas identified by the US National Reading Panel review of over 100,000 research studies. They include: phonemic awareness, phonics, fluency, vocabulary, and comprehension. You could also inquire about their explicit teaching of oral language.
