Ina V.S. Mullis, Michael O. Martin, and Jenny Liu
Overview
In addition to measuring trends in students’ achievement in reading comprehension, PIRLS has the important role of collecting information about students’ home and school contexts for learning to read. Decades of educational research, including four assessment cycles of PIRLS, have found that the extent of students’ opportunities to learn and the effectiveness of their learning environments substantially impact their reading achievement, with more opportunities and supportive environments associated with higher achievement.
In each PIRLS assessment to date (2001, 2006, 2011, and 2016), the high-quality measure of students’ reading achievement together with extensive information about students’ in- and out-of-school experiences while learning to read have provided an important resource for research into how to improve reading education. More recently, with trend data from the series of PIRLS assessments, relating changes in students’ achievement to changes in educational policies or practices can be a powerful source of evidence for determining if the approaches are beneficial.
As with each previous assessment, the goal of PIRLS 2021 is to build on the existing research base and make PIRLS data even more useful in helping educational policymakers and practitioners raise students’ levels of achievement. This involves assigning a high priority to continuing to collect some information to maintain and extend trends, but also means making sure to “keep up with the times” and address new areas of research and policy relevance.
The PIRLS 2021 Context Questionnaire Framework articulates the information to be collected via the PIRLS 2021 Questionnaires, providing brief rationales. The framework is organized according to five broad influences on students’ reading development: home contexts, school contexts, classroom contexts, student attributes, and national contexts.
The Data Collection Instruments
Consistent with prior assessments, PIRLS 2021 collects the data about students’ contexts for learning to read through questionnaires administered to various participants in the education process, including the students’ parents and caregivers, the principals of the students’ schools, and their reading teachers. The students themselves also complete a questionnaire. Taken together, the information from these four questionnaires provides the data for learning about students’ home, school, and classroom experiences. For national contexts, PIRLS 2021 will update the PIRLS encyclopedia that is published with each assessment to describe each country’s approach to reading education. As with previous assessments, representatives from each participating country provide the contents of the PIRLS 2021 Encyclopedia. They complete a questionnaire about reading education policies at the national level and contribute a chapter to the encyclopedia describing the country’s reading curriculum and instructional practices.
The four PIRLS 2021 questionnaires that provide data that can link to each student’s reading achievement are described below.
- The Home Questionnaire, entitled the Learning to Read Survey, is addressed to the parents or primary caregivers of each student taking part in the PIRLS 2021 data collection. This short 10-to-15-minute questionnaire solicits information on the home context, such as languages spoken in the home, parents’ reading activities and attitudes toward reading, and parents’ education and occupation. The questionnaire also collects data on the students’ early childhood education, early literacy and numeracy activities, and the child’s reading readiness at the beginning of primary school.
- The Teacher Questionnaire, completed by students’ reading teachers, gathers information about classroom contexts for reading instruction, such as characteristics of the class, reading instructional time, and instructional approaches. The questionnaire also asks about teacher characteristics, such as their career satisfaction, education, and recent professional development activities. This questionnaire requires about 35 minutes to complete.
- The School Questionnaire, completed by the principal of each participating school, asks about school characteristics, such as student demographics, the school environment, and the availability of school resources and technology. It is designed to take about 30 minutes.
- The Student Questionnaire, given to all students once they have completed the reading assessment, collects information on students’ home environment, such as books in the home and other home resources for learning; student experiences in school, including feelings of school belonging and whether they are victims of bullying; and students’ reading instruction. Students also are asked about their out-of-school reading habits and attitudes toward reading. The student questionnaire requires up to 30 minutes to complete.
The Development Process
In characterizing students’ educational contexts for learning to read, PIRLS focuses on policy relevant and malleable attributes of the home and school that can be used to help interpret differing levels of reading achievement across countries as well as trends from assessment to assessment.
To summarize questionnaire data more reliably, PIRLS has the practice of developing sets of questions related to a topic of interest and combining the results into scales rather than reporting the results of the questions one by one. This practice has evolved from the early assessments and PIRLS 2011 introduced using item response theory (IRT) methods to construct the scales.1 A number of the PIRLS scales measure constructs that are positively related to student achievement, such as parents’ attention to early literacy activities and school safety. First in PIRLS 2016, and now in PIRLS 2021, work continues on improving the content and measurement properties of the context questionnaire scales.
The TIMSS & PIRLS International Study Center works with the PIRLS Questionnaire Development Group (QDG) and National Research Coordinators (NRCs) from the participating countries to update the context questionnaire framework and the questionnaires for each successive PIRLS assessment. This includes addressing new topics, refining scales, and deleting topics that are no longer useful. Development for PIRLS 2021 began in February 2018, when the PIRLS 2021 NRCs from the participating countries made suggestions for updating the questionnaires. Staff at the TIMSS & PIRLS International Study Center drafted an updated context questionnaire framework and updated the home, school, teacher, and student questionnaires based on the NRCs’ recommendations. The PIRLS 2021 QDG reviewed the updated framework and questionnaires at its first meeting in February 2019. Subsequently, the TIMSS & PIRLS International Study Center staff revised the framework for online review by the NRCs prior to publication. The questionnaires were revised and reviewed by the NRCs prior to field testing at their third meeting in June 2019. After the field test, the QDG and the NRCs will review and finalize the questionnaires for the PIRLS 2021 data collection.
The remaining sections of the chapter provide the
specifications for the PIRLS 2021 context questionnaire data collection.