PIRLS 2021 International Results in Reading

About PIRLS 2021

---

Overview of PIRLS

IEA’s PIRLS (Progress in International Reading Literacy Study) is an ongoing international assessment program of students’ reading achievement in their fourth year of schooling—an important transition point in their development as readers. By this time in their education, students typically have learned how to read and are now reading to learn. Conducted every five years since 2001, PIRLS is recognized as the global standard for assessing trends in reading achievement at the fourth grade. PIRLS 2021 was the fifth assessment cycle, providing 20 years of trend results.

PIRLS and TIMSS are directed by IEA’s TIMSS & PIRLS International Study Center at Boston College in close cooperation with the IEA Amsterdam and IEA Hamburg offices. IEA is an independent international cooperative of national research institutions and government agencies that pioneered international assessments of student achievement in the 1960s to gain a deeper understanding of policy effects across countries’ different education systems. IEA has been conducting international assessments of reading literacy and the factors associated with proficient reading comprehension in countries around the world for about 60 years.

Transitioning to Digital Assessment in PIRLS 2021

While ensuring a solid basis for trend comparisons over time, PIRLS continuously evolves with each cycle by capitalizing on advances in technology and measurement methodology to improve the assessments. Simultaneously, PIRLS pioneers new approaches to reading assessment as the internet’s ever-increasing pace of information growth constantly changes the nature of reading comprehension to encompass new online reading literacy skills.

PIRLS 2021 incorporated two major advances in international reading assessment at the fourth grade:

• Transitioned to digital assessment (discussed here)
• Implemented a group adaptive design (see later section Implementing the Group Adaptive Design in PIRLS 2021).

In a digital assessment, measurement can be improved through more engaging and interactive assessment materials and procedures. Aside from the advantages of a more interactive assessment, activities related to operational procedures (e.g., the digital equivalents of printing and sending materials to schools) can be accomplished with even greater consistency and efficiency once the move to digital assessments has been accomplished.

PIRLS 2021 developed a state-of-the-art user interface for the digital assessment where students can freely navigate through the texts and activate a panel that presents the items (see The Amazing Octopus and The Empty Pot). In addition to incorporating texts with interactive features, the digital assessment included innovative ePIRLS tasks (see Oceans) as a continuation of the groundbreaking work begun in 2016 to assess reading comprehension in a simulated online environment. The PIRLS 2021 digital assessment systems included capabilities for text and item translation and localization, test delivery (formerly printing), administration to students, and data delivery for scoring.

In PIRLS 2021, 26 countries and 7 benchmarking entities transitioned to digital assessment as their primary mode of data collection, while also administering the paper-based trend texts replicated from PIRLS 2016 to a “bridge” sample. The United States administered the PIRLS 2021 digital assessment and the PIRLS 2021 paper bridge assessment. The United States opted to report the paper bridge scores. The other 31 countries and 1 benchmarking entity continued to administer the full assessment using paper booklets.

Exhibit 1 provides a list of the PIRLS 2021 participants and indicates whether their results are based on digital or paper data. Altogether, there were 57 countries in PIRLS 2021, including some distinct education systems within countries that have always participated separately throughout IEA’s long history (e.g., the French- and Dutch-speaking parts of Belgium as well as Hong Kong SAR). In addition, PIRLS 2021 included 8 benchmarking participants, mainly regions of countries that also participated in PIRLS.

The PIRLS 2021 Reading Assessment Framework

PIRLS 2021 assessed reading comprehension in accordance with the PIRLS 2021 Reading Assessment Framework (Chapter 1 in PIRLS 2021 Assessment Frameworks). The PIRLS reading assessment framework has been updated with each cycle to keep reading research and education developments at the forefront through reviews by the PIRLS Reading Development Group (RDG) and the National Research Coordinators (NRCs). This maintains PIRLS’ relevance and importance for teaching practice and policy. However, PIRLS is a trend study and the framework’s underlying organization has remained consistent across cycles.

The framework is organized around two overarching purposes for reading: reading for literary experience and reading to acquire and use information. The framework also includes four cross-cutting reading comprehension processes: focus on and retrieve explicitly stated information, make straightforward inferences, interpret and integrate ideas and information, and evaluate and critique content and textual elements.

The PIRLS 2021 Reading Assessment Framework provides information that emphasizes the growing importance of ePIRLS, the world leading assessment of online reading that was successfully launched in 14 countries in 2016 (see ePIRLS 2016 International Results in Online Informational Reading). In the ePIRLS tasks, a teacher avatar guides the students through several simulated multi-modal websites with multiple texts and interactive features to complete school-like assignments about social studies or science topics.

Implementing the Group Adaptive Design in PIRLS 2021

The PIRLS 2021 group adaptive design provides better measurement through better alignment across countries between the assessment difficulty and the students’ levels of reading achievement. The group adaptive design is based on texts and items of three levels of difficulty—difficult, medium, and easy—that are combined into booklets of two difficulty levels (see Chapter 3 of the PIRLS 2021 Assessment Frameworks). The more difficult booklets include difficult and medium texts and items, and the less difficult booklets include easy and medium texts and items. All booklets are administered in each country, but countries whose students have higher reading achievement on average may give the more difficult booklets to a higher percentage of students (70%), and countries whose students have lower average reading achievement may give a higher percentage of their students the less difficult booklets.

Transitioning to the PIRLS 2021 group adaptive design from PIRLS 2016 was a great success because trend blocks could be categorized as easy, medium, or difficult blocks based on data from 2016. The PIRLS 2016 design linked PIRLS blocks of passages and items with less difficult PIRLS Literacy blocks through blocks common to both assessments (see Chapter 3 of PIRLS 2016 Assessment Frameworks). For 2021, PIRLS Literacy blocks contributed content at the “easy” level, the blocks common to both PIRLS and PIRLS Literacy contributed content at the “medium” level, and the PIRLS blocks contributed content at the “difficult” level. Including the newly developed blocks for 2021, the group adaptive design in PIRLS 2021 led to a lower item non-response rate and more precise achievement estimates than the non-adaptive design in PIRLS 2016 (see Chapter 9 in Methods and Procedures: PIRLS 2021 Technical Report).

Providing comprehensive coverage of the PIRLS 2021 Reading Assessment Framework and implementing the group adaptive design resulted in the most comprehensive and complex international reading assessment to date, consisting of 18 text and item sets as well as 5 ePIRLS tasks. Exhibit 2 shows the PIRLS 2021 group adaptive design for the 18 text and item sets, where 9 text and item sets assessed the literary reading purpose, and 9 text and item sets assessed the informational reading purpose. In accordance with the group adaptive design, within the 9 text and item sets for each purpose, 3 text and item sets were difficult, 3 were medium, and 3 were easy.

Developing the new text and item sets for the PIRLS 2021 assessment was a considerable effort. Six new text and item sets were needed to complete the new group adaptive design requirements. To ensure all target levels were successfully met, twice as many—12 text and items sets—were developed for the field test. Also, two ePIRLS tasks were published on the PIRLS 2016 website, so two new tasks for PIRLS 2021 were developed to replace them.

The field test development took nearly two years, including two RDG meetings and three NRC meetings, with one of the NRC meetings specifically devoted to item development. At the NRC item development meeting hosted by Chinese Taipei, 126 representatives from 43 countries drafted more than 600 items.

Despite the initial emergence of COVID-19 in 2020 at the time scheduled for the field test, more than half the countries were able to collect field test data. Therefore, following the field test, the field test data were analyzed, PIRLS 2021 materials were selected, finalized, and assembled, and the countries continued preparations for the PIRLS 2021 main data collection.

PIRLS 2021 Data Collection Successful Despite Disruptions by the COVID-19 Pandemic

PIRLS 2021 is the only international assessment of educational achievement that successfully collected data during COVID-19’s disruption in students’ schooling. Consequently, the PIRLS 2021 International Database provides an extremely rich and valuable data source to research the impact of COVID-19 on teaching and learning reading.

Similar to previous PIRLS assessments, to prepare for PIRLS 2021, the TIMSS & PIRLS International Study Center, IEA Hamburg, and Statistics Canada worked to select a carefully designed random sample of schools within each country and trained countries in data collection procedures designed to yield high quality data. However, as it is well known, many schools around the world faced considerable disruptions to their operations due to COVID-19, with a good number shifting to remote learning or reduced classroom sizes. Even school buildings that remained open often adopted special procedures and often reduced access to prevent the spread of COVID-19.

PIRLS adapted rapidly to the situation and included context questionnaire items specifically targeted to collect information about the challenges faced by the PIRLS 2021 schools and students during COVID-19. The PIRLS 2021 Context Questionnaires can be accessed on the PIRLS 2021 website. Also, the PIRLS 2021 Encyclopedia, which includes a chapter authored by each country describing its reading education, provides information from most countries about how COVID-19 interrupted teaching and learning.

School Operations

At the time of the PIRLS 2021 data collection, there was considerable variation across countries in how primary school operations were affected by the COVID-19 pandemic, with some countries still experiencing school closures and others modifying how they provided in-person instruction.

Exhibit 3 shows the percentages of students in the PIRLS 2021 countries by the number of weeks their principals reported that normal primary school operations were affected by the COVID-19 pandemic. Because principals in the countries with delayed assessments (see Exhibit 5) needed to recall what happened six months previously in the prior 2020-2021 school year to answer the question, the rows for those countries are colored pink. This color coding to distinguish the results based on delayed data collection is also used in subsequent exhibits.

On average, across countries, only 14 percent of the fourth grade students attended schools where normal operations were “not affected” by the COVID-19 pandemic during the 2020-2021 school year. In contrast, 47 percent attended schools where normal operations were affected by the COVID-19 pandemic for “more than 8 weeks” of instruction. The remaining students experienced more moderate disruptions; 10 percent of students attended schools where “less than 2 weeks” of instruction were affected by the COVID-19 pandemic, 15 percent where “2 to 4 weeks” of instruction were affected, and 13 percent where “5 to 8 weeks” of instruction were affected.

Parents’ Perceptions

Exhibit 4 shows parents’ reports of whether or not their child stayed home from school because of the COVID-19 pandemic and their perceptions of the pandemic’s effect on their child’s learning progress. Information was collected using two items in the PIRLS 2021 Home Questionnaire (see “About the Items”). If parents reported that their child did not stay home from school because of the COVID-19 pandemic, they were not asked to respond to the item about perceptions of their child’s learning progress.

Internationally, parents of most students (86%) reported that their child stayed home from school because of the COVID-19 pandemic, with the rest reporting their child did not stay home due to the pandemic (14%) and consequently not being asked any further questions.

Across countries, on average, parents of two-thirds of the students (67%) reported that their child stayed home and that their child’s learning progress was adversely affected by the pandemic—either “a lot” (22%) or “somewhat” (45%). Parents of 19 percent of the students reported that their child stayed home from school because of the COVID-19 pandemic, but their child’s learning progress was “not at all” affected.

Despite the many challenges of conducting a school-based assessment during the COVID-19 pandemic, countries remained committed to participating in PIRLS 2021. Due to the tremendous efforts from all involved, the PIRLS 2021 countries overcame a variety of obstacles to ensure that students could take the PIRLS 2021 assessment. In all, PIRLS 2021 assessed nearly 400,000 students.

As shown in Exhibit 5, most of the countries managed to collect data towards the end of students’ fourth year of schooling according to the original PIRLS 2021 schedule, which was October to December 2020 for Southern Hemisphere countries and February to July 2021 for Northern Hemisphere countries. However, some Northern Hemisphere countries had to delay assessing the cohort of fourth grade students until the beginning of the fifth grade (September to December 2021) and some countries assessed their fourth grade students one year later than originally scheduled (August to December 2021 for the Southern Hemisphere and April to July 2022 for the Northern Hemisphere).

Across the data collection dates, most of the countries assessed fourth grade students toward the end of the school year. The exception is the 14 Northern Hemisphere countries that necessarily had to delay testing and assessed students at the beginning of the fifth grade. For the most part, the students were from the same schools that had been selected for PIRLS 2021, but because of the delay over the summer months the students were 6 months older on average than their PIRLS 2016 counterparts (see later section on Reporting the PIRLS 2021 Achievement Results).

Numbers of Students Assessed

Nationally representative random samples of approximately 4,000 students from 150 to 200 schools participated in PIRLS 2021. PIRLS 2021 collected data from about 400,000 students, 380,000 parents, 20,000 teachers, and 13,000 schools.

A rigorous sampling adjudication provided documentation that almost all the countries met all sampling standards. Of the 57 countries and 8 benchmarking participants, nearly all the countries met the guidelines for coverage of the target population and most met the standards for low exclusion rates (less than 5%). Almost all the countries met or exceeded the school and student participation rate requirements, with only 7 needing to rely on replacement schools to reach the requirement and 4 falling short of the requirements.

In summary, the PIRLS 2021 data are of high quality. It can be said that the pandemic affected almost all countries to some extent, and no assessment can provide data on how students would have performed without COVID-19 affecting schools. No assessment, including PIRLS 2021, can be designed to measure the effects of the COVID-19 pandemic on student achievement and to compare achievement with and without the pandemic. However, PIRLS 2021 provides a unique data source for studying students’ reading achievement and learning experiences around the world during the pandemic.

Summary of Scaling the PIRLS 2021 Data

For more detailed information about scaling the PIRLS 2021 achievement data and links to other references about the methodology, see Chapter 10 (methodology) and Chapter 11 (implementation) in Methods and Procedures: PIRLS 2021 Technical Report.

PIRLS has used well-established psychometric scaling approaches to derive achievement distributions and transform the assessment results of each PIRLS data collection to the PIRLS trend scale. Among these methods, linear scale transformations and linking designs using randomly equivalent samples have been used extensively in past PIRLS cycles for analysis and reporting (described in Chapter 10 of Methods and Procedures: PIRLS 2021 Technical Report). With the transition to the digital environment in PIRLS 2021, it was necessary to adapt analytic procedures and data collection designs to accommodate the change from paper-and-pencil to digital assessment. Accordingly, countries that administered the PIRLS digital assessment implemented a data collection design that involved two student samples: the main sample of about 4,500 students and a second equivalent but smaller “bridge” sample (about 1,500 students). Students in the main sample took the 2021 digital assessment, while the bridge sample was administered the PIRLS 2021 trend items in the PIRLS 2016 paper format. This resulted in equivalent samples of students in each country responding to the trend items in the paper and digital formats, which enabled bridging the trend scale from paper-based PIRLS 2016 to digitally-based PIRLS in 2021. Although the bridge samples were smaller than the digital samples, often the students were in the same schools as those who took the digital assessments. The bridge samples were adjudicated as part of the same process used for all PIRLS 2021 countries and were judged to be the same quality as their digital counterparts.

To ensure that the paper-based and digitally-based assessment results will be reported on the PIRLS trend scale, scaling the PIRLS 2021 data involved the following three steps. First, the usual concurrent calibration approach (described in Chapter 11 of Methods and Procedures: PIRLS 2021 Technical Report) was applied to the paper-based data from PIRLS 2016 and PIRLS 2021, ensuring that the PIRLS 2021 paper data was linked to the PIRLS trend scale. This procedure included all the data from trend countries that administered the paper-based PIRLS 2021 assessment as well as the paper-based bridge data from the digital countries. Second, the digital assessment data from the digital countries was linked to the PIRLS trend scale through population-based linking, which capitalizes on the availability of equivalent samples from the same populations between the digital and bridge samples. Finally, the data from the digital countries, including data from the ePIRLS items, were scaled together to link the ePIRLS data to the PIRLS achievement scale.

Reporting the PIRLS 2021 Achievement Results

Reading achievement results are included in PIRLS 2021 International Results in Reading for all 57 countries and 8 benchmarking entities that participated in PIRLS 2021. Concerns about the comparability of the data resulting from COVID-19 school disruptions and delayed testing complicated reporting the PIRLS 2021 results.

PIRLS and TIMSS have built a reputation for reporting high quality data, but not all data collected meet the expected guidelines. In such cases, PIRLS and TIMSS use annotations to identify results based on data that for some reason fell short of meeting the expected guidelines. The goal is to be clear about issues while still reporting countries’ data. See discussion “Impacts of Modifying the Assessment Schedule on Students’ Achievement” in Countries’ Reading Achievement.

The achievement results for all countries that assessed fourth grade students at the end of the school year are presented according to average achievement in Exhibit 1.1, with the countries that assessed the fourth grade students one year later annotated. Exhibit 1.1 is followed by Exhibit 1.2, which has guidelines for determining significant differences in average reading achievement between the Exhibit 1.1 countries. Exhibit 1.3 includes all the countries presented according to average achievement, with the delayed assessment countries that assessed the fourth grade cohort at the beginning of the fifth grade highlighted in pink.

While PIRLS cannot determine cause and effects, in general there are downward trends in PIRLS 2021 that likely are evidence of the assessment taking place during the COVID-19 pandemic. Because the pandemic was unprecedented in the history of PIRLS trend assessments, the trends between 2016 and 2021 are shown with dotted lines. This should alert researchers that care should be taken when interpreting the PIRLS 2021 results. Similar to the approach used for the PIRLS 2021 achievement data, the trend results for the countries that assessed fourth grade students are in one exhibit, with the “one year later countries” clearly annotated as having a 6-year trend instead of a 5-year trend between 2016 and 2021. Trend results for the countries with delayed assessments at the fifth grade need to be interpreted with great care due to the age difference and are shown in a separate exhibit.

Reporting the PIRLS 2021 Context Questionnaire Data

The PIRLS 2021 Context Questionnaire Framework (see Chapter 2 of PIRLS 2021 Assessment Frameworks) describes the topics covered by the PIRLS 2021 Context Questionnaires. PIRLS 2021 collected extensive data about the contexts for teaching and learning reading through questionnaires administered to students, their parents, teachers, and school principals.

The impact of COVID-19 on the PIRLS 2021 Context Questionnaire data is challenging to evaluate, but PIRLS 2021 did collect a considerable amount of valuable information on multiple levels. The school questionnaire results were only slightly impacted in the countries that delayed assessment to the fifth grade, because most schools had both fourth and fifth grades and the principals were asked to keep the fourth grade and the prior school year (2020–2021) in mind. The delayed assessments had the most impact on the teacher questionnaire data at the fifth grade. However, most countries tried to contact the teachers of the students from the fourth grade and asked teachers to keep the prior school year (2020–2021) in mind when responding to the questionnaire.

The TIMSS & PIRLS International Study Center conducted a series of analyses to establish that there was little or no difference in the responses to the Context Questionnaires between the bulk of the PIRLS 2021 countries that assessed students at the end of fourth grade and the countries with delayed testing of the fourth grade cohort at the beginning of fifth grade.

PIRLS 2021 International Results in Reading includes results for all countries for selected items in the school, home, and student questionnaires (countries with delayed assessments of students in the fifth grade are highlighted in pink). Although COVID-19 impacted data collection, resulting in less questionnaire data included here than in previous assessment cycles, all of the PIRLS 2021 Context Questionnaire data are included in the PIRLS 2021 International Database.

Quality Assurance

Despite the challenges that the COVID-19 pandemic introduced for schools and national research centers responsible for implementing PIRLS 2021, every effort was made to attend to the quality and comparability of the data through careful planning and documentation, cooperation among participating countries, standardized procedures, and rigorous attention to quality control throughout. The assessments were administered to nationally representative and well-documented probability samples of students in each country. Staff from Statistics Canada and IEA Hamburg worked with NRCs on all phases of sampling activities to ensure compliance with sampling and participation requirements, with a few exceptions from compliance annotated in the data exhibits.

IEA Amsterdam worked with the TIMSS & PIRLS International Study Center to manage an extensive series of verification checks to ensure the comparability of translations of the assessment items and questionnaires and to conduct the International Quality Assurance Program of school visits to monitor and report on the administration of the assessment. Together with the TIMSS & PIRLS International Study Center, IEA Hamburg staff worked closely with NRCs to organize data collection operations and to check all data for accuracy and consistency within and across countries.

The extensive efforts to maintain PIRLS’ quality standards during the COVID-19 pandemic were largely successful. Complete documentation of the many technical activities required to conduct PIRLS 2021 is provided in the Methods and Procedures: PIRLS 2021 Technical Report documentation. The volume includes detailed information about the processes used to develop and implement the PIRLS 2021 assessments, including sampling, translation verification, data collection, scaling, linking, and data analysis.