Implementing the group adaptive design in PIRLS 2021 required grouping the assessment passages and items into three levels of difficulty—easy, medium, and difficult—with three literary and three informational passages at each level of difficulty. Of the 18 passages needed for the design, 12 were administered previously in PIRLS 2016 and were available to support the measurement of trends and six were developed and field tested for first time use in PIRLS 2021.

In 2016, the main PIRLS assessment was accompanied by PIRLS Literacy, which was a reading assessment similar to PIRLS in size and scope but less difficult, with shorter and less demanding passages and easier items. Taking PIRLS and PIRLS Literacy together, there were eight passages (four literary and four informational) that appeared only in PIRLS, four shared passages (two for each purpose) that appeared in both PIRLS and PIRLS Literacy, and eight passages (again four for each purpose) that appeared in PIRLS Literacy only. The 12 trend passages for 2021 were chosen from among these passages, with the difficult passages drawn from the PIRLS only passages, the medium passages from the passages shared between PIRLS and PIRLS Literacy (with one exception), and the easy passages from those used in PIRLS Literacy only.

Exhibit 1: Reading purpose and difficulty level for 18 PIRLS 2021 passage

Reading Purpose	Difficulty Level	Passage Label*	Passage Name*
Literary	Difficult	LitD1 (06)	Shiny Straw (06)
		LitD2 (16)	Oliver and the Griffin (16)
		LitD3 (21)	New LitD3 Passage (21)
	Medium	LitM1 (16)	Pemba Sherpa (16)
		LitM2 (21)	New LitM2 Passage (21)
		LitM3 (11)	The Empty Pot (11)
	Easy	LitE1 (21)	New LitE1 Passage (21)
		LitE2 (11)	The Summer My Father Was 10 (11)
		LitE3 (16)	Library Mouse (16)
Informational	Difficult	InfD1 (11)	Where’s the Honey? (11)
		InfD2 (16)	Icelandic Horses (16)
		InfD3 (21)	New InfD3 Passage (21)
	Medium	InfM1 (16)	How Did We Learn to Fly? (16)
		InfM2 (21)	New InfM2 Passage (21)
		InfM3 (06)	Sharks (06)
	Easy	InfE1 (21)	New InfE1 Passage (21)
		InfE2 (11)	Training a Deaf Polar Bear (11)
		InfE3 (16)	Hungry Plant (16)

Exhibit 1 shows how the existing trend passages fit into the purpose-by-difficulty level scheme, and also where the new passages and items belong. Of the six new passages, three will be literary and three informational, with one of each in each of the difficulty categories. The exhibit also includes a passage label for each passage to facilitate assignment of passages to booklets. 

Passage Difficulty LevelTop

For the design to be effective, it is necessary that there be distinct differences between the average difficulties of the passages in each of the passage groups. For example, reasonable difficulty goals in terms of average percent correct across the student population would be 40% for the difficult group, 60% for the medium group, and 80% for the easy group. New passages developed for PIRLS 2021 will aim for these difficulty levels, but there is less flexibility with the existing, trend passages, which make up two-thirds of the passage total. 

Exhibit 2: Average difficulties of existing trend passages from 2016 and target difficulties for 2021 (average percent correct)

Passage Level	Difficulty of Trend Passages from 2016	Target Difficulty for 2021
Difficult	55%	50%
Medium	68%	65%
Easy	80%	80%

As shown in Exhibit 2, the difficulties of the existing medium and difficult trend passages (68% and 55%, respectively) are higher than the long-term goals (60% and 40%, respectively), especially for the difficult passages. However, by combining the existing passages with new passages developed to be closer to the target difficulty, it will be possible to make progress in 2021 toward these long-term goals. Exhibit 2 shows the interim target difficulties for 2021, which should be attainable given the mix of existing and new passages. Further progress will be made toward the long-term goals in PIRLS 2026, as more of the trend passages are replaced by new, more targeted passages. 

Booklet DesignTop

In the PIRLS assessment, each student is randomly assigned a test booklet (or booklet equivalent in digitalPIRLS) consisting of two passages and their items. In PIRLS 2021, the 18 passages are arranged into 18 booklets of two passages each, with each passage appearing in two booklets and paired with a different passage each time. Exhibit 3 summarizes the passage pairs that make up each booklet. The direction of the arrows shows which passage comes first in the booklet. For example, an arrow points from passage InfM1 to LitD1, indicating that these two passages share a booklet, with InfM1 preceding LitD1. Note that when passages of different difficulties are paired in the same booklet, the easier of the two always comes first.

Exhibit 3: Passage pairings for each assessment booklet

The 18 booklets are divided into two levels of difficulty, as follows:

Exhibit 4 shows the passage assignments for the 18 booklets, with booklets 1-9 being the more difficult booklets and booklets 10-18 the less difficult ones. 

• More difficult booklets (9) composed of either two difficult passages or one medium and one difficult passage
• Less difficult booklets (9) composed of two easy passages or one easy and one medium passages.

Exhibit 4: Assessment booklets with passage assignments

Student Assessment Booklets		Part 1	Part 2
More Difficult Booklets	Booklet 1	InfM1 (16)	LitD1 (11)
	Booklet 2	LitD3 (21)	InfD2 (16)
	Booklet 3	LitM1 (16)	InfD1 (11)
	Booklet 4	InfM2 (21)	LitD2 (16)
	Booklet 5	LitD1 (11)	InfD3 (21)
	Booklet 6	LitM2 (21)	InfD2 (16)
	Booklet 7	InfM3 (11)	LitD3 (21)
	Booklet 8	InfD1 (11)	LitD2 (16)
	Booklet 9	LitM3 (11)	InfD3 (21)
Less Difficult Booklets	Booklet 10	LitE1 (21)	InfM1 (16)
	Booklet 11	InfE2 (11)	LitM2 (21)
	Booklet 12	InfE1 (21)	LitE3 (16)
	Booklet 13	LitE2 (11)	InfM2 (21)
	Booklet 14	InfE3 (16)	LitM3 (11)
	Booklet 15	LitE1 (21)	InfE2 (11)
	Booklet 16	LitE3 (16)	InfM3 (11)
	Booklet 17	InfE1 (21)	LitM1 (16)
	Booklet 18	LitE2 (11)	InfE3 (16)

Exhibit 5 also presents the passage assignments for each booklet, this time showing where the 12 existing trend passages belong and where the new passages developed for 2021 will go. 

Exhibit 5: Assessment Booklets with Trend and New Passage Assignments

Student Assessment Booklets		Part 1	Part 2
More Difficult Booklets	Booklet 1	How Did We Learn to Fly? (16)	Shiny Straw (06)
	Booklet 2	New LitD3 Passage (21)	Icelandic Horses (16)
	Booklet 3	Pemba Sherpa (16)	Where’s the Honey? (11)
	Booklet 4	New InfM2 Passage (21)	Oliver and the Griffin (16)
	Booklet 5	Shiny Straw (06)	New InfD3 Passage (21)
	Booklet 6	New LitM2 Passage (21)	Icelandic Horses (16)
	Booklet 7	Sharks (06)	New LitD3 Passage (21)
	Booklet 8	Where’s the Honey? (11)	Oliver and the Griffin (16)
	Booklet 9	The Empty Pot (11)	New InfD3 Passage (21)
Less Difficult Booklets	Booklet 10	New LitE1 Passage (21)	How Did We Learn to Fly? (16)
	Booklet 11	Training a Deaf Polar Bear (11)	New LitM2 Passage (21)
	Booklet 12	New InfE1 Passage (21)	Library Mouse (16)
	Booklet 13	The Summer My Father Was 10 (11)	New InfM2 Passage (21)
	Booklet 14	Hungry Plant (16)	The Empty Pot (11)
	Booklet 15	New LitE1 Passage (21)	Training a Deaf Polar Bear (11)
	Booklet 16	Library Mouse (16)	Sharks (06)
	Booklet 17	New InfE1 Passage (21)	Pemba Sherpa (16)
	Booklet 18	The Summer My Father Was 10 (11)	Hungry Plant (16)

Booklet Assignment within CountriesTop

To ensure that the same assessment is conducted in every country, all 18 passages are distributed in every country, but with varying proportions of the more and less difficult booklets depending on the average reading ability of the student population. This is estimated based on performance in prior PIRLS assessments, or in the field test for countries participating for the first time. Higher performing countries sample proportionally more of the more difficult booklets while lower performing countries sample proportionally more of the less difficult booklets, with the goal of a better match between assessment difficulty and student ability in each country. 

Exhibit 6 illustrates the differential booklet assignment plan for higher, middle, and lower performing countries. Countries with higher average performance, above 550 on the PIRLS achievement scale, would randomly assign proportionally more of the more difficult booklets, e.g., 70%, and fewer of the less difficult booklets, e.g., 30%. Countries with performance between 450 and 550 would assign equal proportions of more and less difficult booklets, and countries with lower average performance, below 450 on the PIRLS scale, would assign proportionally fewer of the more difficult booklets (30%) and more of the less difficult booklets (70%).

Exhibit 6: Booklet assignment plan for higher, middle, and lower performing countries

Integrating ePIRLS with digitalPIRLSTop

The ePIRLS assessment of online informational reading in 2021 consists of five tasks presented by computer or tablet. With the guidance of a teacher avatar, students navigate within and across webpages to answer questions, explain relationships, and interpret and integrate information. Three of the tasks were administered as part of ePIRLS 2016 and are available for measuring trends and two were developed for PIRLS 2021. As a step toward the future further integration of PIRLS and ePIRLS, in digitalPIRLS countries ePIRLS tasks are included in the booklet assignment rotation scheme, as are a number of “hybrid” booklets consisting on one digitalPIRLS informational passage followed by one ePIRLS task. This means that students participating in digitalPIRLS may be presented with one of three booklet types: a regular booklet with two digitalPIRLS passages, an ePIRLS booklet with two ePIRLS tasks, or a hybrid booklet with one digitalPIRLS informational passage followed by one ePIRLS task.

digitalPIRLS Booklet Assignment Rotation SchemesTop

The basic assignment rotation scheme for digitalPIRLS is designed to match that of paperPIRLS, and so has the same 18 passages arranged in the same 18 booklets as paperPIRLS. In paperPIRLS these 18 booklets are distributed among the students in sampled classes using a systematic random assignment process that ensures that the proportions of more and less difficult booklets conform to the rates established for the country. digitalPIRLS follows a similar assignment process for the 18 regular booklets, but also includes provision for the ePIRLS and hybrid booklets.

Similar to the ePIRLS assignment rotation scheme used in 2016, each of the five ePIRLS tasks for 2021 is paired with each of the others, with each task appearing in both the first and second positions in each booklet. This results in 20 distinct ePIRLS booklets. In the hybrid booklets, each of the nine digitalPIRLS informational passages is paired with each of the five ePIRLS tasks, resulting in 45 hybrid booklets. In each one, the digitalPIRLS passage precedes the ePIRLS task. 

Combining the 18 digitalPIRLS booklets, the 20 ePIRLS booklets, and the 45 hybrid booklets results in a total of 83 booklets to be rotated among the sampled students. However, including each booklet type in the rotation at the same rate would result in too few digitalPIRLS booklets relative to the other booklet types, and especially relative to the number of such booklets in paperPIRLS countries. To avoid this imbalance, digitalPIRLS uses a 27 booklet rotation that includes all 18 digitalPIRLS booklets, six ePIRLS booklets, and three hybrid booklets. The assignment of booklets to individual students is conducted by IEA’s WinW3S within school sampling software to ensure accurate implementation of the rotation scheme.

Beginning with a booklet chosen at random by WinW3S, the first iteration of this rotation involves all 18 digitalPIRLS booklets, booklets 1 to 6 of the 20 ePIRLS booklets, and booklets 1 to 3 of the hybrid booklets. The second iteration includes the 18 digitalPIRLS booklets again, together with ePIRLS booklets 7 to 12 and hybrid booklets 4 to 6. This pattern continues with the next iteration including the 18 digitalPIRLS booklets once again, together with ePIRLS booklets 13 to 18 and hybrid booklets 7 to 9, and so on throughout the country’s entire student sample. This booklet assignment scheme results in two thirds of the student sample responding to digitalPIRLS booklets, two ninths to ePIRLS booklets, and one ninth to hybrid booklets. 

The PIRLS 2021 group adaptive design represents a change from the design of previous PIRLS assessments, where booklets of approximately equal difficulty were distributed at random among students in sampled classes with equal probability in each country. By allowing booklets that vary in difficulty to be assigned at country specific rates, the group adaptive design aims to improve the accuracy of measurement in countries participating in PIRLS.

Top