Understanding Performance in Test Taking: The Role of Question Difficulty Order

47 Pages Posted: 14 May 2021

See all articles by Lina Anaya

Lina Anaya

University of Bradford

Nagore Iriberri

University of the Basque Country

Pedro Rey Biel

ESADE University Faculties

Gema Zamarro

University of Arkansas - Department of Education Reform; University of Southern California - Center for Economic and Social Research (CESR)

Date Written: May 2021

Abstract

Standardized assessments are widely used to determine access to educational resources with important consequences for later economic outcomes in life. However, many design features of the tests themselves may lead to psychological reactions influencing performance. In particular, the level of difficulty of the earlier questions in a test may affect performance in later questions. How should we order test questions according to their level of difficulty such that test performance offers an accurate assessment of the test taker's aptitudes and knowledge? We conduct a field experiment with about 19,000 participants in collaboration with an online teaching platform where we randomly assign participants to different orders of difficulty and we find that ordering the questions from easiest to most difficult yields the lowest probability to abandon the test, as well as the highest number of correct answers. Consistent results are found exploiting the random variation of difficulty across test booklets in the Programme for International Student Assessment (PISA), a triannual international test, for the years of 2009, 2012, and 2015, providing additional external validity. We conclude that the order of the difficulty of the questions in tests should be considered carefully, in particular when comparing performance between test-takers who have faced different order of questions.

JEL Classification: C93, D81, I20, J16

Suggested Citation

Anaya, Lina and Iriberri, Nagore and Rey Biel, Pedro and Zamarro, Gema, Understanding Performance in Test Taking: The Role of Question Difficulty Order (May 2021). CEPR Discussion Paper No. DP16099, Available at SSRN: https://ssrn.com/abstract=3846209

Lina Anaya (Contact Author)

University of Bradford ( email )

Emm Lane
Bradford, West Yorkshire Bd9 4JL
United Kingdom

Nagore Iriberri

University of the Basque Country ( email )

Barrio Sarriena s/n
Leioa, Bizkaia 48940
Spain

Pedro Rey Biel

ESADE University Faculties ( email )

Rua General Vitorino, 25 - oitavo andar
Barcelona, Rio Grande do Sul 90810120
Spain

Gema Zamarro

University of Arkansas - Department of Education Reform ( email )

201 Graduate Education Building
Fayetteville, AR 72701
United States

University of Southern California - Center for Economic and Social Research (CESR) ( email )

635 Downey Way
Los Angeles, CA 90089-3332
United States

HOME PAGE: http://works.bepress.com/gema_zamarro/

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
0
Abstract Views
374
PlumX Metrics