Standard Lists of Creative Answers Leave Out Correct Solutions: An Example from PISA 2012

8 Pages Posted: 13 Jul 2016

See all articles by Alexander N. Poddiakov

Alexander N. Poddiakov

National Research University Higher School of Economics (Moscow)

Date Written: July 3, 2016


Interactive complex problem solving is an important 21st century skill (Neubert et al., 2015). Items of the PISA 2012 included interactive problems requiring exploration of a novel device (e.g., a virtual MP3-Player) and ideas for its modifications. The aim of our analysis is to show that the use of “standard lists of creative answers” used to assess creative problem solving can lead to missing some correct solutions. The object of the analysis is the fourth question of the MP3-Player Unit: “Describe how you could change the way the MP3 player works so that there is no need to have the bottom button”. Authors of the item write: “There is no single correct answer, and students may think creatively in devising a solution”. Nonetheless, they limit the number of correct answers to the six that are described in the guidelines. Other responses are not accepted and result in a score of 0 (PISA 2012). We argue that the authors’ list of correct answers is not complete: it includes neither double successive nor simultaneous clicks, even though “double-clicking” is a common ergonomic solution in modern devices. A possible paradoxical reason is that the authors agree entirely with their example of universal rules formulated about 15 years ago, before the era of multi-touch devices: “at no point two buttons have to be pressed simultaneously” (Greiff, 2012, p. 52). However, one can easily show that solutions based on different successive and simultaneous double-clicks should be included in the list of correct answers for the item (Poddiakov, 2012). The paradigm of assessing creative answers in accordance with lists of criteria prepared in advance (“standard lists of creative answers”) seems to be an oxymoron in the context of assessing 21st century skills and needs to be changed.

Keywords: Programme for International Student Assessment (PISA), interactive problems, complex problem solving, creativity, creative thinking

Suggested Citation

Poddiakov, Alexander N., Standard Lists of Creative Answers Leave Out Correct Solutions: An Example from PISA 2012 (July 3, 2016). Available at SSRN: or

Alexander N. Poddiakov (Contact Author)

National Research University Higher School of Economics (Moscow) ( email )

Myasnitskaya street, 20
Moscow, Moscow 119017

Do you have negative results from your research you’d like to share?

Paper statistics

Abstract Views
PlumX Metrics