Taking PISA Seriously: Biases Endemic in Low Stakes Exams

PISA is seen as the gold standard for evaluating educational outcomes worldwide. Yet, being a low-stakes exam, students may not take it seriously resulting in downward biased scores and inaccurate rankings. This paper provides a method to identify and account for non-serious behavior in low-stakes exams by leveraging information in computer- based assessments in PISA 2015. Our method corrects for non-serious behavior by fully imputing scores for items not taken seriously. We compare the scores/rankings calculated by our method to the scores/rankings calculated by giving zero points to skipped items as well as to the scores/rankings calculated by treating skipped items at the end of the exam as if they were not administered, which is the procedure followed by PISA. We show that a country can improve its ranking by up to 15 places by encouraging its own students to take the exam seriously and that the PISA approach corrects for only about half of the bias generated by the non-seriousness.

Files

Metadata

Work Title Taking PISA Seriously: Biases Endemic in Low Stakes Exams
Access
Open Access
Creators
  1. Pelin Aykol
  2. Kala Krishna
  3. Jinwen Wang
Keyword
  1. Low-stakes exams
  2. Computer-based assessments
  3. PISA
  4. Biased rankings
  5. Item response data
License In Copyright (Rights Reserved)
Work Type Article
Publisher
  1. Journal of Labor Research
Publication Date March 26, 2021
Publisher Identifier (DOI)
  1. https://doi.org/10.1007/s12122-021-09317-8
Deposited August 03, 2022

Versions

Analytics

Collections

This resource is currently not in any collection.

Work History

Version 1
published

  • Created
  • Added PISA.pdf
  • Added Creator Kala Krishna
  • Added Creator Pelin Aykol
  • Added Creator Jinwen Wang
  • Published
  • Updated Keyword, Publisher Identifier (DOI), Publication Date Show Changes
    Keyword
    • Low-stakes exams, Computer-based assessments, PISA, Biased rankings, Item response data
    Publisher Identifier (DOI)
    • https://doi.org/10.1007/s12122-021-09317-8
    Publication Date
    • 2021-04-02
    • 2021-03-26
  • Updated Creator Kala Krishna
  • Updated Creator Pelin Aykol
  • Updated