Shedding light on the automated essay scoring process

David Boulanger, Vivekanandan Kumar

Research output: Chapter in Book/Report/Conference proceedingPublished Conference contributionpeer-review

1 Citation (Scopus)

Abstract

This paper explores in depth the suitability of the 2012 Automated Student Assessment Prize (ASAP) contest's essay datasets. It evaluates the potential of deep learning and state-of-the-art NLP tools in automated essay scoring (AES) to predict not only holistic scores but also the finer-grained rubric scores, an area underexplored but essential to provision formative feedback and uncover the AI reasoning behind AES. For comparison purpose, this paper advocates the need for transparency when sharing AES processes and outcomes. Finally, it reveals the insufficiency of ASAP essay datasets to train generalizable AES models by examining the distributions of holistic and rubric scores. Findings show that the strength of agreement between human and machine graders on holistic scores does not translate into similar strength on rubric scores and that the learning made by the machine barely exceeds the performance of a naïve predictor.

Original languageEnglish
Title of host publicationEDM 2019 - Proceedings of the 12th International Conference on Educational Data Mining
EditorsCollin F. Lynch, Agathe Merceron, Michel Desmarais, Roger Nkambou
Pages512-515
Number of pages4
ISBN (Electronic)9781733673600
Publication statusPublished - 2019
Event12th International Conference on Educational Data Mining, EDM 2019 - Montreal, Canada
Duration: 2 Jul. 20195 Jul. 2019

Publication series

NameEDM 2019 - Proceedings of the 12th International Conference on Educational Data Mining

Conference

Conference12th International Conference on Educational Data Mining, EDM 2019
Country/TerritoryCanada
CityMontreal
Period2/07/195/07/19

Keywords

  • Automated essay scoring
  • Automated student assessment prize
  • Deep learning
  • Natural language processing
  • Rubrics

Fingerprint

Dive into the research topics of 'Shedding light on the automated essay scoring process'. Together they form a unique fingerprint.

Cite this