Deep learning in automated essay scoring

David Boulanger, Vivekanandan Kumar

Research output: Chapter in Book/Report/Conference proceedingPublished Conference contributionpeer-review

7 Citations (Scopus)

Abstract

This paper explores the application of deep learning in automated essay scoring (AES). It uses the essay dataset #8 from the Automated Student Assessment Prize competition, hosted by the Kaggle platform, and a state-of-the-art Suite of Automatic Linguistic Analysis Tools (SALAT) to extract 1,463 writing features. A non-linear regressor deep neural network is trained to predict holistic scores on a scale of 10–60. This study shows that deep learning holds the promise to improve significantly the accuracy of AES systems, but that the current dataset and most essay datasets fall short of providing them with enough expertise (hand-graded essays) to exploit that potential. After the tuning of different sets of hyperparameters, the results show that the levels of agreement, as measured by the quadratic weighted kappa metric, obtained on the training, validation, and testing sets are 0.84, 0.63, and 0.58, respectively, while an ensemble (bagging) produced a kappa value of 0.80 on the testing set. Finally, this paper upholds that more than 1,000 hand-graded essays per writing construct would be necessary to adequately train the predictive student models on automated essay scoring, provided that all score categories are equally or fairly represented in the sample dataset.

Original languageEnglish
Title of host publicationIntelligent Tutoring Systems - 14th International Conference, ITS 2018, Proceedings
EditorsJulita Vassileva, Roger Nkambou, Roger Azevedo
Pages294-299
Number of pages6
DOIs
Publication statusPublished - 2018
Event14th International Conference on Intelligent Tutoring Systems, ITS 2018 - Montreal, Canada
Duration: 11 Jun. 201815 Jun. 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume10858 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference14th International Conference on Intelligent Tutoring Systems, ITS 2018
Country/TerritoryCanada
CityMontreal
Period11/06/1815/06/18

Keywords

  • Automated essay scoring
  • Deep learning
  • Writing analytics

Fingerprint

Dive into the research topics of 'Deep learning in automated essay scoring'. Together they form a unique fingerprint.

Cite this