A Weighted Autoscoring System for Discussion Posts in Online Courses

Discussion forums play a vital role in online courses for building community, improving engagement with peers, developing critical thinking, and gaining diverse perspectives on a course topic through asynchronous communication. Given this importance, discussion posts are often used as assessments to evaluate learners’ deeper comprehension and skill development. However, manually scoring discussion posts is slow and becomes increasingly difficult as class sizes grow. This paper presents an autoscoring system for discussion posts considering three weighted aspects – cognitive engagement, topic relevancy, and writing quality, where the weight assigned to each aspect reflects the scoring behavior of the instructor. In the autoscoring system, the score related to cognitive engagement aspect is assigned using a deep learning-based approach grounded in the Interactive, Constructive, Active, and Passive (ICAP) educational framework. The topic relevancy score is assigned using a BERTopic and Sentence-BERT–based semantic similarity method. The writing quality score is assessed through readability, spelling, and profanity check. The final composite score is computed by a weighted linear model of the above three scores, where the weights are estimated by a gradient descent method. With a dataset of manually graded 3,298 posts, the proposed autoscoring system achieved a moderate to high correlation with human scoring, with a Pearson correlation 0.80 and Cohen’s kappa 0.56. The findings suggest that a thoughtful combination of cognitive engagement, topic relevancy, and writing-quality can predict the scoring behavior of a human grader while keeping the process transparent and aligned with instructional intent.