Language Models for Educational Question Generation: Practical Challenges, Personalization Opportunities, and Parameter Optimization

Language models (LMs) are a new technology attracting research interest in many fields, including educational technology, for their ability to generate media, including text, from a prompt. For teachers creating new questions is a time-consuming process yet essential for developing and confirming a learner’s understanding of the material. This research extends existing research on LM-based question generation by investigating what challenges exist when developing a question generator to support both teachers and learners. In addition, to support personalization, the generation of questions at three difficulty levels is investigated. Finally, different settings are systematically evaluated for creating the best possible questions. Using training data from a course, two small LMs, Gemma 7B and Gemma-7B-it (instruction pre-tuned), were trained under 84 experimental settings. 7 prompts styles (10 prompts each), plus 9 human-generated prompts, for each course unit were used for each experimental setting. It was found that considerable manual effort to prepare the training data and extract the generated questions from the LM’s response was required. Depending on the settings, a question with the proper difficulty was generated between 61.5% and 67.4% of the time. Gemma-7B-it improved overall quality but struggled to generate beginner-level questions. Sentences granularity with unit-only training was found to be best. Roleplaying-style prompts created the best questions, and prompt engineering was critical for LM control. Furthermore, LM-generated prompts were found to be better than human-created prompts.