Shimmei, M., & Matsuda, N. (2020). Learning a Policy Primes Quality Control: Towards Evidence-Based Automation of Learning Engineering. In A. Rafferty & J. Whitehill (Eds.), Proceedings of the International Conference on Educational Data Mining (pp. 224-232): EDM.
Abstract: One of the most challenging issues for online courseware engineering is to maintain the quality of instructional components, such as written text, video, and assessments. Learning engineers would like to know how individual instructional components contributed to students’ learning. However, it is a hard task because it requires significant expertise in learning science, learning technology, and subject matter pedagogy. To address this challenge, we propose an innovative application of reinforcement learning (RL) as an assessor of instructional components implemented in given online courseware. After students activities are converted into Markov decision process (MDP), a collection of actions (each corresponds to an instructional component) suggested as a policy is analyzed. As a consequence, the usefulness of individual actions with regards to achieving ideal learning outcomes will be suggested. The proposed RL application is invented for human-in-the-loop learning engineering method called RAFINE. In the RAFINE framework, a machine generates a list of the least contributing instructional components on the given online courseware by interpreting the whole policy. The courseware developers modify those suggested components. As a proof of concept, this paper describes an evaluation study where online learning was simulated on hypothetical online courseware. The results showed that over 90% of ineffective instructional components were correctly identified as ineffective on average.