Automatic grading of answers in knowledge control for “Definition” and “Description” question types
https://doi.org/10.26907/2541-7746.2024.4.580-593
Abstract
The progress in developing an effective automatic knowledge control system is directly associated with creating and implementing a software module for grading answers to test questions formulated in natural language. Previously, an experimental prototype of such a system was designed, and a study was performed where short answers to basic question types provided by students were examined via a pragmatically oriented question-answer text processing algorithm, considering its outputs and exposing flaws. This article introduces the next iteration of the algorithm tailored to handle more complex question types that requires the identification of relations such as “Definition” and “Description.” The key features of the enhanced algorithm were outlined, with a particular focus on the problem of segmenting answers into meaningful chunks, a task for which processing methods have already been found. The results of an experiment based on the developed prototype with obtaining answers from students and a thorough analysis of the instances of the system’s incorrect behavior were discussed.
References
1. Khasianov A., Suleymanov D., Marchenko A. Three agent platform approach for digital education environment. INTED2017 Proc., 2017, pp. 8580–8587. doi: 10.21125/inted.2017.2031.
2. Suleymanov D., Prokopyev N. Development of prototype of natural language answer processor for e-learning. Artificial Intelligence (RCAI 2020) : Proc. 18<sup>th</sup> Russ. Conf. Ser.: Lecture Notes in Computer Science. Vol. 12412. Cham, Springer, 2020, pp. 448–459. doi: 10.1007/978-3-030-59535-7_33.
3. Suleymanov Dzh., Prokopyev N. Experimental prototype of the pragmatically oriented e-assessment algorithm for automatic natural language answer grading. EDULEARN20 Proc., 2020, pp. 1917–1924. doi: 10.21125/edulearn.2020.0617.
4. Suleymanov D.Sh. A system for semantic processing of natural language responses. Ontol. Proekt., 2014, no. 1 (11), pp. 65–77. (In Russian)
5. Burrows S., Gurevych I., Stein B. The eras and trends of automatic short answer grading. Int. J. Artif. Intell. Educ., 2015, vol. 25, pp. 60–117. doi: 10.1007/s40593-014-0026-8.
6. Mishunin O.B., Savinov A.P., Firstov D.I. Current progress and development trends of systems for automatic grading of free answers in natural language. Sovrem. Naukoemkie Tekhnol. Tekh. Nauki, 2016, no. 1, pp. 38–44. (In Russian)
7. Merzlyakov D.A. Generation of regular expressions for automatic grading of open-ended tests. Studencheskii nauchnyi forum – 2013 : V Mezhdunar. studench. nauch. konf. [Student Research Forum – 2013: Proc. V Int. Student Sci. Conf.], 2013. (In Russian)
8. Ziai R., Rudzewitz B. CoMiC: Exploring text segmentation and similarity in the English entrance exams task. CLEF2015 Working Notes. Cappellato L., Ferro N., Jones G.J.F., Juan E.S. (Eds.). Ser.: CEUR Workshop Proceedings, 2015, vol. 1391.
9. Pribadi F.S., Permanasari A.E., Adji T.B. Short answer scoring system using automatic reference answer generation and geometric average normalized-longest common subsequence (GAN-LCS). Educ. Inf. Technol., 2018, vol. 23, no. 6, pp. 2855–2866. doi: 10.1007/s10639-018-9745-z.
10. Xu L.-X., Wang N., Xu L., Li L.-Y. Research of automated assessment of subjective tests based on domain ontology. Multidisciplinary Social Networks Research (MISNC 2014) : Proc. Int. Conf. Ser.: Communications in Computer and Information Science. Vol. 473. Berlin, Heidelberg, Springer, 2014, pp. 445–453. doi: 10.1007/978-3-662-45071-0_37.
11. Kozhevnikov V.A., Sabinin O.Yu. Automatic system for processing answers to open-ended questions in Russian. Inf. Telekommun. Upr., 2018, vol. 11, no. 3, pp. 57–72. doi: 10.18721/JCSTCS.11306. (In Russian)
Review
For citations:
Prokopyev N.A. Automatic grading of answers in knowledge control for “Definition” and “Description” question types. Uchenye Zapiski Kazanskogo Universiteta. Seriya Fiziko-Matematicheskie Nauki. 2024;166(4):580-593. (In Russ.) https://doi.org/10.26907/2541-7746.2024.4.580-593
Views:
208
Email this article 