Towards an improved of teaching practice using Sentiment Analysis in Student Evaluation
Article Sidebar
Main Article Content
Student Evaluation of Teaching (SET) serves as an ad hoc way of assessing teaching effectiveness within higher education institutions. This paper introduces an approach to analyzing sentiments expressed in SET comments using a Large Language Model (LLM). By employing natural language processing techniques, the polarity conveyed by students upon course completion is extracted and analyzed, aiming to furnish educators and stakeholders with valuable insights into teaching quality and areas for improvement in teaching practice. This study showcases the effectiveness of LLMs in sentiment analysis of comments, underscoring their potential to enhance the evaluation process. The development of a prototype tool, collaborative labeling of end-of-course evaluations, and a comparison with LLM-based labeling are experimentally explored. Subsequently, the implications for educational institutions are discussed, and future research directions in this domain are proposed.
Omer K, Jacobs S, Bettger B, Dawson J, Graether S, Murrant C, et al. Evaluating and Improving the Formative Use of Student Evaluations of Teaching. Can J Scholarsh Teach Learn. 2023;14(1):n1. DOI: https://doi.org/10.5206/cjsotlrcacea.2023.1.10960
Mancenido Z. Impact evaluations of teacher preparation practices: Challenges and opportunities for more rigorous research. Rev Educ Res. 2023;00346543231174413. DOI: https://doi.org/10.3102/1883286
Cunningham S, Laundon M, Cathcart A, Bashar MA, Nayak R. First, do no harm: automated detection of abusive comments in student evaluation of teaching surveys. Assess Eval High Educ. 2023;48(3):377–89. DOI: https://doi.org/10.1080/02602938.2022.2081668
Gravestock P, Gregor-Greenleaf E. Student course evaluations: Research, models and trends. Higher Education Quality Council of Ontario Toronto; 2008.
Uttl B. Lessons learned from research on student evaluation of teaching in higher education. Stud Feedback Teach Sch Using Stud Percept Dev Teach Teach. 2021;237–56. DOI: https://doi.org/10.1007/978-3-030-75150-0_15
Gaoa G, Pangb M, Pengc J, Lud Y. A Hierarchical Probe Evaluation Method for Teaching Effect of University Engineering Courses Based on the Keypoints of Knowledge. In: 3rd International Conference on Education, Language and Art (ICELA 2023). Atlantis Press; 2024. p. 376–84. DOI: https://doi.org/10.2991/978-2-38476-214-9_45
Alshammari E. Student evaluation of teaching. Is it valid? J Adv Pharm Educ Res Apr-Jun. 2020;10(2):97.
Röhl S, Bijlsma H, Rollett W. The process model of student feedback on teaching (SFT): A theoretical framework and introductory remarks. Stud Feedback Teach Sch Using Stud Percept Dev Teach Teach. 2021;1–11. DOI: https://doi.org/10.1007/978-3-030-75150-0_1
Jensen E, Dale M, Donnelly PJ, Stone C, Kelly S, Godley A, et al. Toward automated feedback on teacher discourse to enhance teacher learning. In: Proceedings of the 2020 chi conference on human factors in computing systems. 2020. p. 1–13. DOI: https://doi.org/10.1145/3313831.3376418
Mandouit L, Hattie J. Revisiting “The Power of Feedback” from the perspective of the learner. Learn Instr. 2023;84:101718. DOI: https://doi.org/10.1016/j.learninstruc.2022.101718
Lim LA, Dawson S, Gašević D, Joksimović S, Pardo A, Fudge A, et al. Students’ perceptions of, and emotional responses to, personalised learning analytics-based feedback: an exploratory study of four courses. Assess Eval High Educ. 2021;46(3):339–59. DOI: https://doi.org/10.1080/02602938.2020.1782831
Chen S, Nieminen JH. Towards an ecological understanding of student emotions in feedback: a scoping review. Assess Eval High Educ. 2024;1–18. DOI: https://doi.org/10.1080/02602938.2024.2318641
Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) [Internet]. Minneapolis, Minnesota: Association for Computational Linguistics; 2019. p. 4171–86. Available from: https://aclanthology.org/N19-1423
Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. Language Models are Unsupervised Multitask Learners. 2019;
Altrabsheh N, Cocea M, Fallahkhair S. Learning sentiment from students’ feedback for real-time interventions in classrooms. In: International conference on adaptive and intelligent systems. Springer; 2014. p. 40–9. DOI: https://doi.org/10.1007/978-3-319-11298-5_5
Shaikh S, Daudpota SM, Yayilgan SY, Sindhu S. Exploring the potential of large-language models (LLMs) for student feedback sentiment analysis. In: 2023 International Conference on Frontiers of Information Technology (FIT). IEEE; 2023. p. 214–9. DOI: https://doi.org/10.1109/FIT60620.2023.00047
Häkkinen J, Ramadan Z. A Study on the Perception of Feedback with Varying Sentiment Generated Using a Large Language Model. 2023.
Mouronte-López ML, Ceres JS, Columbrans AM. Analysing the sentiments about the education system trough Twitter. Educ Inf Technol. 2023;28(9):10965–94. DOI: https://doi.org/10.1007/s10639-022-11493-8
Shen L, Wang M, Shen R. Affective e-learning: Using “emotional” data to improve learning in pervasive learning environment. J Educ Technol Soc. 2009;12(2):176–89.
Zhou J, Ye J min. Sentiment analysis in education research: a review of journal publications. Interact Learn Environ. 2023;31(3):1252–64. DOI: https://doi.org/10.1080/10494820.2020.1826985
Cutroni L, Paladino A. Peer-ing in: A systematic review and framework of peer review of teaching in higher education. Teach Teach Educ. 2023;133:104302. DOI: https://doi.org/10.1016/j.tate.2023.104302
Zhu JJ, Chang YC, Ku CH, Li SY, Chen CJ. Online critical review classification in response strategy and service provider rating: Algorithms from heuristic processing, sentiment analysis to deep learning. J Bus Res. 2021;129:860–77. DOI: https://doi.org/10.1016/j.jbusres.2020.11.007
Cox A. Exploring the impact of Artificial Intelligence and robots on higher education through literature-based design fictions. Int J Educ Technol High Educ. 2021;18(1):1–19. DOI: https://doi.org/10.1186/s41239-020-00237-8
Tian F, Zheng Q, Zhao R, Chen T, Jia X. Can e-Learner’s emotion be recognized from interactive Chinese texts? In: 2009 13th International Conference on Computer Supported Cooperative Work in Design. 2009. p. 546–51.
Ortigosa A, Martín JM, Carro RM. Sentiment analysis in Facebook and its application to e-learning. Comput Hum Behav. 2014;31:527–41. DOI: https://doi.org/10.1016/j.chb.2013.05.024
Poulos A, Mahony MJ. Effectiveness of feedback: The students’ perspective. Assess Eval High Educ. 2008;33(2):143–54. DOI: https://doi.org/10.1080/02602930601127869
Gan Z, He J, Zhang LJ, Schumacker R. Examining the relationships between feedback practices and learning motivation. Meas Interdiscip Res Perspect. 2023;21(1):38–50. DOI: https://doi.org/10.1080/15366367.2022.2061236
Kanwar A, Sanjeeva M. Student satisfaction survey: A key for quality improvement in the higher education institution. J Innov Entrep. 2022;11(1):27. DOI: https://doi.org/10.1186/s13731-022-00196-6
Giang NTP, Dien TT, Khoa TTM. Sentiment analysis for university students’ feedback. In: Advances in Information and Communication: Proceedings of the 2020 Future of Information and Communication Conference (FICC), Volume 2. Springer; 2020. p. 55–66. DOI: https://doi.org/10.1007/978-3-030-39442-4_5
Dalipi F, Zdravkova K, Ahlgren F. Sentiment analysis of students’ feedback in MOOCs: A systematic literature review. Front Artif Intell. 2021;4:728708. DOI: https://doi.org/10.3389/frai.2021.728708
Neumann M, Linzmayer R. Capturing student feedback and emotions in large computing courses: A sentiment analysis approach. In: Proceedings of the 52nd ACM Technical Symposium on Computer Science Education. 2021. p. 541–7. DOI: https://doi.org/10.1145/3408877.3432403
Ren P, Yang L, Luo F. Automatic scoring of student feedback for teaching evaluation based on aspect-level sentiment analysis. Educ Inf Technol. 2023;28(1):797–814. DOI: https://doi.org/10.1007/s10639-022-11151-z
Katragadda S, Ravi V, Kumar P, Lakshmi GJ. Performance analysis on student feedback using machine learning algorithms. In: 2020 6th international conference on advanced computing and communication systems (ICACCS). IEEE; 2020. p. 1161–3. DOI: https://doi.org/10.1109/ICACCS48705.2020.9074334
Mabunda JGK, Jadhav A, Ajoodha R. Sentiment analysis of student textual feedback to improve teaching. In: Interdisciplinary Research in Technology and Management. CRC Press; 2021. p. 643–51. DOI: https://doi.org/10.1201/9781003202240-100
Reddy SS, Gadiraju M, Maheswara Rao V. Analyzing student reviews on teacher performance using long short-term memory. In: Innovative Data Communication Technologies and Application: Proceedings of ICIDCA 2021. Springer; 2022. p. 539–53. DOI: https://doi.org/10.1007/978-981-16-7167-8_39
Agostini D, Picasso F. Large Language Models for Sustainable Assessment and Feedback in Higher Education: Towards a Pedagogical and Technological Framework. In: Proceedings of the First International Workshop on High-Performance Artificial Intelligence Systems in Education Co-Located with 22nd International Conference of the Italian Association for Artificial Intelligence (AIxIA 2023). 2023.
Parker MJ, Anderson C, Stone C, Oh Y. A large language model approach to educational survey feedback analysis. ArXiv Prepr ArXiv230917447. 2023;
Freyd M. A graphic rating scale for teachers. J Educ Res. 1923;8(5):433–9. DOI: https://doi.org/10.1080/00220671.1923.10879421
Boring A, Ottoboni K. Student evaluations of teaching (mostly) do not measure teaching effectiveness. Sci Res. 2016; DOI: https://doi.org/10.14293/S2199-1006.1.SOR-EDU.AETBZC.v1
Hoel A, Dahl TI. Why bother? Student motivation to participate in student evaluations of teaching. Assess Eval High Educ. 2019;44(3):361–78. DOI: https://doi.org/10.1080/02602938.2018.1511969
Chen Y. Does students’ evaluation of teaching improve teaching quality? Improvement versus the reversal effect. Assess Eval High Educ. 2023;48(8):1195–207. DOI: https://doi.org/10.1080/02602938.2023.2177252
Newman H, Joyner D. Sentiment Analysis of Student Evaluations of Teaching. In: Penstein Rosé C, Martínez-Maldonado R, Hoppe HU, Luckin R, Mavrikis M, Porayska-Pomsta K, et al., editors. Artificial Intelligence in Education. Cham: Springer International Publishing; 2018. p. 246–50.
Dake DK, Gyimah E. Using sentiment analysis to evaluate qualitative students’ responses. Educ Inf Technol. 2023;28(4):4629–47. DOI: https://doi.org/10.1007/s10639-022-11349-1
Falcon S, Leon J. How do teachers engaging messages affect students? A sentiment analysis. Educ Technol Res Dev. 2023;71(4):1503–23. DOI: https://doi.org/10.1007/s11423-023-10230-3
Bird S, Loper E. NLTK: The Natural Language Toolkit. In: Proceedings of the ACL Interactive Poster and Demonstration Sessions [Internet]. Barcelona, Spain: Association for Computational Linguistics; 2004 [cited 2021 Jul 3]. p. 214–7. Available from: https://aclanthology.org/P04-3031
Honnibal M, Montani I. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. Appear. 2018;
Pérez JM, Giudici JC, Luque F. pysentimiento: A Python Toolkit for Sentiment Analysis and SocialNLP tasks. 2021.
Pérez JM, Furman DA, Alonso Alemany L, Luque FM. RoBERTuito: a pre-trained language model for social media text in Spanish. In: Proceedings of the Thirteenth Language Resources and Evaluation Conference [Internet]. Marseille, France: European Language Resources Association; 2022. p. 7235–43. Available from: https://aclanthology.org/2022.lrec-1.785
García-Vega M, Díaz-Galiano M, García-Cumbreras M, Del Arco F, Montejo-Ráez A, Jiménez-Zafra S, et al. Overview of TASS 2020: Introducing emotion detection. In: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020) Co-Located with 36th Conference of the Spanish Society for Natural Language Processing (SEPLN 2020), Málaga, Spain. 2020. p. 163–70.
Hutto C, Gilbert E. VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text. Proc Int AAAI Conf Web Soc Media. 2014 May;8(1):216–25. DOI: https://doi.org/10.1609/icwsm.v8i1.14550
Sanh V, Debut L, Chaumond J, Wolf T. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. ArXiv Prepr ArXiv191001108. 2019;
Adoma AF, Henry NM, Chen W. Comparative analyses of bert, roberta, distilbert, and xlnet for text-based emotion recognition. In: 2020 17th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP). IEEE; 2020. p. 117–21. DOI: https://doi.org/10.1109/ICCWAMTIP51612.2020.9317379
Accepted 2024-06-17
Published 2024-06-20
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Authors grant the journal and Universidad del Valle the economic rights over accepted manuscripts, but may make any reuse they deem appropriate for professional, educational, academic or scientific reasons, in accordance with the terms of the license granted by the journal to all its articles.
Articles will be published under the Creative Commons 4.0 BY-NC-SA licence (Attribution-NonCommercial-ShareAlike).