Contenido principal del artículo

Autores

La evaluación del estudiante sobre la enseñanza (SET) es una forma ad hoc de evaluar la efectividad docente en instituciones de educación superior. En este documento, presentamos un enfoque para analizar los sentimientos expresados en los comentarios de SET utilizando un modelo de lenguaje grande (LLM). Al emplear técnicas de procesamiento de lenguaje natural, extraemos y analizamos los sentimientos expresados por los estudiantes al finalizar el curso, con el objetivo de proporcionar a educadores y administradores ideas valiosas sobre la calidad de la enseñanza y elementos a mejorar de la práctica docente. Nuestro estudio demuestra la efectividad de los LLM en el análisis de sentimientos de los comentarios, resaltando su potencial para mejorar el proceso de evaluación. Nuestros experimentos con un conjunto de datos etiquetados de forma colaborativa demuestran un 93% de precisión en la clasificación de los mensajes. Discutimos las implicaciones de nuestros hallazgos para las instituciones educativas y proponemos futuras direcciones para la investigación en este ámbito.

Jefferson A Peña-Torres, Pontificia Universidad Javeriana, Cali, Colombia

https://orcid.org/0000-0002-3879-3320

1.
Peña-Torres JA. Hacia una mejora en la práctica docente utilizando el Análisis de Sentimiento en la Evaluación de Estudiantes. inycomp [Internet]. 20 de junio de 2024 [citado 16 de julio de 2024];26(2):e-21013759. Disponible en: https://revistaingenieria.univalle.edu.co/index.php/ingenieria_y_competitividad/article/view/13759

Omer K, Jacobs S, Bettger B, Dawson J, Graether S, Murrant C, et al. Evaluating and Improving the Formative Use of Student Evaluations of Teaching. Can J Scholarsh Teach Learn. 2023;14(1):n1. DOI: https://doi.org/10.5206/cjsotlrcacea.2023.1.10960

Mancenido Z. Impact evaluations of teacher preparation practices: Challenges and opportunities for more rigorous research. Rev Educ Res. 2023;00346543231174413. DOI: https://doi.org/10.3102/1883286

Cunningham S, Laundon M, Cathcart A, Bashar MA, Nayak R. First, do no harm: automated detection of abusive comments in student evaluation of teaching surveys. Assess Eval High Educ. 2023;48(3):377–89. DOI: https://doi.org/10.1080/02602938.2022.2081668

Gravestock P, Gregor-Greenleaf E. Student course evaluations: Research, models and trends. Higher Education Quality Council of Ontario Toronto; 2008.

Uttl B. Lessons learned from research on student evaluation of teaching in higher education. Stud Feedback Teach Sch Using Stud Percept Dev Teach Teach. 2021;237–56. DOI: https://doi.org/10.1007/978-3-030-75150-0_15

Gaoa G, Pangb M, Pengc J, Lud Y. A Hierarchical Probe Evaluation Method for Teaching Effect of University Engineering Courses Based on the Keypoints of Knowledge. In: 3rd International Conference on Education, Language and Art (ICELA 2023). Atlantis Press; 2024. p. 376–84. DOI: https://doi.org/10.2991/978-2-38476-214-9_45

Alshammari E. Student evaluation of teaching. Is it valid? J Adv Pharm Educ Res Apr-Jun. 2020;10(2):97.

Röhl S, Bijlsma H, Rollett W. The process model of student feedback on teaching (SFT): A theoretical framework and introductory remarks. Stud Feedback Teach Sch Using Stud Percept Dev Teach Teach. 2021;1–11. DOI: https://doi.org/10.1007/978-3-030-75150-0_1

Jensen E, Dale M, Donnelly PJ, Stone C, Kelly S, Godley A, et al. Toward automated feedback on teacher discourse to enhance teacher learning. In: Proceedings of the 2020 chi conference on human factors in computing systems. 2020. p. 1–13. DOI: https://doi.org/10.1145/3313831.3376418

Mandouit L, Hattie J. Revisiting “The Power of Feedback” from the perspective of the learner. Learn Instr. 2023;84:101718. DOI: https://doi.org/10.1016/j.learninstruc.2022.101718

Lim LA, Dawson S, Gašević D, Joksimović S, Pardo A, Fudge A, et al. Students’ perceptions of, and emotional responses to, personalised learning analytics-based feedback: an exploratory study of four courses. Assess Eval High Educ. 2021;46(3):339–59. DOI: https://doi.org/10.1080/02602938.2020.1782831

Chen S, Nieminen JH. Towards an ecological understanding of student emotions in feedback: a scoping review. Assess Eval High Educ. 2024;1–18. DOI: https://doi.org/10.1080/02602938.2024.2318641

Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) [Internet]. Minneapolis, Minnesota: Association for Computational Linguistics; 2019. p. 4171–86. Available from: https://aclanthology.org/N19-1423

Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. Language Models are Unsupervised Multitask Learners. 2019;

Altrabsheh N, Cocea M, Fallahkhair S. Learning sentiment from students’ feedback for real-time interventions in classrooms. In: International conference on adaptive and intelligent systems. Springer; 2014. p. 40–9. DOI: https://doi.org/10.1007/978-3-319-11298-5_5

Shaikh S, Daudpota SM, Yayilgan SY, Sindhu S. Exploring the potential of large-language models (LLMs) for student feedback sentiment analysis. In: 2023 International Conference on Frontiers of Information Technology (FIT). IEEE; 2023. p. 214–9. DOI: https://doi.org/10.1109/FIT60620.2023.00047

Häkkinen J, Ramadan Z. A Study on the Perception of Feedback with Varying Sentiment Generated Using a Large Language Model. 2023.

Mouronte-López ML, Ceres JS, Columbrans AM. Analysing the sentiments about the education system trough Twitter. Educ Inf Technol. 2023;28(9):10965–94. DOI: https://doi.org/10.1007/s10639-022-11493-8

Shen L, Wang M, Shen R. Affective e-learning: Using “emotional” data to improve learning in pervasive learning environment. J Educ Technol Soc. 2009;12(2):176–89.

Zhou J, Ye J min. Sentiment analysis in education research: a review of journal publications. Interact Learn Environ. 2023;31(3):1252–64. DOI: https://doi.org/10.1080/10494820.2020.1826985

Cutroni L, Paladino A. Peer-ing in: A systematic review and framework of peer review of teaching in higher education. Teach Teach Educ. 2023;133:104302. DOI: https://doi.org/10.1016/j.tate.2023.104302

Zhu JJ, Chang YC, Ku CH, Li SY, Chen CJ. Online critical review classification in response strategy and service provider rating: Algorithms from heuristic processing, sentiment analysis to deep learning. J Bus Res. 2021;129:860–77. DOI: https://doi.org/10.1016/j.jbusres.2020.11.007

Cox A. Exploring the impact of Artificial Intelligence and robots on higher education through literature-based design fictions. Int J Educ Technol High Educ. 2021;18(1):1–19. DOI: https://doi.org/10.1186/s41239-020-00237-8

Tian F, Zheng Q, Zhao R, Chen T, Jia X. Can e-Learner’s emotion be recognized from interactive Chinese texts? In: 2009 13th International Conference on Computer Supported Cooperative Work in Design. 2009. p. 546–51.

Ortigosa A, Martín JM, Carro RM. Sentiment analysis in Facebook and its application to e-learning. Comput Hum Behav. 2014;31:527–41. DOI: https://doi.org/10.1016/j.chb.2013.05.024

Poulos A, Mahony MJ. Effectiveness of feedback: The students’ perspective. Assess Eval High Educ. 2008;33(2):143–54. DOI: https://doi.org/10.1080/02602930601127869

Gan Z, He J, Zhang LJ, Schumacker R. Examining the relationships between feedback practices and learning motivation. Meas Interdiscip Res Perspect. 2023;21(1):38–50. DOI: https://doi.org/10.1080/15366367.2022.2061236

Kanwar A, Sanjeeva M. Student satisfaction survey: A key for quality improvement in the higher education institution. J Innov Entrep. 2022;11(1):27. DOI: https://doi.org/10.1186/s13731-022-00196-6

Giang NTP, Dien TT, Khoa TTM. Sentiment analysis for university students’ feedback. In: Advances in Information and Communication: Proceedings of the 2020 Future of Information and Communication Conference (FICC), Volume 2. Springer; 2020. p. 55–66. DOI: https://doi.org/10.1007/978-3-030-39442-4_5

Dalipi F, Zdravkova K, Ahlgren F. Sentiment analysis of students’ feedback in MOOCs: A systematic literature review. Front Artif Intell. 2021;4:728708. DOI: https://doi.org/10.3389/frai.2021.728708

Neumann M, Linzmayer R. Capturing student feedback and emotions in large computing courses: A sentiment analysis approach. In: Proceedings of the 52nd ACM Technical Symposium on Computer Science Education. 2021. p. 541–7. DOI: https://doi.org/10.1145/3408877.3432403

Ren P, Yang L, Luo F. Automatic scoring of student feedback for teaching evaluation based on aspect-level sentiment analysis. Educ Inf Technol. 2023;28(1):797–814. DOI: https://doi.org/10.1007/s10639-022-11151-z

Katragadda S, Ravi V, Kumar P, Lakshmi GJ. Performance analysis on student feedback using machine learning algorithms. In: 2020 6th international conference on advanced computing and communication systems (ICACCS). IEEE; 2020. p. 1161–3. DOI: https://doi.org/10.1109/ICACCS48705.2020.9074334

Mabunda JGK, Jadhav A, Ajoodha R. Sentiment analysis of student textual feedback to improve teaching. In: Interdisciplinary Research in Technology and Management. CRC Press; 2021. p. 643–51. DOI: https://doi.org/10.1201/9781003202240-100

Reddy SS, Gadiraju M, Maheswara Rao V. Analyzing student reviews on teacher performance using long short-term memory. In: Innovative Data Communication Technologies and Application: Proceedings of ICIDCA 2021. Springer; 2022. p. 539–53. DOI: https://doi.org/10.1007/978-981-16-7167-8_39

Agostini D, Picasso F. Large Language Models for Sustainable Assessment and Feedback in Higher Education: Towards a Pedagogical and Technological Framework. In: Proceedings of the First International Workshop on High-Performance Artificial Intelligence Systems in Education Co-Located with 22nd International Conference of the Italian Association for Artificial Intelligence (AIxIA 2023). 2023.

Parker MJ, Anderson C, Stone C, Oh Y. A large language model approach to educational survey feedback analysis. ArXiv Prepr ArXiv230917447. 2023;

Freyd M. A graphic rating scale for teachers. J Educ Res. 1923;8(5):433–9. DOI: https://doi.org/10.1080/00220671.1923.10879421

Boring A, Ottoboni K. Student evaluations of teaching (mostly) do not measure teaching effectiveness. Sci Res. 2016; DOI: https://doi.org/10.14293/S2199-1006.1.SOR-EDU.AETBZC.v1

Hoel A, Dahl TI. Why bother? Student motivation to participate in student evaluations of teaching. Assess Eval High Educ. 2019;44(3):361–78. DOI: https://doi.org/10.1080/02602938.2018.1511969

Chen Y. Does students’ evaluation of teaching improve teaching quality? Improvement versus the reversal effect. Assess Eval High Educ. 2023;48(8):1195–207. DOI: https://doi.org/10.1080/02602938.2023.2177252

Newman H, Joyner D. Sentiment Analysis of Student Evaluations of Teaching. In: Penstein Rosé C, Martínez-Maldonado R, Hoppe HU, Luckin R, Mavrikis M, Porayska-Pomsta K, et al., editors. Artificial Intelligence in Education. Cham: Springer International Publishing; 2018. p. 246–50.

Dake DK, Gyimah E. Using sentiment analysis to evaluate qualitative students’ responses. Educ Inf Technol. 2023;28(4):4629–47. DOI: https://doi.org/10.1007/s10639-022-11349-1

Falcon S, Leon J. How do teachers engaging messages affect students? A sentiment analysis. Educ Technol Res Dev. 2023;71(4):1503–23. DOI: https://doi.org/10.1007/s11423-023-10230-3

Bird S, Loper E. NLTK: The Natural Language Toolkit. In: Proceedings of the ACL Interactive Poster and Demonstration Sessions [Internet]. Barcelona, Spain: Association for Computational Linguistics; 2004 [cited 2021 Jul 3]. p. 214–7. Available from: https://aclanthology.org/P04-3031

Honnibal M, Montani I. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. Appear. 2018;

Pérez JM, Giudici JC, Luque F. pysentimiento: A Python Toolkit for Sentiment Analysis and SocialNLP tasks. 2021.

Pérez JM, Furman DA, Alonso Alemany L, Luque FM. RoBERTuito: a pre-trained language model for social media text in Spanish. In: Proceedings of the Thirteenth Language Resources and Evaluation Conference [Internet]. Marseille, France: European Language Resources Association; 2022. p. 7235–43. Available from: https://aclanthology.org/2022.lrec-1.785

García-Vega M, Díaz-Galiano M, García-Cumbreras M, Del Arco F, Montejo-Ráez A, Jiménez-Zafra S, et al. Overview of TASS 2020: Introducing emotion detection. In: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020) Co-Located with 36th Conference of the Spanish Society for Natural Language Processing (SEPLN 2020), Málaga, Spain. 2020. p. 163–70.

Hutto C, Gilbert E. VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text. Proc Int AAAI Conf Web Soc Media. 2014 May;8(1):216–25. DOI: https://doi.org/10.1609/icwsm.v8i1.14550

Sanh V, Debut L, Chaumond J, Wolf T. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. ArXiv Prepr ArXiv191001108. 2019;

Adoma AF, Henry NM, Chen W. Comparative analyses of bert, roberta, distilbert, and xlnet for text-based emotion recognition. In: 2020 17th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP). IEEE; 2020. p. 117–21. DOI: https://doi.org/10.1109/ICCWAMTIP51612.2020.9317379

Recibido 2024-04-07
Aceptado 2024-06-17
Publicado 2024-06-20