| C. Troussas, A. Krouska, C. Papakostas, Ph. Mylonas, C. Sgouropoulou |
| Assessing the Impact of Integrating ChatGPT as an Advice Generator in Educational Software |
| 9th South-East Europe Design Automation, Computer Engineering, Computer Networks and Social Media Conference (SEEDA-CECNSM 2024), Athens, Greece, September 20-22, 2024 |
|
ABSTRACT
|
| This paper reports on the study of the integration of ChatGPT as an advisor inside custom educational software developed for Java programming. The software, in cooperation with ChatGPT API, pursues providing real-time, context-specific advice to students for better learning. This work adopted a two-fold evaluation approach to evaluating this integration. First, this study examines the effectiveness of this integration with the help of the Interrupted Time Series Analysis methodology to measure possible improvement in the performance metrics of the students in terms of error rates and task completion times. Second, this work presents a custom-designed questionnaire used to get student perceptions regarding the clarity, usefulness, and impact of ChatGPT's advice, and the level of student satisfaction with the user interface. The key takeaways from this research study are the substantial improvements in performance metrics that were noted quantitatively, with students achieving lower error rates and faster completion times after the intervention of ChatGPT. Qualitatively, learners express their satisfaction with the clarity of advice, which gives them an understanding that works on their learning and confidence in Java programming. These findings point toward the promise of integrating such advanced AI solutions in educational software toward a significant improvement in learning outcomes and the necessity of human-aided continuous user feedback for system refinement.
|
| 20 September, 2024 |
| C. Troussas, A. Krouska, C. Papakostas, Ph. Mylonas, C. Sgouropoulou, "Assessing the Impact of Integrating ChatGPT as an Advice Generator in Educational Software", 9th South-East Europe Design Automation, Computer Engineering, Computer Networks and Social Media Conference (SEEDA-CECNSM 2024), Athens, Greece, September 20-22, 2024 |
[  PDF] [
BibTex] [
Print] [
 Back] |
