Chat Topic Classification for Student Counseling using Text Mining

Abstract

Background and Objectives: Alongside their standard duties, providing guidance and support to students is a crucial responsibility for teachers. The effective counseling of students is enhanced by technology, which improves communication between teachers and students. This study employs text-mining techniques to analyze and categorize chat messages exchanged on social media between teachers and students, demonstrating how technology can facilitate this communication effectively.

Methodology: This research implements a text-mining technique involving natural language processing (NLP) to create a classification model for chat topics. Chat messages used in text mining are collected from 20 students and 10 teachers using the Line and Facebook Messenger apps. Messages from both platforms had similar characteristics and were combined for analysis. The 4,500 chat messages were gathered, and after the cleaning process, 2,610 messages remained. The classification model is accomplished with Term Frequency–Inverse Document Frequency (TF-IDF) and three machine learning methods: random forest (RF), support vector machines (SVM), and logistic regression (LR) to build text classifiers. The model will be used to predict the counseling objectives of students.

Main Results: The evaluation of model performance utilized a 10-fold cross-validation technique due to the small size of the dataset, which helps prevent overfitting. The experimental results showed that the model using the RF technique achieved the highest accuracy among all techniques, with an overall F1 score of 89.55 percent. This was followed by the SVM at 88.68 percent and LR at 88.06 percent. When analyzing the models based on chat topics, the highest F1 score was recorded for the topic titled "Leave," followed by "Urgency," "Score," and "Homework."

Discussions: The RF technique consistently yielded the highest values in all chat topics. These results indicate that the RF technique is the most effective at accurately classifying chats compared to other techniques. Moreover, the evaluation of the technique's performance in this study found that the model's errors were caused by the model identifying many duplicate words across all chat topics. These words are not typically used in data analysis to identify relationships. Thus, future analyses may involve using language experts to eliminate these words.

Conclusions: The research findings can be used to categorize chats and predict their topic for student counseling. These findings can also be used to develop automated communication tools, such as integrated chatbots with e-learning. In addition, the model helps to resolve issues and streamline communication, reducing student wait times. However, the designed system has some limitations. It requires an extensive vocabulary corpus for each type of chat topic to improve the model's accuracy using text-mining techniques. Creating a vocabulary corpus for each type of chat topic necessitates linguistic experts and consumes significant time. Furthermore, the data being analyzed is collected from social media, which includes emerging vocabulary, such as chat language, that presents challenges for the model. Several improvements can be made shortly. For instance, the developed model can be improved using deep learning techniques and engaging linguistic experts to understand word characteristics and chat language better.