Abstract:

This study investigated applicability of natural language processing techniques of opinion mining and measuring linguistic complexity to evaluation of study abroad programs. The polarity of opinions (negative, neutral and positive) demonstrates the failure and success of the program, which will help identify problems in the programs. Opinion mining is a natural language technique for tracking sentiment and mood of sentences. According to El-Halees (2011), opinion miming provides supplementary evaluation data for the traditional course evaluation method.

Given this background, this study examined how well the opinion polarity is measurable statistically on the basis of linguistic features of opinions in terms of sentiment and mood polarities and linguistic complexity, which were derived from classifiers trained by machine learning. Opinion data were compiled from study abroad program evaluation results taken from 350 inbound students from Europe and north America to an Asian country. The evaluation results contained open-ended comments (approximately 6,500 words) as well as the answers in five-point Likert scale for students' satisfaction with the study abroad program. The latter demonstrated that most of the students positively evaluated the interaction with local students.

The polarity of opinions was determined by a faculty at the host university. The opinion data were also annotated with the sentiment polarity (positive or negative), mood polarity (happy or upset), and linguistic complexity (readability) using machine-learning-trained classifiers. Finally, multiple regression analysis was carried out taking these linguistic properties as independent variables, and the manual evaluation results as a dependent variable. The experimental result demonstrated a strong correlation (r = 0.62) between the observed and measured polarities. Thus, this study provided a quantitative method for study abroad program evaluation, which has not been studied thoroughly as pointed out by Savicki et al. (2015). We only had a quantitative evaluation method for students’ readiness for study abroad (Chang et al. 2011). Note that we had various qualitative methods (Association of American Colleges and Universities 2016) using Rubrics (Stevens 2005), for instance.

References:
[1] Association of American Colleges and Universities 2016 Value Rubric Project.
[2] Chang, D.-F. et al. 2011. “Evaluating college students’ perceptions of study abroad using fuzzy logic.” In Huai, H. et al. (eds.) Proceedings of the IADIS International Conference on International Higher Education, pp.19-26.
[3] El-Halees, A. 2011. “Mining opinions in user-generated contents to improve course evaluation.” In J. M. Zain et al. (eds.) Proceedings of Software Engineering and Computer Systems: Second International Conference, Part II, pp.107-115.
[4] Savicki, V. et al. (eds.) 2015. Assessing Study Abroad: Theory, Tools, and Practice. Sterling, VA: Stylus Publishing
[5] Stevens, D. D. et al. 2005. Introduction to Rubrics. An Assessment Tool to Save Grading Time, Convey Effective Feedback and Promote Student Learning. Sterling, VA: Stylus Publishing.

Keywords:

Study abroad program evaluation, natural language processing, opinion mining.