Abstract:

Universities have launched study abroad programs, because study abroad experience will enhance students’ academic and professional goals. While the presence of study abroad experience is crucial, assessment of study abroad experience can identify problems in acquiring students’ study abroad outcomes, which will improve study abroad programs (Engle et al. 2004, Savicki et al. 2015). From the perspective of students, previous research has proposed an evaluation tool for students’ readiness of study abroad (Chang et al. 2011) and an evaluation method for the acquisition of study abroad outcomes (Association of American Colleges and Universities 2016). The former targets students before departing, and the latter after coming back.

Given the lack of evaluation targeting students during study abroad, this study proposes a computer tool that measures a probability for the acquisition of study abroad outcomes. This tool suggests student’s probability of success in terms of 1-to-8 levels (level 1 means “definitely wouldn’t acquire the outcomes, and level 8 is “would acquire the outcomes”) by analyzing students’ answers for open and close questions regarding on-going study abroad experiences. Based on the probability, students can reconsider how they should tackle study abroad dilemmas or problems, which would support students to acquire study abroad outcomes.

Training data were prepared for the multiple regression analysis for measuring the probability levels. The training data were compiled from 332 students who completed one-semester study abroad programs from Japan to English or Chinese-speaking areas. Independent variables were 1-to-8 level scores for the acquisition of study abroad outcomes that were subjectively judged by the students. Dependent variables were answers for 9 multiple-choice questions (a positive-negative two-point Likert-scale question and 8 four-point Likert-scale questions from negative to positive), and the frequency (n > 45) of words in answers for 5 open questions answered in Japanese. The linguistic properties were derived with a text mining tool (Higuchi 2016). The average response rate of these questions was 98.9% (Standard Deviation, 1.7), ranging from 93.1 % to 100.0%.

The multiple regression analysis demonstrated a strong correlation (r = 0.74) in a close test, and the statistical significance (p < 0.05) was observed in 8 dependent variables (3 multiple-choice questions and 5 frequently used words in open-ended questions). This regression model was also examined in a leave-one-out cross validation test, which demonstrated a moderate coefficient (r = 0.57) between the observed and predicted level scores.

References:
[1] Association of American Colleges and Universities 2016 Value Rubric Project.
[2] Chang, D.-F. et al. 2011. Evaluating college students’ perceptions of study abroad using fuzzy logic.
[3] Engle, L. et al. 2004. Assessing language acquisition and intercultural sensitivity development in relation to study abroad program design.
[4] Higuchi, K. 2016. A two-step approach to quantitative content analysis: KH Coder tutorial using Anne of Green Gables (Part I).
[5] Savicki, V. et al. (eds.) 2015. Assessing Study Abroad.

Keywords:

Education support, evaluation tool, study abroad.