Abstract:

Mentoring is defined as the interaction between two persons over an extended period of time (Tolan et al., 2014). School-based Mentoring (SBM) is a form of mentoring that includes structured youth mentoring programs occurring in the school setting with the purpose of promoting positive relationships between students and nonfamilial adults. SBM is popular in the United States, with an estimated 100 million in federal funding supporting these programs in 2011, despite mixed results regarding effectiveness of these programs (DuBois et al., 2011; Lyons & McQuillin, 2019; Wood & Mayo-Wilson, 2012). These programs operate on the assumption that a positive relationship between a student and an adult will foster healthy psychological, social, and academic development for students. However, Wheeler et al. (2010) analysis of three large SBM data sets reported small effects for nonfamilial adult relationships, peer support, perceived scholastic efficacy, truancy, absenteeism, and misconduct in school. Further, in a metanalysis of 6,072 participants, Wood and Mayo-Wilson (2012) reported a small positive effect for SBM on self-esteem and a near zero effect on academic achievement and behavior. A recent SBM study also found mixed effects for student outcomes, with small effects on academic and behavioral outcomes (Lyons & McQuillin, 2019).

Tolan et al. (2014) identified that most SBM literature is not descriptive regarding the implementation of programs or procedures for adherence including mentoring activities. The authors claimed that this lack of detailed reporting impeded the ability for programs to be organized around implementation strategies that strengthen the effect of mentoring. The lack of systematic reporting of mentoring activities limits the ability to study how mentorship moderates the impact on student outcomes, as well as explain mixed results. Despite the lack of documentation, the importance of evaluating the fidelity of implementation is well documented (e.g., Durlak & DuPre, 2008; Kaderavek & Justice, 2010; Nelson et al., 2012). Also, federal funding agencies such as the U.S. Department of Education recognize the need to document treatment fidelity. Treatment fidelity refers to the extent to which the program is implemented as intended (Yeaton & Sechrest, 1981).

Researchers (Gersten et al., 2005; Horner et al., 2005) have recommended ongoing direct observation as the ideal method for measuring treatment fidelity. However, direct observation is not feasible when evaluating large scale 1:1 SBM programs. Also, direct observation during a mentorship session could influence the fidelity itself merely by having an external observer present. Further, direct observations provide only snapshots of the fidelity as compared to a continuous monitoring.

The purpose of this work was to advance data collection methods used to monitor treatment fidelity in SBMs to inform research and evaluation efforts using iC.A.R.E. Mentoring of United Way of Summit County as an illustration. This organization trains and monitors a SBM in two large urban school districts. We outline a method for systematic reporting of interactions during mentoring sessions, map Dane and Schneider’s (1998) multi-dimensional framework for treatment fidelity onto the iC.A.R.E. Progress Report as an example, and illustrate how data transformation strategies can be applied to use the data in further statistical and mixed methods analyses.

Keywords:

School-based mentoring, youth mentorship, treatment fidelity, program evaluation.