Abstract:

Single-case experimental research (SCER) is one of the most commonly used research methods in special education to evaluate the effectiveness of behavioral interventions. Studies employing SCER designs are frequently excluded from meta-analyses of evidence-based practices mainly due to the lack of methodological consensus on the type of treatment effect indices to be used to determine the effect of a treatment on the dependent variable. A number of non-parametric indices have been proposed to calculate treatment effect estimates for data obtained from SCER studies (Parker, Vannest, & Davis, 2011).

The purpose of the present study was to extend the existing evidence on the performance of TauNovlap, Tau-U, Phi, robust Phi, and IRD by comparing and contrasting them in terms of:
(a) size of treatment effect they produce,
(b) agreement on the type of treatment effect,
(c) relationships with one another,
(d) discriminability, and
(e) agreement with visual analysis.

Five research questions were addressed in the present study:
(a) What is the typical range of treatment effect estimates calculated using each method?
(b) To what extend do five methods agree on the type of treatment effect?
(c) What are the relationships among treatment effect estimates calculated using these methods?
(d) How well do five methods discriminate among SSER datasets? and
(e) To what extent do these methods agree with the judgments of visual analysts about existence or non-existence of a change in data patterns?

A total of 200 A-B graphs were used in the present study. The published graphs were scanned or digitally copied from original articles and digitized using the Ungraph data extraction software (Biosoft, 2004). Treatment effect estimates using five non-overlap methods were individually calculated for each graph. Visual analyses of the graphs were conducted by two researchers who had previous experience with SCER and visual analysis.

Findings of the present study showed that overall, four of the five methods, IRD, Tau-U, TauNovlap, and Phi, had very high levels of correlation with each other, and were comparable with respect to mean treatment effect estimates they produced and their agreement with judgments of visual analysts. However, IRD, Tau-U, and TauNovlap showed better discriminability than Phi and robust Phi. Recommendations for future research and practice will be provided.

Keywords:

Single-case experimental research, effect size, treatment effect.