About this paper

Appears in:
Pages: 3198-3202
Publication year: 2021
ISBN: 978-84-09-31267-2
ISSN: 2340-1117
doi: 10.21125/edulearn.2021.0677

Conference name: 13th International Conference on Education and New Learning Technologies
Dates: 5-6 July, 2021
Location: Online Conference

QUANTIFYING THE BORDERLINE CANDIDATE IN STANDARD SETTING

J. McLachlan 1, A. Robertson2, B. Weller1, M. Sawdon3

1University of Central Lancashire (UNITED KINGDOM)
2Cumbria, Northumberland Tyne and Wear NHS Foundation Trust (UNITED KINGDOM)
3University of Sunderland (UNITED KINGDOM)
Background:
Conceptualising the Borderline candidate is one of the most difficult tasks in standard setting. However, it is also central to the process. Here we set out to develop a methodology by which the score of borderline candidates can be retrospectively calculated from the Facility index (the percentage of items answered correctly) of assessment items.

Methods:
We explored performance of all candidates in an academic year in one UK medical school, covering 26 separate assessments. Each assessment had previously been standard set by either Angoff or Borderline Regressions methods. We identified Borderline candidates by reviewing their performance across all assessments in their year. A student was classed as 'Borderline' if they were within 1 Standard Error of Measurement above the pass score, or below the pass score, when a variety of cut-off points were explored experimentally. We plotted the item scores of the Borderline candidates as calculated by each method in comparison with Facility for the whole cohort, and fitted curves to the resulting distributions.

Results:
Borderline candidate scores intercepted the self-plot of all candidate scores at two places - at a facility of 100% and a facility of 20%. These correspond to all candidates getting the item correct and all candidates guessing the outcome. We observed a strong curvilinear distribution showed by Borderline candidates compared to the whole cohort. This relationship was well described by an exponential of the form y ≈ C·exp(F·x), where y is the Facility of Borderline candidates on that Item, x is the observed Item Facility of the whole cohort, and C and F are constants.
We found C and F had similar values under different conditions. Using the typical values for C and F and the observed cohort facility, we could predict the probable Facility for Borderline candidates over the test: in other words, we could calculate the appropriate cut score for Borderline candidates. Differentiating the equation indicates where the assessment ought to be most sensitive.

Conclusions:
This approach can be used to standard-set assessments in their entirety when they are low stakes or norm referenced, in preference to Cohen methods. Where Cohen methods are based on the performance of one candidate (or a very small number of candidates), this exponential method is based on all candidates and all items and is therefore more robust. In high stakes assessments, it can be used to correct values where the Facility is very different from the standard-set value, and its use in this context for the UK General Medical Council proposed national exam. It could also be used to standard set novel items such as Very Short Answer formats, where standard setting panels are unfamiliar with the expected performance of these items.
@InProceedings{MCLACHLAN2021QUA,
author = {McLachlan, J. and Robertson, A. and Weller, B. and Sawdon, M.},
title = {QUANTIFYING THE BORDERLINE CANDIDATE IN STANDARD SETTING},
series = {13th International Conference on Education and New Learning Technologies},
booktitle = {EDULEARN21 Proceedings},
isbn = {978-84-09-31267-2},
issn = {2340-1117},
doi = {10.21125/edulearn.2021.0677},
url = {https://dx.doi.org/10.21125/edulearn.2021.0677},
publisher = {IATED},
location = {Online Conference},
month = {5-6 July, 2021},
year = {2021},
pages = {3198-3202}}
TY - CONF
AU - J. McLachlan AU - A. Robertson AU - B. Weller AU - M. Sawdon
TI - QUANTIFYING THE BORDERLINE CANDIDATE IN STANDARD SETTING
SN - 978-84-09-31267-2/2340-1117
DO - 10.21125/edulearn.2021.0677
PY - 2021
Y1 - 5-6 July, 2021
CI - Online Conference
JO - 13th International Conference on Education and New Learning Technologies
JA - EDULEARN21 Proceedings
SP - 3198
EP - 3202
ER -
J. McLachlan, A. Robertson, B. Weller, M. Sawdon (2021) QUANTIFYING THE BORDERLINE CANDIDATE IN STANDARD SETTING, EDULEARN21 Proceedings, pp. 3198-3202.
User:
Pass: