Abstract:

Increasingly, assessments in schools and colleges are using computer adaptive models. Computer adaptive tests (CATs) have many benefits when compared with traditional paper and pencil assessments. CATs use a student’s response to a question to decide on the next question to offer. If the student gets a question wrong, they may be offered a slightly easier question next time, if they get it right, they may be offered a slightly harder question. This means that assessments can be created that provide questions at an appropriate level of ability for individual students, enabling more questions to be offered within a specific difficulty range. Using CAT, less able students will not be offered questions that are too complex for them, and higher-ability students will not need to spend time answering questions that are too easy before they reach the questions that really stretch them.

Creating robust computer adaptive assessments for use on a large scale, which enable comparisons of performance across schools and populations, is a complex process. It requires extensive trialling of question items. Moreover, item banks need to be very large to allow for an appropriate balance of individual questions across the ability range. Similarly, rigorous statistical methods are used to ensure that the constructs within the assessment are measured appropriately and not prone to gender, cultural or any other kind of bias. Assessments of this kind tend to be used across a wide range of school types and geographical areas and must be psychometrically robust enough to measure accurately under these circumstances.

In addition to using large-scale CATs, there may be certain circumstances when an individual school or college may wish to create a small-scale CAT for use with its own students. In the past, creating such assessments has been complex and expensive, involving commercial solutions and the requirement for a mix of complex technical and statistical skills. However, platforms are now available which can ease the burden of assessment creation, moving the focus to the creation of specific assessments that are not constrained by technical requirements. Similarly, access to the statistics required to calculate the relative difficulty of test items has also been made simpler through the greater availability of free statistics analysis software. Whilst the process of assessment creation has become more straightforward in these respects, CAT designers must still address challenging questions around the curriculum or other constructs to be assessed, the ability range to be covered, and the extent to which questions that have already been written could be repurposed.

In this paper I detail the processes involved in creating a suitable question bank and designing a simple computer adaptive assessment using a freely available platform and statistical analysis environment, within the scope of schools, colleges, or other educational institutions. I also explore some advantages, risks, and considerations around this approach for aspiring CAT designers to think through.

Keywords:

Computer adaptive assessment.