Abstract:

Assessment designers in the Cambridge Digital Assessment Programme were making decisions about the assessment models to use to assess computing systems knowledge and understanding. One option was to use objective questions in an online exam which could be auto-marked using pattern-matching. This piece of responsive research aimed to gather evidence to inform decisions by answering:

1. If exam questions were manipulated to be auto-marked, can the level of demand be maintained?
2. If so, how can the demand be maintained (e.g. which question types could be used)?

The CRAS scale of demand was chosen as the analysis tool; the five dimensions of the scale are Complexity, Resources, Abstraction, Strategy (Task), and Strategy (Response). The analysis using CRAS framework revealed where the demand of the question might be affected. Questions in UK national (GCSE) and international (IGCSE) high stakes Computer Science assessments of computer systems were analysed.

The results showed how demand could be maintained for some, but not all, questions. Demand across the three dimensions of Complexity, Resources and Abstraction could usually be maintained, but demand in the Strategies (Response and Task) would most frequently decrease. For questions where demand could be maintained, the most likely item types to be used for auto-marking would be multiple choice questions of various designs, true / false, text entry, and inline choice (a fill-in-the-blank item where a single text option is chosen from a drop-down menu).

Analysis also uncovered evidence about:
• How different command words used in questions impacted on whether the demand would change or not, and
• the generalisability of the findings to other curriculum subjects.

Keywords:

Assessment, Assessment Design, Automarking, digital.