Abstract:

Introduction:
Becoming a medical doctor in the United Kingdom (UK) requires completion of an undergraduate medical degree followed by postgraduate clinical training. Undergraduate education has become increasingly standardised over recent decades and is quality assured by the UK General Medical Council (GMC). The GMC are currently in the process of introducing a national assessment for all graduating doctors which will be a requirement of a licence to practice in the UK. This assessment for UK graduates will comprise of a knowledge based exam composed of multiple choice questions (MCQ), and a clinical exam organised by medical schools.
The GMC also define UK Medical schools’ curricula, within three main domains of Professional Values and Behaviours, Professional Skills and Professional Knowledge. Within these domains are many learning outcomes (LO) including those relating to sociology, psychology, population health and research methods. These subjects, among others, are highly contextual, and while knowledge is required, it is the evaluation and application of these areas which have a substantive impact on the practice of a doctor. However, these topics will be included within the MCQ component of the new assessment, and are currently included in most UK medical schools' MCQ assessments. There is currently no published evidence for either construct or content validity for these topics assessed in this modality.

Methodology:
MCQs relevant to sociology, psychology, population health or research methods were identified using tags and keywords from a national bank of MCQs from the UK Medical Schools Council Assessment Alliance. They were categorised by LO, topic, content and task. Pooled psychometric data was examined. MCQs were described as “high” performing using a discrimination or point biserial measures ≥0.2. MCQs were examined to see whether they would be considered to contain flaws according to question writing guidelines.

Results:
328 MCQs were identified, of which 113 had been used. 215 MCQs had been rejected during the validation process (65.5%). 26 MCQs assessed psychology (23.0%); 36 population health (31.9%); 47 research methods (41.6%), and 4 sociology (3.5%). Mean overall facility index was 0.55 (sd=0.25), mean discrimination index was 0.15 (0.12), and the mean point biserial was 0.09 (0.12). There were significant differences in facility, discrimination and point biserial measure compared by LO. High performing MCQs generally tested population health or research methods, and were more often knowledge-based. Low performing MCQs were more likely to assess psychology or sociology. Over 40% of MCQs in both groups contained flaws.

Conclusion:
Some areas of population health, sociology, psychology and research methods can be validly assessed using multiple choice questions, particularly the topics of epidemiology, infectious diseases, occupational health, screening or statistics. Sociology is significantly under-represented. Topics included represent only a small fraction of the required knowledge, and gives no opportunity to test application or assimilation of knowledge. Certain topics may be included because they fit this positivist format, and thus the assessment paradigm, rather than having any content validity. There is an urgent need to develop other assessment tools for population health, sociology, psychology and research methods topics, and to publish more evidence of existing assessment methods.

Keywords:

Medicine, assessment, behavioral science, population health, multiple choice question, health professions education.