Abstract:

Conducting a pilot study enables researchers to add experiential knowledge to theoretical understanding. Pilot studies can overcome the predicament of investigating topics of which the researchers have little experiential understanding thus requiring educated guesses to design the study. This is especially true for educational research in the context of COVID-19. In 2020, the COVID-19 pandemic changed the nature of academic life, with face-to-face classes suspended and courses delivered online, using a combination of asynchronous and synchronous modes of instruction. We report on our efforts to pivot our Positive Psychology Team Build Exercise (PPTBE) workshop from face-to-face teams to virtual teams. The delivery of the PPTBE is very different in an online context and we were faced with the challenge of identifying the best way to deliver the workshop and the most appropriate measures to assess the PPTBE in this online context.

The pilot study had three aims:
(1) to gather data to provide guidance for the full study,
(2) to critically interrogate how to implement systems that capture observational and video-recorded data, and
(3) to explore the opportunities for machine-assisted analysis of the captured data.

Traditional pilot studies evaluate the feasibility of the full study’s design by focusing on the participant-level context and do not examine other contextual factors that may affect the ultimate practicality of the full study. We use an implementation-oriented framework to enable a richer understanding of the factors that impede or support the conduct of the subsequent full study.

The structure given by the implementation-oriented framework enables the pilot study to provide insight about possible failure points of the full study, such as where research protocols may not be followed, or whether proposed activities or instruments are inappropriate or overly complex. Information on factors, such as the students’ willingness to participate, the likely response rate and estimations of the resources (finance, staff) needed for the full study, are gathered as part of the pilot. We use the pilot to explore issues with time and resources that may occur during the study (e.g. how much time is required to complete questionnaires and workshop activities) and to highlight any problems with data collection and analysis (e.g. issues in the recording and storing of data; the nature of the data collected and its amenability to automated analysis).

The pilot uses the same software and configuration as the full study. The pilot study consists of eight people, selected from the same group as the full study, a facilitator, a moderator and a data manager. Participants are assigned to one of two groups, one group experience the positive psychology intervention and the other do not.

We measure the implementation outcomes using focus groups for the elements relevant to the participants. These discussions reveal the adequacy of research instruments and uncover items for further exploration. In addition, logistical problems and sensitive or problematic topics that arise may be identified. We also assess the capability of automated techniques to analyse the data, such as speech transcription, speaker diarisation, and parts-of-speech analysis.

Keywords:

Pilot study, implementation-oriented framework, positive psychology, team-building.