Abstract:

This collaborative, explorative project has been investigating the possible role of computational linguistic techniques in providing automated formative feedback on student's written work - as opposed to their use in summative "automated marking". The student work included both traditional essays, and collaborative work using Wikis, by students in the School of Divinity, University of Edinburgh . The project included two phases:

In the first phase, we attempted to identify the criteria that are used in practice by academic staff when marking student work. We approached this by analysing the written feedback provided on samples of student work - this was discussed with the staff and subsequently refined to produce an explicit list of criteria. We then used a number of automated tools (Wordsmith, LIWC) to identify certain "surface" features which distinguished between the "good" and "bad" written work. The paper will describe the identified criteria in detail, and discuss their relationship to the automatically identified features.

In the second phase, we surveyed various computational techniques to determine if they could potentially be used to identify some of the criteria automatically. These techniques included analysis of "surface" features - such as the use of the passive or active voice, or appropriate referencing - as well as "deeper" techniques such as Latent Semantic Analysis (LSA) and "TextTiling". This identified a range of promising approaches, and some of these were tested, using real data from student work. The paper describes these techniques, and results in more detail.

We conclude that:

(1) There is a real potential to produce an automated tool which uses a range of techniques to provide practically useful formative feedback on student's written work.

(2) The academic staff felt that the process of rigorously defining the criteria had been of benefit to their manual marking in the future.

(3) We initially expected to apply the same criteria to the Wikis and the traditional essays. However, we discovered that these require different approaches which led to different styles and marking criteria.