• Version
• Download 242
• File Size 28.54 KB
• File Count 1
• Create Date August 2, 2018
• Last Updated August 2, 2018

Adaptive Comparative Judgment for reliable on-line assessment

Project e-scape1 was commissioned by the UK government to solve a continuing problem with the assessment of project-based coursework. Coursework is widely appreciated as the most valid expressions of learners’, capability –, be that in music, sciences, languages or technology, but despite this, they have consistently proved to be difficult to assess with an acceptable degree of reliability. ,Our approach to solving this problem was highly innovative. Rather than one teacher marking their own students’, portfolios, all teachers collaborate in assessing all the portfolios. Making use of web-connectivity we developed an on-line adaptive methodology in which teachers make simple comparative judgements (comparing the performance in portfolio A and portfolio B). A string of such simple binary judgements, by a connected group of teachers, results [via a Rasch modelling engine] in a rank order of astonishing reliability. Typically our trials have produced reliability statistics of 0.95 or better. The software adapts to the emerging consensus of the judging team, producing the reliability for minimal investment of judging time. ,Assessment projects using this technology have been conducted successfully in Western Australia, Atlanta (USA), Sweden, Israel, Ireland, the UK and most recently in Singapore with the MOE (Humanities and Art Divisions). ,1 See: http://www.gold.ac.uk/teru/projectinfo/projecttitle,5882,en.php

Attached Files