• Version
• Download 166
• File Size 171.56 KB
• File Count 1
• Create Date August 2, 2018
• Last Updated August 2, 2018

Characteristics of Raters: Reliable and Valid Versus Unreliable and Invalid

 ,This study used the multi-faceted Rasch model, classical statistics and verbal aloud protocols (VAP) to analyse raters’, decision-making behaviours on a large-scale oral assessment, involving a stratified sample of 150 Secondary 3 (Grade 9) student performances from Hong Kong. Three different types of raters were used: four Non-native English Teachers, four Native English Speaking Teachers and four Naï,ve Native English Speakers. It was found ,that different raters behaved differently in their rating work. Moreover, the quantitative data obtained from Rasch and classical statistics tended to give the same results and supported the data from VAP. For reliable and valid judgement, four seemingly causal factors were identified: 1) Good understanding of test-takers’, knowledge in terms of lexical range and common world knowledge, 2) TESOL knowledge, 3) Ability to internalise the rating scale in terms of test takers’, performances, 4) Optimal duration of rating (so as to avoid rater fatigue).Three seemingly causal factors contributing to unreliable and invalid judgement were identified: 1) ‘,Applying partial requirements of the rating criteria’, typically involved theinability to separate high and low score points on stress, intonation and vocabulary. 2) ‘,False markers of ability’, i.e., English lexical items commonly ‘,imported’, into Cantonese (students’, mother tongue) which seemed ‘,advanced’, (to raters unfamiliar with the cohort), yet were not truly indicative of high student ability. 3) ‘,Jarring errors’, i.e., minor errors like ‘,go to shopping’, which were so obviously ‘,non-native’, in type as to have a disproportionately ,powerful negative effect on all raters.

Attached Files