I am currently working on a project in which we asked the teachers to rate 3 texts twice. Firstthey rated the three of them on computer and then the same teachers rated them on paper.
We have got two versions of texts, version 1 and versions 2. So a group of teachers rated
version 1 (which have 3 texts) and another group of teacher rated version 2 (which also have
3 texts).
We have got English native teachers and non native teachers. Group 1 of teachers, in which
there were both native and non native teachers, rated the version 1 of the texts. Group 2 of
teachers, which also have got both native and non native teachers, rated the version 2 of
texts.
The data is attached in the excel file. I organized the different data sheets as follows:
1. Sheet V#1 scores (includes all the teachers who used version 1 texts both native and
non-natives) with a total of 22 participants.
2. Sheet V#2 scores includes all the teachers who used version 2 texts both native and
non-natives) with a total of 22 participants.
3. Sheet Natives_V1 (all natives who used version 1 texts) with a total of 11
participants.
4. Sheet Non_natives_V1 (all non-natives who used version 1 texts) with a total of 11
participants.
5. Sheet Natives_V2 (all natives who used version 2 texts) with a total of 11
participants.
6. Sheet Non_Natives_V2 (all non-natives who used version 2 texts) with a total of 11
participants.
7. Sheet Natives_V1&2 includes all the natives together who used both version 1 and 2
with a total of 22 participants.
8. Sheet Non-Natives_V1&2 includes all the natives together who used both version 1
and 2 with a total of 22 participants.
I need to identify, analyze and test the inter-rater reliability of the teachers in the different
groups as explained above. I also need to identify the leniency and harshness of the raters
as well.
For the inter-rater reliability I need to measure the following:
● I need to analyze the data using Cohen’s Kappa inter-rater test.
● I also need to to test for leniency and harshness using multi-faceted rash model.
In summary, I need the following analyses:
1. Cohen’s Kappa inter-rater
2. Multi-faceted rash model
