Friday, October 9, 2009

A closer look at scoring rates

In my blog of 25 August, I described some iterative transformations on scoring rate data from a computer based arithmetic test. I said I would report the results of further iterations, if I liked them, and from the time that has past it should be obvious to anyone reading this that I didn't.

The transformations were based on what I called the scoring rate quotient (SRQ). Essentially I divided the scoring rate for every item in every test session by the mean of all scoring rates for every item in every test session to produce the SRQ for individual session-item combinations and to calculate the mean SRQ for every session and for every item.

To illustrate, imagine two students, A and B, addressing two items, I and J. Imagine in this case that Student B scores at twice the rate of Student A and that Item I is twice as difficult as Item J. The raw scoring rates might look as follows:

 Raw Rates Item I Item J Session Mean Student A 4 8 6 Student B 8 16 12 Item Mean 6 12 9

The scoring rate quotients would then be as follows:

 Quotients Item I Item J Session Mean Student A 0.44 0.89 0.67 Student B 0.89 1.78 1.33 Item Mean 0.67 1.33 1.00

The session quotients can then be used to recalculate the item rates.

 Adjusted Rates Item I Item J Session Mean Student A 6 12 9 Student B 6 12 9 Item Mean 6 12 9

Or the item quotients can then be used to recalculate the session rates.

 Adjusted Rates Item I Item J Session Mean Student A 6 6 6 Student B 12 12 12 Item Mean 9 9 9

Expressing this algebraically, the means are calculated as follows:

 Session Mean A = (RIA + RJA)/2 (1) Session Mean B = (RIB + RJB)/2 Item Mean I = (RIA + RIB)/2 Item Mean J = (RJA + RJB)/2
 Grand Mean A = (RIA + RJA + RIB + RJB)/4 (2)

The scoring rate quotients are then calculated by dividing the raw scoring rates by the grand mean:

 SRQ IA = 4RIA/(RIA + RJA + RIB + RJB) (3) SRQ IB = 4RIB/(RIA + RJA + RIB + RJB) SRQ JA = 4RJA/(RIA + RJA + RIB + RJB) SRQ JB = 4RJB/(RIA + RJA + RIB + RJB)

The session mean quotients are then:

 Session A Mean Quotient = 4(RIA + RJA)/2(RIA + RJA + RIB + RJB) (4) = 2(RIA + RJA)/(RIA + RJA + RIB + RJB) Session B Mean Quotient = 2(RIB + RJB)/(RIA + RJA + RIB + RJB)

The item mean quotients are:

 Item I Mean Quotient = 2(RIA + RIB)/(RIA + RJA + RIB + RJB) Item J Mean Quotient = 2(RJA + RJB)/(RIA + RJA + RIB + RJB)

And the grand mean of the quotients is:

 Grand Mean SRQ = 4(RIA + RJA + RIB + RJB)/4(RIA + RJA + RIB + RJB) (5) = 1

The adjusted item rates are calculated by dividing the raw item rates by the session mean quotients.

 Adj Item IA = RIA(RIA + RJA + RIB + RJB)/2(RIA + RJA) (6) Adj Item JA = RJA(RIA + RJA + RIB + RJB)/2(RIA + RJA) Adj Item IB = RIB(RIA + RJA + RIB + RJB)/2(RIB + RJB) Adj Item IB = RIB(RIA + RJA + RIB + RJB)/2(RIB + RJB)

now we have stipulated that item I is twice as hard as item J so:

 RJA = 2RIA (7) and RJB = 2RIB

So we can re-write expression 6 as:

 Adj Item IA = RIA(RIA + 2RIA + RIB + 2RIB)/2(RIA + 2RIA) = RIA(3RIA + 3RIB)/6RIA = (3RIA + 3RIB)/6 = (RIA + RIB)/2 (8) Adj Item IB = RIB(RIA + 2RIA + RIB + 2RIB)/2(RIB + 2RIB) = RIB(3RIA + 3RIB)/6RIB = (RIA + RIB)/2 so Adj Item IA = Adj Item IB

Thus the adjusted item rate for item I is identical for both sessions, and also equal to the item mean for item I. The same is true for item J.

Of course this is the special case envisaged by Rasch, where all items are completed by all students. It was nice to work through this special case, because, in my mind at least, it indicates that a single pass transformation is sufficient, and that there is no need for multiple iterations.

In my next blog, I shall have a closer look at the more general case where not all items are completed by all students.