Whatever bad things some people say about Microsoft, in the olden days they brought to market a raft of products, which were accessible, easy to use, and useful. MS Access is an example. It may have limitations as a commercial database engine, but as a sketch pad, a tool for collecting one's thought's, it is, in my opinion, hard to beat.
My current task is to design a set of iterations through scoring rate data to render the scoring rate as an objective measure of student ability and item difficulty. The raw data is set out in a single table, as shown in my last blog. On this I have written two queries:
Avg([HAItemB].[rate]) AS AvgOfrate
GROUP BY [HAItemB].[sessidx];
Avg(HAItemB.rate) AS AvgOfrate
GROUP BY HAItemB.item
ORDER BY HAItemB.item;
These queries calculate the average raw scoring rate for each session and each item. The item query looks like this:
A third query calculates the overall mean scoring rate:
SELECT Avg(HAItemB.rate) AS AvgOfrate FROM HAItemB;
The average rate happens to be 18.185, out of a grand total of 14,480 records.
I then joined this query with the two previous queries to calculate the scoring rate quotient (SRQ) for each student session and each item. The results for the above items are shown below.
I then used the session quotients to recalculate the items rates, and the item quotients to recalculate the student/session rates, as proposed in my last blog but one. The table/array below shows this being done for five items in the first session:
And this is where the GUI comes in. I can sit staring at those numbers and thinking about them. At first I could see that a number (Rate) was being divided by two different numbers (ItQ1 and SQ1), and I thought why not save time, multiply them together, and divide Rate by the resulting product? But, to paraphrase Buffy, that would be wrong.
It is the item adjusted session rates (SRateAdj1), which are grouped to form the first pass adjusted session average rates, and the session adjusted item rates (ItRateAdj1) which are grouped to form the first pass adjusted item average rates.
The queries are almost the same as before, except that they are written against the table containing the adjusted rates. So for the sessions we have:
Avg(AdjSesstable1.SRateAdj1) AS AvgOfSRateAdj1
GROUP BY AdjSesstable1.sessidx;
and for items we have:
Avg(AdjSesstable1.ItRateAdj1) AS AvgOfItRateAdj1
GROUP BY AdjSesstable1.item
ORDER BY AdjSesstable1.item;
For completeness, I ran a query to compute the overall adjusted average rates, but guess what? They were identical to each other and to the overall raw mean. I guess a true mathematician would have known that, but I was quite surprised. Anyway, from there it was quite easy to compute the second pass quotients. These are shown for items below, side by side with first pass numbers:
Although we are only looking at five items here, I find these numbers very encouraging. On the first pass, I asked myself the question: Why is the item "1+5" easier than the item "1+1"? Common sense would suggest this was anomalous, cause by the chance happenstance that in this sample, more able students addressed the item "5+1". And after the first iteration, when item rates have been adjusted for the ability of the students addressing them, the estimate of difficulty (given by the reciprocal of SRQ) of the item "1+1" has been increased, while that for "1+5" has been reduced.
I think that's enough for one blog. I'll continue with more iterations tomorrow, and if I like the results, I'll report on them.