Sunday, July 5, 2009

Coding for Rasch

This is my first day back with my main project for two months. This is the page I wanted to write after my 24 April post. The question I want to address is what fields do I need to store data for subsequent Rasch analysis?

Rasch analysis investigates the interaction between test items and test participants. Core fields would therefore include the test item, the participant, and results of the interaction.

Recording the item is pretty easy. If a math student is asked for the product of two and two, the item is 2x2=?

Recording the student is a bit harder. It opens up a whole hornets nest of privacy and data protection issues. However, the analysis does not really require demographic details. All we need to record is that an entity addressing Item A of a session was the same entity addressing Item B in the same session. It does not matter that the same person may later complete another session. From a Rasch perspective, it is a new entity. It may be the same person a little older. It may be the same person with a headache, or feeling tired. It may even be the same person feeling the same; or it may be a different person with similar performance capabilities. All that matters is that data from a specific session is identified for the purpose of the analysis. For the time being I think a date stamp (in milliseconds) will suffice. If the tool ever became really popular, perhaps the date stamp could be augmented by an IP address.

For the interaction details, the Boolean result of the interaction is essential for conventional Rasch analysis, and for my preferred scoring rate metric, the time (in milliseconds) taken to achieve that result, is also essential. That can be converted into a scoring rate for the item, expressed as correct answers per minute (capm).

That will probably do for the main raw data table. That will be the data recorded in each session.

But what information should be served back to the participant? The most ordinary drill and practice program gives the user a raw score as they progress through an activity. Some (such as Mathletics) introduce the element of time, but usually following the model of a conventional speed test. From a psychometric perspective this is fraught with theoretical problems which have been amply discussed in the academic literature, although I don't have any references at my finger tips just now. Few, if any, give a scoring rate on individual items.

We can do that and go a stage further. Data drawn from the raw table can be processed to produce estimations of item difficulty. These estimations can be combined with real time interaction data to feed back estimations of student ability. This information can be displayed for students as they use the tool.

No comments: