Saturday, May 29, 2010

Conundrum

The idea of my Rasch-Itembank open-source project at java.net is to produce a large numeracy estimation item bank, which updates the estimated difficulty parameters in real time, as it is used. To work properly, the client needs to generate items from the item bank according to the latest item difficulty estimation.

The most reliable current working version of my Java Math Test client is a complete cheat. It has absolutely no connection with a database, it is ISP hosted, and it usually works first time, as long as the user has the JRE installed.

A second version, hosted by me, is a partial cheat. It connects to a database to record performance data, but it uses hard coded arrays of items, whose difficulty has been estimated from static data collected in the past. This version is less reliable for many reasons. I don't always have the server turned on, I don't always have the database turned on, and sometimes my IP address changes.

The conundrum is that to get improved estimates of item difficulty, the client needs to be used, and to be used a lot, it needs to be reliable. Using hard coded item arrays, if the connection to the database fails, session data is lost, but the client continues to function, so the user is not pissed off.

However, in this more reliable configuration, a key selling point - that item difficulty and student ability mutually interact in real time, to product the best possible estimates of both - becomes something of a deception.

I think the best compromise solution is to leave reliable versions on my web site, perhaps with the hard coded arrays updated from time to time as more data comes in, and to develop code for the true blue dinky dye version without actually deploying it until data connection reliability can be assured.