De Rerum Natura

Just to show that not all questions behave as nicely as question 9 in the previous post, here is the plot for question 6 in the same exam.The fit is not as good as for question 9. This is expected, there is no reason why the precision error should decay with a perfect exponential behavior. Nonetheless, it still shows a similar decay constant — about six questions. Remember to click on the image to get the larger image in a zoom pane.

To show off the installation of FancyZoom (a trick I learned while visiting the excellent Language Log), I present a graph of the percentage variation in the mean square precision error as a function of the number of questions used to compute it. The image looks small but you can now click on it to obtain a zoomed in version. Try it!

Note how good the fit is to a shifted exponential function of the form:
$$a+b*\exp (-c(n_q–3)).$$
The measurements are the small dots at $n_q=\{\mathrm{4,6,7,10,12}\}$. The fitted values are $a=0.06$, $b=0.2$, and $c=0.43$. The variable $c$ is the decay constant for the variability in the estimate. In particular, if you calculate its inverse $1/c$ you get the number of questions beyond three that will give you less than 33% variability in the estimate. This turns out to be about 2 questions. So ten or twelve questions should be enough for this group of students.

Once again, this suggests that teachers are asking too many questions in their multiple choice exams.