Multiple Choice Quizzes are (a) Ditzy; (b) Irrational (c) (etc.)

Somehow I'm just now catching up with Dafna Linzer's Slate piece on the pathetic flaws in the multiple choice quiz she had to pass in order to become a citizen.  I feel her pain: I served on a committee for a while years ago where we wrote multiple choice questions for the California "baby bar."  It was actually kind of fun: four of us would show up with four drafts and rip each other's work to shreds, a man's manly idea of a good time.  The trick was, of course to sandblast away every hint of ambiguity or potential for misunderstanding before the questions fell into the hands of the young, the litigious and the aspirational.

As I say I enjoyed it but one thing I learned for sure:  at least in the social sciences, it is impossible to write a good multiple choice question.  After agonizing effort, you maybe able to write a pretty-much-unambiguous question but it will be so stylized that it won't teach anybody about anything.  I suppose it might be different elsewhere: if I am asked to select the right value for "pi," I suppose I will know enough to check 3.13159... .  But law, never, and I am  sure the same goes for citizenship.

But then again, that's not the point.  Recall that success in education is learning how to psych the examiner.  So the student is looking, not for the correct answer, but the one the examiner most likely thought was correct.  This is the skill being tested by Ms. Linzer's citizenship test.  If she doubts it, she should read  Chapter 2 of Dixit and Nalebuff's Art of Strategy where they tell you to answer a multiple choice question correctly without even knowing the question  (the answer is "16π.").