Monday, February 11, 2013

My work with Multiple Choice Testing


One of my projects for work was to create an online testing application with the following goals:
Able to quickly generate a test and populate with multiple-choice questions and answers
Able to edit a question while keeping track of the previous question
Able to edit each answer while keeping the same history
Keep track of users
Keep track of users tests they've taken
Grade users tests as they are taken and report to management

This all is fairly easy. The real hard part is defining the different types of multiple-choice questions. At current there are 4 different types.

 1. Standard single answer question

  • This is with only one correct answer out of 2 or more answers. So pick A or B.
  • This question is worth one point.

 2. A single choice question, with multiple possible answers

  • So if you have a questions with 2 or more answers, two or more of them can be correct. 
  • Example question: "Do you know what a bear looks like?"
    Answers: A: Yes
                   B: No, what is a bear
                   C: Maybe, their brown, black or white right?
    In the example both A, and C could be considered correct. But this is a "pick one" question, so choosing either A or C would net you one point.
  • This question is worth only one point as only one answer can be selected.

 3. A multiple choice question (the complexity begins here)

  • The questions will typically have 3 or more answers.
  • Of the answers all of them can be selected.
  • Example question: "What colors are bears?"
    Answers: A: White
                   B: Black
                   C: Brown
                   D: Purple
    In this example A, B and C are all possible correct answers.
  • Because this is a multiple choice question each possible correct answer nets one point for a total of 3 points for this one question. 
  • If the user only picks B and C as their answers they will lose one point and the overall question is then not correctly answered.
In this I refer to a "point" based system. It's finding a balance on how to correctly grade a multiple choice question.

Some other fun test level options are:
Test Category
Group any number of tests into a category for an index display
Total possible wrong answers
Set the number of answers you can have wrong before failing the test
Test attempts
The number of times you can take the test in the selected Retake Interval (difficult issue)
Retake interval
The amount of time between passing the test, or failing it in a select number of attempts.
Multi-part test
Allows to break the test up and add reading material between groups of questions. Each part or section of the test has it's own individual questions and answers.
Test Order (ok this is boring)
Do I really need to explain this???

To explain Test Attempts and the Retake Interval a bit: Because this is testing, a tester should only be able to take the test just so many times before the system stops them. Once you either pass the test or fail the test N times in a selected retake interval, the application will then send an email notification that the test was completed, and give the results. The test, it's questions, and associated answers are all kept and the results will be stored and retained for future review.

A limitation so far is that a test is it's own entity. So each test has it's own questions and answers. There is no shared pool of questions. While this may not be a bad thing in some cases, the ability to "clone" a question and answers into another test might be nice for some cases.

One goal I have is adding another level to this and having it be "org" or group based. So something like a group of school teachers can each have their own index of tests. They could then assign them per student. The issue, how do you know the students before hand? My thought is that a given course would be handed a unique class code to register with so all of their class tests would be shown and not just every test possible.

Let me know if you have any questions or comments about the application. I'm really wanting to get a version of this, or something like it, built and opened up to the public. As is I haven't seen too many applications that can accommodate this.