Introduction
The distribution of points for multiple-response questions requires more attention than the normal multiple-choice questions with one correct answer. This includes not only the standard multiple response, but all question types that need a correct combination of answer options, like the Matching-Matrix, Drag & Drop, Graphic associate questions. With questions that have multiple correct answer options, candidates can choose multiple answer options and decide how many answers they choose. Depending on hte The scoring type selected for the question determines how the points are allocated based on the chosen answers.
Remindo offers four options for the distribution of points (the number of scoring types available depends on the chosen question type):
- Correct answer
- Score mapping
- Quotient rule
- Product line
The different scoring methods determine the number of points a candidate receives for each combination of answers to a question.
As an example, here is a multiple-response question:
Some countries are listed below. Select all European countries:
a. Canada
b. Netherlands
c. Morocco
d. Montenegro
Correct answer
The 'correct answer' option means the following:
"When the candidate chooses exactly the correct answer, he will receive the full number of points assigned to this question. In case of a (partially) incorrect answer, no points will be awarded."
In this case, if the candidate gives the correct answer to the sample question, he gets a point for choosing answer b in combination with choosing answer d. For any other combination the candidate earns no points.
One drawback of this approach is that if a candidate is certain of one answer option but also selects a second one that is incorrect, no points are awarded, even though the answer to the question is partially correct. This reduces the validity of the question.
Score mapping
The 'score mapping' option means:
"When the candidate chooses the correct answer, he gets a point for this. If the candidate chooses a wrong answer, a point will be deducted."
Unlike the ‘correct answer’ option, this option also takes incorrect answers into account. A candidate who ticks all the answers in the example question receives no points (2 points for the correct answers minus 2 points for the incorrect answers = 0 points).
A candidate who selects three answers to the example question (including the two correct answers) scores a total of one point for the question. That is, 2 points for the two correct answers, minus 1 point for one incorrect answer.
The disadvantage of this option is that it does not take into account the number of right or wrong answers in total. For example, if a candidate in the sample question chooses answer b and c, he gets 0 points (while one option is right), if he chooses answer a, b and c, he gets 0 points and if he chooses only wrong answers, he also gets 0 points. This causes the question to have little distinction in this way and is therefore not very effective.
Quotient rule
To overcome the point distribution problem in multiple response questions, there is the quotient rule. The quotient rule uses a formula that takes into account the proportion of correct and incorrect answers.
The formula is as follows:
Score = (PG/(1+PF))
- PG = proportion of correct answers = number of correct answers chosen divided by the number of correct answers in the question
- PF = proportion of incorrect answers = number of incorrect answers chosen divided by the number of incorrect answers in the question
This formula resolves many issues relating to the allocation of points for multiple-correct-answer questions. Selecting this option means: “I first determine how much the candidate’s correct answer is worth, and I do the same for the incorrect answer. The ratio between these two figures gives the score.”
A candidate who chooses answer a, b and c on the sample question with a total score of 1 thus gets a score of 0.25. If the candidate chooses answer a and b, he gets a score of 0.33. It provides a very nuanced picture of the value assigned to the candidate.
However, this rule has one drawback: when the candidate ticks all the answer options, it always results in a score of 0.5. This is therefore more than when the candidate specifically chose answer a and b. This could encourage candidates to tick all the answers anyway, if they are not sure. This can be prevented by choosing multiple answers (limited) in Remindo and specifying a limit for the number of answers the candidate can choose.
Product rule
With the product rule it is not necessary to specify a limit for the number of answer options that can be chosen. In the product rule, the proportion of error is not counted in its entirety, but is compared to the number of correct answers. The formula is as follows:
Score = PGx (1- PF x (1/2 + ((n-k)/n) x (1-1/2)))
- PG = proportion correct = number of correct answers chosen divided by the number of correct answers in the question
- PF = proportion incorrect = the number of incorrectly selected answers divided by the number of incorrect answers in the question
- n = the total number of possible answers
- k = the number of correct answer options
Choosing this option means the following:
How severely an incorrect answer must be penalized depends on the number of correct answers and the number of answer options in total. The comparison between the values of the correct and incorrect answer yields the score.
In the product rule, the extent to which the proportion of error is taken into account and leads to a deduction varies. This makes it less useful for the candidate to tick all the answer options. It leads to a better balanced score.
Different options, different scores
What do the 4 options yield in terms of score when we look at the sample question? The table below shows the results for different answer combinations. There are no bonus points in this example. The candidate can get a maximum of 1 point for this question.
As an example, we use the following multiple-response question with two correct and two incorrect answer options:
Some countries are listed below. Select all European countries:
a. Canada
b. Netherlands
c. Morocco
d. Montenegro
Outcomes Based on a total score of 1 point.
| Answer given | Correct answer | Score mapping | Quotient rule | Product rule |
|---|---|---|---|---|
| b, d | 1 | 1 | 1 | 1 |
| a, b | 0 | 0 | 0,33 | 0,31 |
| a, b, c* | 0 | 0 | 0,25 | 0,13 |
| a, b, c, d** | 0 | 0 | 0,5 | 0,25 |
| b, c | 0 | 0 | 0,33 | 0,31 |
| b, c, d | 0 | 0,5 | 0,67 | 0,63 |
| c, d | 0 | 0 | 0,33 | 0,31 |
* The product rule compares the number of wrong answers with the number of correct answers, therefore using the product rule results in a lower score for these answers.
** In this example, you can see that the effect of the product rule is better than that of the quotient rule. This is an example of a candidate who ticked all the answers.
Pre-indicate how many answers are correct
For multiple response questions, it is possible to indicate in advance how many answers are correct or to leave this blank. It only affects the score distribution if a limit has been set. This is possible in Remindo by choosing multiple answers (limited). When a limit is active, all scoring types can be chosen.
Though communicating in advance to the candidate how many answers are correct also affects the difficulty of the question.
Disclaimer: This text was automatically translated from the Dutch version.