Writing Exam Questions II
This, the second post in a series on writing exam questions, demonstrates good phrasing and shows how to fix some common errors.
You can find an overview of the series at the checklist.
As with the other posts in the series, this represents my personal experience and not the results of a formal study.
Phrasing
I’m using the term phrasing to cover ideas associated with the writing and structuring of the exam questions. A well-written exam question should allow a taker to focus on providing a solution, not parsing the question. Hard to read questions lower the reliability of the exam.
The following points highlight some common mistakes and show ways to fix them.
Ph1: Prefer Direct and Simple
The exam text should be direct and use simple English. Good English comprehension is, of course, important. It is not, however, the thing that we should be measuring in a software developer exam. We’ll start with the following question:
- While prepocessing, compiling, and linking some files together an error message appears
on the console saying that there is an undefined symbol and that the command
ld
failed. What is a likely cause of this error and what might you do to fix it [3].
- An error has occured while building a C++ project. The
ld
command has produced an 'undefined symbol' error.
What is a likely cause of this error [1]. Write one possible fix [2].
The result could easily have stayed as a single sentence, but it would have been a long one. We get simpler sentences by splitting it, and the sentences have different purposes. The first sentence provides useful context, but exam takers are unlikely to need to read it more than once. The second sentence is the meat of the problem. An exam taker, even one with excellent English, is likely to read it more than once to confirm their understanding.
The challenge section has been separated to make it easier for the exam taker to distinguish the problem from what they’re being asked to do. The original sentence has been split into two. This has the added bonus of making the mark weighting clearer to the taker.
Ph2: Make it Concrete
Concrete scenarios are easier to understand than abstract ones. Let’s take the question from above and improve it further:
- Building your C++ project causes the following error:
/usr/bin/ld: /tmp/ccuLawBt.o: in function `main':
x.cpp:(.text+0x10): undefined reference to `f()'
collect2: error: ld returned 1 exit status
What is a likely cause of this error [1]. Write one possible fix [2].
There are two main changes, one of which is subtle, so let’s start with the obvious one. The actual,
concrete error message is now provided. There is no ambiguity about how it will be formatted.
In addition, it’s similar to the error messages takers will have seen while doing their labs. If they
have the required practical experience, this question should be a gift! We are no longer explicit
about the error message coming from ld
. Instead, we are validly assessing the taker’s ability to deduce that
from the error message.
Now, the subtle one - ‘your’. It’s no longer an abstract, random project; It’s the exam taker’s. They have ownership of the project; they will also be expected to take ownership of any fix. I took a course on writing exams for international takers as part of my preparation for VSO and this was probably the piece of advice that stuck most in my mind. In some cases, it might be necessary to create names for individuals or items to make it clearer, e.g. the GEM library. It can on occasion seem a little twee for a US or a UK audience, but they don’t suffer for it and can ignore it. For someone from a culture that is more interested in people rather than getting things done, it makes the question more meaningful.
Still not quite done with this question though…
Ph3: Be Consistent with Directives
- Building your C++ project causes the following error:
/usr/bin/ld: /tmp/ccuLawBt.o: in function `main':
x.cpp:(.text+0x10): undefined reference to `f()'
collect2: error: ld returned 1 exit status
State a likely cause of this error [1]. Describe one possible fix [2].
Here are some suggestions:
- State - just a statement, very short.
- Describe - more details than a statement but remain at a high level.
- Explain in detail - provide interesting low-level details that help us to understand why or how.
- Explain with a code example - demonstrate with something concrete. Trivial syntax errors in the code are typically ignored while marking unless the taker has access to a development environment.
- Explore - show how you would reason about this challenge to create an answer. The final answer is possibly irrelevant; it’s the path taken that’s important.
- Draw - provide a picture, not text (although labels are allowed). Diagrams are often acceptable for other directives, but for this one, a diagram is explicitly required.
- List - give the facts as a bullet list, do not waste time writing it in correct English.
Ph4: No Conflation
Imagine that a sample code has been provided, and the challenge says ‘State the three most likely places where implicit conversions (casts) will occur’. There is a conflation of ideas in the question, although I am sure the question writer was merely trying to be helpful.
The C++20 standard contains the phrase ‘implicit conversion’ more than fifty times. It contains the phrase ‘implicit cast’ zero times. A cast is the syntax for explicitly asking for a conversion to take place. The ‘(casts)’ should be removed because it’s just as likely to be misleading as helpful.
Ph5: Provide an Example
Provide an example answer when useful, e.g.:
- The code below provides the context for this question:
int i = 7;
int* p = &i;
To answer this question you must state the type and value category of the provided expressions. For example, the correct answer for the expression&i
isint*
, prvalue.
State the type and value category of the following expressions:
-
i
-
p
-
&p
-
std::move(p)
-
The choice of example is important as well; it makes the style of answer we’re expecting clear. I would expect the
answer ‘pointer to int’ to be acceptable, but it is made clear that we expect the shorter, more useful answer of
int*
. This distinction would not have been clear if an expression such as x
had been used for the example.
Ph6: Positive Phrasing
Exam questions should be reviewed for hard to read conditions such as double negatives, e.g. ‘select all that are not without errors’. That should, of course, be ‘select all with errors’, and it is, admittedly, an extreme example.
Developers should be used to doing this for their conditionals, e.g. if (!noMoreData)
should at least
raise an eyebrow. Negatives themselves aren’t the enemy, though, and can be the simplest way
of expressing something, e.g. if (!finished)
.