Writing Exam Questions II

October 1, 2021   Martin Jones
Phrasing reliable exams questions.

This, the second post in a series on writing exam questions, demonstrates good phrasing and shows how to fix some common errors.

You can find an overview of the series at the checklist.

As with the other posts in the series, this represents my personal experience and not the results of a formal study.

Phrasing

I’m using the term phrasing to cover ideas associated with the writing and structuring of the exam questions. A well-written exam question should allow a taker to focus on providing a solution, not parsing the question. Hard to read questions lower the reliability of the exam.

The following points highlight some common mistakes and show ways to fix them.

Ph1: Prefer Direct and Simple

The exam text should be direct and use simple English. Good English comprehension is, of course, important. It is not, however, the thing that we should be measuring in a software developer exam. We’ll start with the following question:

  • While prepocessing, compiling, and linking some files together an error message appears on the console saying that there is an undefined symbol and that the command ld failed. What is a likely cause of this error and what might you do to fix it [3].
    • wrong
It’s tough to read that first sentence as a native English speaker. It’s possibly unfairly hard if English is not your primary language. Let’s start our improvements by making the sentences shorter, simpler, and more direct.

  • An error has occured while building a C++ project. The ld command has produced an 'undefined symbol' error.

    What is a likely cause of this error [1]. Write one possible fix [2].
    • better
Details such as ‘preprocessing, compiling, and linking’ and ‘on the console’ have been discarded. They added nothing of relevance.

The result could easily have stayed as a single sentence, but it would have been a long one. We get simpler sentences by splitting it, and the sentences have different purposes. The first sentence provides useful context, but exam takers are unlikely to need to read it more than once. The second sentence is the meat of the problem. An exam taker, even one with excellent English, is likely to read it more than once to confirm their understanding.

The challenge section has been separated to make it easier for the exam taker to distinguish the problem from what they’re being asked to do. The original sentence has been split into two. This has the added bonus of making the mark weighting clearer to the taker.

Ph2: Make it Concrete

Concrete scenarios are easier to understand than abstract ones. Let’s take the question from above and improve it further:

  • Building your C++ project causes the following error:

    /usr/bin/ld: /tmp/ccuLawBt.o: in function `main':
    x.cpp:(.text+0x10): undefined reference to `f()'
    collect2: error: ld returned 1 exit status


    What is a likely cause of this error [1]. Write one possible fix [2].
    • better

There are two main changes, one of which is subtle, so let’s start with the obvious one. The actual, concrete error message is now provided. There is no ambiguity about how it will be formatted. In addition, it’s similar to the error messages takers will have seen while doing their labs. If they have the required practical experience, this question should be a gift! We are no longer explicit about the error message coming from ld. Instead, we are validly assessing the taker’s ability to deduce that from the error message.

Now, the subtle one - ‘your’. It’s no longer an abstract, random project; It’s the exam taker’s. They have ownership of the project; they will also be expected to take ownership of any fix. I took a course on writing exams for international takers as part of my preparation for VSO and this was probably the piece of advice that stuck most in my mind. In some cases, it might be necessary to create names for individuals or items to make it clearer, e.g. the GEM library. It can on occasion seem a little twee for a US or a UK audience, but they don’t suffer for it and can ignore it. For someone from a culture that is more interested in people rather than getting things done, it makes the question more meaningful.

Still not quite done with this question though…

Ph3: Be Consistent with Directives

  • Building your C++ project causes the following error:

    /usr/bin/ld: /tmp/ccuLawBt.o: in function `main':
    x.cpp:(.text+0x10): undefined reference to `f()'
    collect2: error: ld returned 1 exit status


    State a likely cause of this error [1]. Describe one possible fix [2].
    • correct
The directives appear in the challenge. They’re what we’re asking the exam taker to do. In this case, they’re state and describe. You should have a clear, predefined set of directives and use them consistently. During preparation for an exam it might be worth exploring them with the exam takers.

Here are some suggestions:

Ph4: No Conflation

Imagine that a sample code has been provided, and the challenge says ‘State the three most likely places where implicit conversions (casts) will occur’. There is a conflation of ideas in the question, although I am sure the question writer was merely trying to be helpful.

The C++20 standard contains the phrase ‘implicit conversion’ more than fifty times. It contains the phrase ‘implicit cast’ zero times. A cast is the syntax for explicitly asking for a conversion to take place. The ‘(casts)’ should be removed because it’s just as likely to be misleading as helpful.

Ph5: Provide an Example

Provide an example answer when useful, e.g.:

  • The code below provides the context for this question:

    int i = 7;
    int* p = &i;


    To answer this question you must state the type and value category of the provided expressions. For example, the correct answer for the expression &i is int*, prvalue.

    State the type and value category of the following expressions:
    1.  i
    2.  p
    3.  &p
    4.  std::move(p)

correct
It’s far more important that a taker knows the value categories (prvalue, xvalue, etc.) rather than the term ‘value category’. By giving an example of a value category we allow a taker to focus on the crux of the question rather than working out what value category means.

The choice of example is important as well; it makes the style of answer we’re expecting clear. I would expect the answer ‘pointer to int’ to be acceptable, but it is made clear that we expect the shorter, more useful answer of int*. This distinction would not have been clear if an expression such as x had been used for the example.

Ph6: Positive Phrasing

Exam questions should be reviewed for hard to read conditions such as double negatives, e.g. ‘select all that are not without errors’. That should, of course, be ‘select all with errors’, and it is, admittedly, an extreme example.

Developers should be used to doing this for their conditionals, e.g. if (!noMoreData) should at least raise an eyebrow. Negatives themselves aren’t the enemy, though, and can be the simplest way of expressing something, e.g. if (!finished).