Creating Good Exams

When I'm creating an new exam for a course, I typically consider 3 major factors: the category of questions, the content coverage, and the basic comprehension of the exam including using best practices for writing good exam questions.

Bloom's Taxonomy of Learning is a classification of learning objectives that educators use to evaluate their practice and students' performance. At the very bottom is knowledge. A student needs to be able to recall the basic concepts of a discipline to be able to do anything else with the content. However, this is the lowest level. As you move up the taxonomy, the kinds of learning objectives increase in difficulty, as do the skills necessary to meet the objectives. Educators have mapped different kinds of exam questions on to the taxonomy. For simplicity, I usually use 3 exam question classifications:

• Conceptual Understanding Questions: These questions go beyond recall and assess students’ understanding of important concepts. Answer choices to these questions are often based on common student misconceptions or points that students tend to confuse.

• Application Questions: These questions require students to apply their knowledge and understanding to particular situations and contexts. Application questions often ask students to make a decision or choice in a given scenario, connect course content to “real-world” situations, implement procedures or techniques, or predict the outcome of experiments.

• Critical Thinking Questions: These questions operate at the higher levels of thinking, requiring students to analyze relationships among multiple concepts or make evaluations based on particular criteria. These questions require students to engage critical thinking skills (i.e. identifying what the question is really about, recognizing unstated assumptions, prioritizing information for problem-solving. etc).

I aim to have about 1/3rd of questions come from each category of questions. It's important that students can recall the basic concepts, but I'm not that interested in students' ability to memorize and regurgitate my lectures. Application and critical thinking questions are more informative. They allow me to determine if students can actually use the concepts intelligently. 

I also work really hard to make sure there is fair coverage of the material. Exams can't test every concept. They are a sample of the concepts we've covered that allow me to estimate the total amount of information a student can recall and use from the course. But my selection of questions isn't totally random. For example, I try to make the number of questions per lecture proportional to the length of the lecture. I sometimes have short lectures to accommodate in-class tutorials, and those lectures have fewer questions on the exam. If I spent a lot of time on something in lecture, generally it will show up more prominently on the exam.

Finally, I take into account best practices for write clear exam questions. I don't want students to do poorly on the exam because they weren't sure what the question was getting at. I try to use simple grammar and I try to avoid colloquialisms that might be difficult for English Language Learners. I also try to avoid negatives. Sometimes it's hard to avoid, but if I feel like I need to include a negative, like "Which of the following is FALSE", I draw attention to the negative by making it capital letters or underlining it, or bolding it, or all three. 

At the McMaster Symposium on Education & Cognitive, I attended a workshop on how to write good multiple choice exams led by Dr. Joe Kim. Some of the changes I've made based on that workshop is to try to avoid using "None of the Above" or "All of the Above" because they tend to be correct more that 25% of the time (i.e. above chance). In addition, to correctly select "All of the Above" a student only has to recognize two of the options as correct. I've also stopped using options like "Both (A) and (C)". These are more tests of working memory and logic, and it's not fair to evaluate students on their core abilities rather than their understanding of course concepts.

I do other things to make sure my exams are fair and effective means of evaluating students' understanding and skills related to the course content, but those are the big 3 for me. Do you think I'm missing anything?