Please Enter a Search Term
 
"Building a Healthy Tomorrow"  TM
AdmissionsCommunicationsDivision/DisciplineEducationFaculty/StaffLeadershipResearchStudents
Medical Education Scholarship Centre


Teaching/Learning Development - Faculty Development - Faculty Resources - Q&A about Summative and Formative Exams

SUMMATIVE/ FORMATIVE EXAMS:

GENERAL Q&A

 1.     What are the differences and similarities between formative and summative assessment?

Formative assessment is designed to provide ongoing feedback to students in order to improve performance. It can include self-assessment and self-testing such as practice tests and/or study questions with feedback.

Summative assessment is a comprehensive assessment process used to make decisions about pass/fail for the course and the need for remediation.

In the courses that utilize Multiple Choice Questions (MCQ) and Short Answer Questions (SAQ), the questions asked in the formative and summative exams should be similar in content and format. The formative quizzes are available each week via D2L to the students and they receive immediate feedback on their results. The summative exams are written in the computer lab using the Questionmark system in a secure invigilated setting. Students receive results within seven working days of the exam writing.

 

2.     Why do we have summative exams organized at the end of blocks in the courses?

Exams in some courses are scheduled to occur at key points in the progression of the course. Rather than having a single summative examination at the end of the Phase, the integrated material is tested at intervals over the length of the course. This allows for earlier detection and remediation of students experiencing difficulties.

 

3.     Do students receive a mark towards the summative grade by participating in formative quizzes? 

Yes, students are given a participation mark for completing the formative quizzes.

 

4.     What are the advantages of the Questionmark system for our examination process?

Questionmark offers the Faculty of Medicine the ability to offer online secure exams with easier authoring, and powerful reporting and analytic tools.  The Faculty of Medicine can use metadata to tag each question to learning objectives, that in conjunction with other technologies being used by Medicine, offers the ability to generate curriculum and examination blueprints. As with any learning tool, Questionmark has its own strengths and limitations; in our case, the strengths were judged to outweigh the limitations.

 

WRITING QUESTIONS

5.     What general advice can you give to help me develop questions?

The best questions follow these general guidelines

·         Make sure that the question can be answered without looking at the options.

·         If necessary, put most of the information in the question and lead-in to the question. Leave the options short.

·         Focus on testing important concepts

·         Ensure there are 5 responses provided to each question

6.     What are the do's and don'ts of preparing a question?

Students should not be given cues to the correct answer by virtue of a poorly written question. To avoid this:

·         Ensure compatibility between the question and all answer options when it comes to grammar, tense, pluralization, a/an (consonant/vowel), etc.

·         Ensure that all options are plausible

·         Ensure that all options are of similar length

On the other hand, an irrelevant level of difficulty should not be introduced by complex or vague question wording. Clinical scenarios may be complex and in-depth, but the phrasing of questions should not be the most difficult part of an MCQ. To avoid irrelevant difficulty:

·         Avoid questions phrased in the form “All of these EXCEPT” or “Which of these is NOT”.

·         Avoid options such as “all of the above”, “none of the above” or combination options e.g. “A; B; C; A&B; A, B & C”.

·         Avoid using absolutes in options e.g. “Always, Never”.

·         Avoid vague frequency terms in options e.g. “Usually, Often, Rarely”.

·         Avoid long and/or complicated options.

·         Avoid questions having multiple steps that have to be sequentially ordered

7.     Can other types of questions, besides MCQ, be submitted?

Yes, we will accept Fill-in-the-Blank (FITB) and R-Type questions.  It is preferable to have FITB questions with only one word as an answer because all questions are worth only one point.  These questions can be automatically graded using the correct answer provided but will have to be manually reviewed for spelling errors.

 

8.     Are True/False (T/F) or matching questions acceptable?

No, neither is acceptable.

 

9.     Can questions be worth half a point or are all questions worth one point?

All questions or parts of a multi-part question are worth one point.

 

10.   Can I submit a multi-part question? 

Yes. However, each part of the question will be worth 1 point.

 

11.   If I want assistance preparing questions, who can I contact?

Assistance is available from the Medical Education Scholarship Centre. The Teaching Consultant (Steve Shorlin) and Education Specialist for Assessment (Diana Deacon) are available to individuals and groups who would like to improve their item-writing skills. Contact information and resources are available at the Centre’s web site at www.med.mun.ca/MESC .

 

SUBMITTING QUESTIONS

 

12.   How did you determine the formula of "submit 3 formative and 4 summative questions"? Can I submit more?

The number of formative questions requested is based on the frequency of formative quizzes (1 per week). The number of summative questions requested is based on the number that have been requested in the past, given the time limitations for a summative examination. As well, when the summative exam is developed, a reassessment exam is also prepared in the event there are failures and reassessment is required. Faculty can submit more questions in either category but should put them in priority order (all questions will not be on all exams).

 

13.   What is the rationale for the establishment of the current deadlines for submitting our questions? 

The deadlines were established to provide sufficient time for review of questions by a subcommittee of the Phase Assessment Working Group.  Extra time is allotted for rewriting where necessary, entering questions into the Questionmark system, and testing the exam prior to its administration to students.

 

14.   What happens if I miss a deadline for submitting a question?

You will be sent a reminder email just prior to and immediately after the deadline. If no question is received after these reminders, you will be contacted by the UGME Associate Dean and/or the Phase Lead. If you do not submit formative questions, the students will not have the opportunity to assess themselves based on your session(s).  If you do not submit summative questions, the students cannot be examined on your material.

 

15.   If you ask for “x” number of questions per session, and this is what I submit, can some of these be deleted?

Yes, but only if they do not meet the NBME criteria. In this case, we would ask for a rewritten or new question to replace the one that was not used.

 

16.   Can the questions I submit be altered in any way?

Yes, if there are grammatical errors or if the question can be presented in a better format.  For example, an R-Type question may be presented as an A-Type with a drop-down list.

 

REVIEWING QUESTIONS

 

17.   What happens to my exam questions after I submit them to D2l@med.mun.ca?

Questions are first reviewed according to a checklist based on required information (correct answer, objective number, session title, and whether the question is formative or summative) and National Board of Medical Examiner guidelines for writing effective MCQs when they are submitted. Faculty members are contacted if they have submitted questions that lack information or do not meet the NBME guidelines.

At this stage, questions are entered into the appropriate system (D2L for formative quizzes and Questionmark for summative examinations). They are reviewed by a subgroup of the Phase Assessment Working Group that is headed by a member of the Phase Management Team. Faculty are contacted with any suggestions for rewrite that arise from this review.

The subgroup above then selects from the submitted questions according to the priority assigned by the author of the question, the number of hours per session, the objectives tested by each question, and the total number of questions required for the quiz or examination. Formative quizzes are made available in D2L to the students on a weekly basis. Summative examinations go through a review by the Phase Assessment Working Group, and a final review and approval by the Phase Management Team Lead and Phase Management Team Assessment Lead.

 

18.   What criteria are you using to evaluate the questions I submit?

Multiple-choice questions (MCQs) are the most common student assessment tool in the Faculty of Medicine, and they are the method by which some qualifying exams are delivered. The National Board of Medical Examiners (NBME) provides guidelines for writing effective MCQs and we have adopted them as our standards. If a question is determined to not meet these standards, the faculty member who wrote a question may be asked to revise the question.

 

19.   Can my questions be deleted without my permission?

If it is determined to be unacceptable – e.g., T/F or questions with two or more answers – the question will not be added to the question bank. You will be contacted to revise the question or submit another. 

 

FORMATIVE QUIZ PREPARATION

 

20.  Are students given all the formative questions that are submitted?  If not, what happens to those that are not used? 

A maximum of 30 questions are used in each weekly formative quiz, with the number depending on the number of teaching sessions included in the corresponding  period. Unused questions are entered in the formative test bank for future use.

 

21.   Do students get to keep all the formative questions?  If so, will the same formative questions be used each year? 

Students complete the formative quizzes on D2L and the questions are made available to students after the initial deadline for completion of the quiz. The same formative quizzes will not be used each year although some questions may be used again.

 

SUMMATIVE EXAM PREPARATION AND ADMINISTRATION

 

 22.  How do you determine how many and which questions are put on the exam?

The number of questions per session is based on the number of contact hours for the session. The number of questions chosen from a faculty member is proportional to the contact hours of teaching they have done for that exam. Questions are chosen to represent an appropriate and balanced sampling of the objectives. Faculty members are encouraged to prioritize the questions they submit to assist in choosing the questions that are used for their material. If you do not prioritize your questions, the Phase Assessment Working Group will select the questions used.

 

23.   I submitted 4 questions. Only two of them were used on the summative exam. Why weren’t all of my questions used?

        The number of questions used is proportional to the contact hours of teaching that you have done for that exam. Any questions above the number required are used for remediation and/or future exams.

 

24.   If you don't use all my summative questions why can't the students have access to the remainder as formative questions for practice?

        These extra questions may be needed for remediation and/or for future exams, so they are not made available to students as formative questions.

 

25.   What do you do with the extra summative questions I submit that are not used for an exam?

These questions are added to the summative exam question bank for use in remediation and/or future exams.

 

26.     What process is used to approve a summative exam once it has been prepared? 

The prepared summative exam is reviewed by the Phase Assessment Working Group with final review and approval by the Phase Management Team Lead and Phase Management Team Assessment Lead.

27.   As faculty do we have an opportunity to view the exam before it is given to students?

Faculty members can review their questions before the exam is administered. For security and copyright protection, faculty members cannot review the whole exam.

 

28.   Will there be a process in place for selecting questions for exams in future years once there is a large enough bank?  Will I get a request each year to submit questions?  Will duplicates be identified and not entered into the bank a second time?

 A process for selecting questions for exams in future years will evolve as the question bank grows. You will be asked to submit questions for exams but may choose questions that are already in the bank as well as writing new ones. Duplicates will be identified and not entered into the bank a second time.

 

29.   Who invigilates summative exams?  

 As outlined in the Invigilation Policy, summative examinations are invigilated by at least two individuals. These can be faculty members, preferably at least the Phase Lead, or staff members designated by the UGME Office.

 

POST-EXAMINATION REVIEW

 

30.   Will there be an opportunity to review ‘problematic’ questions to see if multiple answers are acceptable or if it is an unfair question that should be credited to all students?  Are there guidelines for such questions?

The exam results and statistical reports will be reviewed by the Assessment Working Group as soon as possible after the examination to identify any problems with the examination, including problematic questions and those challenged by students. This review will follow the guidelines established in the Summative Assessment Policy. The faculty member will be consulted if one of their questions is flagged as problematic so that they can suggest appropriate action on the question (i.e. crediting multiple answers or the whole question).

 

31.   Do I have the opportunity to review my questions for a remedial exam?

Yes, we will use the same process outlined above.

 

32.   Will I get the stats on how students performed on my questions?

Item analysis statistics for your questions are generated by Questionmark and are available to you on request.


Date:     March 17, 2014

 
Copyright © 2017 Memorial University of Newfoundland.   Newfoundland and Labrador, Canada.