Challenges in multisource feedback: intended and unintended outcomes

• Published in: Medical education Vol. 41; no. 6; pp. 583 - 591
• Main Authors: Sargeant, Joan, Mann, Karen, Sinclair, Douglas, Van der Vleuten, Cees, Metsemakers, Job
• Format: Journal Article
• Language: English
• Published: Oxford, UK Blackwell Publishing Ltd 01.06.2007; Blackwell
• Subjects: Alberta; Biological and medical sciences; Clinical Competence - standards; Education, Medical, Continuing; family; Family Practice - standards; feedback; Feedback, Psychological; Female; Health participants; Humans; Male; Medical sciences; Miscellaneous; Nova Scotia; physicians; Physicians, Family - standards; physicians, family, standards/ psychology; Pilot Projects; psychological; Public health. Hygiene; Public health. Hygiene-occupational medicine; questionnaires; standards/ psychology; Surveys and Questionnaires; Canada; Nova Scotia; North America; America; Alberta; Human; Prognosis; Questionnaire; Family study; Health staff; psychological; clinical competence/ standards; physicians; Psychology; standards/ psychology; family practice/ standards; questionnaires; pilot projects; Nova Scotia; Physician; Feedback regulation; Male; Psychometrics; Standards; humans; male; female; feedback; Family environment; Evolution; Female; General practice; family
• ISSN: 0308-0110; 1365-2923
• DOI: 10.1111/j.1365-2923.2007.02769.x

Context  Multisource feedback (MSF) is a type of formative assessment intended to guide learning and performance change. However, in earlier research, some doctors questioned its validity and did not use it for improvement, raising questions about its consequential validity (i.e. its ability to produce intended outcomes related to learning and change). The purpose of this qualitative study was to increase understanding of the consequential validity of MSF by exploring how doctors used their feedback and the conditions influencing this use. Methods  We used interviews with open‐ended questions. We purposefully recruited volunteer participants from 2 groups of family doctors who participated in a pilot assessment of MSF: those who received high (n = 25) and those who received average/lower (n = 44) scores. Results  Respondents included 12 in the higher‐ and 16 in the average/lower‐scoring groups. Fifteen interpreted their feedback as positive (i.e. confirming current practice) and did not make changes. Thirteen interpreted feedback as negative in 1 or more domains (i.e. not confirming their practice and indicating need for change). Seven reported making changes. The most common changes were in patient and team communication; the least common were in clinical competence. Positive influences upon change included receiving specific feedback consistent with other sources of feedback from credible reviewers who were able to observe the subjects. These reviewers were most frequently patients. Discussion  Findings suggest circumstances that may contribute to low consequential validity of MSF for doctors. Implications for practice include enhancing procedural credibility by ensuring reviewers' ability to observe respective behaviours, enhancing feedback usefulness by increasing its specificity, and considering the use of more objective measures of clinical competence.