April 26, 2019 - LA-PMR
ABPMR CertLink Pilot Successful
Three years of groundwork, two years of testing, and a one-year pilot — all to determine whether longitudinal assessment could not only replace the MOC Examination, but also help diplomates learn, improve their performance, and increase their satisfaction with the certification process.
Thanks to the efforts of ABPMR staff and the dedication of our stellar volunteers, the American Board of Physical Medicine and Rehabilitation is announcing today that its pilot was an unqualified success.
“The longitudinal assessment pilot was developed as an MOC quality improvement project for our board,” says Mikaela M Raddatz, PhD, senior psychometrician at the ABPMR. “Four outcome variables were identified to determine the success of the pilot, including performance on the 10-year MOC Examination, performance in CertLink, performance on clone questions, and user satisfaction. The data from the pilot show that the ABPMR longitudinal assessment was successful in all four outcomes.”
In addition to the high user satisfaction results reported in this article, Dr. Raddatz's summary of other outcomes is below.
Diplomates who participated in the ABPMR longitudinal assessment pilot performed significantly better on the MOC Examination than those who did not participate in the pilot. Research shows that tests administered often and spaced out in time promote better retention of information. It is evident that the ABPMR longitudinal assessment assisted pilot participants in preparation for the MOC Examination by answering quarterly questions to promote learning and knowledge retention. Upon additional analysis, it was found that there were no significant differences in Part I or previous MOC Examination scores between pilot participants and physicians who did not participate in the pilot. Therefore, we concluded that the longitudinal assessment was a significant factor in higher performance on the MOC Examination.
Learning through immediate feedback
Feedback has been shown to be a crucial aspect to learning from tests and assessments. After a question has been answered, the ABPMR longitudinal assessment platform provides immediate feedback about the correctness of the answer, as well as a critique that consists of a short paragraph explaining why the right answer is correct and the distractors are incorrect and places this information within the larger context of the topic. Data from the pilot showed that participants spent time reading critiques and performed better on the second attempt of the question.
Learning through spaced-repetition
The ABPMR longitudinal assessment platform emphasizes learning through repeated testing of information. Clone questions, which assess the same content area and are similar to the primary questions, are administered in specific time intervals to promote learning. The ABPMR pilot was designed to include clone questions in the longitudinal assessment, but also on the MOC Examination. Results from the pilot showed that physicians performed better on the CertLink questions on the MOC Examination than in the CertLink platform when seen for the first time. This means that between administrations of the items, physicians learned and were able to retain the new information.
Implementation of CertLink in 2020
Thanks to this strong pilot data supporting the ABPMR quality improvement outcomes, today the ABPMR is also announcing a plan to implement CertLink to replace the MOC Examination beginning in 2020. All diplomates participating in MOC are receiving individualized emails from the ABPMR today, and more information is available on brand new CertLink pages on the ABPMR website.