In Friday morning’s Plenary Session, Dr. Malachai G. Sheahan will present the results of a study that he and his colleagues did to establish metrics for a number of essential skills needed by vascular surgery trainees.
“Over the past fifteen years, we have changed the essential nature of our field. What we do. How we train. Despite this, we have very few tools to assess the overall quality of our most important product, our residents and fellows. While standardized cognitive testing is in place, there exists no national evaluation of technical ability. Our group was formed to design and validate a vascular skills exam,” said Dr. Sheahan, of the Houston Methodist Hospital, DeBakey Heart & Vascular Center, Houston.
He and his colleagues performed this over a 5-year period of model design, assessor training, and pilot exams, following which, formal testing was conducted between October 2012 and December 2014. The timed exam consisted of the following essential vascular surgery skills, according to Dr. Sheahan: an end-to-side (ES) model, a patch angioplasty (PA), and a clock-face (CF) radial suturing skill.
Each examinee in the study was evaluated by two experienced assessors blinded to the trainee’s level. Scoring was performed both on a global assessment (GA) and a more detailed global rating score (GRS). A passing or “competent” grade was defined as a minimum GA score of 3 on each skill.
A total of 243 trainees were examined, who ranged in experience from medical student to senior fellow. Inter-rater reliability (alpha =.89) among the assessors and internal consistency (alpha =.90) for each participant on the three models were both high.
The GRS achieved on each model was able to significantly differentiate junior (pre-MD to PGY2) from senior (PGY3-7) trainees for each of the skills tested, demonstrating construct validity. They also found a strong correlation between GRS and the trainee’s open surgical experience for each model. While no medical student received a passing score (0/43), progressive improvement with experience was demonstrated among integrated vascular residents, with PGY1 at 8%, PGY2 at 40%, PGY3 at 44%, PGY4 at 85%, and PGY5 at 92% passing. Fellows had scores at PGY6 of 56%, and PGY7 of 90%.
Logistic regression analysis demonstrated time spent on vascular rotations (P =.012) and self-reported confidence in vascular skill (P =.041) were independent predictors of a passing score, Dr. Sheahan added.
“Our results suggest that experienced assessors using this exam can effectively evaluate the technical abilities of a vascular trainee.
“The models will help identify distict deficiencies in technical skill and resident outliers who have fallen behind their peers. The goal is to help Program Directors improve their final product.,” Dr. Sheahan concluded. VC
