ChatGPT 4.0 was able to provide a mean accuracy rating of 4.4 on a Likert scale of 5 (1=very poor; 5=excellent) across 15 patient-level questions designed to assess the ability of the artificial intelligence (AI)-driven large language model to provide accurate information on abdominal aortic aneurysms (AAAs).
When tested on physician-level information—questions focused on the five domains of the Society for Vascular Surgery (SVS) AAA practice guidelines— the AI tool returned an accuracy rating of 4.3.
When queried on four questions related to AAA rupture risk, ChatGPT performed only “fairly,” scoring a mean of 3.4.
The data were presented by Daniel Bertges, MD, vascular surgery program director at the University of Vermont, Burlington, during the 2024 Society for Clinical Vascular Surgery (SCVS) Annual Symposium in Scottsdale, Arizona (March 16–20).
In order to perform the testing, a subset of the SVS practice guidelines on AAA were fashioned into 37 questions as Bertges and colleagues sought to query ChatGPT on the physician level. Four additional questions were specifically designed to query for the annual risk of rupture at varying diameters. Ten Board-certified vascular surgeons and one fellow independently graded the accuracy of responses using the Likert scale.
Bertges pointed out one patient-level question posed of ChatGPT in particular: “What should I ask my doctor about my AAA?” The bot returned a score of 4.8, providing such suggested prompts as, “Should I see a specialist, such as a vascular surgeon or a cardiologist?”
“ChatGPT 4.0 provided accurate responses to a variety of patient-level questions regarding AAA,” Bertges told SCVS 2024. “The responses seem well aligned with SVS practice guidelines, except for inaccuracies in quoting the risk of AAA rupture.
“The emergence of generative AI bots presents an opportunity for study of potential applications in patient education and to determine their ability to augment the knowledge base of vascular surgeons.”
