The advent of large language models such as ChatGPT (Open AI) has left little of the world untouched—and a quintet of University of Toronto vascular surgery researchers are bidding to harness this mode of artificial intelligence (AI) and refine it into a working model that is functionally vascular specific, capable of being used in clinical practice, research and education.
And according to research from the group recently published in the Journal of Vascular Surgery-Vascular Insights (JVS-VI), they are making significant strides just 18 months on from the inception of their work.
Tiam Feridooni, MD, a vascular surgery resident at the institution in Toronto, Canada, and colleagues started to develop the model, named VASC.AI, around the same time as ChatGPT-3 was being launched, initially impressed by the mainstream application’s ability to intelligently answer questions but less so by its logical reasoning. Feridooni and colleagues tested the model on questions from the 5th edition of the Vascular Education and Self-Assessment Program (VESAP5), comparing it to the evolving ChatGPT technology in three iterations: ChatGPT-3.5, -4 and -40. Across 244 text-based multiple-choice questions from six VESAP5 modules, VASC.AI significantly outperformed all three by answering 93.8% of questions correctly at the same time as ChatGPT got progressively better through version -40.
As Feridooni describes, VASC.AI is based on ChatGPT’s interface but is combined with retrieval-augmented generation (RAG), an advanced architecture that integrates specialized vascular surgery data into the model, potentially decreasing the generation of incorrect information and enhancing their educational usefulness. That input of vascular data was no small feat. Feridooni and co needed significant resources. That meant copious man hours, as well as funding. Luckily, they enlisted the help of a keen and engaged coder. Monetary help came via the research grant funding of senior author David Szalay, MD.
VASC.AI’s success is rooted in the database, made up of more than 250,000 abstracts, clinical practice guidelines and clinical trials. “The way it works in layman’s terms is we’ve basically told the model you have to draw your answer from this database of information,” explains Feridooni. “By creating this database, it has access to all of the up-to-date information that a vascular surgeon would need to have access to.” The team also put about six or seven months of labor into making sure the model weighs the database’s constituent content appropriately. “That was a process in itself,” Feridooni recalls. “How do we weigh the abstracts versus the clinical trials versus the clinical guidelines? How do we make sure it is drawing from the right guidelines—for instance, if you asked about renal artery stenting, that it would pick out from the guidelines for peripheral vascular disease.”
Going forward, the research team aims to develop a similar interface for patients. “We want to help patients both pre- and postoperatively,” Feridooni explains. “If someone comes in and they learn they have an aneurysm and need surgery, by the time they have had time to digest the news, they are probably halfway home and probably have questions, and you might not see them again until the day of surgery. This will give patients the chance to ask and get accurate responses.” In the postoperative period, Feridooni says such a model could be deployed to aid triage. For example, if a patient has questions about the look and feel of their wounds, the bot could potentially distinguish between the need to visit the clinic or the emergency department.
Ultimately, the researchers want a standalone, vascular-specific large language model. “Hopefully we can gather enough information to train our own vascular-specific model so that is both faster and cheaper, and more accessible to the vascular world,” Feridooni continues, emphasizing that its fundamentals are applicable to other specialties. He sees utility across education, research and clinical practice, describing the model as “democratizing knowledge,” and even has potential as an “unbiased” arbiter in multidisciplinary discussions over how to treat patients with complex disease, he says. “AI is coming, so why not be the leaders in this rather than let it lead us.”
