Brown Neurosurgery Department tests artificial intelligence models on written, oral neurosurgery board exam questions

THE BROWN DAILY HERALD— “The Brown Neurosurgery Department recently published two preprints comparing the performances of Artificial Intelligence Large Language Models ChatGPT, GPT-4 and Google Bard in the neurosurgery written board examinations and the neurosurgery oral board preparatory question bank.

They found that these AI models were able to pass the written exams with ‘flying colors.’ When challenged to answer the more complicated oral exam questions, which require higher-order thinking based on clinical experience and exposure, the models still performed ‘superbly,’ said Ziya Gokaslan, professor and chair of neurosurgery at the Warren Alpert Medical School and neurosurgeon-in-chief at Rhode Island Hospital and The Miriam Hospital…

‘It’s such an exploding story in the world and in medicine,’ said Warren Alpert Professor of Neurosurgery Albert Telfeian, who is also the director of minimally invasive endoscopic spine surgery at RIH and director of pediatric neurosurgery at Hasbro Children’s Hospital.

The project was inspired when fifth-year Neurosurgery Resident and Co-first author Rohaid Ali was studying for his neurosurgery board exam with his close friend from Stanford Medical School, Ian Connolly, another co-first author and 4th year neurosurgery resident at Massachusetts General Hospital. They had seen that ChatGPT was able to pass other standardized exams like the bar examination, and wanted to test whether ChatGPT could answer any of the questions on their exam.

This prompted Ali and Connolly to execute these studies in collaboration with their third co-first author, Oliver Tang ’19 MD’23. They found that GPT-4 was ‘better than the average human test taker’ and ChatGPT and Google Bard were at the ‘level of the average neurosurgery resident who took these mock exams,’ Ali said.

‘One of the most interesting aspects’ of the study was the comparison between the AI models, as there have been ‘very few structured head-to-head comparisons of (them) in any fields,’ said Wael Asaad, associate professor of neurosurgery and neuroscience at Warren Alpert and director of the functional epilepsy and neurosurgery program at RIH. The findings are ‘really exciting beyond just neurosurgery,’ he added…

‘Taking the surgeon out of the equation is not in the foreseeable future,’ said Curt Doberstein, professor of neurosurgery at Warren Alpert and director of cerebrovascular surgery at RIH… ‘We’re at the tip of the iceberg — these things just came out,’ Doberstein said. ‘It’s going to be a process where everybody in science is going to constantly have to learn and adapt to all the new technology and changes that come about.’

‘That’s the exciting part,’ Doberstein added.”

Read the full article Here

Back to News