ChatGPT Defeated Doctors at Diagnosing Illness

Over the summer, researchers gave a failing grade to artificial intelligence when it comes to diagnosing illnesses, finding that a chatbot that read 150 case studies diagnosed the correct condition less than 50% of the time. "ChatGPT in its current form is not accurate as a diagnostic tool," the study in PLOS One noted. The bots must have boned up on their diagnostics since, however, as a new study shows a significant improvement in AI's solo diagnosing skills. The New York Times reports on the research published last month in the JAMA Network Open journal, which details an experiment involving 50 doctors, half paired with a ChatGPT-4 assistant from OpenAI and the other half without. Those without relied on conventional diagnostic methods, including Google and medical reference sites like UpToDate, per a release.

The bots alone were also given the opportunity to read the six case studies on hand and offer their diagnosis and reasoning for coming to their conclusion. Those who graded the entries had no clue if ChatGPT had any involvement. The results: The bots correctly identified the medical condition in question 90% of the time. However, there wasn't a significant difference between the human doctors who used AI and those who didn't, scoring 76% and 74%, respectively. "The LLM [large language model] alone outperformed physicians even when the LLM was available to them, indicating that further development in human-computer interactions is needed to realize the potential of AI in clinical decision support systems," the researchers note.

Per the Times, the case studies in question, which were based on actual patients, had never been published before—meaning ChatGPT couldn't have trained on them and was seeing them for the first time, just like the human doctors. So why did doctors who used ChatGPT not have similar results to the bots working alone? The researchers note that many of the doctors may not have known how to fully maximize AI's full capabilities—but also, they say that humans can be stubborn when it comes to their own opinions. "They didn't listen to AI when AI told them things they didn't agree with," study co-author Adam Rodman tells the Times. Laura Zwaan of Erasmus Medical Center, who wasn't involved with the study, concurs, noting, "People generally are overconfident when they think they are right." (More discoveries stories.)