ChatGPT Passes US Medical Licensing Exams: Study

According to a new study, ChatGPT could score at or near the pass mark of about 60% on the United States Medical Licensing Examination (USMLE), with answers that are consistent, logical, and contain frequent information.

Tiffany Kung and her colleagues from AnsibleHealth, California, USA, tested the performance of ChatGPT on the USMLE, a highly standardized and regulated series of three exams, including Steps 1, 2CK and 3, required for the obtaining a medical license in the United States, according to the study.

Taken by medical students and physicians-in-training, the USMLE assesses knowledge spanning most medical disciplines, ranging from biochemistry to diagnostic reasoning to bioethics.

After reviewing to remove image-based questions from the USMLE, the authors tested the software on 350 of the 376 public questions available as of the June 2022 USMLE release, according to the study.

The authors found that after removing indeterminate responses, ChatGPT scored between 52.4% and 75% on all three USMLE exams, according to the study published in the journal PLOS Digital Health.

The pass mark each year is around 60%.

ChatGPT is a new artificial intelligence (AI) system, known as a large language model (LLM), designed to generate human-like writing by predicting upcoming word sequences.

Unlike most chatbots, ChatGPT cannot search the Internet, according to the study.

Instead, it generates text using word relationships predicted by its internal processes, according to the study.

According to the study, ChatGPT also demonstrated 94.6% concordance across all of its responses and produced at least one meaningful insight, something that was novel, non-obvious, and clinically valid, for 88.9% of its responses.

ChatGPT also outperformed PubMedGPT, a peer model trained exclusively on literature from the biomedical field, which scored 50.8% on an older dataset of USMLE-style questions, according to the study.

While the relatively small size of the entries limited the depth and scope of the analyses, the authors noted that their findings provide insight into the potential of ChatGPT to improve medical education and, ultimately, clinical practice.

For example, they added, AnsibleHealth clinicians are already using ChatGPT to rewrite jargon-rich reports to make it easier for patients to understand.

“Achieving the passing grade for this notoriously difficult expert exam, and doing so without any human reinforcement, marks a notable milestone in the clinical maturation of AI,” the authors said.

Kung added that ChatGPT’s role in this research goes beyond the subject of the study.

“ChatGPT has contributed greatly to the writing of [our] manuscript… We interacted with ChatGPT much like a colleague, asking him to synthesize, simplify and offer counterpoints to ongoing drafts… All co-authors appreciated ChatGPT’s contribution.”