Skip to main content
AAO-HNSF 2023 Annual Meeting & OTO Experience Main banner
Icon Legend
This session is not in your schedule.
This session is in your schedule. Click again to remove it.
Back
Rate ePoster

Patient Safety and Quality Improvement

(083) ChatGPT in Otolaryngology: Assessment of Performance on Board Review Questions

Monday, October 2, 2023
2:45 PM – 3:45 PM East Coast USA Time
Has Audio

    Senior Author(s)

  • Brian E. Benson, MD

    Chair, Department of Otolaryngology/Head & Neck Surgery
    Hackensack Meridian School of Medicine
    Hackensack, New Jersey, United States

    • Presenting Author(s)

  • Ellen Hong

    Medical Student
    Hackensack Meridian School of Medicine
    Yorba Linda, California, United States

    • Co-Author(s)

  • GS

    Guy Stein, N/A

    Hackensack Meridian School of Medicine, USA

  • AR

    Andrea Radevic, N/A

    Hackensack Meridian School of Medicine, USA

  • ES

    Ezra Schneier

    Student
    Hackensack Meridian School of Medicine
    teaneck, New Jersey, United States

  • DW

    David Wassef, MD

    Rutgers New Jersey Medical School, USA

  • Brandon K. Nguyen, MD

    Resident
    Rutgers New Jersey Medical School
    Newark, New Jersey, United States

Disclosure(s):
Brian E. Benson, MD: No relevant relationships to disclose.
Ellen Hong: No relevant relationships to disclose.
Ezra Schneier: No relevant relationships to disclose.


Introduction: ChatGPT is a large language model launched by OpenAI in November 2022 programmed with reinforcement learning techniques. This study aims to evaluate the AI technology's ability to answer otolaryngology board-style questions to assess its potential uses in medical education

Methods: Otolaryngology board-style questions were submitted to ChatGPT from the Board Vitals question bank. In total, 500 questions were submitted. Results were analyzed in terms of ChatGPT's overall performance score with breakdown by subspecialty, the average response time, and the overall percentile as compared to otolaryngology residents and fellows. Additionally, the acceptability of its explanations were evaluated.

Results: ChatGPT’s performance compares unfavorably to otolaryngology residents and fellows. ChatGPT was unable to correctly answer the majority of the otolaryngology board-style questions. Furthermore, the provided justification for its answer choices were also incorrect.

Conclusions: ChatGPT is a developing resource for both patients and practitioners. These results suggest that specialized knowledge and clinical reasoning currently lie beyond ChatGPT’s purview. Further studies are needed to determine ChatGPT’s role in the medical education system.