Skip to main content
AAO-HNSF 2023 Annual Meeting & OTO Experience Main banner
Icon Legend
This session is not in your schedule.
This session is in your schedule. Click again to remove it.

Patient Safety and Quality Improvement

ChatGPT: Assessment of Clinical Applicability in Otolaryngology

Monday, October 2, 2023
4:21 PM - 4:27 PM East Coast USA Time
Location: 202A

    Presenting Author(s)

  • PN

    Peter G. Nagy, MD

    Resident
    Beth Israel Deaconess Medical Center
    Boston, MA, United States

    • Co-Author(s)

  • KT

    Kevin Tie, MD

    Resident
    Beth Israel Deaconess Medical Center
    Boston, MA, United States

  • Ernest D. Gomez, MD, MTR

    Instructor
    Beth Israel Deaconess Medical Center / Harvard Medical School
    Boston, MA, United States

    • Senior Author(s)

  • Christopher D. Brook, MD

    Assistant Professor
    Beth Israel Deaconess Medical Center
    Lincoln, MA, United States

Disclosure(s):
Peter G. Nagy, MD: No relevant relationships to disclose.
Introduction: This project aims to evaluate the potential use of ChatGPT as a clinical adjunct in otolaryngology. The objective is to assess its effectiveness in further characterizing patients' chief complaints and providing accurate diagnosis and management suggestions for various clinical scenarios

Methods: The study design is a simulated clinical evaluation where ChatGPT will receive chief complaints from patients and relevant clinical scenarios. The responses of ChatGPT will then be evaluated by academic otolaryngology faculty using a grading scale (Not Appropriate (NA), Somewhat Appropriate (SA), and Very Appropriate (VA)) to assess accuracy of diagnosis and management suggestions. The outcome measures will be the average grade of the three independent reviewers' evaluation of the responses of ChatGPT. Intraclass Correlation Coefficient (ICC) will be used as the measure of inter-rater reliability (IRR).

Results: For the 10 chief complaints presented to ChatGPT, it generated 5-10 follow up interview questions that were deemed either SA 10% of the time or VA 90%. This was consistent between evaluators (std=0) and there were no cases of NA responses. For the 10 clinical scenarios, diagnosis and next step in management were evaluated separately. Of the 10 clinical scenarios, diagnosis was deemed NA 35%, SA 20%, and VA 45% amongst reviewers. Next step in management was deemed NA 33%, SA 20%, and VA 45% amongst reviewers. ICC was calculated at 0.92.

Conclusions: ChatGPT's advanced language processing capabilities allow for the generation of questions specific to a patient's symptoms, improving the accuracy of diagnoses and ultimately leading to better patient outcomes. Furthermore, its ability to provide concise and evidence-based information on treatment options can assist healthcare providers in making informed decisions on patient management. Overall, the integration of ChatGPT in otolaryngology has the potential to greatly improve the patient-provider interaction and enhance the quality of care but