Skip to main content
AAO-HNSF 2023 Annual Meeting & OTO Experience Main banner
Icon Legend
This session is not in your schedule.
This session is in your schedule. Click again to remove it.
Back
Rate ePoster

Otology/Neurotology

(068) BPPV Information on Google vs AI (ChatGPT)

Monday, October 2, 2023
2:45 PM – 3:45 PM East Coast USA Time
Has Audio

    Presenting Author(s)

  • Jeffrey R. Bellinger, BS

    Medical Student
    Department of Otolaryngology – Head and Neck Surgery, University of Virginia, Charlottesville, VA
    Charlottesville, Virginia, United States

    • Co-Author(s)

  • JD

    Julian S. De La Chapa, MD

    University of Virginia, Department of Otolaryngology - Head and Neck Surgery, USA

  • Minhie Willow Kwak, BA

    Medical Student
    University of Virginia School of Medicine Department of Otolaryngology-Head and Neck Surgery
    Glen Allen, Virginia, United States

  • GR

    Gabriel A. Ramos, BA

    University of Virginia School of Medicine Department of Otolaryngology-Head and Neck Surgery
    Charlottesville, Virginia, United States

  • DM

    Daniel Morrison, MD

    University of Virginia School of Medicine Department of Otolaryngology-Head and Neck Surgery, United States

    • Senior Author(s)

  • Bradley Kesser, MD

    Professor and Vice Chair
    University of Virginia, Department of Otolaryngology - Head & Neck Surgery
    Charlottesville, Virginia, United States

Disclosure(s):
Jeffrey R. Bellinger, BS: No relevant relationships to disclose.
Bradley Kesser, MD: No relevant relationships to disclose.


Introduction: To quantitatively compare online patient education materials found using traditional search engines (Google) versus conversational AI models (ChatGPT) for benign paroxysmal positional vertigo (BPPV).

Methods: The top 30 Google search results for “benign paroxysmal positional vertigo” were compared to the OpenAI conversational AI language model, ChatGPT, responses for 5 common patient questions posed about BPPV in February 2023. Metrics included readability, quality, understandability, and actionability. Questions asked to ChatGPT were scored individually and then also as a combined response to simulate the questions being asked sequentially. Outcome measures included the scores of validated online information metrics including Flesch-Kincaid Grade Level (FKGL), Flesch Reading Ease (FRE), DISCERN instrument score, and Patient Education Materials Assessment Tool for printed materials (PEMAT-P). These metrics were scored by two reviewers. Additionally, to evaluate for inaccurate or outdated information, two experts in the field (BK, DM), an attending otolaryngologist with subspecialty certification in Neurotology, and a second-year fellow in Otology/Neurotology, reviewed the ChatGPT responses for accuracy and currency and scored responses using a 5-point Likert scale.

Results: Mean readability scores, FKGL and FRE, for the Google webpages were 10.7 +/- 2.6 and 46.5 +/- 14.3 respectively. ChatGPT responses had a higher FKGL score of 13.9 +/- 2.5 (P <.001) and lower FRE score of 34.9 +/- 11.2 (P =.005), both corresponding to lower readability. The Google webpages had a DISCERN Part 2 score of 25.4 +/- 7.5 compared to the individual ChatGPT responses with a score of 17.5 +/- 3.9 (P =.001), and the combined ChatGPT responses with a score of 25.0 +/- 0.9 (P=0.928). The average scores of the reviewers for all ChatGPT responses for accuracy was 4.19 +/- 0.82 and 4.31 +/- 0.67 for currency.

Conclusions: The results of this study suggest that the information on ChatGPT for BPPV is more difficult to read, of lower quality, and more difficult to comprehend compared to information on Google searches.