(137) Diagnostic and Management Applications of ChatGPT in Otolaryngology Clinical Scenarios

Introduction: With public interest at a high, ChatGPT, among other large language models, may become an increasingly utilized tool for both patients and clinicians. Its efficacy in diagnosis and treatment for specialty care, such as otolaryngology, has not been well evaluated.

Methods: ChatGPT 4.0 was queried for diagnoses and management plans for twenty physician-written clinical vignettes in otolaryngology. Attending physicians then rated the difficulty of the clinical vignettes and agreement with the differential diagnoses and management plans of ChatGPT responses on a 5-point Likert scale. Summary statistics were calculated. Univariate linear regression was then performed between vignette difficulty and quality of the diagnoses and management plans.

Results: Six attending physicians completed the survey (33% response rate). Overall, vignettes were rated as very easy to neutral difficulty (range of median score: 1.00 – 3.50; overall median 2.00). There was high agreement with the differential diagnosis provided by ChatGPT (range of median score: 3.50 – 5.00; overall median: 5.00). There was also high agreement with treatment plans (range of median score: 3.50 – 5.00; overall median: 5.00). There was no association between vignette difficulty and agreement with differential diagnosis or treatment. There was a significant association between agreement with diagnosis and agreement with treatment (r = 0.33; 0.21 – 0.69, 95% C.I.; p < 0.001).

Conclusions: Generative artificial intelligence models like ChatGPT are being rapidly adopted in medicine. Performance with curated, easy to moderate difficulty otolaryngology scenarios indicate high agreement with physicians for diagnosis and management. However, a decreased quality in diagnosis is associated with decreased quality in management. Further research is necessary on ChatGPT’s ability to handle unstructured clinical information.