(084) ChatGPT versus WebMD

Introduction: Chatbots have been previously explored as a method of healthcare information to patients. ChatGPT is a new large language model Artificial Intelligence chatbot that is trained to participate in a conversational dialogue with a human user. We evaluate the quality of medical information provided by ChatGPT compared to search engine results.

Methods: 36 common otolaryngology diagnoses were entered into ChatGPT as, “I was just diagnosed with (medical condition), can you tell me more about it?” The corresponding informational articles were identified on websites known for high-quality patient education content. ChatGPT’s responses were then compared to the traditional website content.

Results: ChatGPT was able to provide correct medical information regarding otolaryngic diagnoses in a conversational format that compared favorably to the content available on high-quality websites. Corresponding website articles did not always provide comprehensive information, such as risk factors or causes. For some diagnoses, web articles did not exist at all.

Conclusions: This study demonstrates that medical information obtained from AI chatbots compares favorably to that obtained from high-quality websites. However, it is important to ensure the quality of medical information provided by AI programs, which are known to routinely provide incorrect information.