AI Chatbots Give Wrong Medical Advice 50% of the Time, BMJ Open Study Finds

Gate News message, April 15 — A study recently published in BMJ Open found that five major AI chatbots provided problematic medical advice in approximately 50% of cases, with nearly 20% of responses classified as highly problematic. Researchers from the U.S., Canada, and the U.K. tested ChatGPT, Gemini, Meta AI, Grok, and DeepSeek using 10 questions across five health topics.

The chatbots performed better on closed questions and topics such as vaccines and cancer, but struggled with open-ended questions and subjects like stem cells and nutrition. None of the tested chatbots produced a complete and accurate reference list.

The study's authors warned that public-facing chatbots could amplify misinformation, as these tools are not licensed to provide medical advice and can present flawed answers with unwarranted certainty.