Studies find AI chatbots get health questions wrong about half the time
What happened: A study published in BMJ Open in April 2026 asked five free chatbots (ChatGPT, Gemini, Grok, Meta AI and DeepSeek) 50 health questions each about cancer, vaccines, stem cells, nutrition and fitness. Two experts rated every answer.
- About half the answers were judged problematic, and nearly 20% highly problematic.
- Nutrition and fitness answers were the weakest. Vaccines and cancer were handled best, but still went wrong about a quarter of the time.
- The chatbots’ reference lists were unreliable, with wrong authors, broken links and made-up papers.
- Only 2 of the 250 questions were refused.
The researchers deliberately asked tricky questions, so everyday error rates may be lower. The versions tested were from early 2025.
A second finding: A study in Nature Medicine (February 2026) found chatbots alone picked the right medical answer about 95% of the time, but real people using them got it right less than 35% of the time, no better than people who didn’t use AI. Knowing what to ask, and which part of the answer matters, is hard.
Why it matters for you: AI answers sound confident and caring, which makes wrong advice easy to believe, especially on diet, home remedies and supplements.
How to use AI for health safely:
- Use it to understand a report or prepare questions for your doctor, not to decide treatment.
- Check claims with a doctor or an official health website, and treat AI-given references as leads to verify.
- Never change medicines or doses because a chatbot said so.
- Remove your name and ID numbers before pasting reports into a chatbot.