Are you comfortable sharing your deepest health secrets with a robot? Probably not, right? But millions are doing just that, and the consequences could be more serious than you think. Giving your healthcare information to a chatbot like ChatGPT might seem convenient, but it's a minefield of privacy risks and potential misinformation.
OpenAI reports that a staggering 230 million people each week are turning to ChatGPT for health and wellness advice. They envision the chatbot as a friendly guide through the daunting world of insurance paperwork and a helpful tool for becoming a better self-advocate. In return, they hope you'll entrust it with your most sensitive data: diagnoses, medications, lab results, the whole shebang. But here's the kicker: while chatting with an AI might feel like a doctor's visit, it's definitely not the same thing. Tech companies aren't held to the same ethical and legal standards as medical professionals. Experts are urging caution, and rightfully so.
The health and wellness sector has rapidly become a key battleground for AI development, representing a pivotal test of how readily people will integrate these technologies into their lives. Recently, industry giants have made significant strides into the medical arena. OpenAI unveiled ChatGPT Health, a dedicated section within ChatGPT designed for health-related queries, promising a more secure and personalized user experience. Meanwhile, Anthropic introduced Claude for Healthcare, a "HIPAA-ready" product aimed at hospitals, healthcare providers, and individual consumers. Notably, Google, despite possessing one of the most advanced AI tools in Gemini, has focused on updating its MedGemma medical AI model for developers, rather than launching a consumer-facing chatbot.
OpenAI is actively encouraging users to share sensitive data – medical records, lab results, and even data from fitness apps like Apple Health, Peloton, Weight Watchers, and MyFitnessPal – with ChatGPT Health, promising deeper, more personalized insights. They assure users that their health data will be kept confidential, won't be used to train AI models, and that stringent measures are in place to ensure its security and privacy. ChatGPT Health conversations are supposedly isolated within a separate part of the app, allowing users to review or delete their Health "memories" at any time.
To further bolster user confidence, OpenAI launched a similar product with even tighter security protocols almost simultaneously: ChatGPT for Healthcare. This tool, part of a broader suite of products geared towards businesses, hospitals, and clinicians, aims to streamline administrative tasks, draft clinical letters and discharge summaries, and assist physicians in gathering the latest medical evidence for improved patient care. Like other enterprise-grade offerings, it boasts enhanced protections compared to the consumer-facing version, particularly for free users, and is designed to comply with the stringent privacy obligations of the healthcare sector. Given the similar names and launch dates, it's easy to see how many – including people I spoke to for this story – could mistakenly assume that the consumer product offers the same level of protection as its clinically oriented counterpart.
But here's where it gets controversial... Even if you trust a company's promises to protect your data, they can always change their mind. And this is the part most people miss...
Regardless of the security assurances provided, they are far from foolproof. Experts warn that users of tools like ChatGPT Health have limited recourse against breaches or unauthorized use beyond what's outlined in the terms of use and privacy policies. With most states lacking comprehensive privacy laws and no overarching federal law, data protection for AI tools like ChatGPT Health largely hinges on the promises made by companies in their privacy policies and terms of use, according to Sara Gerke, a law professor at the University of Illinois Urbana-Champaign.
Even if you believe a company's vow to safeguard your data – OpenAI claims to encrypt Health data by default – they could always change their mind. Hannah van Kolfschooten, a researcher in digital health law at the University of Basel in Switzerland, points out that while ChatGPT's current terms of use state that data will remain confidential and won't be used for model training, these terms are subject to change without legal repercussions. "You will have to trust that ChatGPT does not do so." Carmel Shachar, an assistant clinical professor of law at Harvard Law School, echoes this sentiment: "There’s very limited protection. Some of it is their word, but they could always go back and change their privacy practices."
Furthermore, assurances of compliance with data protection laws like HIPAA shouldn't be taken as gospel, Shachar warns. While such compliance is a positive indicator, it carries little weight if a company that voluntarily complies fails to do so. Voluntarily complying isn't the same as being legally bound. "The value of HIPAA is that if you mess up, there’s enforcement."
There's a reason why medicine is a heavily regulated field – and it's not just privacy. Mistakes can be dangerous, even deadly. Numerous examples illustrate chatbots confidently dispensing false or misleading health information. For instance, a man developed a rare condition after ChatGPT suggested replacing salt with sodium bromide, a historical sedative. Google's AI Overviews once wrongly advised pancreatic cancer patients to avoid high-fat foods – the opposite of what they should be doing.
To mitigate these risks, OpenAI emphasizes that its consumer-facing tool is designed for use in conjunction with physicians, not for diagnosis or treatment. Tools intended for diagnosis and treatment are classified as medical devices and are subject to stricter regulations, including clinical trials and safety monitoring. Although OpenAI acknowledges that a major use case for ChatGPT is supporting users' health and well-being, the company's claim that it is not a medical device carries significant weight with regulators, according to Gerke. "The manufacturer’s stated intended use is a key factor in the medical device classification," she explains, meaning companies can largely avoid oversight by stating that their tools aren't for medical use, even if they are being used for medical purposes. This highlights the regulatory challenges posed by technologies like chatbots.
For now, this disclaimer keeps ChatGPT Health outside the purview of regulators like the FDA. However, van Kolfschooten argues that it's reasonable to question whether such tools should be classified and regulated as medical devices. It's crucial to consider how the tool is being used, not just what the company claims. OpenAI suggested that people could use ChatGPT Health to interpret lab results, track health behavior, and reason through treatment decisions. If a product performs these functions, one could reasonably argue that it falls under the US definition of a medical device, suggesting that Europe's stronger regulatory framework may be the reason why it's not yet available there.
OpenAI has invested considerable effort in demonstrating ChatGPT's medical capabilities and encouraging users to utilize it for health queries, even while claiming it's not intended for diagnosis or treatment. They highlighted health as a major use case when launching GPT-5, and CEO Sam Altman even invited a cancer patient and her husband on stage to discuss how the tool helped her understand her diagnosis. The company assesses ChatGPT's medical prowess using HealthBench, a benchmark developed with over 260 physicians across dozens of specialties, which "tests how well AI models perform in realistic health scenarios." However, critics point out that HealthBench lacks transparency. Other studies – often small, limited, or conducted by the company itself – suggest ChatGPT's medical potential, showing that it can pass medical licensing exams, communicate better with patients, outperform doctors at diagnosing illness, and help doctors make fewer mistakes when used as a tool.
Van Kolfschooten argues that OpenAI's efforts to present ChatGPT Health as an authoritative source of health information could undermine any disclaimers it includes, telling users not to use it for medical purposes. "When a system feels personalized and has this aura of authority, medical disclaimers will not necessarily challenge people’s trust in the system."
Companies like OpenAI and Anthropic are vying for dominance in what they perceive as the next major market for AI. The figures showing how many people are already using AI chatbots for health suggest they might be onto something. Given the stark health inequalities and difficulties many face in accessing even basic care, this could be a positive development – if that trust is well-placed. We trust our private information with healthcare providers because the profession has earned that trust. It remains to be seen whether an industry known for its rapid pace and disruptive innovation has earned the same level of trust.
So, what do you think? Are you comfortable entrusting your health data to an AI chatbot? Do you believe the potential benefits outweigh the risks? Or are we rushing headfirst into a future where our most sensitive information is vulnerable to misuse and misinformation? Share your thoughts in the comments below. I'm genuinely curious to hear your perspective!
