Almost three in 10 Britons now use AI chatbots to get health advice. And UK medical-policy experts are concerned that this could create confusion, undermine trust in GPs and other healthcare workers, and make it more difficult to take an informed decision about your health.
The warning, from The Academy of Medical Sciences, comes as Open AI rolls out ChatGPT Health in the US. It allows users to “securely connect” their medical records and wellness apps to “ground conversations” in their own health information, so that the bot can “take a more active role” in their health.
ChatGPT Health is not yet available in the UK but, with more than 300 million people worldwide regularly directing their health-related questions to ChatGPT, it could swiftly become a “democratising force in healthcare”, Tanzeem Choudhury, a health-innovation expert at New York’s Cornell University told CBS News. But there are definite “risks and trade-offs”. Yvonne Doyle, former medical direction of Public Health England, agrees: “Used well,” an AI tool such as this can “help direct people to trusted sources” but it can also mislead “convincingly” and “exploit uncertainty”.
‘Zero awareness of fallibility’
People have been using the internet to answer their questions for decades, but the “danger” with an AI chatbot is that it responds with “fluent, emotionally convincing paragraphs” and “remembers context”, making it “sound more ‘human’ than many humans”, said Celina Cartoto and Anthony Kelly on RTE. Yet it has “zero awareness of its own fallibility”, always providing an answer, “even when one doesn’t exist”. People need to remember that, though it “speaks like a doctor, listens like a therapist and remembers like a friend”, it is actually “none of them”.
We know that all AI chatbots can “hallucinate” and provide “misleading, inaccurate or even dangerous” information, said CBS News. Open AI has been careful to state that ChatGPT Health is designed to support, not replace doctors, but it has not responded to questions about whether it is “equipped to recognise when to advise a user to seek help from a medical professional”. It seems that “onus is on the user”.
Even before the rollout of the Health feature, there were indications that users could become dangerously reliant on ChatGPT for medical advice. Last month, a 55-year-old US pastor sued OpenAI, claiming its bot “brought him to the brink of death”, by “repeatedly and insistently” misdiagnosing his symptoms of near-fatal pulmonary embolism, said the BBC. An OpenAI spokesperson told CBS News that “ChatGPT is not a doctor and should never be a substitute for medical care, diagnosis or treatment”.
AI can be a “useful educational tool” for health information, said Leana Wen, an emergency doctor in the US, and a clinical associate professor at George Washington University, told CNN. It can “explain medical terminology” and “compare treatment options at a high level”, helping “people prepare questions” for their doctor and be “more engaged” in their care. But “it has important limitations” and “should not replace clinical judgement” by a medically qualified human.
Privacy concerns
There is also the question of privacy. How risky is it to share so much of your individual medical information with an AI tool, such as ChatGPT Health? Open AI has said that Health has “enhanced privacy” and encryption, and that conversations used in the Health tab won’t flow into non-Health chats. But it’s no accident that Health has not yet been rolled out in the UK, or the EU, where data protection and digital privacy laws are stricter than in the US.
“If you are providing data directly to a technology company that is not providing any healthcare services, then it is buyer beware,” Bradley Malin, a biomedical informatics expert, told Time. OpenAI says it will protect your data but “what exactly does that mean”?
It all comes down to whether or not “you trust OpenAI to keep to its word”, Robert Wachter, a medic who has written extensively about AI and healthcare, told the magazine. OpenAI does have “really strong corporate interest in not screwing this up”. Its brand is “dependent” on the user “feeling comfortable” in sharing their information, and any sort of data breach would end that trust.
The end of doctors?
The increasing use of AI programmes for health queries is becoming “existentially threatening” for medics, Jonathan Chen, a doctor at Stanford University Medical Center, told The New York Times. What is our purpose going to be now? Should we “get out of the way and let a computer take over”?
Not for a while yet, Lee Schwamm, associate dean for digital strategy and transformation at Yale School of Medicine, told the paper. And that’s because chatbots can’t yet think critically enough about the health information people are providing – and more importantly, not providing – them with. AI is “really good at matching patterns and making predictions”. But it can only do so “based on the data that it is given about the patient. It has no way to extract” what may be crucial further information itself.
Besides, patients still need a human connection. If you’re going to be told you have a serious health issue, or even how much longer you have to live, “you want to look someone in the eye”.