Health products powered by artificial intelligence are streaming into our lives, from virtual doctor apps to wearable sensors and drugstore chatbots. IBM boasted that its AI could "outthink cancer." Others say computer systems that read X-rays will make radiologists obsolete.

"There's nothing that I've seen in my 30-plus years studying medicine that could be as impactful and transformative [as AI]," said cardiologist Eric Topol, executive vice president of Scripps Research. AI can help doctors interpret MRIs of the heart, CT scans of the head and photographs of the back of the eye, and could take over many mundane medical chores, freeing doctors to spend more time talking to patients.

Even the US Food and Drug Administration - which has approved more than 40 AI products in the past five years - says "the potential of digital health is nothing short of revolutionary."

Yet many industry experts fear AI-based products won't be able to match the hype.

Doctors and consumer advocates fear that the tech industry, which lives by the mantra "fail fast and fix it later," is putting patients at risk.

Early experiments in AI provide a reason for caution, said Mildred Cho, a professor of pediatrics at Stanford's Center for Biomedical Ethics.

Systems developed in one hospital often flop when deployed in a different facility, Cho said. Software has been shown to discriminate against minorities. AI systems sometimes learn to make predictions based on factors that have less to do with disease than the brand of MRI machine used, the time a blood test is taken or whether a patient was visited by a chaplain.

In one case, AI software incorrectly concluded that people with pneumonia were less likely to die if they had asthma - an error that could have led doctors to deprive asthma patients of the extra care they need.

"It's only a matter of time before something like this leads to a serious health problem," said Steven Nissen, chairman of cardiology at the Cleveland Clinic.

Medical AI, which pulled in US$1.6 billion (HK$12.48 billion) in venture capital funding in the third quarter of 2019 alone, is "nearly at the peak of inflated expectations," concluded a report from the research company Gartner.

"As the reality gets tested, there will likely be a rough slide into the trough of disillusionment."

That reality check could come in the form of disappointing results when AI products are ushered into the real world. Even Topol, the author of Deep Medicine: How Artificial Intelligence Can Make Healthcare Human Again, acknowledges that many AI products are little more than hot air. "It's a mixed bag," he said.

Experts such as Bob Kocher, a partner at the venture capital firm Venrock, are blunter. "Most AI products have little evidence to support them," Kocher said.

Some risks won't become apparent until an AI system has been used by large numbers of patients.

"We're going to keep discovering a whole bunch of risks and unintended consequences of using AI on medical data," he said.

None of the AI products in the market have been tested in randomized clinical trials, the strongest source of medical evidence, Topol said.

The first and only randomized trial of an AI system - which discovered that colonoscopy with computer-aided diagnosis found more small polyps than standard colonoscopy - was published online last October.

Few tech startups publish their research in peer-reviewed journals, which allow other scientists to scrutinize their work, according to an article in the European Journal of Clinical Investigation.

Such "stealth research" - described only in press releases or promotional events - often overstates a company's accomplishments.

And although developers may boast about the accuracy of their devices, experts note that AI models are mostly tested on computers, not in hospitals. Using unproven software "may make patients into unwitting guinea pigs," said Ron Li, medical informatics director for AI clinical integration at Stanford Health Care.

New kid on the block

AI systems that learn to recognize patterns in data are often described as "black boxes" because even their developers don't know how they have reached their conclusions. Given that AI is so new - and many of its risks unknown - the field needs careful oversight, said Pilar Ossorio, a professor of law and bioethics at the University of Wisconsin-Madison.

There's been little research on whether the 320,000 medical apps now in use actually improve health, according to a report on AI published on December 17 by the National Academy of Medicine.

"Almost none of the stuff marketed to patients really works," said Ezekiel Emanuel, professor of medical ethics and health policy in the Perelman School of Medicine at the University of Pennsylvania.

Analysts say that AI developers have little interest in conducting expensive and time-consuming trials.

"It's not the main concern of these firms to submit themselves to rigorous evaluation that would be published in a peer-reviewed journal," said Joachim Roski, a principal at Booz Allen Hamilton, a technology consulting firm.

But Oren Etzioni, chief executive officer at the Allen Institute for AI, said developers have a financial incentive to make sure their products are safe.

"If failing fast means a whole bunch of people will die, I don't think we want to fail fast," Etzioni said.

"Nobody is going to be happy, including investors, if people die or are severely hurt."

When good algorithms go bad

Some AI devices are more carefully tested than others. An AI-powered screening tool for diabetic eye disease was studied in 900 patients at 10 primary care offices before being approved in 2018.

The manufacturer, IDx Technologies, worked with the FDA for eight years to get the product right, said Michael Abramoff, the company's founder and executive chairman. The test, sold as IDx-DR, screens patients for diabetic retinopathy, a leading cause of blindness, and refers high-risk patients to eye specialists, who make a definitive diagnosis.

Yet some AI-based innovations intended to improve care have had the opposite effect.

A Canadian company, for example, developed AI software to predict a person's risk of Alzheimer's based on their speech. Predictions were more accurate for some patients than others. "Difficulty finding the right word may be due to unfamiliarity with English, rather than to cognitive impairment," said co-author Frank Rudzicz, an associate professor of computer science at the University of Toronto.

Doctors at New York's Mount Sinai Hospital hoped AI could help them use chest X-rays to predict which patients were at high risk of pneumonia. Although the system made accurate predictions from X-rays shot at Mount Sinai, the technology flopped when tested on images taken at other hospitals.

Eventually, researchers realized the computer had merely learned to tell the difference between that hospital's portable chest X-rays - taken at a patient's bedside - with those taken in the radiology department. Doctors tend to use portable chest X-rays for patients too sick to leave their room, so it's not surprising that these patients had a greater risk of lung infection.

Google-owned DeepMind has created an AI-based mobile app that can predict which patients will develop acute kidney failure up to 48 hours in advance. A post on its website described the app as a "game-changer."

But the system also produced two false alarms for every correct result, according to a study in Nature.

In view of the risks involved, doctors need to protect their patients' interests, said Vikas Saini, a cardiologist and president of the non-profit Lown Institute.

"While it is the job of entrepreneurs to think big and take risks," he said, "it is the job of doctors to protect their patients."

Kaiser Health News (TNS)

Is AI putting patients at risk?