Texas hospital struggles to make IBM's Watson cure cancer

If IBM is looking for a new application for its Watson machine learning tools, it might consider putting health care providers' procurement and systems integration woes ahead of curing cancer.

After spending more than four years and US$62 million on its Oncology Expert Advisor project, the M.D. Anderson Cancer Center at the University of Texas still looking for answers in all those areas.

The fall-out from that project has now prompted the resignation of the cancer center's president, Ronald DePinho, the Wall Street Journal reported Thursday.

The university recently published an internal audit report into the procurement processes that led it to hand almost $40 million to IBM and over $21 million to PwC for work on the project, almost all of it without board approval.

It noted that the scope of its review was limited to contracting and procurement practices and compliance issues, and did not cover project management and system development activities.

The audit "should not be interpreted as an opinion on the scientific basis or functional capabilities of the system in its current state," because a separate review of those aspects of the project is being conducted by an external consultant, it said.

The Oncology Expert Advisor was intended to take in medical literature, treatment options, research data and patient records, using IBM's Watson machine learning algorithms to provide clinical advice to help oncologists in the community to provide the same level of care as doctors at the M.D. Anderson Cancer Center. That never happened.

The contract with IBM was first signed in June 2012, and extended 12 times. But as of September 2016, according to the audit, "The system is not in clinical use and has not been piloted outside of M.D. Anderson." Furthermore, IBM ended its support for the pilot and demonstration systems on Sept. 1, 2016. IBM has agreed with the university that the system "is not ready for human investigational or clinical use, and its use in the treatment of patients is prohibited," according to the audit report.

IBM officials were not immediately available to comment, but told the Wall Street Journal that the initial pilot project was a success.

Part of the problem seems to be that M.D. Anderson swapped its old electronic medical records system, ClinicStation, for a new one, Epic, in 2016, and Watson never learned to read the new record format.

"OEA has not been updated to integrate with the current system," the audit noted.

Mission creep also led to the project increasing considerably in scope. Initially focused on advising on myelodysplastic syndrome (MDS) leukemia treatment, it expanded after eight months to include five additional types of leukemia, then two years later to include lung cancer. PwC's work to integrate it with clinical and other databases also expanded as part of M.D. Anderson's Translational Research Accelerator, initially known as its Big Data Initiative (BDI).

The auditor focused criticism on compliance processes. It reviewed six non-competitive procurements worth a total of $51.4 million, finding two that were not formally justified and approved as exclusive non-competitive acquisitions, one contract amendment that was correctly executed, and one change order that was not signed by an authorized party. The project as a whole did not follow required IT governance processes.

Responding to the audit, university chancellor Willam McRaven acknowledged the auditor's finding that contractors were paid regardless of whether the original contracted scope-of-work was completed.

However, he wrote, "The research and development nature of the work inevitably led to goals and expectations that shifted over time, often making original contracts moot in terms of specified deliverables. To the extent that targets and expectations shifted over time, the incomplete documentary record makes it impossible to determine whether or not the revised milestones were met."

M.D. Anderson must improve its systems to record the evidence necessary to justify payments to contractors on projects that evolve over time, he added.

IBM's Watson Oncology tool is doing just fine at Memorial Sloan Kettering Cancer Center in New York.

At M.D. Anderson Cancer Center, though, maybe IBM's best chance to pick up some extra business is with its Watson Regulatory Compliance Analytics tool.