The Cerner electronic health record rollout went down last Wednesday for three hours affecting 95,000 users. Adverse events linked to the outage have not been confirmed.
This outage is the second reported since the previous outage March 3 that nearly halted health care and treatment services at the agency’s new EHR test site in Spokane, WA.
Subscribe to our weekly newsletter and stay up to date.
The outage was reportedly traced back to three Oracle databases that went down impacting EHR systems across the Defense Department and VA.
Oracle agreed to purchase Cerner in December 2021 for $28 billion shortly after VA affirmed the agency would continue ahead with its rollout of the new EHR at other locations that same month.
After the December purchase deal was struck:
“[B]oth companies said the transaction would improve the availability of technologies such as cloud, artificial intelligence and machine learning to federal agencies, and that the goal would be to focus on delivering zero unplanned downtime for Cerner systems running on Oracle’s Gen2 cloud.”
There has since been at least two outages within VA, one of which was severe.
Last week’s EHR system outage impacted the Defense Department, Coast Guard, and three Veterans Affairs medical centers. The nearly three-hour outage prevented clinicians from updated and accessing medical records starting around 5:00 p.m. ET according to an IT ticket.
The EHR deployed by VA is called Cerner Millennium, while the DOD and Coast Guard system is called Cerner MHS Genesis.
The disruption was traced back to a bug in one Oracle server that took down three of its databases that serve as the underpinning of the systems affected. There is an ongoing investigation into the “root” cause of the bug.
A VA official specifically disclaimed the outage was not due to deployment efforts. In a press conference, Terry Adirim, VA EHRM Integration Office program executive director, did not address how the pending purchase of Cerner by Oracle is impacting deployment if at all.
“Affected clinicians were unable to log into EHRM applications or retrieve EHRM data to legacy applications,” according to an IT ticket detailing the issue obtained by Nextgov. The system failed over to a “read only” backup and was fully restored shortly before 8:00 p.m.
Clinicians were instructed to create paper records during the outage to record medical information that could not be entered into the EHR.
The EHR system at Mann-Grandstaff VA Medical Center in Spokane, WA went offline March 3 after an update led to possible data corruption. The outage may have lasted to March 5.
Mann-Grandstaff VA is the test site for the Cerner Millennium system.
An investigation by Spokesman-Review revealed the Cerner system has seen multiple outages since its launch October 2020. The incident March 3 was “unlike previous episodes insofar as all data” in multiple software programs “may be corrupted.”
Robert Fischer, director of the VA Medical Center, told staff, “Assume all electronic patient data is corrupted/inaccurate.” Fisher told clinicians they should “provide only those healthcare services you are comfortable providing assuming all electronic sources of data are unreliable.”
Veterans were unable to fill prescriptions, even by mail, and clinicians were told to “make every effort to limit” ordering medications, laboratory tests and imaging studies like X-rays.
So much for “zero” unplanned downtime. Thanks Oracle.
Cerner was awarded the $10 billion contract in 2018 to modernize the existing electronic health record system without VA considering other bids. The software was supposed to work effectively with the similar system Cerner developed for the Defense Department.
The Cerner rollout of Millennium has seen repeated delays. It is now estimated to cost $21 billion, and the agency’s full adoption of Millennium has reportedly been hamstrung with multiple delays due to the pandemic and other challenges.
In July 2021, the IG published the results of its audit of the Millennium project that found the agency failed to properly report costs and funding needs associated with the new EHR system. VA apparently was not being transparent with taxpayers about the cost of the system.
EHR Rollout Allegations Audited
In February 2022, the Government Accountability Office found VA had failed to ensure data transferred during the rollout of its new EHR met clinicians’ needs.
VA did not properly monitor the accessibility, accuracy and appropriateness of clinical information as it was transferred in segments from one system to another.
Other problems included:
- VA identified errors in allergy, medication, and immunization data, which may pose risks for patient safety.
- VA started making reports from the new system available without consulting all users, so VA can’t be sure the reports meet their needs.
In March 2022, the IG published the results of its audits into the success and challenges faced by the Cerner Millennium rollout from October 2020 to date. Three different reports addressed care coordination, ticketing, and medication management.
The findings can be summed up, “The OIG is concerned that deployment of the new
EHR without resolution of the deficiencies presents risks to patient safety.” IG should be outraged, not concerned, about the avoidance of common sense when rolling out a system not ready for prime time, as will be discussed below.
Examples of substantiated allegations include allegations suggesting veterans’ lives are knowingly being placed at risk in favor of moving forward with a buggy and problem riddled rollout of a not-quite-ready-for-prime-time EHR system.
These are the worst offenders in my view that were substantiated and remain unresolved by the date of release of the IG report (March 17, 2022):
- Cerner service desk support staff did not communicate ticket status to end users.
- Cerner service desk support staff closed tickets prior to resolution.
- Medication lists were not accurately imported into the new EHR.
- The new EHR was not configured to support future clinic orders and automatically discontinued them. Providers were not notified of the discontinuance.
- The way patient record flags displayed or failed to display in parts of the new EHR raised safety concerns.
- Facility data reported to the national high-risk dashboard for patients with active High Risk for Suicide patient record flags had been inconsistent since the facility transitioned to the new EHR.
- Deficiencies in the data migration process caused incorrect names, genders, and contact information in the new EHR for some patients.
- New EHR issues caused delays in scheduling primary care appointments.
- Referrals were being lost or not addressed because of difficulties in the referral management processes in the new EHR.
- Some laboratory orders were “disappearing” and never reached the facility laboratory.
- Laboratory orders were “disappearing” from the providers’ view at times, affecting coordination of care, as providers were unable to tell what had been ordered and if there was duplication in orders from other providers.
- Veterans were unable to access the patient portal and use secure messaging to contact their care teams in the new EHR.
As an attorney who has worked on medical malpractice cases, I can tell you the above list of glitches, bugs, and missing information puts lives at risk.
But, it does not take a law license to know that “disappearing” lab orders, improperly canceled medication prescriptions, and botched referrals can be the difference between life and death in certain circumstances.
If a veteran is injured as a result of these errors, the person may consider speaking with legal counsel to investigate.
The IG report into Cerner ticketing practices is worth reviewing.
It makes one wonder if Cerner has the same commitment to veterans as VA espouses. Hopefully, VA did not select a vendor with irreconcilably different values.
If so, let’s hope the duo get a divorce before they have kids.
Oracle To Buy Cerner
Reports that Oracle was considering buying Cerner began circulating in mid December 2021 after VA Secretary Denis McDonough indicated the agency would proceed in its implementation of the Cerner Millennium system.
December 3, 2021, VA announced plans to move forward with the halted Cerner program, and that the agency would solve its problems with more management. It would also start deploying the software to network locations that server VA facilities in Ohio, Indiana, and Michigan.
The Oracle purchase offer of Cerner was formally announced by both companies December 20, 2021, in an all-cash tender worth $28.3 billion. Cerner will be Oracle’s anchor asset, which Oracle intends to expand using the company’s already global footprint.
As of March 2022, the purchase offer was extended. The companies appear to be waiting for approval of the deal by the Department of Justice and Federal Trade Commission.
At the center of the dollar signs are tens of millions of health records of soldiers and veterans contained within the Cerner data systems. Who now owns that data or the profit from data derivatives only possible through aggregation of electronic health systems?
Should it be taxpayers who paid to collate the data? How about the veterans whose private health information is in the system?
eMarketer reported the purchase was not only the largest ever for Oracle, but “Even though Oracle is a legacy tech company, it has fallen behind tech giants like Microsoft, Google, and Amazon when it comes to cloud computing—this deal would change that.”
The Cerner acquisition is apparently aimed to transform healthcare. Oracle will address potential treatment delays “transform[ing] healthcare delivery by providing medical professionals with better information – enabling them to make better treatment decisions resulting in better outcomes.”
There is big money behind the purchase, which targets the data within the databases managed by Cerner. Note that Cerner is not called an electronic health record company but a “data analytics firm.”
Oracle – Cerner Acquisition Press Release
According to a Cerner press release about the purchase:
“Working together, Cerner and Oracle have the capacity to transform healthcare delivery by providing medical professionals with better information—enabling them to make better treatment decisions resulting in better patient outcomes,” said Larry Ellison, Chairman and Chief Technology Officer, Oracle. “With this acquisition, Oracle’s corporate mission expands to assume the responsibility to provide our overworked medical professionals with a new generation of easier-to-use digital tools that enable access to information via a hands-free voice interface to secure cloud applications. This new generation of medical information systems promises to lower the administrative workload burdening our medical professionals, improve patient privacy and outcomes, and lower overall healthcare costs.”
“We expect this acquisition to be immediately accretive to Oracle’s earnings on a non-GAAP basis in the first full fiscal year after closing—and contribute substantially more to earnings in the second fiscal year and thereafter,” said Safra Catz, Chief Executive Officer, Oracle. “Healthcare is the largest and most important vertical market in the world—$3.8 trillion last year in the United States alone. Oracle’s revenue growth rate has already been increasing this year—Cerner will be a huge additional revenue growth engine for years to come as we expand its business into many more countries throughout the world. That’s exactly the growth strategy we adopted when we bought NetSuite—except the Cerner revenue opportunity is even larger.”
“Cerner has been a leader in helping digitize medical care and now it’s time to realize the real promise of that work with the care delivery tools that get information to the right caregivers at the right time,” said David Feinberg, President and Chief Executive Officer, Cerner. “Joining Oracle as a dedicated Industry Business Unit provides an unprecedented opportunity to accelerate our work modernizing electronic health records (EHR), improving the caregiver experience, and enabling more connected, high-quality and efficient patient care. We are also very excited that Oracle is committed to maintaining and growing our community presence, including in the Kansas City area.”
“Oracle’s Autonomous Database, low-code development tools, and Voice Digital Assistant user interface enables us to rapidly modernize Cerner’s systems and move them to our Gen2 Cloud,” said Mike Sicilia, Executive Vice President, Vertical Industries, Oracle. “This can be done very quickly because Cerner’s largest business and most important clinical system already runs on the Oracle Database. No change required there. What will change is the user interface. We will make Cerner’s systems much easier to learn and use by making Oracle’s hands-free Voice Digital Assistant the primary interface to Cerner’s clinical systems. This will allow medical professionals to spend less time typing on computer keyboards and more time caring for patients.”
The final implementation of the sale has seen some delays.
Veterans Affairs Buyers’ Remorse?
The EHR community within VA has seemingly been opposed to the change from the open-source solution used by VA for many decades to the new Cerner solution the agency does not and cannot control.
VA still largely relies on VistA, a system developed in-house and launched in 1978. VistA was considered a nimble system that allowed local customization depending on feedback from clinicians and other stakeholders.
VistA is widely considered the best EHR in the world based on a 2016 Medscape survey of 15,000 physicians. By comparison, the system used by the Defense Department, AHLTA, was considered the worst. The Cerner system is considered the middle of the EHR pack.
VistA is open-source and costs zero to license. Cerner is quite expensive to implement, now an estimated $21 billion.
The concerning sentiment was well encapsulated in a scathing article by Spokesman-Review reporter Orion Donovan-Smith republished in Stars and Stripes December 2021 after VA affirmed its intent to stick with the Cerner platform.
Congress charged DOD and VA to develop a health records system with “full interoperability” between the health systems. This paralleled the timeframe when the Obama administration rolled out the Virtual Life Electronic Record (VLER) for servicemembers.
“The goal there was for DOD and VA to agree upon an electronic health record system that they could both run,” said Roger Baker, who served as VA’s chief information officer from 2009 to 2013. Baker was VA’s point person in the effort.
“Our view at VA was that because of all the complexities of government medical care, the VistA system was the logical place to end up, and DOD’s view was that VistA was absolutely unacceptable,” Baker said. “We wrestled through that for three years, and that’s why the program failed in the end.”
Baker stepped down from the position following numerous Congressional hearings and investigations into challenges VA faced when rolling out many different IT systems.
Baker reportedly served on a Cerner advisory board from 2017 to 2018. (This led up to Secretary David Shulkin’s stalled negotiations to contract with Cerner prior to his now infamous ouster by subordinate agency officials within VA Public Affairs.)
The DOD moved forward with its Cerner solution in February 2017 at Fairchild Air Force Base outside Spokane, WA. Soon after, many of the current challenges facing Mann-Grandstaff were reported at Fairchild.
March 2017, President Donald Trump put his son-in-law Jared Kushner in charge of a then new White House office aimed to implement new business ideas into government starting with VA. Kushner supposedly gave officials 2 weeks to come up with a solution that would allow a smooth transition of service records from the DOD to VA.
Apparently, Kushner did not ask what veterans had done for decades without modern X Box level technology – print out the records and carry or mail them to VA to be digitized. Baker indicated the real problem is DOD relies on paper records that must be printed and then scanned into the VA system.
The solution officials came to was to push VA to adopt the same system implemented by the Pentagon arguing the VistA system was obsolete, anyway.
VA Secretary Shulkin Ouster
Kushner was successful in pushing the agenda of Trump’s Mar-a-Lago resort trio, a doctor, a lawyer, and chairman of Marvel Entertainment. ProPublica revealed significant deal making within IT sectors to essentially pimp out veteran data to Silicone Valley.
Insiders suggest Shulkin and others came to conclude the Cerner solution was “neither operationally effective nor operationally suitable.” Shulkin was ousted with the help of Trump insiders working within VA who reportedly went as far as to block Shulkin’s phone to prevent him from speaking directly to the press.
Shulkin was undermined when IG investigated his careless and improper use of agency funds for travel and other miscellaneous issues.
His replacement Robert Wilkie ultimately signed the $10 billion contact with Cerner, and the rest is history.
The Meagher Misconception
Enter Ed Meagher. He served as the VA deputy chief information officer and chief technology officer from 2001 to 2006.
Meagher argues the biggest misconception in this mess was that VistA is broken and needed to be replaced.
“That couldn’t be further from the truth,” Meagher said, pointing to a recent VA pilot program that showed VistA can operate faster and at lower cost through cloud computing.
Even though VistA relies on relatively old coding language, and it can be difficult to integrate with modern software, Baker argues the challenges were overblown. It suggests the problems with VistA were simply a red herring used to push the adoption of the Cerner system.
“In order to sell this massive program, VA had to create a rationale,” Baker said. “VistA has its problems, but if you look at what has been sold to Congress, those problems have been trumped up.”
One Big Problem
Unlike Cerner’s solutions, the VistA platform was built to provide veterans with quality and timely access to care. While not perfect, the system was not built to maximize profit or recover losses.
Cerner, on the other hand, built out its platforms to bill insurance companies.
“A lot of the reason that the Cerner record is failing out in Spokane is that, love it or hate it, the VistA system is the highest-rated electronic health record system as far as doctors are concerned,” Baker said. “And if you’re going to replace that system, you’ve got to figure out why do they want it.”
Problems at Fairchild have persisted through September 2021 when unresolved critical incidents that “could result in mission failure” were confirmed by the Government Accountability Office.
So, why do veterans really want a Cerner solution engineered to bill insurance companies?
While the agency is steadfast in its support of the Cerner solution, some lawmakers are questioning the premise of the change. Rep. Matt Rosendale of Montana is charged with oversight of the Cerner rollout within the House Committee on Veterans Affairs, is skeptical.
In a hearing, Rep. Rosendale concluded, “The idea that the VA and DOD need the same company’s software to exchange medical records is, frankly, ridiculous.” In the same hearing, he said, “It’s unacceptable to be risking medical errors while bugs get worked out in a live environment.”
Most American’s likely agree. Can one imagine a Senator allowing a dysfunctional EHR to replace their own existing system by their own highly paid physician?
To solve the problems, VA announced in December it would simply add a new layer of management with a program executive director who is a DOD official.
Has anyone noticed DOD seems to be all over the change forced onto VA while many VA officials, at least those working directly with the Cerner product, oppose the change?
Meagher said McDonough seems to think that “through heroic management and good speechifying we’re going to convince everybody to take an inferior product to be half as productive.”
“I’ve never seen adding new layers of bureaucracy make a program more effective,” Baker said in response to the move.
“What VA is apparently doing is, ‘We can’t let this out, because then Congress is going to realize that we’re not doing very well and they’re going to cut our money, and so the last thing we want to do is admit any kind of a mistake,” Baker said.
Even Rep. Debbie Wasserman Shultz (D-Fl) has been critical of the lack of transparency in a hearing last October.
“We’ve had a long string of leaders at the VA tell us that the intention is to finish in 10 years, the intention is to be transparent, the intention is to not affect patient safety, and none of that ever happens.”
Baker is not optimistic of the Cerner move giving it a 10% chance of replacing VistA. If it does, he says it will cost $30 billion and take 15 to 20 years.
Rep. Mike Bost (R-Il) has now urged McDonough and Remy to reconsider. Of course, that was before Oracle purchased Cerner, which introduces another layer of complication to an already complicated move.
“The VA electronic health record modernization effort has become a series of broken promises,” Bost said in a statement. “The serious issues with the new system in Spokane are undeniable. Yet, VA has been unwilling to face the music and make the fundamental changes that are necessary to fix them.”
Fish Or Cut Bait?
Where are you at on the transition? Should we stick it out with Cerner, now Oracle, or should we stick a fork in the transition?
The funds already paid are likely a sunk cost at this point, anyway.
I think the writing is on the wall. Baker is probably accurate in his assessment.
At what point is Cerner in breach of contract if the glitches are not fixed?
What is the extent the American taxpayer is on the hook for negligence claims for veterans injured due to the glitchy EHR?
VA knows the system does not work as advertised that that it puts veteran’s lives at risk when the dysfunction is bad.
Would a civilian hospital system adopt and then rollout a known dysfunctional EHR prior to the bugs being worked out? Would Senators tolerate such a change in their own hospital?