Spectrum of Conditions and Vendors

With the adoption of digital health over the last decade, medical records have moved from being mostly on paper to being nearly completely digitized. Electronic Health Record vendors as well as healthcare focused data analytics firms have been steadily increasing their predictive capabilities for clinical and operational improvements with these EHR datasets. In this article, I’ll focus on the clinical applications, covering the spectrum of conditions and providers. I’ll also be referencing and sharing a catalogue I created for care providers to be able to to identify available analytics tools. There are many examples emerging that demonstrate significant improvements for health systems able to leverage EHR data:

• North Oaks Health System lowered mortality rates by 18%
• University of Pennsylvania Health System was able to detect sepsis 12-hours before onset
• NorthShore University HealthSystem meaningfully lowered readmission rates

Digging into the clinical applications of predictive analytics, the top issues that are being addresses are predicting unplanned readmissions, sepsis risk, and inpatient mortality. The full list of conditions can be found at www.automated.health.

A caveat to the data gathered is that although there are numerous site-specific projects described in news briefs about tools built on some of the larger vendor’s platforms, the firms often do not provide detailed information online about their predictive analytics capabilities. For instance, as noted in the intro, one of Epic’s clients, NorthShore University HealthSystem, used the EHR platform to develop CAPE, a ‘Clinical Analytics Predictive Engine,’ that assesses risk for cardiac arrest, 30-day readmission and mortality. Another hospital, North Oaks Health System, implemented CCA, a Clinical Care Advisory tool developed by Epic, that helped lower mortality rates by 18%. A difficulty in conducting a review such as this, is that while these tools are discoverable in various writeups, they are not comprehensively detailed by the vendors themselves.

With that as an observation on gaps in the available data, roughly five vendors provide the largest collection of solutions, with other vendors providing a smaller range of products.

What Goes Into The Predictions, And How Good Are They?

When making predictions, the full breadth of data points across a patient’s EHR records is evaluated. This includes Encounters, Lab Tests, Medications, Vital Signs, Procedures, Notes, and Diagnoses. AI within the healthcare industry is seeking to address the ‘black box’ nature behind many models that can make it difficult to for a person reviewing the output to understand the ‘thought process’ involved. In their work on EHR analytics, the Google AI team took steps towards explainability. In a model for inpatient mortality, they present a proof-of-concept schematic that calls out the elements in a patient’s chart that contributed to the prediction. In the example showcased in the writeup, the schematic shows that the model considered medication (Vancomycin, Metronidazole), physician notes (metastatic breast cancer, R Lung malignant, R Lung empyema, R Lung pleurx), a radiology report (left pleural effusion, right pleural effusion), and a pulmonary consult (complicated pleural space, increased loculted effusion) as important data points. Finding ways to increase interpretability is a key requirement for making the predictions most useful to clinicians.

How well do these predictive solutions in fact work? To answer this we’ll look at some of the metrics provided. Dascena Insight, a tool which “autonomously forecasts sepsis onset using only vital sign data and is compatible with all major EHR systems,” achieved sensitivity of 90% and specificity of 90% in a randomized clinical trial. Since the true positive rate = sensitivity (TPR), and the false positive rate = 1- specificity (FPR), the numbers translate into a 10% FPR and a 90% TPR, and indicate that the model does a good job of separating the two classes (at risk, not at risk). A variety of publications estimate sepsis rates in ICUs at around 25-30%. To mirror those numbers, let’s imagine we have a population of 100 people in an ICU, of which 30 have sepsis and 70 do not. With the Dascena Insight values provided, there would be 34 alerts sent out, of which 27 would be true positives, and 7 would be false alarms.

Unfortunately, many of the vendors do not publish performance metrics on their sites, so while I’ve been mainly seeking to cover available products, with the lack of vendor data, let’s take a look at the metrics from a research group, Google AI. In the work referenced above, ‘Deep Learning for Electronic Health Records’, Google researchers obtained the following results for the area-under-the-receiver-operating curve. A score of 0.5 has no predictive value and is equivalent to random guesses, approaching 1 demonstrates increasing accuracy, and 1 is perfect (which is typically never achieved in practice).

• Predicting if patients will stay long in the hospital: 0.86
• Predicting inpatient mortality: 0.95
• Predicting unexpected readmissions: 0.77

While acknowledging that these were retrospective studies, and that it’s important to recognize that actual performance against live data would potentially be lower for a variety of reasons, from this data it’s clear that the models have significant predictive value.

Given the relative newness in applying advances in AI to EHR data, joint interest by research institutions and hospital systems, and the already visible strong performance gains over prior approaches across a range of conditions, I think we’ll be seeing an ongoing increase in capabilities over the coming year which will be a welcome development for providers and patients alike.

Written by Trevor z Hallstein, Healthcare Product Lead at Swish Labs