Trustworthy and High-Performance Question Answering for Electronic Health Records

ABSTRACT Patients accumulate large volumes of information in their electronic health record (EHR), and finding this information often proves difficult, especially given the usability issues associated with EHRs. One of the most intuitive means of encoding information needs is with natural language questions thus automatic question

answering (QA) methods provide an intuitive interface for quickly accessing patient information. An emerging and promising solution to this QA problem is the use of large language models (LLMs), which are natural language processing (NLP) models trained on significant amounts of natural language text. LLMs

provide an excellent building block for training high-performance QA systems and have been shown to be excellent at answering medical questions However, LLM-based models have also been shown to provide completely false information (“hallucinations”). When applied to EHR QA, this could result in clinicians making

diagnosis and treatment decisions based on invalid information about their patient, easily leading to harms. The naïve use of such systems in a clinical environment is thus seen as unacceptable to some. Further, LLMs are not well-suited to the structured data found in EHRs as the language models are trained only on unstructured text.

On the other hand, there are systems designed from the ground up to be trustworthy and well-suited to the clinical environment. As part of our preliminary work, we developed the quEHRy system that understands when a question is outside its capabilities and thus rarely returns a wrong answer. There is still much room for

improvement for quEHRy, however, as it currently only answers questions for structured data and its understanding of medical concepts requires improvement. In other words, the strengths and weaknesses of quEHRy are well-complemented by LLM-based QA systems (and vice versa). The key question this proposal tries

to answer, then, is how to combine such systems to achieve a single EHR QA system that is both trustworthy and high-performance. To this end, we hypothesize it is possible to create hybrid methods leveraging the power of LLMs while maintaining the trustworthiness of carefully-designed systems like quEHRy. Further, this both requires and

enables QA over both structured and unstructured EHR data, so we will investigate ways to better understand the alignment between structured and unstructured data, and how this can be leveraged for EHR QA.