Zoom Meeting: https://umass-amherst.zoom.us/j/91805727476
Abstract:
Natural Language Processing(NLP) methods have been widely used for information extraction(IE) from unstructured natural language texts. These NLP systems use machine learning models that are trained on named entity recognition (NER), relation extraction (RI), context-based question answering (CQA) and text classification datasets. In the past few decades, several such systems have been proposed for safety-critical domains such as social-media-based public health monitoring, drug efficacy surveillance and pharmacovigilance. Failure of a deployed system in such domains can have potentially catastrophic negative costs. Safe and responsible deployment of these systems should ensure that the risk to human subjects is minimized, and provide deployment-time monitoring controls to the end-user.
Therefore, in addition to having good evaluation performance, we posit that safety-critical IE systems should necessarily have the following three properties. (1) They should be able to produce a calibrated confidence score for their predictions. (2) They should be able to provide interpretation for their predictions. And, (3) they should not cause any additional harm to users who contributed to their development. In this work, we first provide a case study that designs a compliant IE system for an IE problem with a simple output structure, Anti-Coagulant Reversal Agent Administration Extraction (ACRAA). ACRAA is a text classification problem which allows us to interpret the aforementioned requirements in the context of machine learning. Next, we look at structured prediction problems that have a more complex output structure. We describe and release the MADE 1.0 (UMass Medication and Adverse Drug Event) dataset. MADE 1.0 is a dataset of annotated electronic health records (EHR) which is used to train and evaluate structured output IE systems like NER, RI and CQA. To improve the performance of NER methods on clinical texts, we propose two Conditional Random Field (CRF) based output layers for neural networks and provide inference methods for them. To improve the calibration of structured prediction systems, we propose a calibration scheme that can provide calibration for any predefined output entity of interest. This calibration scheme can also use global features and uncertainty estimates to improve the performance of the underlying model. Lastly, for our remaining work, we outline a scheme for token-based importance scores to provide interpretability for IE systems.
Advisor: Hong Yu