COVIDDX: AI-based clinical decision support system for learning COVID-19 disease representations from multimodal patient data

The COVID-19 pandemic has affected the world on a global scale, infecting nearly 68 million people across the world, with over 1.5 million fatalities as of December 2020. A cost-effective early-screening strategy is crucial to prevent new outbreaks and to curtail the rapid spread. Chest X-ray images have been widely used to diagnose various lung conditions such as pneumonia, emphysema, broken ribs and cancer. In this work, we explore the utility of chest X-ray images and available expert-written diagnosis reports, for training neural network models to learn disease representations for diagnosis of COVID-19. A manually curated dataset consisting of 450 chest X-rays of COVID-19 patients and 2,000 non-COVID cases, along with their diagnosis reports were collected from reputed online sources. Convolutional neural network models were trained on this multimodal dataset, for prediction of COVID-19 induced pneumonia. A comprehensive clinical decision support system powered by ensemble deep learning models (CADNN) is designed and deployed on the web. The system also provides a relevance feedback mechanism through which it learns multimodal COVID-19 representations for supporting clinical decisions.