**Note: As of 3.26.2017, a publicly available dataset on COVID-19 chest X-ray called Covidx is just released to public by University of Waterloo, we are still working on the deep learning model for this dataset.
Coronavirus Pandemic:As a live update from CNN, COVID-19, previously known as novel Coronavirus, has killed more than 4607 people and infected over 124,581 people in 144 countries. (Pettersson et al., 2020) This number is continuing to grow every day. In the United States, the COVID has been spreading rapidly. Currently, 21 states have declared an emergency over the spread of Coronavirus and they reported over 1,000 confirmed cases, and over two dozen fatalities attributed to COVID-19.
Pneumonia problem:WHO also officially calls COVID-19 a pandemic (a global health emergency). According to them, COVID-19 are a large family of viruses that cause illness ranging from the common cold to more severe diseases such as pneumonia, severe acute respiratory syndrome, and even death. Elderly patients and patients with pre-existing conditions are considered at the highest risk of mortality by Coronavirus.
Chest X-ray:COVID-19 can first be screened for flu-like symptoms for mild cases. For the more serious case with pneumonia-like symptoms, a chest x-ray is a golden standard for physicians and radiologists to check for the infection caused by the virus. An x-ray imaging will allow your doctor to see your lungs, heart and blood vessels to help determine if you have pneumonia. When interpreting the x-ray, the radiologist will look for white spots in the lungs (called infiltrates) that identify an infection. This exam, together with other vital signs such as temperature, or flu-like symptoms, will also help doctors determine whether a patient is infected with COVID-19 or other pneumonia-related diseases. The standard procedure of pneumonia diagnosis involves a radiologist reviewing chest x-ray images and send the result report to a patient’s primary care physician (PCP), who then will discuss the results with the patient.
Fig 2: Current chest X-ray diagnosis vs. noval process with PneumoScan.ai
The problem with a long wait time for chest X-ray result:A survey by the University of Michigan shows that patients usually expect the result came back after 2-3 days a chest X-ray test for pneumonia. (Crist, 2017) However, the average wait time for the patients is 11 days (2 weeks). This long delay happens because radiologists usually need at least 20 minutes to review the X-ray while the number of images keeps stacking up after each operation day of the clinic. Patients usually have to frequently call in with their clinic to check for the result after 5 days of not hearing back. Source: Radiologists want patients to get test results faster With the patient's expectations of timely results in mind, the cutting edge technology of deep learning such as computer vision can be used to speed up patients’ wait time for pneumonia lab results; patients with positive pneumonia lab results can receive appropriate care sooner, especially in this critical time of the COVID-19 pandemic. PneumoScan.ai is a web application with an integrated machine learning model where radiologists can upload x-ray images for the machine learning model to determine with high accuracy if the x-ray image indicates pneumonia.
What it does
New research has found that an artificial intelligence (AI) radiology platform such as our PneumoScan.ai can dramatically reduce the patient’s wait time significantly, cutting the average delay from 11 days to less than 3 days. (Mauro et al., 2019) It is due to the fact that an AI system can review, highlight the pneumonia sign and classify each X-ray image all in less than 10 seconds (comparing the radiologist’s 20 minutes that we mentioned earlier), and it can do that same task effortlessly for 24 hours without taking a break. This time cut is especially critical in the time amid the pandemic of COVID-19. With this spreading rate, it will be overwhelming for radiologists to review a massive number of chest X-ray images of potential COVID-19 infected patients. With the assistance of PneumoScan.ai, it can automatically highlight the suspected signs of pneumonia for the radiologists and speed up the process of chest X-ray review. Therefore, more COVID-19 positive-tested patients will get their result back faster and receive appropriate care sooner to prevent the spread of the virus.
How we built it
Dataset:For this project, we are going to use a dataset available at Kaggle consisting of 5433 training data points, 624 validation data points and 16 test data points. Chest X-ray images (anterior-posterior) were selected from retrospective cohorts of pediatric patients of one to five years old from Guangzhou Women and Children’s Medical Center, Guangzhou. All chest X-ray imaging was performed as part of patients’ routine clinical care. Link to dataset: Chest X-Ray Images (Pneumonia) Data Processing: The image data is loaded from the training directory and apply some image augmentation to generate different variants of the images using preprocessing libraries from Pytorch. Model building:
Fig 3: Overview of the deep learning Pytorch framework for Chest- X-ray image localization
This project’s goal is to draw class activation heatmaps on suspected signs of pneumonia and then classify chest x-ray images as “Pneumonia” or “Normal”. After initializing the model, we load the pre-trained Resnet-152 available from Torchvision for transfer learning. ResNet-152 provides the state-of-art feature extraction since it is trained on a big dataset of ImageNet.
Resnet-152:ResNet-152, as the name sounds, consists of 152 convolutional layers. Due to its very deep network, the layers are arranged in a series of Residual blocks. These Residual blocks skip connections to help prevent the vanishing gradients which are a common problem with networks with deep architecture like ours. Resnet also supports Global Average Pooling Layer which is essential for our attention layer later on. Link about Resnet from Pytorch hub: https://pytorch.org/hub/pytorch_vision_resnet/
Global Average Pooling Layer:The Global Average Pooling layer replaces the fully connected layers at the end of the common convolutional-layer architecture. The Global Average Pooling layer helps reduce a large number of parameters at the end of Resnet-152, this helps prevent over-fitting which is a common problem of using fully-connected layers on deep networks.
Attention Layer:For the attention layer to draw the heatmap, we use the global average pooling layer proposed in Zhou et al. (2016) Global average pooling layer explicitly enables the convolutional neural network (CNN) to have remarkable localization ability.
Results:We achieve 97% accuracy on the training dataset and 80% on the testing dataset.
Web development:The trained weights of the deep learning Pytorch model for the chest X-ray image localization are used to predict pneumonia in a form of Django backend web app pneumoscan.ai. This web app is hosted at this domain pneumoscan.site. While the minimal front-end of this web app is done using HTML, CSS, Jquery, Bootstrap.
Use case:Radiologists can upload single or multiple X-ray images to our web app to get accurate results within a few seconds. Currently, anyone can access this service without any authentication needed. In future, radiologists required to sign-in first to use our service and they can also avail other added services like “save results in history” and “download results”.
Demo:The working demo of our model can be seen in the demo section of our website. In the demo section, we have provided eight sample X-rays images and by just clicking on them, visitors can see results of the respective sample image.
Fig 4: Main libraries used for this project
The packages required for this project are as follows:
• Torch (torch.nn, torch.optim, torchvision, torchvision.transforms)
Challenges I ran into
This hackathon project was altogether was a very different experience for us which challenged us throughout this project when two of our initial and main developers in Pytorch dropped out. So, this is the first time we all were working with the PyTorch and deploying it into a web-app. Writing code for the prediction in a form of Django backend was a little bit of challenging and confusing at the initial stage. But in the end model.eval() saved our lives from death. Deployment of this large size website was also challenging for us as most of hosting platforms like Heroku doesn’t allow to exceed more than 500 Mb.
Accomplishments that I'm proud of
We manage to finish the project in such a limited time of 2 weeks in our free time from school and work. Two of our initial and main developers in Pytorch dropped out and we still keep striving to submit on time while learning and developing at the same time. We are really satisfied and proud of our final product for the hackathon.
What we learned
Through this project, we learn to develop a complicated image-recognition deep learning model using Pytorch. We also learn the process of developing a mini data science project from finding dataset to training the deep learning model and finally deploy & integrate it into a web-app. This project can’t be done without the efforts and collaboration from a team with such diverse backgrounds in technical skills.
Fig 5: Developing team
What's next for PneumoScan?
- After this competition, we will work on improving the performance of the platform, preferably by reading more scientific literature on state-of-art deep learning models implemented for radiology.
- We also plan to add the bound box around the suspected area of infection on top of the heatmap to make the output image more interpretable for the radiologists.
- In many pieces of literature, they mentioned developing the NLP model on radiology report with other structured variables such as age, race, gender, temperature... and integrating it with the computer vision model of chest X-ray to give the expert radiologist’s level of diagnosis. (Irvin et al., 2019; Mauro et al., 2019) We may try to implement that as we move further with the project in the future.
- With the improved results, we will publish these findings and methodologies in a user-interface journal so that it can be reviewed by expert computer scientists and radiologists in the field.
- Eventually, we will expand our classes to include more pneumonia-related diseases such as atelectasis, cardiomegaly, effusion, infiltration, etc. so that this platform can be widely used by the radiologists for general diagnosis even after the COVID-19 pandemics is over. Our end goal is to make this tool a scalable that can be used in all the radiology clinic across the globe, even in the rural area with limited access to the internet like those in Southeast Asia or Africa.
Crist, C. (2017, November 30). Radiologists want patients to get test results faster. Retrieved from https://www.reuters.com/article/us-radiology-results-timeliness/radiologists-want-patients-to-get-test-results-faster-idUSKBN1DH2R6
Irvin, Jeremy & Rajpurkar, Pranav & Ko, Michael & Yu, Yifan & Ciurea-Ilcus, Silviana & Chute, Chris & Marklund, Henrik & Haghgoo, Behzad & Ball, Robyn & Shpanskaya, Katie & Seekins, Jayne & Mong, David & Halabi, Safwan & Sandberg, Jesse & Jones, Ricky & Larson, David & Langlotz, Curtis & Patel, Bhavik & Lungren, Matthew & Ng, Andrew. (2019). CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison.
Kent, J. (2019, September 30). Artificial Intelligence System Analyzes Chest X-Rays in 10 Seconds. Retrieved from https://healthitanalytics.com/news/artificial-intelligence-system-analyzes-chest-x-rays-in-10-secondsLambert, J. (2020, March 11). What WHO calling the coronavirus outbreak a pandemic means. Retrieved from https://www.sciencenews.org/article/coronavirus-outbreak-who-pandemic
Mauro Annarumma, Samuel J. Withey, Robert J. Bakewell, Emanuele Pesce, Vicky Goh, Giovanni Montana. (2019). Automated Triaging of Adult Chest Radiographs with Deep Artificial Neural Networks. Radiology; 180921 DOI: 10.1148/radiol.2018180921
Pettersson, H., Manley, B., & Hernadez, S. (2020, March 16). Tracking coronavirus' global spread. Retrieved March 16, 2020, from https://www.cnn.com/interactive/2020/health/coronavirus-maps-and-cases/
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016). Learning Deep Features for Discriminative Localization. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). doi: 10.1109/cvpr.2016.319
Try It out
- Facebook Hackathon: AI
- Pandemic Response Hackathon
- COVID-19 Global Hackathon 1.0
- HackCOVID 2
matplotlib, numpy, pil, pytorch1.0.1, torchvision0.2.2