Coronavirus Disease 2019 (COVID-19) was first detected in December 2019 and has spread rapidly in most cities and countries worldwide.1) Despite the expiration of the public health emergency declaration,2) the number of patients hospitalized for COVID-19 continues to increase, resulting in over six million deaths globally.3) Faced with this global health emergency, patients with various comorbidities, including cancer, have shown life-threatening outcomes after COVID-19 infection.4) Some patients with comorbidities stemming from various treatments might cause more immunosuppressed status.5) Patients with cancer have a 2.25-fold higher risk of mortality, including increased rates of hospital admission or mortality, compared to patients without cancer.6) Considering the immunosuppressed status of some vulnerable populations on virus infections, previous studies aimed to provide precise tools for screening or predicting prognosis of individuals with underlying health conditions such as cancer. However, the limitation in recruiting vulnerable patients, which resulted in an uneven dataset in the analysis, hindered the provision of consistent outcomes for screening risk factors or predicting prognostic conditions in previous studies.7) Despite these limitations, in clinical practice there is a consistent need for improved diagnostic or predictive methods for COVID-19 in patients with deteriorating health conditions. In particular, pre-existing comorbidities are well-known risk factors closely associated with increased mortality among COVID-19 infected patients.8) Nevertheless, the heterogeneous levels of data granularity regarding vulnerable health conditions, such as cancer, within specific subpopulations among the collected samples restricted the generation of accurate estimates since current statistical methods provided insufficient prognostic or diagnostic information.9) Still, as the outcome severity is greater in patients with cancer than in patients without cancer infected with COVID-19,5) there is an unmet need for screening or predicting outcomes in patients with cancer, especially compared to other vulnerable patients with COVID-19 infection and various other health conditions, despite existing imbalanced dataset issues.
As artificial intelligence (AI) has contributed to clinical decision-making and disease diagnosis,10) it has supported real- time inference for health-risk alerts and prediction of health outcomes.10) With significant discriminatory insights into AI in healthcare, during the pandemic era, various patient data were used in AI studies to provide efficient AI models and precise predictors of COVID-19 infection.11,12) However, as in previous studies, the performance of AI models was significantly influenced by imbalanced data, which could not be exempt from unbalanced datasets in comorbidities.7,13) Therefore, it is necessary to evaluate the impact of imbalanced datasets affected by comorbidity diversity on the performance of AI models and predictors among patients infected with COVID-19. In particular, considering the difficulties of enrolling vulnerable patients in trials, such those with cancer,14) it is necessary to assess the effect of including patients with cancer among those with COVID-19 on the performance of AI models through comparison with datasets without cancer.
Therefore, the current systematic review was conducted to evaluate the performance of AI models and predictors of COVID-19 in patients with pre-existing comorbidities, comparing those with and without data of patients with cancer.
This systematic review was conducted based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.15)
The PubMed and Cochrane Library databases were searched for eligible articles up until May 2023. A manual search was conducted to identify studies evaluating AI models and to provide important predictors or values for clinical outcomes, such as mortality, severity, hospitalization, and mechanical ventilation, for datasets including patients with or without cancer. Titles and abstracts were distinguished using the following terms to categorize the associated text: “Cancer”, “COVID-19”, “AI”, “Cardiovascular disease (CVD)”, “diabetes”, or “mortality”, “severity”, “hospitalization”, or “mechanical ventilation.”
The investigators initially evaluated titles and abstracts to identify potentially relevant studies. To qualify for inclusion in this study, studies were required to meet the following criteria: 1) studies using data from patients with or without cancer diagnosed with COVID-19; 2) studies using AI, such as AI, machine learning, and deep learning; and 3) studies using AI models to predict mortality, severity, hospitalization, or mechanical ventilation. Preprinted works, abstracts, reviews, systematic reviews, meta-analyses, books, and animal studies were excluded. Our study incorporated various studies aimed at predicting the severity of Covid-19. The definition of severity in this context varied across the included studies. Disagreements between the two investigators were mutually resolved.
Two investigators extracted data from the selected literature. Initially, information was included regarding the first author, source of data, number of patients included in the data, inclusion or exclusion criteria, endpoints, types of AI models, performance metrics of the AI models, and important predictors or values. If AI models were developed during a study, we classified these new AI models as “self-developed model.” In addition, based on the data source, datasets used in individual studies were classified as “hospital data” and “public data” if the source was hospital based or publicly available, respectively. These measures involved a comparison between studies that included datasets containing information about patients with cancer (referred to as a “cancer study”) and studies that excluded patients with cancer from their datasets (referred to as a “non-cancer study”). The comparison was based on the predicted values, including mortality, severity, hospitalization, and mechanical ventilation. These “cancer study” or “non-cancer study” groups comprised patients diagnosed with COVID-19 and had one or more comorbidities. The current study measured the performance metrics of the AI models, which included the evaluation of models used more than twice. Additionally, the five most important variables, including underlying diseases that served as predictors of mortality and severity, were evaluated using the included data. Important predictors are variables with a high ranking for prediction among the variables in the prognostic prediction model. We collected the top five variables based on age, sex, or underlying diseases that appeared in the included articles. Underlying diseases were classified based on “cardiovascular”, “endocrine”, “respiratory”, “gastrointestinal”, “psychological”, “neurological”, “cancer”, or “others.” The variables of the underlying diseases were categorized according to the relevant diagnosis or treatment. If the diagnosis or treatment was not specifically included in any of the underlying disease categories, the variable was included in the “other” category.
To compare the performance of the AI model between cancer and non-cancer studies based on the AUC range, the proportion of included studies was described as a percentile. The effect size for the AUC of the machine learning models, expressed as the mean difference and standard deviation, was calculated. A heat map was created to illustrate the important variables and analyze the frequency of studies stipulating important underlying diseases, according to age and sex, as the top five important variables. In addition, the percentages were described based on comorbidities in both cancer and non-cancer studies. Analyses were conducted using Microsoft Excel and R software (version 4.3.1).
From the search results, we obtained 1629 articles from PubMed and 213 studies from the Cochrane Library. After eliminating forty duplicate studies, an additional 1,334 research articles were excluded following the screening of titles and abstracts, in accordance with the inclusion and exclusion criteria for the present study. Finally, sixty-three suitable articles16-78) were included in the analysis, which were divided into forty- five articles16-60) of cancer studies and eighteen articles61-78) of non-cancer studies (Fig. 1).
The basic characteristics of the sixty-three included studies are shown in Tables 1 and 2. A total of 9,284,777 patients from the cancer study and 1,095,679 patients from the non-cancer study were included in the analysis. Among the cancer studies, twenty-eight16,19,20,22,24,27,28,31,33,35,36,39-50,54-58) used only hospital data to evaluate AI models or important predictors, whereas one study59) included both hospital and public data. Sixteen studies17,18,21,23,25,26,29,30,32,34,37,38,51-53,60) used public data for predicting COVID-19 infection among cancer studies. As predictive values, thirty-one studies16-46) predicted mortality using AI models, eight studies17,20,22,41,47-55) evaluated severity as the final outcome among cancer studies, and four studies predicted hospitalization.
Among the eighteen studies61-78) that evaluated AI models in patients without cancer infected with COVID-19 (Table 2), fifteen studies61,63-65,67,69-78) used hospital data, and three studies62,66,68) used public data. As predictive values, eleven studies61- 71) predicted mortality, whereas five studies71-75) predicted the severity of COVID-19. Based on the diversity of patient data among cancer studies, 58.9 percent of patients had urinary diseases such as urinary tract infections, kidney stones, interstitial cystitis, kidney failure, urethritis, whereas only 0.03 percent of patients had gastrointestinal diseases as comorbidities. Furthermore, the highest proportion of patients had cardiovascular disease (37.02 percent) as a comorbidity among non-cancer studies; and psychological diseases were not identified among non-cancer studies (Fig. 2).
For cancer and non-cancer studies, the performance metrics of the AI models were demonstrated using AUC, accuracy, sensitivity, specificity, and F1 score (Tables 3 and 4). Among the forty-two studies16,17,19-28,30-42,44-60) providing performance metrics in cancer studies, forty studies16,17,20-28,31-33,35-60) provided AUC values with the AI model (Table 3). Eighteen non-cancer studies60-77) provided performance metrics of the AI models, including the AUC value (Table 4).
To predict mortality, the AUC values of AI models in cancer studies showed various levels compared with non-cancer studies (Fig. 3a). Majority of non-cancer studies (54.55 percent) showed AUC levels of AI models over 0.90, whereas 30.77 percent of cancer studies showed AUC values in the same range as that for predicting mortality. For predicting severity, compared to non-cancer studies, a larger proportion of cancer studies (20 percent vs 33.33 percent, respectively) provided AI models with AUC values between 0.90 and 1.00 of COVID-19 infection (Fig. 3b). For predicting hospitalization, 66.67 percent of studies showed the AUC value from 0.90 to 1.00 among non-cancer studies, while 28.57 percent of studies showed AUC level of AI models in the same range within cancer studies (Fig. 3c). For non-cancer studies, only one study provided an AUC level of AI model (AUC 0.80~0.89) (Fig. 3d) predicting hospitalization. For predicting mortality (3.85 percent), severity (8.33 percent), and hospitalization (14.29 percent), only cancer studies showed AUC values between 0.50 and 0.69. Additionally, based on the predicted values for mortality and severity, support vector machine (SVM) showed the highest AUC compared to other models such as random forest (RF) or extreme gradient boosting (XGboost) (Supplementary Fig. 1).
To predict the mortality and severity of COVID-19 in both cancer and non-cancer studies, age was ranked as the most important value compared to other predictors, such as psychological or neurological diseases (Supplementary Fig. 2). In cancer studies, cardiovascular disease was indicated as the most or second most important value for predicting severity, whereas in non-cancer studies, no study indicated cardiovascular disease as an important value (Supplementary Fig. 2). Furthermore, despite the inclusion of data from patients with cancer no studies have demonstrated cancer as an important predictor of severity.
We conducted a systematic review to evaluate AI models that predict mortality, severity, hospitalization, mechanical ventilation, and other relevant predictors by comparing cancer and non-cancer studies. According to the current study, majority of non-cancer studies appear to exhibit AUC values ranging between 0.8 and 1, whereas cancer studies demonstrate more diverse AUC values including values of <0.65. Although a higher level of AUC represents better performance of AI models to distinguish between positive and negative scores, among cancer studies, the AUC values of one could promote the overfitting of data with a small sample for specific categories.79) Furthermore, the data imbalance of comorbidities shown in cancer studies might also contribute to the low levels and inconsistency of AUC values among cancer studies compared with non-cancer studies. Because the classification of included data can improve the outcome of AI models with inter- and intra-observer variability, the degree of data imbalance, defined as the ratio of the sample size of the minority class to that of the majority class, could also influence the model performance.80) Under- or overrepresentation of categories of included datasets, such as cancer studies in the current study, are potential sources of class imbalance among the patient data collected and the diversity of model performance.81) In particular, when including the data from patients with cancer, the uncertainty of cancer-specific risk factors, including balanced datasets in cancer studies for accurate prediction of outcomes such as mortality or severity of COVID-19 infection, could be more challenging than in non-cancer studies.81,82) According to Lara et al., low prevalence of certain conditions such as patients with cancer infected by COVID-19 with concurrent medical problems might hinder the collection of more representative data to provide a balanced dataset.83) Furthermore, mitigating the errors of overfitting caused by fewer datasets available for some categories, including uneven data of patients with cancer, could also affect the low metrics of AI model performance among some cancer studies.81,82) Considering the close association between high level of AUC and improved performance of AI models, AUC values consistently high in non-cancer studies from 0.80 to over 0.90 in the current study might reflect more balanced datasets used for improved prediction.84) The diversity of database constructions related to COVID-19 infections,83) especially data of patients with existing comorbidities, might cause unequal AI model performance, such as AUC values. Therefore, we still need more balanced datasets to provide consistent and improved model performance for predicting clinical outcomes of COVID-19 infections among cancer studies. Furthermore, we constructed a forest plot based on the AUC values obtained from the models applied to patients with and without cancer. In addition, based on the current investigation, RF and XGBoost were employed to predict COVID-19 infection among the included studies to predict mortality and severity among machine learning models, with SVM showing the highest AUC value. SVM exhibited a trend toward the highest AUC value. However, it is difficult to definitively conclude that SVM were the best performing models, as each study utilized data from different populations and the usage frequency of particular model varied.
Additionally, the importance of each predictive indicator in cancer and non-cancer studies was evaluated in the current study. Age was equally important for all predictive indicators among the included studies that predicted the clinical outcomes of COVID-19. A previous systematic review investigating the association between various predictive factors and the risk of mortality due to COVID-19 demonstrated findings similar to those of our study.86) The results revealed an increased susceptibility to COVID-19-related mortality with advancing age (OR: 2.61, 95 percent CI: 1.75-3.47; HR: 1.31, 95 percent CI: 1.11-1.51).86) Since the onset of COVID-19, older age has been recognized as a risk factor.87) In particular, patients with various comorbidities, including cancer, are exposed to various types of medications, thereby suppressing the immune system and may invoke vulnerability to COVID-19 infection.88) Age-related alterations in the immune system affect many aspects, leading to a decrease in pathogen immunity with increased age.89) Aging is associated with high morbidity and mortality due to various infections and a significant decrease in vaccine efficacy.89) In the recently announced COVID-19 and Cancer Consortium (CCC19) cohort,90) the median age of patients with cancer and COVID-19 was 66 years old, with 56 percent aged 65 years or older. The TERAVOLT cohort study on patients with thoracic malignancies and COVID-19 revealed a close association between age and increased risk of mortality (OR 1.88, 95 percent CI 1.0-3.6).91) However, the exact cause of this association is unclear, and further research is needed on its interaction with age in the context of COVID-19.
Cardiovascular disease (CVD) has also been demonstrated as an important factor across all predictive indicators in cancer studies, whereas in non-cancer cases, it has been shown as a significant predictor of mortality. Among other comorbidities, CVD has been an independent predictor of mortality.92) This suggests that CVD is an independent risk factor for viral acquisition with serious consequences; therefore, the cumulative risk may be higher in patients with CVD.93) Increased concerns and treatment of patients with various comorbidities such as cancer could increase the CVD burden with increasing blood pressure and relevant diseases.93) Momtazmanesh et al.94) also indicated that preexisting and newly developed CVDs are common in patients with COVID-19 and are associated with increased severity and mortality in these patients. A previous systematic review of the mortality and severity of COVID-19 also demonstrated that CVD was associated with an increased risk of deteriorated outcomes in patients with COVID-19.95) Therefore, CVDs play an important role in the outcome of patients with COVID-19, which require careful consideration and management in clinical practice.
Our study has several limitations. First, there is a possibility of overlooked studies due to the search methodology used. Specific keywords were employed to search for relevant articles. Although our search keywords provided effective results in achieving the study objectives, there is a risk that important materials did not emerge in our search queries. Second, the interpretation of our results should proceed with caution because the judgment criteria between severe and non-severe patients were not uniform. Third, we excluded deep learning when constructing the forest plots because no deep learning methods had two or more AUC values. Therefore, future work is required to collect and analyze more relevant resources, necessitating further studies on the presentation of predictor importance.
In conclusion, the current systematic review demonstrated diverse AUC values in cancer studies compared with non-cancer studies. Among cancer studies, under- and over-representation of data on comorbidities has been reported. Considering that the AUC values were influenced by the dataset balance, more data should be applied to develop or evaluate AI models predicting clinical outcomes such as mortality or severity of COVID-19 in patients with various comorbidities, as well as predictors.
The authors have no conflicts of interest to declare with regards to the contents of this study.