Hidden-mortality risk among patients deemed “low-risk” following high-risk operations

In the surgical field, growing attention has been paid to the identification of patients with a particularly high predisposition to postoperative morbidity and mortality following high-risk operations. The hepatopancreatic operation is one of the most complex surgical procedures, and it places a heavy burden on individuals and public health systems. The latest statistics reveal that an estimated 40.5% of patients undergoing hepatopancreatic operation have experienced at least 1 postoperative complication, and about 1.3% of these patients died within 30 days (1). Given the high burden associated with high-risk operations, research efforts have shifted toward the identification and management of patients at the highest risk of morbidity and mortality, but have overlooked the hidden risk among patients labeled “low risk” by multiple prognostic tools (2,3). Whether such hidden risks merit special consideration or can be overlooked is currently an open question and the subject of ongoing debate.

Sahara et al. (4) took the first step in seeking to answer this question by characterizing the incidence of death among patients who underwent hepatopancreatic operation and were deemed to have a low estimated morbidity and mortality risk, and they further developed a classification-tree model to predict “unpredicted death” within 30 days. Relative to the traditional National Surgical Quality Improvement Program Estimated Probability (NSQIP EP) model (5), this classification-tree model showed decent prediction performance as reflected by the area under the curve (AUC), a summarized accuracy index. Given the limited number of deaths, the authors trained the model using 10-fold cross validation to predict 30-day mortality (4). For practical reasons, the application of this prognostic model in surgery units has far-reaching implications for the timely identification of seemingly safe patients who might benefit from critical care after undergoing a high-risk hepatopancreatic operation mainly due to their poor short-term survival outcomes. We applaud the authors’ efforts to extend their previous work; however, we wish to comment on the data interpretation and extrapolation, as both merit further exploration and special discussion.

It is widely recognized that the occurrence of postoperative mortality following high-risk operations is a multistep, multifactorial process that involves patient demographics and laboratory values, and performance measures and interventions during the preoperative, perioperative, and postoperative periods. With such a large number of potential factors, it can be a challenging task for traditional statistical regression models that address possible nonlinear, collinear, and interactive relationships when predicting death outcomes (6,7). For example, the high degree of collinearity among laboratory values can lead to unstable estimates of prediction. In this context, determining the thrifty panel of contributing factors and characterizing their internal actions represents a practical strategy for ensuring informed consent and shared decision making among patients and surgeons.

To achieve this goal, a vast amount of resources and exhaustive endeavors have been devoted to developing predictive or prognostic models, but the majority of these models hinge on traditional statistical methods, which assume the relative independence of examined factors that is often not tenable in real scenarios (8). A practical way to overcome this challenge is the adoption of more advanced big data-based machine-learning techniques, which was the approach adopted by Sahara et al. (4). To clearly delineate the relationship between preoperative characteristics and 30-day mortality rate, Sahara et al. (4) employed the classification-tree algorithm to identify and optimize potential contributing factors according to their relative importance. The presentation of the classification-tree model is easy to interpret and objective, but it is not without its limitations. In theory, the reproducibility of the classification-tree model is highly sensitive, as a very small change in the data can lead to a large change in the tree structure. Given that a single classification tree is often a weak learner, a number of single trees, known as a random forest, is recommended for better prediction.

To date, dozens of machine-learning algorithms have been proposed for both continuous and categorical outcomes, such as the support vector machine, K-nearest neighbor, and gradient-boosting machine. Given the relative strengths and weaknesses of each machine-learning algorithm, consideration must be given to which one is preferable. For the sake of accuracy and to enhance model performance, one situation is to feed all possible factors to available machine-learning algorithms and ensemble them as a hard or soft voting classifier (9). Doing so would increase the vitality of the prognostic model for 30-day mortality estimation and highlight the necessity of reassessing mortality risks among patients deemed low risk after undergoing a high-risk hepatopancreatic operation.

Further, the prediction performance of the machine-learning model developed by Sahara et al. (4) was appraised based on the AUC alone. In the medical literature, the receiver operating characteristic analysis is widely used to quantify the discrimination accuracy of a diagnostic test or prediction model. Sensitivity and specificity constitute the basic metrics of accuracy; however, they are seriously limited by comparing the accuracies of competing tests (10). To comprehensively evaluate prediction performance, besides the AUC, 4 other metrics are recommended; that is, accuracy, precision, recall, and F1 score (11). By definition, accuracy refers to the rate of correct prediction, precision measures the ability to target actual positive observations, and recall reflects the capability to predict actual positivity correctly. The F1 score is calculated as the harmonic mean between precision and recall, and it takes both false positives and false negatives into account. Thus, a comprehensive assessment of prediction performance from various aspects should be encouraged to facilitate external validations of the machine-learning model.

In closing, the prognostic model based on classification-tree algorithms by Sahara et al. (4) is undoubtedly an important contribution to survival improvement among patients deemed low-risk following high-risk operations in the surgical field, and it is worthy of public attention.

Acknowledgments

Funding: None.

Footnote

Provenance and Peer Review: This article was commissioned by the editorial office, Hepatobiliary Surgery and Nutrition. The article did not undergo external peer review.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://hbsn.amegroups.com/article/view/10.21037/hbsn-2022-08/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Merath K, Chen Q, Bagante F, et al. Synergistic Effects of Perioperative Complications on 30-Day Mortality Following Hepatopancreatic Surgery. J Gastrointest Surg 2018;22:1715-23. [Crossref] [PubMed]
2. Zhao X, Ma Y, Dong X, et al. Molecular characterization of circulating tumor cells in pancreatic ductal adenocarcinoma: potential diagnostic and prognostic significance in clinical practice. Hepatobiliary Surg Nutr 2021;10:796-810. [Crossref] [PubMed]
3. Moris D, Shaw BI, Ong C, et al. A simple scoring system to estimate perioperative mortality following liver resection for primary liver malignancy-the Hepatectomy Risk Score (HeRS). Hepatobiliary Surg Nutr 2021;10:315-24. [Crossref] [PubMed]
4. Sahara K, Paredes AZ, Tsilimigras DI, et al. Machine learning predicts unpredicted deaths with high accuracy following hepatopancreatic surgery. Hepatobiliary Surg Nutr 2021;10:20-30. [Crossref] [PubMed]
5. Bilimoria KY, Liu Y, Paruch JL, et al. Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg 2013;217:833-42.e1-3.
6. Zhao Y, Naumova EN, Bobb JF, et al. Joint Associations of Multiple Dietary Components With Cardiovascular Disease Risk: A Machine-Learning Approach. Am J Epidemiol 2021;190:1353-65. [Crossref] [PubMed]
7. Hu W, Yang H, Xu H, et al. Radiomics based on artificial intelligence in liver diseases: where we are? Gastroenterol Rep (Oxf) 2020;8:90-7. [Crossref] [PubMed]
8. Spreafico C. A review about methods for supporting failure risks analysis in eco-assessment. Environ Monit Assess 2021;193:439. [Crossref] [PubMed]
9. Peppes N, Daskalakis E, Alexakis T, et al. Performance of Machine Learning-Based Multi-Model Voting Ensemble Methods for Network Threat Detection in Agriculture 4.0. Sensors (Basel) 2021;21:7475. [Crossref] [PubMed]
10. Obuchowski NA, Bullen JA. Receiver operating characteristic (ROC) curves: review of methods with applications in diagnostic medicine. Phys Med Biol 2018;63:07TR01. [Crossref] [PubMed]
11. Jin J, Wang H, Peng F, et al. Prognostic significance of preoperative Naples prognostic score on short- and long-term outcomes after pancreatoduodenectomy for ampullary carcinoma. Hepatobiliary Surg Nutr 2021;10:825-38. [Crossref] [PubMed]