Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Dr. Answer AI for prostate cancer: Clinical outcome prediction model and service

  • Mi Jung Rho,

    Roles Investigation, Methodology, Writing – original draft, Writing – review & editing

    Affiliation Catholic Cancer Research Institute, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea

  • Jihwan Park,

    Roles Data curation, Methodology, Project administration, Validation, Writing – original draft, Writing – review & editing

    Affiliations Department of Biomedicine & Health Sciences, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea, Department of Urology, Seoul St. Mary’s Hospital, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea

  • Hyong Woo Moon,

    Roles Conceptualization, Validation, Writing – review & editing

    Affiliation Department of Urology, Seoul St. Mary’s Hospital, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea

  • Chanjung Lee,

    Roles Data curation, Formal analysis, Methodology, Software, Visualization

    Affiliation LifeSemantics, Seoul, Republic of Korea

  • Sejin Nam,

    Roles Software, Visualization

    Affiliation LifeSemantics, Seoul, Republic of Korea

  • Dongbum Kim,

    Roles Software, Visualization

    Affiliation LifeSemantics, Seoul, Republic of Korea

  • Choung-Soo Kim,

    Roles Resources, Validation

    Affiliation Department of Urology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Republic of Korea

  • Seong Soo Jeon,

    Roles Resources, Validation

    Affiliation Department of Urology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea

  • Minyong Kang,

    Roles Resources, Validation

    Affiliations Department of Urology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea, Department of Health Sciences and Technology, SAIHST, Sungkyunkwan University, Seoul, Republic of Korea

  • Ji Youl Lee

    Roles Conceptualization, Funding acquisition, Supervision, Validation, Writing – review & editing

    uroljy@catholic.ac.kr

    Affiliation Department of Urology, Seoul St. Mary’s Hospital, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea

Dr. Answer AI for prostate cancer: Clinical outcome prediction model and service

  • Mi Jung Rho, 
  • Jihwan Park, 
  • Hyong Woo Moon, 
  • Chanjung Lee, 
  • Sejin Nam, 
  • Dongbum Kim, 
  • Choung-Soo Kim, 
  • Seong Soo Jeon, 
  • Minyong Kang, 
  • Ji Youl Lee
PLOS
x

Abstract

Objectives

The importance of clinical outcome prediction models using artificial intelligence (AI) is being emphasized owing to the increasing necessity of developing a clinical decision support system (CDSS) employing AI. Therefore, in this study, we proposed a “Dr. Answer” AI software based on the clinical outcome prediction model for prostate cancer treated with radical prostatectomy.

Methods

The Dr. Answer AI was developed based on a clinical outcome prediction model, with a user-friendly interface. We used 7,128 clinical data of prostate cancer treated with radical prostatectomy from three hospitals. An outcome prediction model was developed to calculate the probability of occurrence of 1) tumor, node, and metastasis (TNM) staging, 2) extracapsular extension, 3) seminal vesicle invasion, and 4) lymph node metastasis. Random forest and k-nearest neighbors algorithms were used, and the proposed system was compared with previous algorithms.

Results

Random forest exhibited good performance for TNM staging (recall value: 76.98%), while k-nearest neighbors exhibited good performance for extracapsular extension, seminal vesicle invasion, and lymph node metastasis (80.24%, 98.67%, and 95.45%, respectively). The Dr. Answer AI software consisted of three primary service structures: 1) patient information, 2) clinical outcome prediction, and outcomes according to the National Comprehensive Cancer Network guideline.

Conclusion

The proposed clinical outcome prediction model could function as an effective CDSS, supporting the decisions of the physicians, while enabling the patients to understand their treatment outcomes. The Dr. Answer AI software for prostate cancer helps the doctors to explain the treatment outcomes to the patients, allowing the patients to be more confident about their treatment plans.

Introduction

Currently, it has become easier to collect and use electronic medical record (EMR) data from multiple hospitals because of the increased availability of multi-center clinical data provided by hospitals. In addition, owing to the growing necessity for developing a clinical decision support system (CDSS) that employs artificial intelligence (AI), the importance of predictive models using AI has been emphasized. Therefore, several researchers have attempted to develop predictive models using AI in their studies on prostate cancer (PCa) [13].

In South Korea, a large-scale government-supported project employing large-scale multi-organization data has also been initiated in 2018 by the National IT Industry Promotion Agency (NIPA) [4]. The “Korea Data and Software-driven Hospitals (K-Dash)” is a collaborative consortium of 26 hospitals and 22 companies that are developing “Dr. Answer”, which is an AI solution for eight diseases such as cardio-cerebral vascular disease, heart disease, breast cancer, colon cancer, PCa, dementia, epilepsy, and childhood incurable genetic diseases. The “PROstate Medical Intelligence System Enterprise-Clinical, Imaging, and Pathology (PROMISE CLIP)” is one of the NIPA-supported projects for PCa undertaken by K-Dash. This project has been addressing the medical requirements of PCa since April 1, 2018.

PCa is difficult to diagnose and requires complicated treatments depending on the condition of a patient. Additionally, it is difficult to provide detailed explanations of the prognosis to the patients. Thus, an AI-based technique for predicting the outcome of PCa is suitable as a CDSS for both physicians as well as PCa patients. The PROMISE CLIP project has developed four prediction models as follows: (1) Prediction of treatment outcomes after a definitive surgery, which helps to define an ideal patient population for aggressive follow-up or early postoperative ancillary treatment. (2) Prediction of pathologic outcomes to help patients and clinicians to choose an optimal treatment option. (3) Accurate interpretation of multiparametric magnetic resonance imaging (MRI). (4) Precise digital pathology of PCa to improve accuracy, reduce human error, and increase reproducibility [5].

In this study, we attempted to develop an outcome prediction model using the PCa clinical data of patients treated with radical prostatectomy. In addition, we developed a CDSS according to the clinical outcome prediction model for PCa.

Methods

Target user

The Dr. Answer AI software (SW) for prostate cancer is targeted towards patients diagnosed with PCa before undergoing a radical prostatectomy. The patients could be provided with a tumor, node, and metastasis (TMN) staging service and be informed of the PCa-related risk factors before the radical prostatectomy. Thereafter, they could determine the direction of treatment and also predict the outcome of the radical prostatectomy.

Clinical outcome variables of clinical outcome prediction model

In this study, the clinical outcomes indicated the probability of the occurrence of 1) TNM classification, 2) extracapsular extension (ECE), 3) SVI, and 4) lymph node metastasis. They were determined with the assistance of three urologists based on the importance of clinical outcomes.

The PCa is staged using the TNM classification of cancer staging. It separately assesses the tumor, lymph nodes, and secondary cancer (metastases: M). In this study, we used the TNM classification as the first prediction outcome.

ECE refers to the growth or spread of tumor cells outside the lymph node capsule [6]. Models predicting extracapsular extension in PCa that are based on nomograms have been proposed earlier [7,8]. The occurrence of ECE in regional lymph nodes is one of the high-risk adverse pathological features in patients suffering from oropharyngeal squamous cell carcinoma [8]. In this study, ECE was used as the second prediction outcome.

SVI refers to the presence of PCa in the areolar connective tissue, which are present around the SVs and outside the prostate [9]. Previous studies have provided definitions of SVI: 1) PCa penetrating the muscular wall of the SV, 2) PCa must involve the extraprostatic portion of the SV, 3) cancer in the prostate and SV is usually directly contiguous and 4) a minority of tumors have noncontiguous metastases to the SV [10]. We have used SVI as the third prediction outcome.

Finally, metastatic lymph nodes are those lymph nodes that contain cancer cells, which can enable the cancer to spread from the primary tumor to other locations. It has been reported that the presence of lymph node metastases is an important independent prognostic factor for predicting deaths due to cancer. Moreover, previous studies have discovered that the lymph node metastases are associated with tumor recurrence and cancer-specific mortality [11]. A majority of cancer-specific deaths occur as a result of metastasis [12]. Thus, the metastatic lymph nodes are important in our prediction model.

Algorithm based on random forest and k-nearest neighbors

In this work, algorithms, such as random forest (RF) and k-nearest neighbors (KNN), and logistic regression were applied to develop the Dr. Answer AI SW for PCa. Weka 3.8.3 and Python 3.7 were used as the machine learning programs. 12 clinical variables were used as input data (as listed in Table 1), including the age at diagnosis, family history, and Gleason score.

Since its introduction in 2001, RF has been a popular technique that involves the aggregation of several decision trees, and has provided a reduction in the variance as compared to that using single decision trees [13]. The overall prediction obtained using RF is an average of the predictions from individual trees. Moreover, RF predicts the test data using an unweighted average over the collection by aggregating the built trees. RF has generalizability and robustness in overall performance and does not result in overfitting problems [14].

Meanwhile, KNN is a simple algorithm, first proposed in the early 1970s, that stores all the cases and classifies new cases based on a similarity measure. It is an effective model that is widely employed for data mining [15], and it functions effectively as a nonparametric model for classification and regression [16]. It is also commonly used in research on PCa [15,1719].

Here, the data were divided into two sets, i.e., the training dataset and the test dataset. The seven training datasets were matched to three test datasets, and the study achieved a prediction performance over 10-fold cross validation.

Ethics

The study was performed in accordance with the Declaration of Helsinki and was approved by the Institutional Review Board of Catholic University (IRB number: KC18SNDI0512), Samsung Medical Center (IRB number: SMC201807069001), Bundang Seoul University Hospital (IRB number: B1808486102), and Asan Medical Center (IRB number: 2018–0963). We waived the requirement for informed patient consent from IRB board.

Results

Input variables for clinical outcome prediction model

To develop the clinical outcome prediction model, 12 input variables related to PCa were used; the variables were 1) age at diagnosis, 2) BMI, 3) marital status, 4) education, 5) smoking, 6) alcohol, 7) family history of PCa, 8) initial prostate specific antigen (PSA) value, 9) Gleason grade group, 10) maximum positive core, total cores, 11) high-grade prostatic intraepithelial neoplasia(HGPIN), 12) core ratio(Number of positive cores / Total cores).

We collected 7,128 cases of anonymized PCa patients after the radical prostatectomy treatment: 1,723 were from hospital C, 2,751 from hospital S, and 2,654 from hospital A (Table 2). These three university hospitals are located in Seoul, South Korea. The Gleason mean ranges from 6.89–7.13, and the initial PSA mean ranges from 7.79–14.14. Most of the PCa patients were in the age group of 60–70 years.

There were missing values for each outcome variable; the number of missing values were: 1) TNM classification (T = 1,847, N = 1,898, M = 1,896), 2) ECE (n = 260), 3) SVI (n = 218), and 4) lymph node metastasis (n = 533) (Table 3). Finally, each algorithm was developed with the final cases that excluded the missing values for each variable, as follows: 1) TNM classification (T = 5,281, N = 5,230, M = 5,232), 2) ECE (n = 6,868), 3) SVI (n = 6,910), and 4) lymph node metastasis (n = 6,595).

The final analysis method was chosen by comparing each of the algorithms (Table 4). The RF exhibited a good performance in TNM staging (recall: 76.98%). Furthermore, KNN exhibited a good performance for ECE, SVI, and lymph node metastasis (80.24%, 98.67%, and 95.45%, respectively). RF and KNN also exhibited a good performance using the synthetic minority oversampling technique (SMOTE), over 10-fold cross-validation. The SMOTE is an over-sampling method and an algorithm used in the framework of learning from imbalanced data [20,21]. Since its introduction in 2002, it has proven to be successful in various domains. In this study, there were small samples of occurrence of 1) TNM classification, 2) ECE, 3) SVI, and 4) Lymph node metastasis, and thus, this data was imbalanced.

SW structure and function

The Dr. Answer AI SW for PCa was developed according to our clinical outcome prediction model. It consists of three primary service structures: patient information, clinical outcome prediction, and outcome according to the National Comprehensive Cancer Network (NCCN) guidelines (Fig 1).

The patient information service structure could check the name, age at diagnosis, and family history of the patient. The clinical outcome prediction service structure could check clinical outcome results through computer tomography (CT), bone scan, and MRI images. There was a probability of occurrence in TNM staging, ECE, SVI, and lymph node metastasis.

The clinical outcomes were combined, and the patients were divided into risk groups according to the NCCN guidelines. This service provides an initial guideline for therapies.

Service process

The service shown in section 1 of Fig 2 provides the basic patient information, consisting of four values: 1) age at the time of diagnosis, 2) BMI, 2) drinking status, and 4) smoking preference. The service shown in section 2 of the figure provides the EMR lab results, such as PSA, Atypical Small Acinar Proliferation(ASAP), PIN, TNM, Gleason score, biopsy score, CT, bone scan, and MRI images.

The service shown in the third section provides a trend graph of the lab results of PSA time series. Thereafter, this service provided the probability of occurrence of 1) TNM staging, 2) ECE, 3) SVI, and 4) lymph node metastasis according to our prediction algorithm. The probability of occurrence is provided using a color graph in section 4 of Fig 2. Finally, this service presented the risk group, expected survival of the patients, and initial therapy guidelines as per the NCCN guidelines.

The service received the patient data from the EMR and provided the results on a screen (Fig 3). In the case of CT, bone scan, and MRI images, the lab results could be found in detail using the conclusion button on the service page. The data recorded in this service were automatically recalled and saved during a product launch and shutdown. After using the service, the X button of the program was pressed to terminate the service.

Patients’ experience

Our SW is targeted towards patients that have been diagnosed with PCa and have not undergone radical prostatectomy. Before radical prostatectomy, the patient can be provided with the TMN staging service and be informed of the risk factors related to PCa. This can help in determining the course of treatment and predict the outcome of the radical prostatectomy. The results of TMN staging and PCa related risk factors were expressed using a proportion. Understanding these results could be beneficial to PCa patients; the Dr. Answer AI SW for PCa provided results to the patients using a website and printed reports.

Discussion

In this study, an outcome prediction model was developed using the multicenter clinical data of PCa with radical prostatectomy. The proposed SW, which is based on the clinical outcome prediction model for PCa treated with radical prostatectomy, is a part of the PROMISE CLIP-initiated Dr. Answer project [4], which began to address the medical demands of PCa. The proposed SW is targeted towards patients diagnosed with PCa, who have not undergone radical prostatectomy, as well as physicians. LifeSemantics Corp., which has an experience of developing a mHealth management platform for patients [2224], has contributed to the development of the Dr. Answer AI SW for PCa.

PCa is difficult to diagnose and it requires complicated treatments depending on the condition of the patient. Moreover, it is difficult to explain the prognosis to the patients as the physicians, including the Korean physicians, may not have sufficient time to explain the treatment outcomes in detail. It has been reported that the shortage of the physicians’ time is a negative factor that causes overcrowding in outpatient healthcare [25]. According to the results of a previous survey, the average treatment time per patient in a Korean general hospital is 6.2 min. However, this should be increased to at least 8.9 min to increase patient satisfaction [26]. Therefore, there is a need for a CDSS that can provide effective treatment for PCa and support the short treatment time. The proposed SW could address these issues.

In addition, it has been reported that the patients are more satisfied when the physicians employ a patient-centered approach during consultations [27], which could be enabled using the proposed technique for predicting PCa outcomes. This could improve the patient satisfaction levels during consultations and treatments.

The clinical outcome prediction model could also function as a CDSS, supporting the decisions of doctors and helping the patients in understanding their treatment outcome. Furthermore, the Dr. Answer AI SW for PCa also enabled the doctors to clearly explain the treatment outcomes to patients; thus, allowing patients to have confidence in their treatment plans. The results of our service, indicating the TMN staging and PCa-related risk factors, have been provided using proportions and graphics; in addition, the patients can obtain a printed report directly from the physician as well as access the reports on the website of the Dr. Answer AI SW for PCa. Thus, the patients can check the PCa-related risk factors and treatment outcomes at any time, which puts them at ease during future treatments. Furthermore, the patients can determine the course of treatment and predict the outcome of radical prostatectomy. According to previous studies, the PCa decision aids have been reported to increase knowledge, reduce conflicts, and increase the active role of a patient during the decision-making process [2830]. This is because the decision aids provide the user with detailed information and also help them to determine their preferred treatment [31]. While diverse CDSSs and PCa decision aids have been developed owing to the evolution of technology, the proposed clinical outcome prediction model could assist patients in the PCa treatment decision and compliance.

The proposed model is an integrated one based on AI, that employs the RF and KNN algorithms. In this study, we focused on the probability of occurrence of 1) TNM staging, 2) ECE, 3) SVI, and 4) lymph node metastasis. Each variable is important when predicting the PCa prognosis and treatments. According to a systematic review, ECE, SVI, and lymph node metastasis were statistically significant for BCR free survival [32]. Previous predictive models and related studies on each of the variables have not provided combined results [6,7,3335]. In addition, previous research provided oncologic outcomes such as survival rate and biochemical recurrence. Although survival rate and biochemical recurrence are important outcomes, there are limitations in explaining the treatment outcomes to the patients in detail. We developed an integrated clinical prognosis prediction model that can predict various treatment outcomes such as 1) TNM staging, 2) ECE, 3) SVI, and 4) lymph node metastasis. The predictive rate of each variable is also high, which is considered to be useful clinically. Thus, the Dr. Answer AI SW for prostate cancer helps the doctors to explain the treatment outcomes to patients, instilling confidence in patients regarding their treatment plans.

However, there are certain limitations in this study. The SMOTE model was used because of the imbalanced data and the limited outcome of the sample data. Future studies should focus on developing the model using balanced data such that sizeable outcome data can be collected. We plan to test the utility of the proposed system via user questionnaires for patients and physicians, and update our SW based on the results. Finally, we focused on probability of occurrence of 1) TNM staging, 2) ECE, 3) SVI, and 4) lymph node metastasis as a clinical outcome. Future studies should focus on oncologic outcomes including survival rate and biochemical recurrence.

Despite these limitations, the clinical outcome prediction model could effectively act as a CDSS and support the decisions of the physicians as well as support the patients to understand their treatment outcomes. The Dr. Answer AI SW for PCa enables the doctors to explain the treatment outcome to the patients, thereby instilling confidence in patients regarding their treatment plans.

References

  1. 1. Auffenberg GB, Ghani KR, Ramani S, Usoro E, Denton B, Rogers C, et al. askMUSIC: leveraging a clinical registry to develop a new machine learning model to inform patients of prostate cancer treatments chosen by similar men. 2019; 75:901–907. https://doi.org/10.1016/j.eururo.2018.09.050 pmid:30318331
  2. 2. Guinney J, Wang T, Laajala TD, Winner KK, Bare JC, Neto EC, et al. Prediction of overall survival for patients with metastatic castration-resistant prostate cancer: development of a prognostic model through a crowdsourced challenge with open clinical trial data. 2017; 18:132–142. https://doi.org/10.1016/S1470-2045(16)30560-5 pmid:27864015
  3. 3. Yadav KK. How AI Is Optimizing the Detection and Management of Prostate Cancer. JIp 2018; 9:19–19. https://doi.org/10.1109/MPUL.2018.2866354
  4. 4. NIPA. Doctor Answer National IT Industry Promotion Agency 2018 [cited]. Available from: http://dranswer.kr
  5. 5. Park J, Rho MJ, Park YH, Jung CK, Chong Y, Kim C-S, et al. PROMISE CLIP Project: A Retrospective, Multicenter Study for Prostate Cancer that Integrates Clinical, Imaging and Pathology Data. 2019; 9:2982. https://doi.org/10.3390/app9152982
  6. 6. Gooch J, King TA, Eaton A, Dengel L, Stempel M, Corben AD, et al. The extent of extracapsular extension may influence the need for axillary lymph node dissection in patients with T1–T2 breast cancer. 2014; 21:2897–2903. https://doi.org/10.1245/s10434-014-3752-0 pmid:24777858
  7. 7. Ohori M, Kattan MW, Koh H, Maru N, Slawin KM, Shariat S, et al. Predicting the presence and side of extracapsular extension: a nomogram for staging prostate cancer. 2004; 171:1844–1849. https://doi.org/10.1097/01.ju.0000121693.05077.3d
  8. 8. Hararah MK, Stokes WA, Jones BL, Oweida A, Ding D, McDermott J, et al. Nomogram for preoperative prediction of nodal extracapsular extension or positive surgical margins in oropharyngeal squamous cell carcinoma. 2018; 83:73–80. https://doi.org/10.1016/j.oraloncology.2018.06.005 pmid:30098782
  9. 9. Jewett H, Eggleston J, Yawn DJ. Radical prostatectomy in the management of carcinoma of the prostate: probable causes of some therapeutic failures. TJou 1972; 107:1034–1040. https://doi.org/10.1016/S0022-5347(17)61201-1
  10. 10. Potter SR, Epstein JI, Partin AW. Seminal vesicle invasion by prostate cancer: prognostic significance and therapeutic implications. JRiu 2000; 2:190. pmid:16985773
  11. 11. Jameson JL, De Groot LJ. Endocrinology: Adult and Pediatric: Elsevier Health Sciences; 2010.
  12. 12. Brower V. Researchers tackle metastasis, cancer’s last frontier. Journal of the National Cancer Institute. 2007; 99:109–111. pmid:17227994
  13. 13. Couronné R, Probst P, Boulesteix A-L. Random forest versus logistic regression: a large-scale benchmark experiment. JBb 2018; 19:270. https://doi.org/10.1186/s12859-018-2264-5 pmid:30016950
  14. 14. Yang X, Wu W, Yan B, Wang H, Zhou K, Liu KJ. Infrared image super-resolution with parallel random Forest. IJoPP 2018; 46:838–858. https://doi.org/10.1007/s10766-017-0551-9
  15. 15. Bouazza SH, Hamdi N, Zeroual A, Auhmani K. Gene-expression-based cancer classification through feature selection with KNN and SVM classifiers. 2015 Intelligent Systems and Computer Vision (ISCV): IEEE; 2015. pp. 1–6.
  16. 16. Maillo J, Ramírez S, Triguero I, Herrera FJ. kNN-IS: An Iterative Spark-based design of the k-Nearest Neighbors classifier for big data. K-BS 2017; 117:3–15. https://doi.org/10.1016/j.knosys.2016.06.012
  17. 17. Jensen C, Carl J, Boesen L, Langkilde NC, Østergaard LR. Assessment of prostate cancer prognostic Gleason grade group using zonal-specific features extracted from biparametric MRI using a KNN classifier. JJoacmp 2019; 20:146–153. https://doi.org/10.1002/acm2.12542 pmid:30712281
  18. 18. Boutilier JJ, Lee T, Craig T, Sharpe MB, Chan TC. Models for predicting objective function weights in prostate cancer IMRT. JMp 2015; 42:1586–1595. https://doi.org/10.1118/1.4914140
  19. 19. Acar E, Leblebici A, Ellidokuz BE, Başbınar Y, Kaya GÇ. Machine learning for differentiating metastatic and completely responded sclerotic bone lesion in prostate cancer: A retrospective radiomics study. JTBJoR 2019; 92:20190286. https://doi.org/10.1259/bjr.20190286
  20. 20. Fernández A, Garcia S, Herrera F, Chawla NV. SMOTE for learning from imbalanced data: progress and challenges, marking the 15-year anniversary. JJoair 2018; 61:863–905. https://doi.org/10.1613/jair.1.11192
  21. 21. Han H, Wang W-Y, Mao B-H. Borderline-SMOTE: a new over-sampling method in imbalanced data sets learning. International conference on intelligent computing: Springer; 2005. pp. 878–887.
  22. 22. Rho MJ, PARK J, NA E, JEONG J-E, KIM JK, Kim D-J, et al. Types of Problematic Smartphone Use based on Psychiatric Symptoms. 2019:12287.
  23. 23. Soh JY, Cha WC, Chang DK, Hwang JH, Kim K, Rha M, et al. Development and validation of a multidisciplinary mobile care system for patients with advanced gastrointestinal cancer: interventional observation study. 2018; 6:e115. https://doi.org/10.2196/mhealth.9363 pmid:29735478
  24. 24. Kwon H, Lee S, Jung EJ, Kim S, Lee J-K, Kim DK, et al. An mHealth management platform for patients with chronic obstructive pulmonary disease (efil breath): randomized controlled trial. 2018; 6:e10502. https://doi.org/10.2196/10502 pmid:30143475
  25. 25. Bahadori M, Teymourzadeh E, Ravangard R, Raadabadi M. Factors affecting the overcrowding in outpatient healthcare. JJoe 2017; 6 https://doi.org/10.4103/2277-9531.204742 pmid:28546986
  26. 26. Lee CH, Lim H, Kim Y, Yoon S, Park YS, Kim S, et al. Analysis of New Patient’s Willingness to Pay Additional Costs for Securing Satisfactory Consultation Time. 2017; 27:39. https://doi.org/10.4332/KJHPA.2017.27.1.39
  27. 27. Marvel MK, Epstein RM, Flowers K, Beckman HB. Soliciting the patient’s agenda: have we improved? JJ 1999; 281:283–287. https://doi.org/10.1001/jama.281.3.283
  28. 28. Volk RJ, Hawley ST, Kneuper S, Holden EW, Stroud LA, Cooper CP, et al. Trials of decision aids for prostate cancer screening: a systematic review. 2007; 33:428–434.e411. https://doi.org/10.1016/j.amepre.2007.07.030
  29. 29. Kassan EC, Williams RM, Kelly SP, Barry SA, Penek S, Fishman MB, et al. Men’s use of an Internet-based decision aid for prostate cancer screening. 2012; 17:677–697. https://doi.org/10.1080/10810730.2011.579688
  30. 30. Taylor KL, Davis KM, Lamond T, Williams RM, Schwartz MD, Lawrence W, et al. Use and evaluation of a CD-ROM-based decision aid for prostate cancer treatment decisions. 2010; 36:130–140. https://doi.org/10.1080/08964289.2010.525263
  31. 31. Clarke G, Eubanks D, Reid E, Kelleher C, O’connor E, DeBar LL, et al. Overcoming Depression on the Internet (ODIN)(2): a randomized trial of a self-help depression skills program with reminders. 2005; 7:e16. https://doi.org/10.2196/jmir.7.2.e16 pmid:15998607
  32. 32. Liu H, Zhou H, Yan L, Ye T, Lu H, Sun X, et al. Prognostic significance of six clinicopathological features for biochemical recurrence after radical prostatectomy: a systematic review and meta-analysis. 2018; 9:32238.
  33. 33. Datta K, Muders M, Zhang H, Tindall DJ. Mechanism of lymph node metastasis in prostate cancer. JFo 2010; 6:823–836. https://doi.org/10.2217/fon.10.33
  34. 34. Wang L, Hricak H, Kattan MW, Chen HN, Kuroiwa K, Eisenberg HF, et al. Prediction of seminal vesicle invasion in prostate cancer: incremental value of adding endorectal MR imaging to the Kattan nomogram. 2007; 242:182–188. https://doi.org/10.1148/radiol.2421051254
  35. 35. Kim JK, Yook IH, Choi MJ, Lee JS, Park YH, Lee JY, et al. A Performance Comparison on the Machine Learning Classifiers in Predictive Pathology Staging of Prostate Cancer. 2017; 245:1273–1273.