Machine learning model better identifies patients for pharmacist intervention to reduce hospitalization risk in a large outpatient population

Introduction

Background

Healthcare costs in the United States have been rising in recent years, with the coronavirus disease 2019 (COVID-19) pandemic exacerbating the problem (1,2). This is stressing individuals in need of care as well as the systems and personnel providing the care (3). Preventable hospitalizations and adverse drug reactions (ADRs) are a significant source of healthcare expenditures in the United States and internationally (4,5). ADRs can be defined broadly as undesirable effects of pharmacotherapy. Examples of ADRs include allergic reactions or toxicity experienced when taking one or more drugs (6). These reactions can result in a patient being hospitalized (7). Pharmacists are well positioned to address the problem of ADRs and the associated risk of hospitalization (8).

Pharmacists are healthcare professionals with extensive training in pharmacotherapy. Pharmacists are also among the most accessible healthcare professionals (9). Pharmacists have demonstrated the ability to practice in various capacities, including in telepharmacy settings. Telepharmacy has expanded since the COVID-19 pandemic, and patients with access to telepharmacy services have lower hospitalization rates compared to those who do not (10,11). Various patient factors can impact the risk of hospitalization (12). Some factors, such as healthcare literacy, may be addressed by the pharmacist (13). While access has increased, efficiently identifying patients most likely to benefit from a pharmacist’s evaluation and consultation remains difficult. Patients at a high risk of hospitalization are often prioritized for healthcare interventions. Efficiently utilizing pharmacists can save on healthcare costs directly by optimizing workforce deployment and indirectly by preventing ADRs and associated hospitalizations (11). Advances in artificial intelligence (AI) and machine learning (ML) algorithms have shown promise in supporting clinical decision-making and patient management, which may also be applied to optimize the selection of patients for pharmacist consultation and intervention (14-16).

Rationale and knowledge gap

Advanced AI and ML algorithms allow for the analysis and identification of patient characteristics associated with an outcome of interest in large datasets (17). Predictive models that are generated can be trained and optimized on these datasets and then validated on independent populations. These validated models can then be used to predict the outcomes of interest in new populations and are implemented for patient population management (18). These tools may be useful in increasing the efficiency of care delivery in various settings (19).

RxLive^® (St. Petersburg, FL, USA) is a novel, value-based pharmacy system that allows remote pharmacotherapy delivery by clinical pharmacists directly to patients. This platform is utilized by provider organizations to better manage and deliver pharmacotherapy to their patients. RxLive connects providers, patients, and pharmacists to improve access and quality of pharmaceutical care while aggregating key patient data points supporting predictive model generation. Previously, access to RxLive services was associated with reduced hospitalizations in outpatient populations (11). The ability to gather and monitor patient metrics, such as medication adherence, may have contributed to the reduced hospitalization rate observed (20). The number of prescribed medications is often the key metric used when determining patient eligibility for pharmacist consultation or medication therapy management as polypharmacy is associated with poor clinical outcomes and elevated drug costs (21-23). While polypharmacy is a relevant metric to consider when deciding if a patient may benefit from pharmacist intervention, other metrics may allow for more precise patient selection and predicted risk of experiencing adverse outcomes such as hospitalization.

Applying an ML approach to stratify hospitalization risk and assist in selecting those high-risk patients for pharmacist consultation has never been attempted. This approach may have a role in optimizing pharmacist utilization in outpatient populations. This is increasingly relevant as telepharmacy becomes a more widely implemented pharmacotherapy delivery platform. As ML and other AI modalities become available, this approach may be leveraged to better serve patients, improve outcomes, and utilize healthcare resources.

Objective

The current study applied ML, a subset of AI, to identify patient metrics associated with hospitalization in a large outpatient population. An ML model was generated utilizing these metrics to predict hospitalization rates in outpatient populations. This ML model was then deployed to predict hospitalization rates in a deidentified patient population. The hospitalization rate observed using this predictive ML model was compared to the rate observed in a group selected only using polypharmacy. This represents the first use of AI and ML for patient selection for pharmacist consultation. This work has wide-spanning implications for incorporating ML predictive algorithms into multiple aspects of patient management to improve outcomes and save resources. We present this article in accordance with the TRIPOD reporting checklist (available at https://jmai.amegroups.com/article/view/10.21037/jmai-24-183/rc).

Methods

Cohort selection for model development

The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Institutional Review Board (No. IORG0010260, application No. EXM454435) and individual consent for this retrospective analysis was waived. Two large, outpatient medical groups located in New York and Florida were included in this study. Deidentified patient data were collected on a rolling basis from these sites between December 1, 2021 and August 10, 2022, initially to form a training dataset and subsequently to validate the ML model. Data from 9,797 patients were utilized for training and validation of the ML model. The demographics of these patients are shown in Table 1. Race and ethnicity data were not uniformly collected in the patient population and not included in the current model. All patients were insured via a Medicare Advantage plan. The patient population was primarily an urban population with some suburban and rural participants as well. The top 25 most common patient diagnoses in the study population can be found in Table 2. A complete list of diagnoses associated with the patients in this population can be found in table available at https://cdn.amegroups.cn/static/public/jmai-24-183-1.xlsx.

ML model development

After curating the data obtained from RxLive and the Bureau of Labor Statistics (BLS), several models (logical regression, Random Forest classification, and XGBoost Classification) were trained using the Amazon Web Services Sagemaker tool (Seattle, WA, USA). Python (Wilmington, DE, USA) version 3.10 was used for all analyses and model development. The “leave time out” method was used for dividing the dataset into training and validation. The validation step is necessary to accurately evaluate the model’s performance because when making a prediction, the model is not exposed to data points from the future time period. The training dataset (from December 2021 to May 2022) had predictive variables incorporated. A complete list of predictive variables can be found on Table 3. The validation dataset (June 2022 to August 2022) was used to evaluate the model performance and to avoid overfitting.

Data preparation and study design

The training dataset was constructed from specific patient metrics collected on a 90-day sliding window based on when the patient entered the study. Examples of patient metrics included previous hospital admissions, drug costs, and the number of active prescribed and over-the-counter medications. These metrics were collected before December 2021 and then used to develop a predictive ML model to stratify patients based on hospitalization risk. The 2010 census population and income data were obtained from the BLS data to capture additional socioeconomic factors using the patients’ zip code (24). The training dataset was curated in a way to train a binary classification model and hence the outcome variable was chosen to be 1 if a patient is hospitalized in the next 90 days and zero if not. All categorical variables were one-hot encoded (converted to 0 and 1 type of features) and continuous variables were used wherever possible. The entire dataset was constructed and used to “train” the predictive ML model from December 2021 through May 2022. Model validation was completed between June and August 2022. The true positive rate for admission within the next 90 days was compared to the false positive rate using area under the curve (AUC) validation. A hospitalization was defined as any unplanned inpatient admission. Admissions to long-term care and hospice were not included in the study.

The ML model was then deployed in the large outpatient study population. Patients were organized into two groups. Each group consisted of the top 15% of patients considered “high risk” for hospitalization based on either polypharmacy or having the highest hospitalization risk as determined by the ML model using multiple patient metrics. Polypharmacy is defined as patients with 8 or more active medications as reported by SureScripts^® (Arlington, VA, USA). Ninety-day hospitalizations were recorded for patients selected by the validated ML model and polypharmacy groups for comparison.

Statistical analysis

Model performance was assessed using the metric AUC of the receiver operating characteristic (ROC) curve. The ROC curve is the true positive rate and false positive rate of the model across all the cutoffs possible for the model. The AUC of the training dataset and validation dataset were monitored simultaneously to assess model performance and avoid overfitting. An AUC value of above 0.8 for the validation dataset was determined to have adequate predictive power for deployment. Statistical analysis was performed within the Python Scipy package (version 1.10.0). Two-tailed Student t-tests were utilized for analysis. Statistical significance was defined as a P value <0.05.

Results

ML model validation

The XGBoost model had the best performance regarding the AUC of the training and validation dataset as shown in Figure 1. An AUC value of 0.845 from the training dataset and of 0.815 from the validation dataset indicates that the model is not overfitted. It also indicates a high predictive power of differentiation of hospitalization and non-hospitalization from the model inputs. Previous hospitalization, number of active medications, anticoagulant use, and inferred income based on zip code were among the metrics most predictive of a patient experiencing a hospitalization within 90 days.

Figure 1 ML model ROC training and validation curves. The blue and orange ROC curves shown represent true positive rates and false positive rates for various probability cutoffs in the training and validation data sets. The curves for training and validation are similar, indicating the model is neither overfitted nor underfitted. The validation dataset AUC is more than 0.8, indicating confidence that the ML model has predictive power. AUC, area under the curve; ML, machine learning; ROC, receiver operating characteristic.

Impact of utilizing an ML model versus polypharmacy alone to predict hospitalization rates

Figure 2 shows the comparison of 90-day hospitalizations per patient in the top 15% of patients utilizing polypharmacy alone versus an ML model approach. The mean 90-day hospitalization rate for the top 15% of patients (n=1,469) selected based only on polypharmacy was 0.120 (SD =0.013) hospitalization per patient. The 90-day hospitalization rate of the top 15% (n=1,469) of patients as selected by the ML model was 0.285 (SD =0.027) hospitalizations per patient (two tailed t-test, P<0.001). The incorporation of the predictive ML model was validated during a 3-month collection period. The model demonstrated an 80% accuracy rate (AUC >0.8) (Figure 1). The ML model predicted high risk patients and improved the true positive hospitalization rate by a factor of 3.5 while reducing false positive rate 48%. Some patients selected by the ML model were also present in the polypharmacy group as both groups were selected from the same dataset. Table 4 contains the true and false positives associated with this model.

Figure 2 Comparison of predicted hospitalizations per patient using polypharmacy alone vs. ML model approach. *, P<0.05. The ML model demonstrated superior ability to predict hospitalizations in the patient population compared to polypharmacy alone. ML, machine learning.

Discussion

Key findings

The ML model outperformed using polypharmacy alone to predict hospitalization risk in our study population. Using this novel ML model allowed for more robust prediction of hospitalization rate in the study population compared to using polypharmacy alone. This represents the first study that examines the impact of utilizing an ML model to better predict hospitalization risk in a large outpatient population. This information can be used in patient selection processes for pharmacist intervention. At the ML model’s inception, there was concern that the impact of the model would be region-specific. Utilizing this model in a large population with diverse payor coverage and patient demographics, this represents a starting point that can continue to be iterated on for increased generalizability in even larger populations.

Explanation of findings

The use of AI and ML modalities has been reported in the pharmacy space but remains mostly unexplored (25). Improving patient selection with AI/ML approaches has been reported in different healthcare applications (15,26). Due to the average hospitalization cost in the United States, combined with the accessibility of pharmacists, selecting patients for pharmacist intervention has the potential to save healthcare dollars while improving pharmacist utilization efficiency. Traditional selection criteria for intervention, such as medication number, does not capture the entire complexity of a patient’s clinical disposition. ML models allow multiple patient characteristics to be incorporated into making clinical decisions and allow for more sophisticated and complex selection processes to be utilized. This can allow for better targeting for pharmacist intervention which can decrease risk for hospitalization. Better prediction of adverse healthcare outcomes allows for optimized deployment of healthcare resources to prevent the predicted poor outcome. This ML model is a first step toward implementing emerging technological approaches to improve patient care, lower healthcare costs, and conserve healthcare professional bandwidth in an increasingly stressed environment. An ML approach and associated precision care may also lower barriers to healthcare access for some at-risk populations by increasing the operating efficiency of existing healthcare infrastructure.

Strengths and weaknesses

While the current study represents a compelling initial observation of the potential for ML models being utilized in the pharmacy space, some aspects may be impacting the results. Patients in the study population may have a clinical disposition outside of the metrics considered that impacted the reported hospitalization rate. While the ML model was utilized in a large population, this still represents a small fraction of the national population. Increasing and diversifying future study populations would further elucidate the robustness of the current model for selecting patients in the pharmacy space. Given the population observed, the ability of this ML model to be deployed in other patient populations, such as pediatric patients, needs to be further investigated. Utilizing updated census data would also benefit the model and ensure good performance in future iterations.

Conclusions

This study exhibits the capability of an ML model to better predict hospitalizations in a large outpatient population. Expanding this study population with improved patient demographic documentation to refine the model remains the primary future direction of this work. Incorporation of race and ethnic data will make the model more refined and generalizable. As healthcare continues to embrace a “big data” approach to increase efficiency while also improving outcomes, more data will become available to refine the current model used in this study. Further developments in the AI space will likely continue to drive this approach to resource allocation in healthcare. As pharmacists continue to be the most accessible healthcare professionals in the United States, efficiently deploying their expertise to improve outcomes will increase in importance. The current study represents the first look at how ML and its improved predictive capabilities can be leveraged to better focus pharmacy resources and potentially improve healthcare outcomes.

Acknowledgments

Funding: None.

Footnote

Reporting Checklist: The authors have completed the TRIPOD reporting checklist. Available at https://jmai.amegroups.com/article/view/10.21037/jmai-24-183/rc

Data Sharing Statement: Available at https://jmai.amegroups.com/article/view/10.21037/jmai-24-183/dss

Peer Review File: Available at https://jmai.amegroups.com/article/view/10.21037/jmai-24-183/prf

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (Available at https://jmai.amegroups.com/article/view/10.21037/jmai-24-183/coif). E.H. is employed by RxLive Inc. Y.X. is employed by Premier Strategy Consulting LLC. K.E. was a shareholder of RxLive Inc. while this research was performed. The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Institutional Review Board (No. IORG0010260, application No. EXM454435) and individual consent for this retrospective analysis was waived.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Martin AB, Hartman M, Lassman D, et al. National Health Care Spending In 2019: Steady Growth For The Fourth Consecutive Year. Health Aff (Millwood) 2021;40:14-24. [Crossref] [PubMed]
2. Di Fusco M, Shea KM, Lin J, et al. Health outcomes and economic burden of hospitalized COVID-19 patients in the United States. J Med Econ 2021;24:308-17. [Crossref] [PubMed]
3. Shrank WH, DeParle NA, Gottlieb S, et al. Health Costs And Financing: Challenges And Strategies For A New Administration. Health Aff (Millwood) 2021;40:235-42. [Crossref] [PubMed]
4. Formica D, Sultana J, Cutroneo PM, et al. The economic burden of preventable adverse drug reactions: a systematic review of observational studies. Expert Opin Drug Saf 2018;17:681-95. [Crossref] [PubMed]
5. Lim R, Ellett LMK, Semple S, et al. The Extent of Medication-Related Hospital Admissions in Australia: A Review from 1988 to 2021. Drug Saf 2022;45:249-57. [Crossref] [PubMed]
6. Zazzara MB, Palmer K, Vetrano DL, et al. Adverse drug reactions in older adults: a narrative review of the literature. Eur Geriatr Med 2021;12:463-73. [Crossref] [PubMed]
7. Coleman JJ, Pontefract SK. Adverse drug reactions. Clin Med (Lond) 2016;16:481-5. [Crossref] [PubMed]
8. Lee H, Ryu K, Sohn Y, et al. Impact on Patient Outcomes of Pharmacist Participation in Multidisciplinary Critical Care Teams: A Systematic Review and Meta-Analysis. Crit Care Med 2019;47:1243-50. [Crossref] [PubMed]
9. Valliant SN, Burbage SC, Pathak S, et al. Pharmacists as accessible health care providers: quantifying the opportunity. J Manag Care Spec Pharm 2022;28:85-90. [Crossref] [PubMed]
10. Pantasri T. Expanded roles of community pharmacists in COVID-19: A scoping literature review. J Am Pharm Assoc (2003) 2022;62:649-57. [Crossref] [PubMed]
11. Hefti E, Wei B, Engelen K. Access to Telepharmacy Services May Reduce Hospital Admissions in Outpatient Populations During the COVID-19 Pandemic. Telemed J E Health 2022;28:1324-31. [Crossref] [PubMed]
12. Bhowmik T, Eluru N. A comprehensive county level model to identify factors affecting hospital capacity and predict future hospital demand. Sci Rep 2021;11:23098. [Crossref] [PubMed]
13. Baker DW, Parker RM, Williams MV, et al. Health literacy and the risk of hospital admission. J Gen Intern Med 1998;13:791-8. [Crossref] [PubMed]
14. Hassan N, Slight R, Morgan G, et al. Road map for clinicians to develop and evaluate AI predictive models to inform clinical decision-making. BMJ Health Care Inform 2023;30:e100784. [Crossref] [PubMed]
15. Cui W, Finkelstein J. Using Machine Learning to Identify No-Show Telemedicine Encounters in a New York City Hospital. Stud Health Technol Inform 2022;295:328-31. [Crossref] [PubMed]
16. Salman OH, Taha Z, Alsabah MQ, et al. A review on utilizing machine learning technology in the fields of electronic emergency triage and patient priority systems in telemedicine: Coherent taxonomy, motivations, open research challenges and recommendations for intelligent future work. Comput Methods Programs Biomed 2021;209:106357. [Crossref] [PubMed]
17. Ngiam KY, Khor IW. Big data and machine learning algorithms for health-care delivery. Lancet Oncol 2019;20:e262-73. [Crossref] [PubMed]
18. Choudhury A, Asan O. Role of Artificial Intelligence in Patient Safety Outcomes: Systematic Literature Review. JMIR Med Inform 2020;8:e18599. [Crossref] [PubMed]
19. Ben-Israel D, Jacobs WB, Casha S, et al. The impact of machine learning on patient care: A systematic review. Artif Intell Med 2020;103:101785. [Crossref] [PubMed]
20. Hefti E, Xie Y, Engelen K. An Analysis of Medication Adherence in a Large Outpatient Population During the COVID-19 Pandemic Using a Novel Value-Based Pharmacy System. Telemed J E Health 2024;30:556-62. [Crossref] [PubMed]
21. Frazier SC. Health outcomes and polypharmacy in elderly individuals: an integrated literature review. J Gerontol Nurs 2005;31:4-11. [Crossref] [PubMed]
22. Fried TR, O'Leary J, Towle V, et al. Health outcomes associated with polypharmacy in community-dwelling older adults: a systematic review. J Am Geriatr Soc 2014;62:2261-72. [Crossref] [PubMed]
23. Kojima G, Bell C, Tamura B, et al. Reducing cost by reducing polypharmacy: the polypharmacy outcomes project. J Am Med Dir Assoc 2012;13:818.e11-5. [Crossref] [PubMed]
24. Labor Force Statistics from the Current Population Survey [database on the Internet]. United States Department of Labor. 2022. Available online: https://www.bls.gov/cps/tables.htm/. Accessed: 12/29/2022.
25. Corny J, Rajkumar A, Martin O, et al. A machine learning-based clinical decision support system to identify prescriptions with a high risk of medication error. J Am Med Inform Assoc 2020;27:1688-94. [Crossref] [PubMed]
26. Lopez CD, Ding J, Trofa DP, et al. Machine Learning Model Developed to Aid in Patient Selection for Outpatient Total Joint Arthroplasty. Arthroplast Today 2022;13:13-23. [Crossref] [PubMed]