Unlocking the potential: ChatGPT and the quest for reliable blood transfusion information

Introduction

Blood transfusion, a critical supportive medical procedure that consistently saves numerous lives, remains a cornerstone of modern healthcare. While widely employed as a supportive therapeutic intervention, it is imperative to acknowledge its inherent risks. These include the transmission of infectious agents, concerns related to iron overload, acute allergic reactions, hemolytic reactions, hyperkalemia, as well as conditions such as transfusion-related circulatory overload and transfusion-related acute lung injury (1). It is worth highlighting that the transfusion of incorrectly matched blood components emerges as the primary cause behind adverse incidents associated with blood transfusion therapy (2). The varying perceptions and understandings of blood transfusion risks among the general public and healthcare professionals contribute to the complexity of this situation (3). These differing perspectives within the healthcare community can significantly influence clinical decision-making regarding the necessity and safety of blood transfusions. Patients frequently turn to online sources for information to self-educate about blood transfusion (4). In these scenarios, interactive online artificial intelligence (AI) platforms like Chat Generative Pre-Trained Transformer (ChatGPT) play a revolutionary role in patient engagement with the internet. In this study, our objective was to evaluate the reliability of responses provided by ChatGPT to questions related to blood transfusion. To assess the reliability of ChatGPT’s knowledge, we utilized the expertise of hospitalists who meticulously analyzed its responses to rate them on Likert scale and contributed to our conclusions. This evaluation aims to shed light on AI’s capabilities and limitations in healthcare.

Methods

A 20-question survey covering general blood transfusion-related questions was created on Google document and sent to 12 consenting hospitalists (inpatient physicians) working at the same facility. To ensure the comprehensiveness and accuracy of our survey, we drew upon the official guidelines provided by the NHS Blood and Transplant (NHSBT) organization. The NHSBT’s dedicated page on blood transfusion frequently asked questions (FAQs) served as a foundational reference (5). Additionally, we enhanced our questionnaire by incorporating FAQs encountered in our clinical practice. These questions were derived from direct interactions with patients, caregivers, and healthcare professionals involved in blood transfusion procedures. By using both the NHSBT guidelines and our clinical encounters, we aimed to create a survey that addresses the concerns of patients comprehensively and reflects the dynamic nature of real-world scenarios. The survey represented commonly asked questions by the patients and relatives before consenting to the transfusion. These questions were entered in the English language into ChatGPT version 3.5 in August 2023, and the responses were compiled in the same Google document. Twelve expert hospitalists who consented to the survey evaluated these responses on a Likert scale of Poor, Fair, Good, Very Good, and Excellent. The responses were then analyzed for reliability by calculating Cronbach’s alpha (α) for all the questions using an online statistical tool (6). The consensus on the use of ChatGPT as a source of information or its ability to replace human consent before blood transfusion was evaluated and shortcomings were discussed.

Results

Out of 240 total quality responses, the most common response was “Very Good” (n=129, 52.8%), followed by “Excellent” (n=65, 27.1%), “Good” (n=36, 15%), “Fair” (n=8, 3.3%) and “Poor” (n=2, 0.8%) (Figure 1). Only Questions 15 and 19 received a “Poor” quality grade response by one expert, whereas in 15 out of 20 questions, ≥50% of experts recorded a response of “Very Good”. The questions were divided into three categories “General Concern”, “Procedure-related questions”, and “Technical Questions”. It was found that in all general concern and Procedure-related questions, ≥50% of experts responded “Excellent” or “Very Good”, whereas none of them gave a “Poor” quality response. But in Technical questions, two out of eight have received a “Poor” quality response from one expert. The Cronbach’s α of the cumulative responses is 0.78 suggesting that the items have an acceptable level of consistency in terms of reliability. Question-wise analysis for reliability can be found in Appendix 1. As α >0.7 is considered acceptable in consistency, hence all the responses were acceptable. However, the higher the value co-relates with stronger reliability. Considering this, technical questions requiring the application of medical knowledge were found to have a lesser α (≤0.75) value. ChatGPT generated responses to the survey can be found Appendix 2.

Figure 1 Clustered bar graph representing the assigned quality grades to each question by our expert hospitalists.

Discussion

This is the first study that emphasizes the potential of ChatGPT as an informational resource for patients in the context of blood transfusion which can be employed by patients to educate themselves on the topic of blood transfusion possibly reducing physician burden.

These findings align with a recent study that affirmed ChatGPT responses’ appropriateness when addressing bariatric surgery questions (7). The majority of responses received ratings of “Very Good” or “Excellent”, with 90% of experts expressing the opinion that the generated responses exhibited a level of comparability, to some extent, with those provided by practicing clinicians. The quality of responses produced by ChatGPT may be attributed to its utilization of non-supervised pre-training methods, which enable ChatGPT to access a comprehensive corpus of text-based digital information and formulate responses in a conversational style (8). Nevertheless, experts have identified several factors that could potentially restrict the widespread applicability of ChatGPT in the medical domain. For instance, there is a need to incorporate references and links to established societal guidelines to enhance and contextualize responses. Additionally, it is advisable to include information about local services, such as nearby educational seminars and peer support groups. Such resources may potentially exhibit correlations with decreased hesitancy toward blood transfusions, mirroring recommendations made in the context of bariatric surgery (9,10).

These findings necessitate careful consideration within the context of several limitations. Firstly, it is essential to highlight that ChatGPT was not specifically designed for medical purposes. Additionally, due to ChatGPT’s training data originating from 2021, some responses needed to be more consistent with recent medical guidelines. With medical guidelines ever evolving relying on the information from older ChatGPT versions is one of the limitations of AI in assisting in disseminating up to date information (11). Moreover, the subsequent release of an updated version, ChatGPT 4.0, after the survey may have improved response quality. Future research endeavors should aim to validate these findings with a more comprehensive sample of expert participants.

Patients and medical professionals should be aware of the limitations of current AI before using it as their sole informational resource. AI can potentially act as a bridge for the patient and the doctor, making consultations easier and helping the patient comprehend medical concerns and treatments. This can potentially reduce the doctors’ burden, enhancing healthcare quality, and facilitating an efficient healthcare system. Most researchers and doctors are still learning about the use of AI in medicine. It’s hard to predict the ethical concerns we will face with AI until its application is more widespread. Both humans and AI have their strengths and weaknesses, and if they work together, they can aim towards improving of health care.

In conclusion, this study highlights the potential of ChatGPT as a valuable informational resource for patients undergoing blood transfusion, indicative of the growing integration of AI platforms in healthcare, potentially transforming how patients access online information sources.

Acknowledgments

Funding: None.

Footnote

Peer Review File: Available at https://jmai.amegroups.com/article/view/10.21037/jmai-24-14/prf

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://jmai.amegroups.com/article/view/10.21037/jmai-24-14/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Alsharidah AS, Alsuhaibani HA, Almansour BS, et al. Perceptions About Blood Transfusion Therapy Among the General Public and Healthcare Professionals in the Qassim Region of Saudi Arabia. J Blood Med 2021;12:139-45. [Crossref] [PubMed]
2. Suddock JT, Crookston KP. Transfusion Reactions. In: StatPearls. Treasure Island (FL): StatPearls Publishing; August 8, 2023.
3. Kaur P, Kaur G, Kaur R, et al. Assessment of Impact of Training in Improving Knowledge of Blood Transfusion among Clinicians. Transfus Med Hemother 2014;41:222-6. [Crossref] [PubMed]
4. Molina-Arrebola MA, Fernández-Guerrero E, Aguirre-Ortega FJ, et al. Digital resources for transfusion education. J Educ Health Promot 2020;9:173. [Crossref] [PubMed]
5. NHS Blood and Transplant [Internet]. [cited 2024 Mar 12]. Transfusion FAQs. Available online: https://www.nhsbt.nhs.uk/what-we-do/blood-services/blood-transfusion/transfusion-faqs/
6. T-test, chi-square, ANOVA, regression, correlation [Internet]. [cited 2024 Mar 12]. Available online: https://datatab.net/statistics-calculator/reliability-analysis/cronbachs-alpha-calculator
7. Moazzam Z, Lima HA, Endo Y, et al. A Paradigm Shift: Online Artificial Intelligence Platforms as an Informational Resource in Bariatric Surgery. Obes Surg 2023;33:2611-4. [Crossref] [PubMed]
8. Introducing ChatGPT. Accessed September 13, 2023. Available online: https://openai.com/blog/chatgpt
9. Shinan-Altman S, Sandbank GK, Natarevich-Katzav H, et al. Self-management After Bariatric Surgery: a Comparison Between Support Group Participants and Patients Receiving Individual Dietary Follow-Up. Obes Surg 2023;33:826-35. [Crossref] [PubMed]
10. Eaton L, Walsh C, Magnuson T, et al. On-line bariatric surgery information session as effective as in-person information session. Surg Obes Relat Dis 2012;8:225-9; discussion 229. [Crossref] [PubMed]
11. JohnsonDGoodmanRPatrinelyJAssessing the Accuracy and Reliability of AI-Generated Medical Responses: An Evaluation of the Chat-GPT Model. Res Sq2023. doi: .10.21203/rs.3.rs-2566942/v1