A concise framework for fairness: navigating disparate impact in healthcare AI

Introduction

The integration of artificial intelligence (AI) in healthcare has the potential to transform patient care. However, concerns about fairness, transparency, and potential biases have been raised (1). Bias, a systematic error in decision-making processes, can stem from various sources in an AI framework, including sample bias, outcome bias, and biases introduced during data handling and feature engineering (2). If an AI system is trained on biased data, it may make biased decisions, leading to suboptimal care and disparate impact on certain protected groups (3,4). As AI plays an increasingly significant role in healthcare decision-making, prioritizing fairness and mitigating biases becomes crucial.

Previous frameworks, such as the one proposed by Rajkomar et al., have outlined foundational principles for ensuring fairness in the design, deployment, and evaluation of machine learning models to advance health equity (5). Their framework focuses on identifying fairness challenges and promoting accountability in AI systems. Building on these concepts, our framework extends this work by offering a more detailed, actionable approach tailored specifically to the healthcare domain. We address fairness comprehensively across all stages of the AI lifecycle, focusing on key aspects such as data representation, label quality, fairness metrics, and methods to determine fairness for specific concepts. By addressing these components, we aim to provide a proposed framework for researchers to consider when developing equitable AI systems in healthcare.

A concise framework for ensuring fairness in healthcare AI

A comprehensive and systematic approach to model development and deployment is crucial to effectively address potential biases and ensure fairness in healthcare AI systems. Here, we propose a concise approach encompassing all stages of the AI lifecycle, from data collection and preprocessing to model selection, evaluation, and ongoing monitoring. Figure 1 presents a step-by-step framework that can guide researchers, developers, and healthcare professionals in their efforts to create fair and equitable AI models (4,6).

• Data representation: assess the representativeness of the training data. Underrepresentation of specific populations can lead to sample bias and unreliable predictions for those groups. Ensure the data accurately represents the diverse population the AI system will serve by collecting and integrating data from underrepresented groups where feasible. To address these challenges, conducting data analyses to identify demographic gaps, utilizing fairness metrics to quantify representation, and applying stratified sampling or re-weighting techniques can ensure proper representation. To mitigate biases arising from data engineering, preprocessing methods such as feature scaling, encoding, and handling of missing data must be implemented to prevent disproportionate effects on any group. For smaller hospitals or clinics with limited resources, practical approaches include leveraging publicly available datasets to supplement local data and collaborating with regional or national networks to share de-identified data. Integrating feedback from diverse stakeholders, including patients, caregivers, and community representatives, is also critical to ensure relevance and fairness in data collection strategies. Structured mechanisms to gather feedback could include surveys, focus groups and advisory panels to better address potential bias and concerns across diverse populations.
• Outcome labels: inaccuracies in the outcome labels used for training can indirectly influence fairness by introducing systemic errors. These inaccuracies, arising from misdefined outcomes or measurement errors, can lead to unintended disparities in model predictions (6). Outcome bias occurs when labels reflect underlying systemic biases present in the data. Implementation of fair AI systems requires rigorous validation of outcome labels to eliminate systemic errors and ensure clinical relevance.
• Feature and transformation bias: mitigate biases during feature engineering by carefully handling missing values and combining categories. When collecting sensitive attributes, such as race and ethnicity indicators, consider specialized methods and their implications for fairness. For example, data that is self-reported can vary in consistency, while third-party recorded data might introduce observer bias (6). Furthermore, inference from other data sources can also perpetuate existing stereotypes. Addressing these issues by implementing transparency measures and validating data accuracy is crucial for reducing biases introduced during data collection and preprocessing.
• Model selection: evaluate potential models based on both accuracy and fairness metrics that are relevant to the intended clinical application. Select models that balance performance and fairness, considering the potential impact on different patient populations and the specific goals of the AI system (4).
• Fairness metrics: define fairness for the specific context and select relevant metrics, such as False Positive/Group Size (FP/GS) Parity, False Discovery Rate (FDR) Parity, False Positive Rate (FPR) Parity, Recall Parity, False Negative/Group Size (FN/GS) Parity, False Omission Rate (FOR) Parity, and False Negative Rate (FNR) Parity (6). Incorporate fairness in model selection through various approaches:
  • Performance versus equitability: compare the equity of models with similar overall performance. Evaluate the trade-offs between performance and fairness, such as balancing FDR Parity and Recall Parity, to explicitly show these trade-offs.
  • Subgroup performance: consider models that perform best for specific subgroups (e.g., different races or sexes) and those that perform consistently across groups.
  • Equity penalty: develop a model selection parameter that penalizes performance based on deviation from equity, aggregating equity across multiple groups. This approach provides options for the final model, balancing overall performance and equity measures. Unlike traditional model selection, which is based solely on performance metrics, the final choice involves a judgment call reflecting the dual goals of accuracy and equity (6).
• While the fairness metrics listed above pertain to binary classification tasks, there are other tasks such as segmentation or time-to-event prediction that do not fall under this classification for which bias would need to be evaluated differently. In multi-class classification, metrics such as equalized odds difference or average absolute odds difference can be used to ensure equitable treatment across all classes. In tasks like segmentation, fairness can be assessed using spatially aware metrics that account for differences amongst demographic groups. For time-to-event predictions, fairness can be assessed using survival-specific metrics that assess disparities.
• Evaluate and iterate: assess the selected models using the chosen fairness metrics. Analyze the trade-offs between performance and fairness, and iteratively refine the models to improve both aspects simultaneously.
• Deployment: after deploying the model, ensure equitable use by providing clear guidelines on intended use and application of models. Engage with clinicians and administrators to gather feedback on fairness in real-world settings. Facilitate training sessions to help users understand model’s capabilities and limitations to ensure compliance with ethical standards.
• Monitoring: regularly monitor the model for data drift, which refers to changes in data distribution over time that can degrade model performance. Data drift can occur due to various factors, such as changes in population demographics, genetic variation, imaging techniques, disease prevalence, and social determinants of health (4). Continuously assess the model’s predictions against new data to ensure accuracy and reliability. If data drift is detected, retrain the model with updated data to maintain performance and fairness (4). Establish a feedback loop with stakeholders to gather insights on model outcomes and potential biases, ensuring improvements align with demands of the real-world. Additionally, after clinical deployment, there is poor longitudinal follow-up to determine whether certain AI recommendations were correct. Establishing systems to continuously monitor model performance, even when reference standards (e.g., expert annotations or patient outcomes) are unavailable can help evaluate whether AI predictions align with expected outcomes. Alternative methods to assess the accuracy of the AI model’s recommendations could include retrospective audits of AI recommendations, surrogate markers of performance, or clinician feedback.

Figure 1 Flowchart of the proposed framework for ensuring fairness in healthcare artificial intelligence.

Suggestions for selecting fairness metrics for specific contexts

Defining fairness in healthcare AI applications involves understanding the project’s goals and the potential impacts of the model’s decisions on different subgroups. The choice of fairness metrics often depends on the type of intervention and the intended use. When selecting fairness metrics, it is essential to consider the potential harms associated with different types of errors and how these harms may disproportionately affect certain subgroups. When choosing fairness metrics, the following guidance based on the context could be considered:

• For screening tools, it may be appropriate to prioritize metrics that focus on minimizing false negatives, such as FNR Parity and FN/GS Parity. This is because false negatives in screening can lead to delayed diagnosis and treatment, which may have serious consequences for patient outcomes.
• For diagnostic tools, it may be beneficial to balance metrics that consider both false positives and false negatives, such as FPR Parity, FNR Parity, and Recall Parity. This approach ensures that the tool is accurate in identifying the condition while minimizing the harms associated with both types of errors.
• For prognostic tools, it may be suitable to consider metrics that focus on the reliability of positive predictions, such as FDR Parity and Recall Parity. This is because false positives in prognostic tools can lead to unnecessary interventions or distress for patients, while false negatives may result in missed opportunities for early intervention.
• For resource allocation, it may be appropriate to consider metrics that account for group sizes, such as FP/GS Parity and FN/GS Parity. This ensures that the chances of being incorrectly included or excluded from receiving resources are similar across subgroups, promoting equitable access to care.
• For patient education, it may be beneficial to prioritize metrics that minimize false positives and false negatives, such as FPR Parity, FOR Parity, and Recall Parity. This approach ensures that patients receive accurate information about their conditions and the likelihood of different outcomes, empowering them to make informed decisions about their health.

Achieving parity on all fairness metrics simultaneously may not always be possible, especially when the prevalence of the condition differs across subgroups (6). In such cases, stakeholders should prioritize the most relevant fairness metrics based on the specific context and the potential impact on different subgroups. By considering appropriate fairness metrics and iteratively developing, evaluating, and refining models, healthcare organizations can create AI systems that promote equity and fairness (Table 1).

Conclusions

The integration of AI in healthcare has the potential to revolutionize patient care, but it must be approached with a commitment to fairness and equity. By adopting a comprehensive framework for bias mitigation and fairness promotion, healthcare organizations and AI developers can create AI systems that enhance care quality while avoiding the perpetuation of disparities. To fully realize AI’s potential in healthcare, organizations must engage with patients and stakeholders, understand their needs and concerns, and prioritize fairness as a core value.

Acknowledgments

None.

Footnote

Peer Review File: Available at https://jmai.amegroups.com/article/view/10.21037/jmai-24-438/prf

Funding: None.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://jmai.amegroups.com/article/view/10.21037/jmai-24-438/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Char DS, Shah NH, Magnus D. Implementing Machine Learning in Health Care - Addressing Ethical Challenges. N Engl J Med 2018;378:981-3. [Crossref] [PubMed]
2. Ferrara E. Fairness and Bias in Artificial Intelligence: A Brief Survey of Sources, Impacts, and Mitigation Strategies. Sci 2023;6:3. [Crossref]
3. Gianfrancesco MA, Tamang S, Yazdany J, et al. Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data. JAMA Intern Med 2018;178:1544-7. [Crossref] [PubMed]
4. Chen RJ, Wang JJ, Williamson DFK, et al. Algorithmic fairness in artificial intelligence for medicine and healthcare. Nat Biomed Eng 2023;7:719-42. [Crossref] [PubMed]
5. Rajkomar A, Hardt M, Howell MD, et al. Ensuring Fairness in Machine Learning to Advance Health Equity. Ann Intern Med 2018;169:866-72. [Crossref] [PubMed]
6. Rodolfa K, Saleiro P, Ghani R. Chapter 11: Bias and Fairness. In: Big data and social science: data science methods and tools for research and practice. Second edition. Boca Raton, FL: CRC Press; 2021.