1.1 Cardiology

AI has the potential to transform cardiology by enhancing diagnostic accuracy, supporting personalized treatment, and contributing to improved patient care [18,19]. Specifically, AI algorithms have proven effective in analyzing cardiovascular data to aid in the early detection and diagnosis of conditions such as arrhythmias, heart failure, and coronary artery disease [20,21]. For instance, machine learning models have shown high precision in interpreting echocardiograms, at times surpassing human experts in diagnostic speed and accuracy [22]. Additionally, AI holds potential for managing cardiovascular risk by integrating diverse patient data to predict individual risk factors [23]. It also contributes to the development of personalized treatment plans by analyzing patient data and predicting responses to various treatment modalities, optimizing therapeutic decisions [24]. In oncology, AI tools like MesoNet have been used to predict patient survival by analyzing whole-slide digitized images. These tools have demonstrated greater accuracy in predicting patient survival compared to current pathology practices, offering new insights and potentially guiding better treatment decisions [25]. However, the integration of AI in clinical practice requires rigorous validation and careful ethical considerations to prioritize patient safety and uphold data security [26]. The continuous evolution of AI technologies holds promise for advancing diagnostic tools and therapeutic strategies, potentially improving precision and predictive capabilities in whole-person care. This could include recommending lifestyle changes tailored to individual contexts alongside medications adjusted to their genomic profiles [27]. Moreover, AI could play a role in supporting health equity by facilitating access to advanced cardiac care in underserved areas through telemedicine platforms and remote monitoring, which could help reduce disparities in cardiovascular health outcomes [28].

1.2 Ophthalmology

AI has demonstrated significant potential in ophthalmology by improving diagnostic accuracy for common eye diseases and streamlining clinical workflows. Deep learning models have been particularly effective in analyzing complex visual data from imaging techniques such as fundus photography and optical coherence tomography (OCT) [29]. These models are very good at diagnosing common eye diseases like diabetic retinopathy, glaucoma, and age-related macular degeneration, sometimes achieving expert-level sensitivity and specificity [30]. Automated analysis using AI not only speeds up the diagnostic process but also reduces human error, facilitating earlier and more precise interventions [31]. Furthermore, AI’s predictive capabilities are being harnessed to forecast disease progression, which is crucial for conditions like glaucoma, where early detection can prevent severe vision loss [6]. Using AI tools in clinical practice also includes using AI-driven decision support systems that help plan treatments by guessing how different types of treatments will work [32]. Despite these advancements, challenges remain, including data privacy concerns, the need for large annotated datasets for training algorithms, and the integration of AI into existing clinical workflows [33]. However, ongoing research and collaboration between AI technologists and ophthalmic experts are likely to overcome these obstacles, solidifying AI’s role in modern ophthalmology [34]. This further opens the frontier for the expansion of access to ophthalmological care in existing deserts through technology-enabled care models and non-specialist operators at the point of service, thereby addressing disparities in eye care availability and quality.

1.3 Dermatology

AI is advancing dermatology by improving diagnostic accuracy, personalizing treatments, and streamlining patient management. Deep learning models, such as CNNs, are effective in diagnosing skin cancer, demonstrating performance comparable to dermatologists in identifying melanomas and other skin conditions [35,36]. Moreover, AI applications in dermatology extend beyond cancer detection to managing chronic conditions like psoriasis and atopic dermatitis. These algorithms assist in monitoring disease progression and treatment response, enhancing decision-making in telemedicine and clinical workflows [37]. AI systems are also utilized in cosmetic dermatology, optimizing treatment recommendations based on individual facial analysis [38]. Ensuring the efficacy and safety of AI tools requires rigorous validation processes with diverse datasets to mitigate bias and enhance generalizability [14]. Integration into clinical workflows and user training is crucial to maximizing the benefits of AI in dermatological practice [39]. As AI continues to evolve, continuous collaboration between technologists and clinicians is essential to address ethical considerations and improve patient outcomes in dermatology. Importantly, AI can democratize access to high-quality dermatological care by enabling remote consultations and diagnostics, thus reducing barriers for individuals in remote or underserved regions.

1.4 Neurology

AI has contributed to advancements in neurology by improving diagnostic accuracy and supporting research efforts. Techniques like machine learning and deep learning assist in analyzing imaging data for disorders such as Alzheimer’s and epilepsy [5,40]. These technologies enhance the interpretation of MRI and CT scans, often achieving higher accuracy than traditional methods [6]. In treatment, AI algorithms personalize therapies based on patient data, optimizing outcomes for diseases like Parkinson’s [41]. Additionally, AI has advanced neuroprosthetics by enabling adaptive brain-machine interfaces (BMIs), which, when combined with physiotherapy, show potential to improve motor function in patients with severe motor disabilities [42]. Predictive analytics in healthcare highlight the importance of transparent algorithms for reliable validation and monitoring, which support applications in neurology, such as tracking disease progression in conditions like multiple sclerosis [43]. AI’s applications extend to research, facilitating the understanding of complex neurological phenomena and the development of innovative treatments [44]. For instance, StrokeSight, an EEG-based system, demonstrates how AI can improve neurology by providing fast, affordable stroke diagnoses and personalized treatments using deep learning [45]. Overall, AI’s integration into neurology not only enhances clinical practices but also opens new avenues to address rare diseases and develop more finely tuned medications tailored to individual patients [46].

1.5 Radiology and cancer treatments

AI is advancing radiology and cancer treatment by enhancing imaging accuracy, enabling personalized therapies, and improving diagnostic workflows. Deep learning algorithms have shown significant progress in interpreting medical images, supporting the early detection and diagnosis of cancers such as breast and lung cancer [47]. Machine learning models assist radiologists by identifying complex patterns in imaging data, aiding in faster and more efficient diagnoses [48]. In cancer treatment, AI assists in formulating personalized treatment plans based on patient data and predictive analytics, which can predict treatment outcomes and suggest optimal therapies [49]. AI applications in radiology extend to prognostic evaluations, predicting disease progression and survival rates, refining treatment protocols and developing follow-up strategies [50]. The development of AI systems capable of outperforming human experts in tasks like breast cancer prediction highlights their potential in clinical settings. These advancements are supported by large datasets and strict validation processes, ensuring reliability and clinical applicability [51].

1.6 Emergency medicine and critical care

AI stands to significantly transform emergency medicine and critical care by enhancing patient triage, treatment efficacy, and time to diagnosis. As AI-powered tools like diagnostic algorithms have been incredibly good at the early identification of conditions like sepsis and its progression, there is the potential to improve patient outcomes [52,53]. The prediction of patient outcomes has also positioned AI to facilitate the optimization of critical resource allocation. During the COVID-19 pandemic, AI-based self-triage tools had a critical role to play in predicting cases and hospitalizations at the population level [54]. AI-powered chatbots and virtual assistants are streamlining patient intake and symptom assessment [55,56]. Furthermore, AI’s role in triaging patients based on urgency has revolutionized care prioritization, improving ED operations and patient flow [57,58]. These advancements underscore AI’s potential to not only enhance emergency medical services but also pave the way for a more efficient, accurate, patient-centered healthcare system that extends care beyond traditional brick-and-mortar facilities.

2.1 Sources of bias

Bias in AI healthcare systems can arise at multiple stages of the machine learning (ML) pipeline, necessitating phase-specific mitigation strategies. To address this comprehensively, researchers have developed three primary approaches, each targeting a distinct phase of the pipeline: (i) pre-processing, which focuses on reducing bias in the input data before model training [59]; (ii) in-processing, which applies fairness constraints during the training phase [60]; and (iii) post-processing, which modifies model outputs to enhance fairness [61]. We adopt this three-stage framework to capture the end-to-end process of data collection, model development, and decision-making in AI healthcare systems. This structure, widely used in the fairness literature, supports systematic bias analysis and enables targeted detection and mitigation strategies at each stage [59–64]. The division also facilitates clearer reasoning about when and how fairness interventions can be most effectively applied throughout the ML pipeline.

A critical first step in designing effective interventions is to identify the underlying sources of bias at each stage of the ML pipeline. This section outlines those sources, with a summary provided in Table 1 and detailed discussions presented in the following sections.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summary of biases across each stage of the machine learning (ML) pipeline.

https://doi.org/10.1371/journal.pdig.0000864.t001

2.2 Potential consequences of biases

Biases in AI systems, particularly in healthcare, can have profound and far-reaching consequences. These biases usually stem from the data on which the AI systems are trained, the design of the algorithms themselves, and the contexts in which they are deployed (as discussed in Sect 2.1). Here are several potential consequences of biases in AI systems in healthcare:

Misdiagnosis and inequitable healthcare outcomes: AI integration in healthcare brings significant advancements but also risks of misdiagnosis and inequitable outcomes due to biased algorithms. AI systems often rely on non-representative data, leading to biased decision-making. For instance, Obermeyer highlighted that an algorithm used in healthcare disproportionately favored white patients over black patients because it used healthcare costs as a proxy for healthcare needs, indirectly embedding racial biases in its predictions [12]. Moreover, these biases in AI can exacerbate existing healthcare disparities. Rajkomar and others discussed how AI applications, if not carefully designed and monitored, could inherit and amplify socioeconomic and racial disparities. This is particularly problematic in diagnostics, where AI systems are trained predominantly on data from specific demographic groups, potentially leading to poorer diagnostic accuracy for underrepresented groups [4]. This was evident in a study by Adamson and Smith, which found that dermatology AI systems demonstrated lower accuracy rates in skin lesion diagnosis for dark-skinned individuals compared to those with lighter skin [14]. Such biased AI not only risks misdiagnosis but also contributes to inequitable healthcare outcomes by potentially steering healthcare resources away from those who may need them most. Vayena and his team emphasized the ethical imperative to ensure that AI tools in healthcare are developed with consideration for fairness and equity, advocating for diverse and inclusive data sets and algorithmic transparency to mitigate these biases [73].

Loss of trust in healthcare systems: Recent studies have echoed the concern that the deployment of biased AI in healthcare could significantly undermine public trust in medical systems. Trust is foundational to the patient-provider relationship and is crucial for the effective delivery of healthcare services [74]. When AI tools exhibit bias, whether in diagnosis, treatment recommendations, or patient management, they can lead to misdirected care, fostering distrust among patients, particularly in marginalized communities [4]. Kherbache and his team pointed out that when patients perceive AI-driven processes as opaque or unfair, their trust in the overall healthcare system may decline [75]. Incidents of AI failures that receive public attention can exacerbate this erosion of trust, leading patients to question the reliability and ethics of using AI in medical decision-making. Researchers like Kerasidou argue that trust is not only about the accuracy of AI but also its alignment with ethical principles that govern healthcare, such as beneficence and non-maleficence [76]. Furthermore, the lack of transparency in how AI models make decisions can be a major barrier to trust. Blease’s study [77] suggests that without clear communication about how AI tools contribute to healthcare decisions, patients may become skeptical of diagnoses and treatments, fearing that their personal healthcare data could be misused or misunderstood. This potential trust deficit could have severe implications, not just for individual health outcomes but also for public health at large, as mistrust in healthcare systems can lead to lower rates of healthcare utilization, vaccine hesitancy, and poor adherence to medical advice [78]. Veinot highlights the need for healthcare systems to maintain high standards of accountability and transparency as they integrate AI technologies to mitigate such risks [79].

Legal and ethical implications: The use of biased AI in healthcare not only poses clinical risks but also entails significant legal and ethical implications. Deploying biased AI systems could lead to legal breaches of anti-discrimination laws. In the United States, for example, the Civil Rights Act and the Americans with Disabilities Act set legal standards that could be violated by biased AI algorithms, which fail to provide equitable care across different patient demographics [80]. Ethically, biased AI conflicts with the fundamental medical ethics principles of justice and non-maleficence, demanding fairness and avoidance of harm, respectively [81]. Moreover, biased algorithms in healthcare could potentially expose medical practices and institutions to litigation related to malpractice or negligence, especially if these algorithms contribute to substandard care outcomes [82]. For instance, if an AI system were to consistently provide inferior diagnostic support for certain racial groups, this could be viewed as a form of systemic negligence or malpractice [73]. Additionally, the ethical implications extend to the breach of patient trust and the compromise of patient autonomy. The ethical medical practice relies heavily on the principles of informed consent and respect for patient’s autonomy—principles that are challenged by opaque AI systems that do not make their decision-making processes or inherent biases clear to patients or practitioners [77]. These potential legal and ethical failures underscore the necessity for rigorous oversight and transparent development processes for AI in healthcare, aiming to ensure that these technologies adhere to both existing legal frameworks and ethical standards of practice.

Resource misallocation: The deployment of biased AI in healthcare settings can lead to resource misallocation, a critical issue that impacts both the efficiency and fairness of medical services. Biased algorithms may misdirect resources by prioritizing certain groups over others based on flawed data inputs or biased training procedures. For example, a study by Obermeyer demonstrated how an algorithm used for managing healthcare resources inadvertently favored healthier white patients over sicker black patients due to biased data inputs that did not accurately reflect patient needs [12]. This misallocation can exacerbate existing healthcare disparities by diverting necessary medical attention and resources away from those who are most in need. As Chen pointed out, such disparities are not just a matter of clinical outcomes but are deeply tied to social and economic inequalities that AI tools can inadvertently perpetuate [13]. Furthermore, biased AI can influence the allocation of resources within healthcare facilities, potentially resulting in inefficiencies that strain healthcare systems. This includes misallocating medical staff, diagnostic tools, and hospital beds, which can degrade the quality of care delivered while increasing wait times and healthcare costs [4]. Resource misallocation also raises ethical questions about fairness and equity in healthcare provisioning. It challenges the ethical principle of justice, which demands that healthcare resources be distributed based on need rather than biased algorithms [81]. This ethical breach can lead to further mistrust and reluctance among underserved populations to engage with healthcare systems, perpetuating a cycle of disadvantage.

Stifling innovation: The presence of bias in AI systems used in healthcare not only affects the accuracy and fairness of medical services but can also stifle innovation. When AI models are made using biased data, they might not accurately reflect the different needs of the population. This could make these innovations less useful and applicable to a wider range of demographic groups. Liang talks about how biased training datasets in AI development make it harder for these systems to work with different types of data. This could stymie innovation by preventing it from solving larger or more complex health problems that affect many people. Moreover, reliance on biased AI can deter investment in developing new technologies that are inclusive and equitable. Potential investors may be cautious about funding projects that might not meet regulatory standards for fairness or could lead to public backlash — a concern highlighted by Corbett-Davies and Goel, who note the legal and social implications of deploying biased AI [83]. In addition, the perpetuation of bias in AI could solidify existing disparities in healthcare innovation. As Benjamin notes, if innovation is directed predominantly at solving problems specific to well-represented groups, less common but equally pressing issues in underrepresented groups are neglected, thereby limiting the scope and impact of technological advancements in healthcare [84]. The studies of Rajkomar [4], Hardt [85], and Howell [86] made things even more complicated by pointing out that biased algorithms can make it harder to find new treatments that work for everyone. This feeds a cycle of innovation that helps groups that are already overrepresented in data sets more than others. This restricted focus on innovation not only affects the equity of healthcare delivery but also limits the development of truly innovative, comprehensive healthcare solutions. These factors combined suggest that biased AI not only hinders the progression of medical technologies but also potentially locks the healthcare sector into a cycle of uneven innovation where only the needs of the majority are systematically addressed.

2.3 Fairness metrics and trade-offs

Biases in AI healthcare systems, as discussed in the previous sections, can significantly impact both the quality and equity of healthcare delivery. To address these challenges, fairness metrics provide a structured framework for evaluating and mitigating disparities in AI model predictions. Metrics such as Statistical Parity [87], Equal Opportunity [88], and Equalized Odds [89] serve as foundational tools for detecting bias. However, these metrics often reflect distinct fairness priorities, leading to inherent trade-offs when applied simultaneously [90]. For example, optimizing Statistical Parity to ensure equal outcomes across groups may conflict with Equal Opportunity, which focuses on ensuring equal access to positive outcomes for individuals who qualify. These conflicts underscore the complexity of achieving fairness in healthcare AI.

Furthermore, tensions arise not only between fairness metrics but also between different dimensions of fairness, such as group fairness and individual fairness. While group fairness seeks equitable treatment across demographic groups, individual fairness emphasizes similar outcomes for individuals with comparable features. These trade-offs become even more challenging in healthcare scenarios where fairness must often be balanced with performance, such as ensuring accurate predictions while maintaining equity. By navigating these conflicts, we aim to design AI systems that uphold fairness as a core principle in healthcare. To provide a clear understanding, we introduce these three key fairness metrics along with their mathematical foundations.

Statistical parity (SP): Statistical Parity [87] ensures that the probability of a positive predicted outcome is equal across different groups defined by a sensitive attribute. Mathematically, it is defined as:

(1)

where is the predicted outcome and S is the sensitive attribute. Statistical Parity is satisfied when SP= 0, indicating no disparity between the groups.

Equal opportunity (EO): Equal Opportunity [88] ensures that individuals who are likely to achieve the advantaged outcome (e.g., repaying a loan, being admitted to college) have an equal chance of receiving the positive prediction, regardless of their sensitive attribute (e.g., race, gender). Mathematically, it is defined as:

(2)

where is the binary prediction, S is the sensitive attribute, and Y = 1 represents the advantaged outcome. Equal Opportunity is satisfied when EO = 0 indicates no disparity in predictions for the advantaged outcome across sensitive groups.

Equalized odds (Eq.Odds): Equalized Odds [89] ensures that for each ground truth label (Y = y), the predicted outcome is equally likely across groups defined by the sensitive attribute (S). Mathematically:

(3)

where is the predicted outcome, S is the sensitive attribute, and Y is the ground truth label. Equalized Odds is satisfied when Eq.Odds = 0, indicating no disparity across groups for all ground truth labels.

These group fairness metrics Statistical Parity, Equal Opportunity, and Equalized Odds reflect fundamentally different fairness priorities, leading to inherent trade-offs when attempting to satisfy them simultaneously. Statistical Parity requires predictions to be independent of sensitive attributes, ensuring equal positive outcome rates across groups, regardless of differences in their underlying characteristics, which can sometimes lead to disparities in how individuals within a group are treated. Consider a healthcare scenario where males have a higher prevalence of heart disease compared to females, enforcing Statistical Parity would mean predicting the same proportion of males and females as being at risk. However, this might conflict with Equal Opportunity, which requires equal true positive rates across groups. Achieving Statistical Parity in this scenario may lower the true positive rate for females, as increasing their positive predictions could include many individuals without heart disease. This highlights how prioritizing Statistical Parity can undermine Equal Opportunity by creating disparities in prediction accuracy within groups.

To achieve Equalized Odds, a model must ensure that true positive and false positive rates are the same across groups, which often requires adjusting prediction thresholds differently for each group. For example, if females are less likely to have heart disease compared to males, the model might lower the prediction threshold for females to classify more of them as “at risk" increasing the true positive rate for females. At the same time, it might raise the threshold for males to reduce false positives. While these adjustments equalize predictive performance across groups, they disrupt Statistical Parity by creating unequal proportions of positive predictions for males and females, reflecting the differences in the underlying likelihood of heart disease between the groups. This trade-off shows how achieving Equalized Odds can violate Statistical Parity by altering the balance of positive predictions across groups.

The trade-offs between these fairness metrics are formalized in the Impossibility Theorem [90], which demonstrates that it is often impossible to satisfy all fairness criteria simultaneously. For instance, achieving Statistical Parity might require selecting more from a group with a lower base rate, sacrificing Equalized Odds or Equal Opportunity, while prioritizing Equal Opportunity or Equalized Odds might allow selection rates to differ across groups, violating Statistical Parity. These inherent conflicts require careful consideration when deciding which fairness metric to apply. Ultimately, the choice of fairness metric should depend on the specific context and the ethical or practical priorities of the application. For instance, in critical healthcare scenarios like diagnosing heart disease, Equal Opportunity or Equalized Odds might be prioritized to ensure equal access to accurate treatment and minimize disparities in predictive performance. On the other hand, Statistical Parity could be more relevant in preventive care programs, where equitable outreach and representation are paramount goals [91].

The conflict between fairness metrics can also be seen in the difference between individual and group fairness, which focuses on fairness at different levels. Group fairness [92] aims to ensure equitable treatment across different subgroups defined by sensitive attributes, i.e., group fairness aims to ensure that the outputs of ML algorithms are independent of the sensitive attribute. In contrast, individual fairness [93] enforces fairness at a finer granularity by ensuring that similar individuals receive similar predictions based on their input features. Unfortunately, the simultaneous achievement of both maxima is practically impossible due to the trade-offs involved. Fig 1 illustrates this trade-off in the context of organ allocation, where gender is the sensitive attribute, and medical urgency determines priority. In the performance-driven result (left side of the Fig 1), organs are allocated solely based on urgency and likelihood of survival, prioritizing patients P₁, P₂, P₃, and P₄ due to their higher need and better survival prospects, while rejecting P₅, P₆, P₇, and P₈ because of their lower urgency or expected benefit. This approach maximizes utility by allocating resources to those expected to achieve the greatest overall benefit, such as improved survival rates and long-term health outcomes. However, it may inadvertently perpetuate systemic disparities between subgroups. The group fairness-driven result (right side of the Fig 1) adjusts the decision boundary to ensure equal approval rates for males and females, resulting in females P₇ and P₈ being approved despite their lower urgency, while males P₃ and P₄, with higher urgency, are denied. While this adjustment addresses equity at the group level, it compromises individual fairness, as highlighted by the purple rectangle, where individuals near the decision boundary disproportionately bear the group fair loss of the entire subgroup.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Illustration of the trade-off between performance-driven result and group fairness-driven result in the context of organ allocation, where gender is the sensitive attribute.

https://doi.org/10.1371/journal.pdig.0000864.g001

Building on the discussion of trade-offs between individual and group fairness, the tension between fairness and performance further illustrates these challenges. In contexts such as organ allocation, performance-driven approaches prioritize outcomes for individuals with the highest urgency or expected benefit. However, enforcing group fairness constraints such as ensuring equal approval rates across subgroups defined by sensitive attributes like gender can lead to a reduction in overall performance. For example, in the organ allocation scenario, ensuring that approval rates are equal for males and females might result in allocating organs to individuals like P₇ and P₈ (females, lower urgency) while denying organs to P₃ and P₄ (males, higher urgency). This adjustment ensures equity at the group level but comes at the cost of reduced overall utility, as fewer lives may be saved or long-term health outcomes may be compromised. This trade-off highlights the inherent difficulty of simultaneously achieving fairness and optimal performance, as prioritizing one objective often necessitates sacrifices in the other.

3.1 Bias detection methods

Bias detection is a crucial step in identifying and addressing inequities in AI systems, especially in healthcare given its high-stakes nature. This section discusses various tools and techniques for detecting bias, categorized by their roles in the pre-processing, in-processing, and post-processing stages of the machine learning (ML) pipeline. The aim is to provide a clear and actionable understanding of bias detection.

3.2 Mitigation strategies

Mitigating biases in AI systems within healthcare is crucial for ensuring equitable treatment across all patient demographics. This section outlines key strategies to address and reduce biases, including the use of diverse and representative data and fairness-aware approaches, thereby enhancing the fairness and accuracy of AI models.

4.1 Ethical considerations

The integration of AI in healthcare raises significant ethical challenges, including privacy concerns, algorithmic bias and accountability. One critical gap lies in ensuring patient data privacy while enabling AI models to be trained on diverse datasets. Ensuring the confidentiality and security of this data against breaches is paramount [117]. Current approaches, such as anonymization, may not adequately prevent re-identification risks, particularly when combined with external data sources [118]. Future research should explore privacy-preserving techniques, such as federated learning and differential privacy, to enable secure training of AI systems without compromising data confidentiality [119]. Algorithmic bias is another major concern. AI models trained on non-representative data can perpetuate systemic inequalities, leading to unfair treatment outcomes for underrepresented groups [4]. Solutions like adversarial debiasing and fairness-aware algorithms should be further developed and rigorously tested in healthcare settings [120]. Furthermore, the deployment of AI in healthcare must take into account the potential displacement of healthcare professionals, raising concerns about job security and the loss of valuable human expertise and empathy in patient care [121]. Lastly, accountability for AI-driven decisions in healthcare is a critical ethical issue. It is critical to define who is responsible—the developers, the users, or the AI itself—when an AI system’s decision results in patient harm [122].

4.2 Legal considerations

Legal considerations in AI healthcare systems encompass intellectual property rights, liability for errors, regulatory compliance, and policy gaps that hinder bias mitigation. A notable gap exists in the legal frameworks governing intellectual property rights as there is an ongoing debate over whether AI-generated medical inventions can be patented and who owns the rights to AI-generated data and algorithms [82]. Future research should focus on developing frameworks that equitably address the rights and interests of developers, institutions, and patients. Furthermore, liability issues arise when AI systems make erroneous decisions that could harm patients. Determining whether the healthcare provider, the software developer, or another party is liable requires intricate legal analysis and possibly new legal frameworks [123]. Solutions such as liability insurance for AI developers and a staggered approach to liability, from strict liability for high-risk applications to simple fault-based liability for consumers, could address this gap [124]. Policy gaps further hinder bias mitigation. Future directions also include promoting policy changes that enforce transparency and fairness in data collection and algorithmic processing in healthcare applications [81]. Governments and institutions should implement policies mandating fairness audits and bias detection tools to ensure equitable outcomes. AI applications in healthcare must adhere to existing healthcare regulations, such as the Health Insurance Portability and Accountability Act (HIPAA) in the U.S., which governs the privacy and security of patient data, and the General Data Protection Regulation (GDPR) in the EU, which sets stringent guidelines for data protection and privacy [125]. As AI technologies evolve, there may also be a need for specific regulations that address AI’s unique aspects in healthcare to ensure that these innovations are safely and effectively integrated into medical practice [126]. Lastly, issues of informed consent are magnified with AI, as patients must understand the role of AI in their care, including how their data will be used and the implications of AI-driven decisions [122].

4.3 Transparency and interpretability

The lack of transparency and interpretability in AI models remains a major obstacle to fairness in healthcare. Many advanced models, such as deep neural networks, function as "black boxes," making their decision-making processes hard to understand and verify. This lack of clarity can lead to mistrust among healthcare providers and patients, as well as unidentified biases in outcomes [127]. For instance, an AI diagnostic tool might predict a higher risk of disease for one patient over another with nearly identical medical histories, without providing any rationale for the differing assessments. To address this, future research should focus on developing “glass-box” approaches that are inherently interpretable, such as decision trees or linear models, fostering greater trust among healthcare providers and patients [127]. Future research should focus on adopting model-agnostic explanation frameworks like LIME [97] or SHAP [128], which provide insights into the decision-making processes of any AI model, regardless of its underlying architecture. Developing these tools will enable clinicians to better understand and oversee AI-driven decisions, potentially leading to more equitable healthcare outcomes. Additionally, implementing model documentation standards like Model Cards can provide essential transparency by detailing the capabilities and limitations of AI models to all stakeholders [100].

4.4 Diversity and representation

The lack of diversity in training datasets is a significant research gap that impacts AI fairness in healthcare. Models trained on data that underrepresents certain demographic groups risk perpetuating biases, thereby worsening healthcare inequalities [4]. For example, a skin cancer detection AI trained predominantly on lighter skin tones may fail to accurately identify conditions on darker skin, highlighting the impact of underrepresentation in training data on healthcare outcomes. To mitigate this issue, techniques for synthetically augmenting underrepresented classes in datasets or incentivizing the collection of more comprehensive and inclusive data must be developed [129]. Additionally, implementing algorithmic fairness approaches during model training, such as fairness constraints or adversarial debiasing, can help mitigate biases introduced by skewed datasets [62]. Studies suggest that geospatial AI models, when applied to geographic information systems (GIS), can uncover regions with inadequate healthcare services, thus guiding interventions to those most in need [79]. However, predictive analytics must be carefully designed to ensure they do not perpetuate biases present in underlying data [12]. Integrating feedback mechanisms from healthcare providers into AI systems offers a pathway to refine these models, ensuring they adapt to serve diverse populations effectively. The future direction in bridging this gap involves developing AI technologies that are not only predictive but also prescriptive, providing actionable insights to improve universal healthcare accessibility [4].

4.5 Interdisciplinary approaches

The integration of AI in healthcare requires interdisciplinary approaches that combine insights from medical ethics, data science, social sciences, and clinical practice to comprehensively address fairness [130]. However, current AI applications often lack this multidimensional perspective, resulting in solutions that are technically adequate but socially or ethically inappropriate [81]. For example, an AI diagnostic tool developed with only technical expertise might fail to account for socio-economic factors, leading to misdiagnoses in underserved communities that lack access to quality healthcare infrastructure. Future research should focus on frameworks that incorporate not only technical accuracy but also ethical, legal, and social implications from the outset, ensuring that AI systems are developed with a holistic view of fairness [79]. These collaborative frameworks can help prevent the exacerbation of existing disparities and promote a more equitable distribution of healthcare resources [13].

4.6 Longitudinal effects

The longitudinal effects of AI systems on healthcare fairness remain underexplored, with existing studies often focusing on short-term outcomes and immediate biases in algorithmic decision-making [131]. Understanding how these technologies affect health disparities over time is crucial, as initial biases can amplify if not properly addressed [4]. For example, an AI system predicting patient readmissions might keep giving more resources to well-funded hospitals because they had better outcomes in the past, making the gap worse for less-supported hospitals over time. Future research should involve continuous monitoring of AI systems post-deployment to assess their impact on various demographic groups across different time intervals [132]. This approach will help identify evolving biases and enable timely modifications to algorithms, thereby ensuring more equitable healthcare outcomes [12]. Additionally, incorporating feedback loops that allow healthcare providers to report disparities can further refine AI applications in real-world settings.