The elusiveness of a standard definition of fairness in ML models

Moral concerns over algorithmic fairness originate from a much-chronicled history of socioeconomic disparities and inequalities that certain populations in the West have experienced for centuries. As the interest of this journal is health care, we center comprehensions of unfairness on the phenomenon of health inequities, i.e., unjust (and, therefore, legally and ethically unacceptable) practices in health care distributional space that prevent or discourage the availability of desirable health care services to historically marginalized subgroups [15]. Health inequities do not only conjure up health inequalities. They especially denote, because they are caused by, historically evolved biases or prejudices that unjustly favor one individual or group over another [16–17]. Models that therefore rely on variables or computational formulae that sustain such inequities are patently unfair and, therefore, unjust.

Space only allows a brief comment on why contemporary approaches towards developing a useful and operational conception of fairness are complicated by philosophical, political, and socioeconomic disagreements. Nevertheless, these philosophical and political controversies over fairness explain, or so we believe, its “afterthought” status in the ethical guidelines ML literature, especially as it might occur among American scholars but also among data scientists throughout the West.

We begin by examining perhaps the most common strategy for achieving fairness in the literature. Called “fairness through unawareness,” this strategy seeks to eliminate prejudices or biases from datasets that unfairly disadvantage certain subpopulations at risk for discrimination [18–20]. A good example is the August 2022 U.S. Department of Health and Human Services (DHHS) proposal to extend Section 1557 of the Affordable Care Act’s antidiscrimination requirements to clinical algorithms if the algorithm’s outputs result in discrimination from the use of race or ethnicity variables [21]. This concern about models using protected or sensitive variables in discriminatory ways is pre-eminent among bias mitigation sensibilities and plays a central role in the FTC’s understanding of fairness [21,22].

A major problem with this strategy, however, is that a protected or sensitive variable can have significant predictive value in certain clinical contexts, such as age in sepsis prediction models [23] or race in cancer screening practices [24]. It could therefore be clinically counter-productive to purge certain models of protected attributes if they could dramatically improve the model’s predictive accuracy or the assignment of treatment options (thus prompting the reverse notion of “fairness through awareness” [20]). Furthermore, deleting a protected variable to advance the justice interests of one group might cause a deterioration in predictive accuracy for another group and therefore preserve allegations of discrimination (albeit now against a majority group!) [18,22]. Yet, another problem with a blanket removal of protected attributes is that certain models can nevertheless “infer” the protected attribute, even if it is eliminated, and continue to maintain its discriminatory impact [25,26]. Consequently, “fairness through unawareness” is not always effective, possible, or even ethically warranted.

A second conceptualization of fairness proposes not only equal predictive accuracy across groups but also equalized error rates across groups [13,18]. This fairness intuition—called “fairness as equalized odds”—was a prominent criticism of the much-discussed COMPAS recidivism prediction model inquiry reported by ProPublica in 2016 [27]. COMPAS is a widely used model in the U.S. judicial system that provides courts with likely-to-reoffend scores for individuals coming up for parole or who are coming up for trial. Receiving a high COMPAS score among the former means they will be denied parole and spend longer time in jail. Receiving a high score among the latter means they will be denied bond and remain in jail until their case comes up for trial [28].

What captured the interest of social justice scholars when the ProPublica report appeared was that the model’s true positive or likely-to-reoffend scores were equally accurate between white and Black individuals. However, its false positive scores, i.e., likely-to-reoffend predictions that were wrong, heavily disadvantaged Black individuals by way of their having to spend more jail time. Alternatively, its unlikely-to-reoffend (or false negative) ratings heavily advantaged white persons by way of their being awarded bond or parole [27,29,30]. Remarkably and as data scientists proceeded to demonstrate, it might be mathematically impossible to satisfy both fairness goals simultaneously—in the COMPAS case, a nondiscriminatory accuracy score plus a nondiscriminatory impact rating [31,32]. As discussed in Part II, this implies that clinician users and model developers might need to collaborate and agree on which of their comprehensions of fairness will be algorithmically overriding (e.g., predictive accuracy versus group error rates) and/or how to make certain adjustments in the model that satisfy their mutually negotiated vision of fairness.

This suggests a third and all-inclusive problem with fairness comprehensions. Developers of models that are presumably “fair” should be held accountable for their deliverables, and they therefore should be willing to open their models and data training sets to analytical scrutiny, such as the DHHS proposal mentioned above might require. Yet, some developers in the United States have balked at this suggestion of “fairness through transparency,” claiming that their models are proprietary and protected from public scrutiny [30,33,34]. Their pushback is largely prompted by their companies having heavily invested in research and development and fearing that their competitors will benefit from their cost outlays and be better positioned in the marketplace [35]. Thus far, courts and other regulatory agencies have refused to take a position on fairness through transparency, possibly at least owing to the immense lobbying pressures that large IT corporations can exert on state and federal legislators in protecting their business interests [36].

A fourth and very common recommendation for achieving fairness is by readjusting the model’s computational networks through re-weighting its sensitive or protected variables so that an historically oppressed group is treated “the same” as the majority or historically favored group. For example, one might evaluate the fairness of a model by changing the racial identity, age, sex, or residence location of members of the various sub-populations to see whether majority or historically advantaged group members would receive the same or a lesser level of benefits if they were counterfactually re-identified as minority persons [37].

This fourth approach to achieving fairness—which is sometimes labeled “fairness as demographic or statistical parity” [18]—stipulates that the likelihood of a positive outcome, like receiving a life-saving medication in times of acute medication scarcity, should be the same whether an individual is or isn’t in a protected group. It captures the essence of non-discrimination whereby equals—or people who are considered equals—are treated as equals [38].

The problem that fairness as demographic parity presents is when people do not present as equals in a morally obvious way, and moral disagreement arises about whether 1) they or their circumstances should be “equalized” and, if so, 2) what form that “equalizing” should take. Consider the following hypothetical scenario:

Case 1. A new and powerful strain of COVID has appeared. Hospitals once again find themselves without sufficient resources and begin developing ML models to help with triage-based care allocation computations. Early testing of the algorithms indicates they are easily able to distinguish patients who are economically better off and are usually in better pre-morbid health from ones who are poorly off. Not surprisingly, the models typically predict better outcomes and hence recommend more aggressive care for the former. Some model developers, however, argue that the models’ distributional weights should be adjusted to favor impoverished, especially homeless, patients. They believe that these populations have largely been victimized by historically unjust social structures and practices that explain their dire life circumstances, their heightened susceptibility to the virus, and their poorer prognosis. Other data scientists disagree, arguing that everyone is responsible for his or her station in life, and everyone should be treated similarly (or according to what they can pay for). Yet a third group of data scientists promotes an entirely different view: The algorithm, they argue, should promote social utility, not fairness. For them, the model’s allocational weights should be adjusted to favor essential workers like health professionals, food producers and distributors, transit workers, and sanitation personnel.

In this scenario, the first group argues that demographic parity should be achieved by, actually another, i.e., a fifth, definition of “fairness as distributive equity.” They believe it a social justice imperative to provide tangible benefits or advantages directly to groups who have historically suffered from unfair social practices and policies [15]. Those advantages would help to “equalize” their right to social goods as compared to the claims of majority group members. But they are opposed by another group who reject providing an outright benefit to the less well-off and instead favor a sixth notion of fairness: as “distributive equality” where everyone is treated the same [39]. Whereas the equity group might insist that a predictive model favorably weight a protected category like income or race—because they may token social inequity—the equality group might want to delete or de-emphasize such variables and simply have the model predict an outcome (and its associated treatment recommendation) on purely clinical criteria. The equality group can argue that the equity group advocates policies that are unfair to other socioeconomic groups and might repudiate any model that is weighted to favor the less well-off [40]. Notice, too, the third group problematically believes that fairness should not figure prominently in the algorithmic weighting affecting allocation as they favor a social utility approach.

An attempt to remediate the equity versus equality debate invites a seventh fairness characterization as “equal opportunity fairness.” Equal opportunity fairness, at least as it has developed in the United States, repudiates the practice of algorithmic models being re-calibrated to give certain job, credit card, or medical school applicants a direct or immediate algorithmic/mathematical “boost” that advantages them over others [39–41]. Indeed, United States courts, especially the U.S. Supreme Court, have ruled that certain direct and immediate advantaging practices, such as hiring quotas or a seat in a medical school class that favors members of an historically disadvantaged group, may be discriminatory to other competing groups [40–43]. Instead, this fairness approach focuses on barriers to opportunity, like sub-standard education, housing, or poor access to health care, and seeks to improve these health determinants so as to equalize “opportunity of access” for everyone. As such, this approach might not affect the moral decision making of model developers because algorithms do not directly engage in social justice efforts that seek to remediate various social determinants of health like income, race, education, etc. In any event, continuing public health research on health inequities suggests the ongoing failure of opportunity remediation on health disparities in the U.S., e.g., that disease rates for diabetes, hypertension, asthma, and stroke are worsening in Black populations as compared to whites [44].

How, then, might a morally conscientious ML development team and a clinician user group that is collaborating either in a research or business/purchasing capacity decide on all this, which suggests an eighth approach to characterizing fairness, namely, “fairness as the outcome of democratic processes” [45–47]. Perhaps the most popular strategy for achieving a democratic voice bearing on fairness in algorithmic development is to recruit culturally diverse teams that will periodically examine models for bias that then “enable the platform designers (and others) to identify ethical, and sometimes legal, issues and values that were not foreseen at the design stage, and to take additional measures to address these properly.” [46] But related problems immediately arise. First, even if development team members display some modicum of cultural diversity, there is no guarantee that they would bring an ethically sufficient range of ideological diversity to problems like those presented by the above. Furthermore, even if they did, we would still be without a way to respond to questions like what metrics should inform us as to when “diversity” has been achieved in a team, how one defines the essential properties of “culture” to ensure they are adequately represented among team members, and, most concerning, how to settle fairness disputes among team members.

What the above suggests—and its fairness characterizations and associated problems are summarized in Table 1—is that although pursuing a universal or one-size-fits-all definition of fairness is intellectually and pragmatically attractive, the elusive nature of that definition strongly suggests that fairness only makes sense in a particular context with a specific use or application. Indeed, the dense contextuality of fairness might partially explain the paucity of regulatory oversight in the United States, as well as why “the field lacks a design methodology to help AI design teams of members from different backgrounds brainstorm and surface potential fairness issues during the design stage.” [45, p.32].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Fairness definitions mentioned in this paper and their barriers.

https://doi.org/10.1371/journal.pdig.0000386.t001

Nevertheless and despite this dizzying array of fairness manifestations, no one in the model developer to user chain of operation wants to deploy a blatantly unfair model, at least because it could cause immense harm through misdiagnoses, inappropriate treatment recommendations, or civil rights violations like denial of benefits or entitlements. In addition to the harm any of these adversities might cause, they can also expose developers and clinicians to lawsuits, federal investigations, and along the way, considerable reputational loss [48].

Consequently, although it appears reasonable to task model developers with creating fair models—since they collect and curate the data, develop the model’s computational system, etc.—we endorse Xivuri’s and Twinomurinzi’s recent work on “the need to involve the society affected by the model in the development process and establish a platform for all AI stakeholders to participate.” [49] Indeed, the collaborations they envision go beyond just a developer-clinician collaboration and require “bring[ing] customer collaboration into the design development and deployment of responsible AI. Engage customer advisory councils, drawn from a broad cross-section of our customer base in the product development lifecycle to gain feedback around our development themes related to AI and ML.” [49]

Part II of this essay discusses how various moments in these collaborations might proceed, especially in instances where model developers, clinician users, and the lay public might disagree over a model’s fairness and, hence, need to create or negotiate justifiable conceptions of fairness that inform the model’s computations.

Moral character and fairness in ML development

The above suggests that an absence of regulatory guidance that might provide reliable moral co-ordinates for fairness analyses combined with a bewildering array of fairness definitions or characterizations explain why fairness might figure as an afterthought. That afterthought status can be present not only among professional ethics groups but among model developers themselves, whose data scientists typically hail from engineering training that has paid little attention to it [50]. As noted above, however, no model developer desires to create an obviously unfair model and as we discuss below, many model developers will go to considerable lengths to ensure that their models are fair. Indeed, some model developers have been keenly concerned about algorithmic fairness for years and have developed a host of bias mitigation tools to improve it such as IBM’s AI Fairness 360, Microsoft’s open-source toolkit Fairlearn, the University of Chicago’s Aequitas, Google’s TCAV, or Oracle’s Skater [51,52].

Despite these fairness interventions, however, clinician users might nevertheless wonder how committed model developers are to algorithmic fairness and social justice. Kate Kaye, for example, has noted that model developers occasionally use fairness tools “at the end of a development process, as a last box to check before a product or machine-learning model is shipped.” [53] The moral ideal, of course, would be to contemplate and analyze biasing traps or pitfalls at the inception of a project and then maintain that critical posture throughout the model’s development and deployment. But as Tricia Griffin and her colleagues recently reported on their study of moral agency in data scientists, one of her interviewees noted that “we’re not wondering if what we’re doing is ethical or not” [50]. That experience could then be compounded by the way many data scientists working on a model can feel morally disengaged from its intentions and impact, especially if management appears disinterested about fairness in the model’s project design [54]. Of course, a third factor that might dilute fairness considerations in model development is that model developers often work for companies that are largely motivated by profit maximization pressures that can subordinate fairness considerations to returns on investment. More than one of Griffin’s respondents noted how the promise of money, fame or “their personal brand” could and sometimes do obscure or taint workers’ ethical sensibilities [50].

A fourth factor, however, that calls for the kind of thoughtful collaboration on fairness that Xivuri and Twinomurinzi [49] encourage is perhaps the most interesting: When specific use cases present moral perplexities that invite multiple stakeholder perspectives on whether or not a model is fair or unfair. For example, consider the following three hypothetical cases:

Case 2. Patients who are scheduled for clinic appointments and then don’t appear can represent a significant revenue loss to an institution. At a meeting with clinical leadership, a hospital’s informatics group announces that it has been working with a commercial AI developer to create an ML model that predicts these “no-shows.” Once the model predicts a likely no-show, it will then double-book the slot so as to ensure that worker productivity (and related hospital income) is not lost. Although double-booking is not uncommon in the United States, some personnel at the clinic oppose the plan, pointing out that when the model is wrong, clinicians’ workloads will be stressed, patients’ wait times will increase, and their clinical visits will be (problematically) shortened. Clinicians note that this will especially and unfairly affect less well-off persons who are more likely to miss appointments for various reasons, often beyond their control. The model developer and the hospital’s executive leadership robustly disagree. They argue that if a certain segment of the hospital’s patient population is statistically likely to miss appointments, then they should bear whatever burdens result. Moreover, executive leadership argues that the hospital’s precarious financial position demands that all potential sources of revenue loss be mitigated as much as possible and that the financial survival of the hospital trumps the inconvenience of a longer wait time for certain patients. [55–57]

Case 3. A mortality prediction model has come into widespread use across the United States. Originally used to identify patients who need more intense or a frankly different program of care, ethically astute clinical personnel have begun to suspect that the model’s mortality predictions have become self-fulfilling prophecies. That is, once the model delivers a high probability prediction of an inpatient’s demise, some clinical personnel aggressively initiate conversations on discontinuing or withholding further intensive care measures with the patient or his or her family members in the hope of persuading them that such care would be “futile.” Some staff are deeply distressed by this, which they feel especially discriminates against the less well-off who oftentimes have meager, if any, health insurance. They argue that the model can reflect abundant historical bias: The life circumstances of less well-off persons dispose them to heightened illness acuity while their reduced financial circumstances might have historically prevented their accessing needed care. Furthermore, the medical records of less well-off persons are known to be particularly prone to inaccuracies, which can reduce a mortality prediction model’s accuracy. Nevertheless, other staff argue the model’s value in conserving revenues and reducing caregiver stress by preventing “futile” care allocations [58,59].

Case 4. A mental health facility has been approached by a generative AI developer whose model can considerably reduce the staff’s time spent in documentation. The model can generate patient summaries, clinical notes, treatment recommendations, and insurance billings as well as respond to patients’ questions and concerns. While the model is attractive to the facility’s business owners in terms of its relieving staff from these tasks and conserving revenues, some clinicians are fearful of confidentiality breaches. They argue that mental illness can be very stigmatizing and that the developer who owns the model will have access to extremely sensitive data. If that data is breached, released without warrant, or inadvertently placed on a public database, the psychological harms to the facility’s patients could be devastating. The facility’s shareholders and the clinical staff find themselves at an impasse over whether the model’s use should be adopted [60,61].

Cases like these illustrate the need for ML developers and at least their clinician users to exert as much moral insight and perspective as possible so as to identify relevant fairness/unfairness “flashpoints.” For example, should a clinic or hospital’s leadership even consider a double-booking algorithm since it will impose discomfort and burden on certain patient groups? Alternatively, if the clinical entity’s financial survival is truly at stake, developers and clinician users might collaborate on modifying the model. For example, the model might be reconfigured to double-book likely no shows who only require short visits or visits that might not require a physician’s attention [56]. Or the model might “examine” the clinic’s weekly workflow and double-book patients during predictably light periods or down-times. It might even build in a few minutes between all that day’s patient appointments to give clinicians a rest or some extra time that can be used to fully attend to patients who have been double-booked [57].

Another important factor in model developer-user collaborations is how model developers should be transparent as to how their datasets are created, curated, validated, implemented, and shared, as a failure to do this would taint any convincing justification that the model is “fair” [10,62–64]. Ideally, developers would exhibit heightened ethical awareness by producing algorithms whose “fair” outcomes are particularly sensitive towards minority groups or the sub-populations affected by the model [17]. They would be willing to explain and justify why a particular data set was used and what bias mitigation strategies were deployed and why. Developers and users might subscribe to policies like the United Kingdom’s Medicines and Healthcare Products Regulatory Agency’s stipulation: namely, that all entities selling products that leverage artificial intelligence (AI) demonstrate that their training and validation sets use data that are adequately representative of all patient populations in a user’s catchment area [11]. Or they might deploy quality assessment tools like the quality assessment of diagnostic accuracy studies (QUADAS-AI) to evaluate data sets for risk of bias [65].

In this regard, case study 3 reflects the challenge of inaccurate data, especially among socioeconomically marginalized populations. As an alternative, rather than use the model to urge the withholding or discontinuation of care straightaway, a model developer-clinical user collaboration might rethink the model to identify these (likely) terminally ill patients for more aggressive but compassionate end-of-life conversations [59]. The model might recommend and facilitate a referral to palliative care clinicians and even suggest to them certain talking points, relevant to the patient’s social, medical, or spiritual history, around which to build a compassionate end of life conversation.

Extending beyond the model developer-clinician user collaboration, the ethical characteristics of a commitment to fairness inevitably entail a willingness to engage in public conversations. For example, the AI Policy Forum that is convened by the MIT Schwarzman College of Computing brings together scientists, technologists, policymakers, and business leaders to discuss the societal challenges of artificial intelligence [66]. Similarly, The World Economic Forum has convened a Global Future Council on Artificial Intelligence whose participants identified practical interventions for companies to employ to assure AI Fairness [67]. And the United States FTC has held hearings on predictive analytics and guidelines for businesses using artificial intelligence [9,68]. Some commentators would like to see evaluations and reports resulting from meetings like these being directed to a newly created national office, perhaps like the FDA, which has a particular interest in human welfare especially concerning the less well-off [29]. Somewhat like the U.S. Centers for Disease Control and Prevention, that office might issue advice or directives to model developers and users on best practices.

These kinds of public conversations would be especially valuable in instances such as case study 4, which implicates serious confidentiality risks. Relevant questions, at least in the United States, might involve a public debate on whether or not patients should be informed that the confidentiality of their information cannot be guaranteed despite promises of anonymity [60,61]. A public airing of views in a case like this might provoke legislative action that would legally hold model developers as a “business associate” of the mental health facility. In American (HIPAA) law, a legally designated business associate is as obligated to protect the confidentiality of information which it receives as are the clinicians who source the material [69]. Furthermore, a clinician user group might only want to contract with model developers who can demonstrate robust and elaborate data security precautions.

In any case, the task of maintaining a sensitivity towards fairness will continue as long as ML models evolve and are deployed. New technology, new practice standards, and shifting healthcare demographics will persistently affect algorithmic accuracy and value over the lifetime of a model. Just as a one-size-fits-all fairness definition seems unlikely, a one-time, cross-sectional assessment and remediation of a model will not ensure the model’s fairness over its lifetime.

Towards a lifetime ethics audit of ML models

A model satisfying certain technical and ethical criteria at its launch does not guarantee its long-term value. Commentators largely agree that evaluative processes should be in place for the life of the model for various reasons: inaccuracies due to bias or programming error might unexpectedly materialize (or the model doesn’t perform as intended); technologies and their operational environments might change; demographic patterns or patient populations might shift in ways that compromise accuracy and fairness; and never-ending system upgrades will require a continuing work relationship between model developers and users [29,68]. Indeed, after-market surveillance may enable increased trust in ML models in medicine, such as occurs in FDA long-term surveillance of drugs and medical devices in the U.S.

An element in a lifetime audit oversight intervention, for example, might be “transfer learning.” Transfer learning is a method that begins with an already trained model, say in image recognition, and enables a newer model to recognize original images that are re-produced via new imaging technologies [70]. The objective is essentially to repurpose an older model so as to keep pace with technological advances that require a newer one. Its key advantage is reducing costs because retraining a model from scratch can be prohibitively expensive. Nevertheless, there will come a time when a model reaches the end of its life, e.g., the training data are too outdated, model users require more complex computations, the physical infrastructure no longer supports the model, etc., and the model can no longer be transferred. Such moments are not without moral dimensions as clinician users will need to decide when the needs of their patients require the abandonment of older models.

The impact of machine learning models on patients introduces another consideration in the lifetime audit: the need for impact assessments. Such assessments would evaluate the model’s impact on users, its risks, the transparency and accessibility of its data sets, and its explainability and generalizability (and their limits) [29]. Ideally, risk assessment efforts would begin at the earliest stage of the model developer-clinician user relationship, as the above anticipates the distinct possibility of disagreement over whether a model’s design or outputs are fair or unfair. Thus, Krishna, et al underline the need for strategy and governance that include “principles, policies, and standards; roles and responsibilities; control processes and procedures” extending through all facets of the model development throughout its life cycle [71]. Important considerations that an impact assessment would include is a thoughtful analysis of the kinds of risk the model might pose to the end user; the degree to which senior management acknowledges and appreciates the presumptive risks involved; and a governance structure that adjudicates and manages the risks emanating from the model.

Importantly, impact assessments must include the lay public’s impression and assessment of the model’s purpose, the conditions and limits of its use, how its data are collected, curated, and shared, and how its deliverables are economically and ethically received [62,63,72]. However, this would initially require a significant effort to educate the public so that they be informed consumers and critical voices of ML models—a challenge that thus far remains sadly unattended [73].

All of these recommendations are ambitious, especially given the nascent stage of ML use in healthcare facilities, and all of them admit complex sub-tasks whose details will likely precipitate a host of valuative conflicts reflecting Western cultures’ moral pluralism. But to the extent that ML models may become standard of care technologies, they need to be vetted with an eye toward ethical uniformity bearing on fairness and will need to be monitored for unanticipated biases throughout their lifetime. Such an audit plausibly applies to any of the cases discussed above.