Inclusion criteria

Studies published in English were included if they met the following three criteria.

1. The studies were economic evaluations, including cost-effectiveness analysis, cost-utility analysis, cost-benefit analysis and cost-consequence analysis. They all look at costs in monetary terms and differ in how the outcomes of the interventions are measured and compared. Cost-effectiveness analysis compares the benefits (for one specific outcome) relative to the costs in alternative interventions [11]. The outcome selected could be narrow such as full-time employment rates or may be broad (such as number of life years gained). Similar to cost-effectiveness analysis, cost utility analysis compares the benefits relative to the costs, however, it translates the health benefits into a common metric quality-adjusted life years (QALYs) such that the value for money of different interventions can be easily compared. The results are shown in forms such as cost per QALY gained. Cost-benefit analysis, on the other hand, provides net benefits of an intervention (benefits minus costs) by converting all the outcomes into monetary terms (e.g., an additional working day is worth $350). Therefore, if an intervention costs $5,000 and increases two working days, the net benefits calculated in the cost-benefit analysis is $4,300. Cost-consequence analysis considers the impact on a broad range of outcomes but does not attempt to combine outcomes into a single metric, and simply presents the changes in costs and outcomes separately without comparing them relative to each other (e.g., costs = $5000, benefits = 1 life saved & 5% higher employment rate).
2. The study only included social interventions which occurred outside healthcare settings and were carried out by individuals who are not registered healthcare professionals. Adopted from the World Health Organisation classification of disability assistance and support [12], the types of interventions that may be included in our study are community support and independent living; residential support services; respite services; support in education and employment; community access, assistance animals; information and advice services; and other support to caregivers and the assistive devices and technologies. Further details of the types of services and their definitions are outlined in the S1 File.
3. The study focused on people with disability and people with health conditions or impairments that may lead to disability, including sensory/speech (e.g., hearing loss), neurological (e.g., multiple sclerosis), physical (e.g., incomplete use of arm), intellectual (e.g., development delay), cognitive (e.g., dementia), and psychosocial disabilities (e.g., autism) or their caregivers. The categories of disability included in this study are detailed in the S2 File. Although disability is usually defined as having functioning difficulties that last for six months or more [10], previous studies often did not make such a distinction. Therefore, the inclusion criteria were kept broad rather than limited to health conditions that were expected to last six months or more.

Search strategy

Different electronic databases were searched for studies published between 1 January 2005 and 31 August 2019, and an updated search was conducted to cover studies published between 1 September 2019 and 20 October 2021. The databases included PubMed, AgeLine, Business Source Complete, CINAHL Plus, Communication and Mass Media Complete, EconLit, and SPORTDiscus with Full Text. The reference lists of included studies were hand-searched to identify any further studies that met the inclusion criteria. We excluded studies prior to 2005 because recent studies were more likely to reflect better the current knowledge in the economic evaluation methods and be more relevant for current policy making.

The search strategy was developed based on the inclusion criteria listed above. Medical Subject Headings (MeSH) and keywords were used to indicate the type of economic evaluation, disability and intervention (see S3 and S4 Files).

Search results were managed using Covidence online software. At the importing stage, the software removed duplicates across the databases. Two authors independently screened titles and abstracts for relevance. The third author resolved conflicts if they occurred. The two reviewers then independently assessed the full text of the selected studies. Conflicting decisions were resolved between the two primary reviewers; when consensus could not be reached, the third reviewer acted as an adjudicator.

Data extraction

A data extraction form was used to extract data on the intervention information (study design and methodology, settings, detailed intervention, and control description) and economic evaluation information (outcomes and QALYs, study perspective in terms of the relevant stakeholders considered, and the cost considered [11, 13], sensitivity analysis, subgroup analysis).

Two authors extracted the key information independently, including the cost items, the type of economic evaluation, perspective, time horizon and sample size. The results of the extraction were then compared. Conflicting decisions were resolved where possible between the two primary reviewers. The third reviewer acted as an adjudicator if conflicts were not resolved.

Information less likely to be subject to differences of opinion (i.e. interventions, countries, age and population) was extracted only by the first author. The second author completed an independent extraction of these items on a 10% random sample. A third reviewer independently compared the results of these two processes. Dual extractions were not performed in these areas because no more than 10% of substantial errors occurred [14].

We assessed the reporting completeness of studies using the 24-item Consolidated Health Economic Evaluation Reporting Standards (CHEERS) checklist [15]. CHEERS was intended to guide rather than assess reporting thus there is no explicit rating scale. The checklist was adapted by the authors to categorise each CHEERS item into: “Complete”, “Partially complete”, “Not complete”, and “Not applicable” to assess the reporting completeness of each CHEERS item–see the S5 File for further details. Two authors carried out the CHEERS reporting completeness assessment independently for all included studies. Conflicts were firstly resolved between the two authors. If no consensus was reached, a third author adjudicated.

Scope of the economic evaluations

Fig 1 shows a flow diagram of the process applied in study selection. In the initial search, we identified 1,630 articles and selected for inclusion 20 articles. In the updated search covering the period two years after the initial search, we identified 183 articles and selected for inclusion one article. Three articles were added from a hand-search of the references of the included articles. We included 24 articles in the end.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. A PRISMA flow chart of studies included and excluded at each stage.

https://doi.org/10.1371/journal.pone.0278930.g001

We first describe the scope of the evidence. Among the 24 studies, six intervention areas were identified with 10 focused on employment interventions [16–25], and six on community support and independent living interventions [26–31]. Three studies related to support for caregivers [32–34], two studies on anti-stigma campaigns that may reduce discrimination [35, 36], another two on residential support [37, 38], and one on assistance animals [39].

Of the various types of disabilities, nearly half of the studies focused on interventions targeting people with mental illnesses [16–19, 21, 24, 32, 33, 35, 36]. Four studies were about people with dementia [27, 28, 31, 34], three about people with musculoskeletal disorders [22, 23, 25], and two about people with intellectual disabilities [37, 38]. There was only one study each about stroke [26], cerebral palsy [29], multiple sclerosis [30], vision loss [39] and autism [20]. Further details can be found in S2 Table.

As shown in Table 1, the volume of studies has not increased over time. Six studies were conducted between 2005 and 2009, 10 between 2010 and 2014, and eight between 2015 and 2021. However, the type of economic evaluation conducted did change over time. Even though some health funding bodies recommend cost-utility analysis (in which the outcomes are commonly measured using a single metric that incorporates both quantity and quality of life, i.e, QALYs) for comparability, it has not gained popularity until recently. In the early years of 2005 and 2009, only one study conducted a cost-utility analysis [32]. The other studies were cost-benefit analysis (i.e, converting outcomes to monetary terms) [16, 38, 39], cost-effectiveness analysis (i.e., using natural unit outcomes) [32, 34], or cost-consequences analysis (i.e., reporting outcomes and costs separately) [37]. Between 2010 and 2014, cost-effectiveness analysis became the most popular economic evaluation type that eight out of ten studies published in this period used either cost-effectiveness analysis alone [28, 35] or further alongside with cost-benefit analysis [18, 23, 36] or cost-utility analysis [20, 22, 23, 33]. From 2015 to 2021, cost-utility analysis became the dominating economic evaluation method, and seven out of eight studies used this method either alone [25, 29] or with cost-benefit analysis [19, 21, 26, 27, 31]. We also found that among the 24 studies, three studies only performed cost-utility analysis [25, 29, 30].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summary of the key characteristics of the included studies.

https://doi.org/10.1371/journal.pone.0278930.t001

Methodology assessment

Perspectives and costs.

Economic evaluations are often conducted from a particular perspective, which defines what cost implications are considered relevant for the funding decision. Some decision makers may want to ignore the cost implications for some payers. For example, if a health sector perspective is adopted, only costs incurred by the health sector are considered. This is important because whether costs are included ore excluded can change the perceived cost effectiveness of the intervention. The cost items and the payers (when stated) are documented in Table 2. The first two columns present the first author of the study and the type of intervention. The area with the dots indicates the cost items included in each study, and the area with the tick indicates the payer. For example, Squire et al. [22] (study #1 in the table) evaluated an employment intervention from the societal perspective. The medical and non-medical cost items were listed with the green dots indicating the cost items that had been included. From the columns to the right, we see that the author considered costs borne by the government and the employer.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Cost items and payers by perspective.

https://doi.org/10.1371/journal.pone.0278930.t002

We firstly assessed whether the studies reported the perspective and what perspectives they adopted. In our study, we found that nine studies did not mention the perspective(s) adopted [17, 20, 26, 34–39]. Among the studies that reported the perspectives, very few reported their perspective to the health sector perspective separately which is the preferred perspective by some government funding bodies. Ten studies adopted the broadest societal perspective [19, 21–23, 25, 27, 29, 30, 32, 33]; eight studies reported taking the health sector and social service perspectives [18, 20, 22, 24, 27, 30, 31], four reported taking the health sector perspective alone [22, 28–30]. Two studies also took the social insurer perspective [19, 23]. One study each also added the employer [22], voluntary [32] and individual [32] perspectives. We found that studies on community support interventions were more likely to adopt a health sector perspective, which does not consider productivity costs.

Among the 10 studies taking the societal perspective, all of them included intervention costs (n = 10), but it was not always clear who the payer was or was likely to be. All 10 studies included general practitioner costs [19, 21–23, 25, 27, 29, 30, 32, 33]; only three studies [25, 27, 30] explicitly stated that hospital emergency costs were included. Productivity costs (related to work) for people with disability were included in seven studies, but only two included productivity costs for caregivers. We can see from Table 2 that government costs were more frequently considered than costs to others, even when the societal perspective was being taken. In only three studies were cost implications for charities, insurers, employers, or individuals considered [22, 25, 32].

Among the studies that adopted the perspective of the health sector and social services (n = 8) or the health sector (n = 4), the medical costs included were highly variable. More than 75% included the GP costs [20, 22, 24, 25, 27–32], hospital in-patient costs [18, 20, 24, 25, 27–31], hospital outpatient costs, [18, 20, 22, 24, 25, 27–32] and allied health costs [18, 20, 22, 24, 25, 27–32]. Only five of the eight studies reported social services costs when taking the perspective of the health sector and social services [18, 20, 24, 27, 31, 32].

Among the six studies with a time horizon longer than a year, only three [20, 32, 39] applied a discount rate to reflect preferences for benefits today rather than in the future.

Reporting completeness

Information on the percentage and number of items with complete reporting on the CHEERS checklist is provided in the last column of the S1 Table. Eight studies (33%) complete at least 80% of the relevant reporting items [16, 19, 23, 27, 28, 31–33]. In general, the reporting was more complete for studies published more recently and those focusing on either employment, community support or caregiver support interventions.

The key attributes of each CHEERS item and the number of studies that complete, partially complete, do not complete or are not applicable against each item standard are presented in the S2 Table.

Findings and discussion

Economic evaluations are widely used for assisting funding decisions. However, we found it challenging to use the evidence included in our systematic review to compare the relative cost-effectiveness of different interventions and thus serve the purpose of assisting funding decisions for social interventions for people with disability. Two main concerns should be noticed.

First, the evidence was limited in number and scope. To allow governments to make evidence-based funding decisions, a broad picture of cost-effectiveness of interventions that cover support to different life aspects and people with different restrictions should be available. However, our findings show that half of the evidence were focused on people with mental illnesses, which was a much higher percentage than the estimated 1 in 8 people with mental disorders globally [40].

We are also aware that some interventions targeting a range of social determinants of health for people with disability (e.g., information and communication, education or physical environment) [41] had not been evaluated. Without evaluating different social interventions for people with disability and then comparing the results, policymakers may not be likely to gain an idea of which areas should be prioritised to be funded when resources are limited.

Second, the existing evidence is highly variable in how both outcomes and costs were measured and thus lacks comparability. For decision makers, it is difficult to decide if they should prioritise an intervention that costs $40,000 per QALY or the one that costs $3,000 per additional day of working. Although cost-utility analysis have gained more popularity in recent years, the widely used health utility instruments primarily focus on physical health and thus are likely to omit the broad benefits that go beyond health [42]. For example, transitioning from institutional to community living may improve not only mental health, but also overall well-being through a greater sense of independence and control over one’s life. There has been heated debate about whether we should use QALYs to quantify health outcomes in disability interventions [43]. One primary concern is that the QALYs calculation may devalue interventions that do not aim to bring back “full health or functioning” to the person but instead support them in achieving high levels of wellbeing despite their current conditions [43]. People with disability may be highly satisfied in reaching a particular functional level instead of paying the additional mental, physical and emotional burden needed to achieve full functioning (or full health). In this case, the interpretation of the cost-utility analysis results that use QALYs for social interventions for people with disability may be misleading.

We found that only two studies included were based solely from the perspective of the health sector. This may be attributable to the fact that the costs of such social interventions are likely spread to sectors other than health. However, for the studies which adopted a wider perspective than health sector, we found that the cost items included were highly heterogeneous. For example, some studies included the productivity costs of informal caregivers in the societal perspective while others did not, which could potentially impact the total amount of the costs as cost to informal caring may be substantial. In most of the cases where some items may be relevant but missing, we are not sure if they were excluded because they had no impact [32, 33] or because there was no data [16]. Therefore, even when studies presented similar results (e.g., $7,000/QALY), it remained unclear whether the interventions had the same value for money.

The above issues indicate that after a decade of Drummond, Weatherly and other authors publishing their concerns on economic evaluations, similar challenges remain in conducting economic evaluations for social interventions for people with disability. Unlike clinical interventions for which QALYs have become the gold-standard measurement for outcomes and relevant costs are often confined to the health sector [44], social interventions often consist of a mixture of different types of services, and there is less consensus on how to measure the outcomes and costs involved consistently. The lack of consistency across the studies included in this review makes it hard for policymakers to compare and draw clear conclusions on the relative cost-effectiveness of the interventions. More economic evaluation studies are required to inform priorities for funding disability interventions; meanwhile, more considerations and guidance are needed to ensure comparability across studies.

Recommendation for future research

Given the challenges and concerns discussed above, we believe that the research community should endeavour to further develop ways to ensure comparable evidence is available to inform evidence-based policymaking. Recommendations of areas for improvement are highlighted as follows:

Outcome measures: Researchers should justify the choice of measures to be chosen for the calculation of QALYs and non-QALYs outcomes. In the short run, when doing economic evaluations for social interventions for people with disability, researchers should consider using both the QALYs (in cost-utility analysis) for comparisons and non-QALYs outcomes (in cost-effectiveness analysis) if the available outcome measures to calculate QALYs are not likely to capture the benefits of the interventions. Consequently, when policymakers review the evidence, they can both compare the cost-effectiveness across different interventions and be aware of the benefits beyond health. In the long run, consideration should also be given to developing and using a validated disability-specific well-being instrument that could be more sensitive and covers broader outcomes, including important life areas such as independence, safety and sense of belonging with less emphasis being placed on their levels of functioning. In this case, researchers could use this well-being instrument to produce evidence that reflects intervention impact beyond health and is comparable across studies.

Perspective: The health sector perspective is recommended by some government funding bodies for economic evaluation. However, as we found in this study, the costs and benefits of social interventions for people with disability are most likely to go beyond health. As a result, we recommend that researchers consider incorporating both the health sector and social perspectives when doing economic evaluations in this area. The former is the preferred perspective by some government funding bodies, while the latter would capture more costs and impacts. Researchers should also be clear about the likely impacts on costs in sectors other than health so that policymakers could understand the implications of taking a narrow health sector perspective.

Costs: Researchers should make explicit which costs were considered, and why some costs that maybe relevant were not included. Using a comprehensive standardised costing template can help collect costing information, report on costing decisions and justify the costing perspective in a more consistent way. If researchers reported zero costs or stated and justified non-relevant cost items, this would significantly enhance transparency and comparability in cost reporting. Future studies should also consider costs particularly relevant to the disability sector, such as costs of informal caring or those incurred by charities/non-government organisations, to reduce the likelihood of cost-shifting by only considering a government cost perspective.

Strength and limitations

This study is the first systematic review of the economic evaluations of social interventions for people with disability. It describes the heterogeneity of the methodologies used in the current economic evidence base and how the evidence is distributed across types of disability, intervention areas, and geographic locations. The review was undertaken using a comprehensive search strategy following a registered protocol. While this review provides critical evidence to help inform future research in this field, there are a number of limitations.

First, studies not in English were excluded. This could lead to the potential exclusion of studies, omitting particular relevant studies from non-English countries. Second, while we included broad search terms to capture as many relevant studies as possible, many studies did not report whether health conditions and/or impairments were likely to last more than six months. Therefore, some studies may have included people with an impairment but not disability. However, because the health conditions and impairments mentioned are closely related to disability, the economic evaluations in these studies are still likely to be valuable for decision-making in this field.

Lastly, this review evaluated what evidence was (and was not) considered in an economic evaluation reporting. It did not attempt to evaluate the quality of the evidence used, for example, the potential for bias in the estimated effect sizes. While most of the included economic evaluations were conducted alongside a randomised control trial, and the quality of the trial can also affect the quality of the economic evaluation, evaluating this dimension was out of scope for the current review.