Source of data and participants

Data for the development of the prediction model was from the Adult Social Care department of Manchester City Council, administrative data owned by the local authority as a public body, not research data. The Performance, Research and Intelligence team of the council harness the data generated from adult service users’ contacts with the social care system, alongside data from other council departments and local National Health Service (NHS), able to be accessed by them under devolved arrangements between health and social care [16]. There are around 70,000 contacts with Adult Social Care per-year in the city, from a population of over 424,992 adult residents. The data contain the social care information on all adults who access services in the Manchester City local authority area, 32 constituency wards comprising the central urban district within the Greater Manchester region, which covers 10 local authorities in Northwest England.

The study was reviewed and approved by an institutional review board (ethics committee) before the study began (Health Research Authority, Social Care REC: Ref: 23/IEC08/0012). University research staff were granted access to the dataset (from 31/08/2023) in the council’s secure environment for analysis (researchers went to the data, the data did not come to them), complying with established practice in this type of data access for research [17]. Authors had no access to information that could identify individual participants (data subjects) during or after data collection.

We identified a cohort of older people, aged 60 years and above, from the dataset who had initial contact, care assessments [1], or reviews from the council between 1^st April 2021 and 31^st March 2023, inclusive. Those under Deprivation of Liberty Safeguards [18] were excluded, due to their non-capacity to consent to council privacy notices concerning access to their data. The index date was taken as the date of first contact assessment for all individuals, with their data followed up until they had the outcome of interest or the end of the cohort period, of 31 March 2023, whichever occurred first.

Outcomes

The primary outcome was whether older people remained at home (i.e., they were not permanently admitted to a care home) at the end of the cohort period, i.e., as of 31 March 2023. Remaining at home and not entering permanent residential or nursing home care is a key marker of success for social care at home [19] and thus was used as an indicator of older people’s strengths in mitigating any threats to their wellbeing. We also re-ran this model with this outcome reversed, investigating predictors of permanent placement in a care home, an indicator of vulnerability and threats to wellbeing, through the cohort period. This variable was the obverse of the primary outcome, in that it reflects an adverse outcome, albeit entering a care home can be a positive choice for older people and their families. Of all care home admissions in the council dataset, those where the admission was permanent and not temporary, i.e., for respite after hospital discharge or during COVID-19 aftercare arrangements, were used to indicate this outcome of interest.

Candidate predictor variables

We selected, a-priori, potential candidate predictor variables that we hypothesised would be linked to adverse outcomes, like care home admission, or to be indicative of potential threats to older people’s wellbeing, and from the literature [9,20–23]. These 27 candidate data items (86 predictor parameters) were ones for which we first aimed to investigate whether variables were available in the local authority systems. They are summarised in Box 1. We defined these variables according to data recorded at, or prior to the index date of each individual (i.e., those variables that are available for the first assessments individuals have with the local authority Adult Social Care department). We determined which of these variables were available and could be extracted into a database for analysis, along with reasons for non-availability, through discussions with council data analysts. Additionally, we collapsed categories of some variables in cases of low levels of a categorical variable, and to permit more parsimonious model estimation. Our choice of variables to take forward into the final model analyses were also governed by whether they would be available, or relevant, at the point the prediction model would be applied in practice. For example, service use during the cohort period is a response to circumstances identified at index assessment (in other words, an outcome), so is not relevant here in predicting future needs. In Box 1 we also identify the items that were taken forward as candidates for final model development/analysis, after we had completed this process of combining categories and removing variables irrelevant to prediction.

Sample size

As the dataset created was a fixed sample size, calculations were used to determine the maximum number of candidate predictors (prediction parameters) that could be considered for model development (i.e., prior to any data driven predictor selection) [26]. These calculations were performed in the R package pmsampsize [27]. For inputs into these calculations, we used a conservatively proposed c-statistic of 0.7 (i.e., level of discrimination predictive performance that we expected the models to reach), a target shrinkage of 0.9, and an outcome prevalence of 0.96 (remaining at home) and 0.04 (those in care homes) taken from our development data.

For our fixed available sample size (20,218) the sample size calculations indicated we could consider a maximum of 46 candidate predictor parameters (which results in a minimum required sample size of 19,770 and 20,064 for the primary and secondary outcome, respectively). In our final analyses (Tables 1 and 2) we eventually considered 31 candidate predictor parameters.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Prediction model of ‘remaining at home’ (as opposed to entering care homes).

https://doi.org/10.1371/journal.pone.0328330.t001

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Prediction model of risk of care home admission (not ‘remaining at home’).

https://doi.org/10.1371/journal.pone.0328330.t002

Missing data

The percentage of missing values across the variable categories included in analysis varied between 0 and 76%. S1 Table in the Supplemental Materials gives the missing data rates of each variable. Many older people had no scores for some variables, as unless the local authority progressed their initial contact to a full Care Act [1] assessment, these variables were not measured. Data was inspected using graphical plots generated by the VIM 6.2.2 package [28], no pattern of missing data differentiated any socio-demographic group of older people. We used multiple imputation to analyse five multiple imputed datasets. The multiple imputation procedure was repeated in each iteration of the cross-validation loops. Methodologists currently regard multiple imputation as the preferred technique for management of missing data because it improves accuracy and statistical power relative to alternative missing data techniques. Graphical checks were performed on continuous variables to observe the relationship between each predictor and outcome variable. Incomplete variables were imputed under fully conditional specification, using the default settings of the mice 3.0 package [29]. Fully conditional specification was used, as in the mice package an imputation method is chosen dependent on the measurement level of each variable and a fully conditional specification is considered suitable where linear relationships are observed between each continuous predictor variable and the outcome variable. Visual plots were inspected to ensure each variable had achieved acceptable imputation. The parameters of substantive interest were estimated in each imputed dataset separately and combined using Rubin’s rules [30].

Statistical analysis methods

As recommended [26], we used all the data for model development and internal validation using cross-validation. This is an iterative process where the full dataset was split into five random folds of equal size. In each step, a fold was left out with the models developed on the remaining folds that was then validated on the excluded fold; this process then repeats at each stage.

For model development, two models were developed. First, we fitted a logistic regression model to the binary outcome of whether older people remained at home versus not (i.e., admitted permanently to care home or other care facility) within two years of the first index contact. Second, we fitted a logistic regression model to this outcome (reversed), treated as a binary outcome of placed in a care home (vs. not placed in care home) after first index contact assessment. For all modelling, we considered all covariates simultaneously as main effects and model assumptions were tested post-estimation.

For validation of each model (in each cycle of the cross-validation process), predictive performance was assessed using metrics of calibration (agreement between observed and expected outcomes) and discrimination (ability of the models to differentiate those who will experience the outcomes from those who will not). We provided calibration plots, assessments of calibration-in-the-large and calibration slope, and c-statistics for discrimination [31]. All performance results were summarised across the cycles of the cross-validation processes, including calculation of 95% confidence intervals. For graphical summaries of predictive performance (calibration plots), we report the calibration plots for each of five iterations in the cross-validation analyses separately. Calibration plots show the observed outcome proportion against the expected outcome proportion from the model, with a perfectly calibrated model having a calibration plot along the y = x line.

All analyses were performed using R version 4.3.1 “Beagle Scouts” [32], along with the packages: broom [33], car [34], dplyr [35], forcats [36], mice [37], predRupdate [38], readxl [39], tidyverse [40], VIM [41], and writexl [42].

Overall study cohort

There were 20,218 older people overall in the study cohort who had received at least a contact assessment from the local authority with many (75%) going on to receive subsequent care needs assessments and reviews.

Baseline characteristics

Table 3 shows the baseline characteristics of the cohort. The mean age of older service users was 77 years. The majority (58%) were female and White British/Irish (62%). Manchester City residents were living in areas of high deprivation (mean IMD of 44) with 40 per cent living as housing association tenants, and 38 per cent as owner occupiers. In terms of functioning at index assessment, there were significant difficulties with Activities of Daily Living experienced by older users in the domains of eating (78% had difficulties, unable to function without help), personal hygiene (81%), managing the toilet (57%), being appropriately clothed (70%), and maintaining the home (74%). Almost 30 percent had an unpaid family carer they could call on for support. In terms of the Primary Support Reason they approached the social care authority for, mostly this was for personal care (40%) and access and mobility (31%). Around 9 per cent of older users had a diagnosis of dementia.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Demographic and other characteristics of cohort (n = 20,218) for model development and validation. Values are numbers (percentages) unless stated otherwise.

https://doi.org/10.1371/journal.pone.0328330.t003

Predictor variables

The variables eventually included in the models were therefore ones for which information was available and relevant to the concept of identifying risks of future harms (or strengths) in a prediction model. Therefore, those unavailable, above, were excluded from the models. Also, all service use variables (Box 1) were excluded from the modelling, as they were not relevant for predicting future outcomes, as individuals’ service use is itself an (intermediate) outcome. Tables 1 and 2 outline all eventual predictor variables and their candidate predictor parameters included in the final model analyses.

Model evaluation

Tables 1 and 2 show the coefficients of our final models for remaining at home or entering care homes respectively. For the model of remaining at home (Table 1) the following were significant predictors: Ethnicity Asian or Asian British, Black or Black British (more likely to remain at home); Living in area of high deprivation (more likely to remain at home); living in residential care or ‘other’ tenancy – including unstable accommodation (less likely to remain at home); No access to carer (less likely to remain at home)’ Primary Support Reason of needing physical or mental health support (less likely to remain at home) or support to carer (more likely to remain at home). For the model of entering care homes (Table 2) the following were significant predictors: Ethnicity Black or Black British (less likely to enter care homes) or other non-white ethnicity (more likely to enter care homes); living in private tenancy (more likely to enter care homes); living in residential care or other tenancy (more likely to enter care homes); Activities of Daily Living (ADL) unable to eat and keep the home habitable without difficulty (more likely to enter care homes); Primary Support Reason being need for physical, mental health support or support with memory and cognition (more likely to enter care homes).

Both models were robust as to assumptions for logistic regression, i.e., binary, independence of variables, large enough sample size, linearity of predictors and log odds, and multicollinearity.

Model validation

Tables 4 presents the predictive performance results from the internal cross-validation process for each model. It can be seen, in the data reflecting the full models, that calibration-in-the-large/pooled c-statistic (calibration intercept = 0) is good and no evidence of overfitting (calibration slope inside range 0.9-1.1). Low Brier scores indicated good predictive accuracy.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Models of ‘remaining at home’ and entering care homes: Validation – calibration indices^a.

https://doi.org/10.1371/journal.pone.0328330.t004

S1 Fig in supplementary information, presents calibration plots of observed vs predicted probability for the models of remaining at home in each of the five cycles of the cross-validation process. These show that the models were well calibrated across the full risk range.

Developing a predictive index

Results from these analyses were shared with the council and in an interactive, forum-theatre style immersive workshop event run by our public involvement and engagement partners [43]. An audience of stakeholders (council data intelligence officers, service managers, social workers, charity members, researchers, and older people drawing on social care) deliberated on the form a social care predictive index could take. It was envisaged that this could be to aid professional judgement in individual cases and/or support service planning or forecasting at a population level. It could also put the older person in control of their care, in conversations with professionals, by using the data from the index as a feedback mechanism to guide joint decision-making. An index was thought to be broadly acceptable to stakeholders, but with some caveats. Basing an index on the ‘right’ data was viewed as important. It was thought professionals must “read into” the tool the “right things”, which should include such data as: age, housing situation, how much support older people need and details about their health. There was concern and a little distrust of data and how it might be used, e.g., by central government. The idea of the ‘strengths’ part of the project was seen as worthwhile – “to build on what people can do, not what they can’t”. The data used for the index should also be accurate, and there was concern at data becoming dated: “would it reflect present circumstances or those six months ago?”

Comprehensive analysis of the data would produce a tool that would hopefully “give you an idea of what the outcomes might be and what you might ultimately be looking at facing in the future and being able to put plans into place in advance of reaching that point”. But assurances of data security were seen as important and the index should be seen only as a support to professional judgement, with “computers and data being only tools at the hands of human beings”

We developed the index with the feedback from the workshop event in mind. From our analyses, we determined the contributions of each predictor variable to the outcome from the model of risk of care home admission. We multiplied each covariate by the corresponding coefficient estimate and summed across these to get the linear predictor, then applied the inverse-logit function to obtain the predicted risks [44] for older people with different characteristics, in an exemplar interactive index. We used R and a Shiny package to create this [45]. Our exemplar index, predicting individuals’ risk of care home admission, is openly accessible (at: https://scs-10.shinyapps.io/VisableRev/), giving an output of predicted risk, interpreted as the degree of risk of being in a care home if the person has the circumstances considered. From this index, S2 and S3 Fig in the Supplemental Materials show two examples of potential use of the index in future practice by council team leaders or social workers, highlighting the probability of care home admission, for hypothetical cases of relatively high and low risk respectively. These examples are given to adhere to reporting guidelines for prediction modelling (TRIPOD) that clearly state that authors should give examples of how to make predictions from the model.

Supplementary S2 Fig shows a hypothetical case of a 70-year woman from a relatively deprived area of the inner city (IMD = 46) living in a private tenancy, with ‘other’ (non-white) ethnicity. In terms of need, they would have been assessed as not able to feed themselves or look after their home, but were able to bathe and groom, use the toilet and dress themselves. Their records would show that they did not have access to a carer and their Primary Support Reason for accessing social care would be for ‘mental health support’. They would be known to have a dementia diagnosis. The Figure shows how one could make a prediction for such a hypothetical person, who would have entered the social care system, with these items of information available in their record. For this hypothetical case, there would be a 12% probability of them being admitted to a care home, two years after their initial social care assessment.

Supplementary S2 Fig shows a hypothetical case of a 61-year-old white man, an owner-occupier from a well-resourced outer-city area (IMD = 15.51). In terms of need, they would have been assessed as able to undertake all daily living activities but would have difficulties looking after their home. Their records would show that they did have access to a carer and their Primary Support Reason for accessing social care would be for ‘access and mobility’. They would be known to not have a dementia diagnosis. The Figure shows how one could make a prediction for such a hypothetical person, who would have entered the social care system, with these items of information available in their record. For this hypothetical case, there would only be a 1% probability of them being admitted to a care home, two years after their initial social care assessment.

It is not recommended that this prototype index yet be used to support operational decision making in individual cases in local authority Adult Social Care departments. The analyses presented here require further external validation in other local authorities before this is appropriate. Work is planned on an external validation in other local authorities in the UK, the results of which will be presented separately. Work is also underway to investigate how the index could potentially be reconfigured to assist service planning at a population level, rather than individual risk profiling.