The PCOC Symptom Assessment Scale (SAS): A valid measure for daily use at point of care and in palliative care programs

Background Very few measures are used successfully as part of routine care within national palliative care outcome programs. Only a handful of studies examine these measures. The aim of this study is to evaluate the validity of a measure used in a national outcomes program: the Palliative Care Outcomes Collaboration Symptom Assessment Scale (PCOC SAS). Methods A retrospective multi-site cohort study with secondary analysis of routinely collected patient-level data to assess PCOC SAS’s internal consistency, construct validity, reliability, interpretability, acceptability and sensitivity. The analyses used two sets, with data collected by inpatient and community palliative care services registered with the Australian national PCOC. Results Dataset one included 1,117 patients receiving palliative care from 21 services. Dataset two included 5,294 patients receiving palliative care from 119 PCOC services. PCOC SAS demonstrated the ability to detect and discriminate distress by palliative care phase, functional status and diagnosis. Excellent and good convergent and discriminant validity were demonstrated. Fair through to substantial inter-rater and intra-rater reliability levels were evidenced. Sufficient interpretability resulted along with necessary levels of acceptability and sensitivity. Conclusion PCOC SAS is a valid and reliable patient-reported outcome measure suitable for use in routine clinical care with patients requiring palliative and or end-of-life care, including in national outcomes programs.


Introduction
validating measures was used [6], drawing upon previously and recently published approaches within the context of palliative and end-of-life care [7,8]. This study therefore involved the evaluation of: internal consistency, construct validity, reliability (agreement, inter-rater and intra-rater reliability), interpretability, sensitivity and acceptability (Table 1).

Materials
PCOC SAS is a patient-reported scale used to measure subjective aspects related to health. It evaluates perceived distress. Although designed to be used by patients from any disease or treatment group and age, it can be used by proxies [9]. The scale assesses eight dimensions: pain, insomnia (difficulty sleeping), nausea, bowel problems, appetite problems, breathing problems, fatigue, and an 'other' item, which may be added to the measure. Table 1. A brief explanation of the key psychometric properties examined in this study.

Brief explanation
Internal consistency Internal consistency examines how items (e.g., questions) in a measure (e.g., PCOC SAS) are associated (or correlate) with each other. Internal consistency provides an indication of the level of coherence of a measure. It also provides an indication of whether there are redundant items in a measure. From a clinical perspective, good internal consistency for measures that examine a single concept is important. This is because measures with high internal consistency may place unnecessary burden on patients, carers and staff (as the measure may have redundant questions in it). While measures with a low internal consistency may mean that a number of different concepts are being measured by the scale, and this may make it more challenging to use as part of routine clinical care.

Construct validity
Construct validity investigates whether a measure examines the concept (or construct) it intends to. Convergent and discriminant validity are part of construct validity. Convergent validity tests how closely items correlate. Discriminant (or divergent) validity recognises that unrelated items should have low correlations. Theory or hypothesis testing is part of examining these types of validity. This is because what we need to measure may not always be observed directly. Establishing a clear hypothesis or theory to test from the beginning is important as it helps reduce the risk of bias. If the relationship between the theory and the measure is not apparent in the results from the study, then the measure does not measure what it intends to measure (and it has poor construct validity) or the theory tested was incorrect. Reporting negative findings and reviewing the theories that were tested in research is an important part of developing outcome measures. This helps build knowledge, allows for critical appraisal of research results, and it ensures honesty in research reporting and conduct.

Reliability
Concerns whether a measure is able to produce reproducible and consistent results. Intrarater (test-retest) reliability involves the same person repeating the measure. Inter-rater reliability examines agreement between different raters (e.g., a carer and a staff member).

Interpretability
Provides an indication of the extent to which someone (the patient, the carer, staff) can derive meaning from the numerical scores in the measure. This is important when it comes to ensuring the measure informs clinical care.

Acceptability
Examines how agreeable the measure is to the user (the patient, the carer, staff). Low levels of acceptability may result in a measure not being used, it being used incorrectly and missing data.

Sensitivity
Examines whether the measure can detect differences between groups. For example, whether the measure can detect differences between different diagnostic groups or those with advanced versus early-stage disease.
For each of the eight dimensions, there are 11 levels in the response options. The response options range from 'absent' to 'severe' distress. A score of 0 indicates that the patient is distress free, meaning that the score >0 effectively identifies distress for each domain in the scale. In order to assist patients to discriminate between these 11 response options reliably, the PCOC SAS response options have been grouped into six intensity categories. Each category has a corresponding descriptor, colour and facial expression. Higher total scores and higher individual item scores represent higher levels of distress [S1 File].
In relation to the use of PCOC SAS as part of routine care, all Australian PCOC-registered services use the measure daily within inpatient settings (including daily in residential aged care facilities) and at each contact within community settings. Its use is bolstered by a national education program, online educational materials, and calibration sessions, supported by a clinical manual and information brochure [10]. PCOC SAS and its accompanying information brochure is available in 15 languages: Arabic, Chinese (simplified and traditional), Croatian, English, Filipino, Germany, Greek, Hindi, Italian, Macedonian, Russian, Serbian, Somali, Spanish and Vietnamese. Throughout Australia, PCOC SAS scores are either collected using paper or electronic means (e.g., portable handheld devices). PCOC provides IT software to participating services to help address any technological barriers to its use at point of care.

Population and settings
All services registered with PCOC were deemed eligible to participate in this study. A smaller set of services were approached to participate in a sub-study to allow for a more thorough examination of the measure. These sub-study services were services able to adequately represent national palliative care services and the population currently accessing palliative care (including public and private providers). A further criterion was that services with well-established procedures for recording who completed symptom distress assessments could participate. Hospital-based services that only offered consult-liaison services were excluded due to the advisory nature of the service.

Data collection
Demographic information collected for patients at baseline included: age, sex, country of birth, preferred language, Indigenous status, and primary diagnosis. Clinical patient information collected included: primary diagnosis, palliative care phase [11,12], Australia-modified Karnofsky Performance Status (AKPS) [13], the Palliative Care Problem Severity Score (PCPSS) [14], the Resource Utilisation Group-Activities of Daily Living (RUG-ADL) [12] and PCOC SAS [5]. Data was collected for a four-week period commencing October 2017.

Ethics
Ethical approval was granted for the PCOC program by the University of Wollongong and Illawarra Shoalhaven Local Health District Health and Medical Human Research Ethics Committee (reference no. HE2006/045). This involved an amendment to the original PCOC ethics application to ensure approval of the analysis of patient and proxy data, which was not routinely submitted to PCOC although routinely documented by some services. As only routinely collected, de-identified, aggregated clinical data were used in this study, separate participant consent was not necessary.

Analysis
Secondary analysis of patient-level data collected as part of routine clinical practice was completed. Descriptive statistics were derived to describe the patient sample, the range of individual item PCOC SAS scores, the PCOC SAS total score, and missing values.
Internal consistency. We estimated the internal consistency of PCOC SAS using Cronbach's α for the total score and individual items. Due to the multi-dimensional nature of the measure, and before analysis, we lowered the normally accepted threshold for good internal consistency from 0.8 to 0.6 [15].
Construct validity. For construct validity, we examined PCOC SAS against phase and functional status. We hypothesised that patients in an 'unstable' or 'deteriorating' phase would have higher PCOC SAS total scores, as compared to those in 'stable' and 'terminal' phase. In line with established definitions, a stable palliative care phase was defined as the patient's symptoms and problems were adequately controlled by established management. An unstable palliative care phase was when an urgent change in the patient's plan of care or their emergency treatment was required due to a new problem or an escalation of an existing patient or carer problem. A patient was defined as being in a deteriorating palliative care phase when their care plan was adequate but periodic review was required due to gradual functional decline and or a worsening of existing problems, and or the development of new but expected problems (either patient or carer problems). A terminal palliative care phase was when the death of the patient was likely in a matter of days [11,14]. Also, we hypothesised that patients with a lower AKPS [13] who would have lower PCOC SAS scores as compared to those with higher AKPS scores.
For convergent and discriminant validity (part of construct validity), we used Spearman's correlation coefficient to correlate PCPSS with PCOC SAS. We hypothesised that a high correlation (r>0.70) would be evident for the pain items on both measures. Given the differences between the measures (PCOC SAS examines distress and the PCPSS examines symptom severity), we anticipated mid-range correlations between the total scores. Mid-range correlations were anticipated between the PCPSS 'other' item and the PCOC SAS total score, excluding the pain item. Mid-range correlations were defined in advance of analysis as r = >0.5 0.7.
Reliability. For inter-rater reliability, we analysed independent patient and proxy ratings completed �48 hours when the patient was in the stable phase. For patients in an unstable or deteriorating phase, this was analysed between patient and proxy ratings completed �24 hours of each other. Cohen's weighted kappa was calculated for reliability. For agreement, both exact agreement and a tolerance agreement, reflecting the proportion of cases where proxy or patients' ratings were equal to or within +1 or −1 of the score, were calculated. Cohen's Weighted Kappa was calculated to ascertain the level of chance-corrected agreement. A preestablished criterion of agreement was used: poor k<0.00, slight k = 0.00−0.20, fair k = 0.21 −0.40, moderate k = 0.41−0.60, substantial k = 0.61−0.80, almost-perfect k = 0.81−1.00 [16].
We hypothesised that test-retest reliability (intra-reliability) would be demonstrated in a patient with no change in their phase within one episode of care, with analysis by phase undertaken. Pearson's correlation coefficient was used to examine incidences where PCOC SAS was completed twice �48 hours when the patient was in a stable phase. Analysis of the instances where assessments were completed �24 hours for when the patient was in an unstable or deteriorating phases. Patients in terminal phases were excluded (due to the likelihood of a larger proportion of proxy reported assessments).
Interpretability. A Flesch-Kincaid readability test [17] was completed to identify the years of education required to complete PCOC SAS. In line with pre-established standards, sufficient interpretability was evident if someone with the reading skills of a �12-year old were required [18]. Pre-established standards to define what was acceptable for the extent of missing items were used, with items with values of >4% deemed to be insufficient [15].
Acceptability. Acceptability was examined by analysing the distribution of the PCOC SAS total scores for each patient (median, interquartile range (IQR)). Acceptability was hypothesised to be sufficient if PCOC SAS total scores were well distributed and median scores were near the mid-point of the scale. Floor or ceiling effects would be evident if more than 15% of respondents achieved the lowest or highest possible score, respectively [6]. Acceptability was examined by phase, individual items and total scores. (The levels of missing data, which may also be an indication of acceptability, is reported in the interpretability section.) Sensitivity. In relation to sensitivity, we hypothesised that PCOC SAS would discriminate between individual symptom distress scores by diagnosis and phase. We anticipated that as compared to all other patients diagnosed with cancer, higher distress scores would be found for patients diagnosed with: lung cancer or COPD for the breathing item, colorectal cancer for

PLOS ONE
The PCOC Symptom Assessment Scale (SAS): A valid measure for routine use in palliative and end-of-life care the bowel item, head and neck cancer for pain, pancreatic cancer for pain, gynaecological cancer for pain, bone and soft tissue cancers for pain, and that there would be more reports of distress related to fatigue in deteriorating and unstable phases.

Palliative care service and subject characteristics
Two sets of data were derived for this analysis. Dataset one included 1,117 patients receiving palliative care from 21 services. Of the 21 services, 12 were inpatient services and nine provided services in the community. Of the 1,117 patients, the majority (n = 819, 72.4%) were receiving care in the community, with just over a quarter (n = 320, 28.6%) receiving care in an inpatient setting. Dataset two included 5,294 patients receiving palliative care from 119 services. Of the 119 services, 39 services were providing palliative care in community and 80 inpatient services. In a few instances, patients received care in both the inpatient and hospital setting and so are counted against both, meaning that percentages will not sum to 100% (Table 2). Across both datasets, patients with a primary diagnosis of cancer formed the majority. Approximately 1 in every 5 patients had a non-malignant primary diagnosis with respiratory failure being the most common, followed by those diagnosed with cardiovascular disease. The full range of individual item PCOC SAS scores and the PCOC SAS total scores were evident in the full and smaller datasets. The patient-reported values for the smaller dataset are conveyed in Table 3. Limited missing values were found for all items (i.e., �4%), except for insomnia (Table 3).

PLOS ONE
The PCOC Symptom Assessment Scale (SAS): A valid measure for routine use in palliative and end-of-life care

Internal consistency
With rounding conventions applied, the PCOC SAS demonstrated good internal consistency with a Cronbach α of 0.59 for patient ratings, and 0.62 when all assessments evaluated, that is with patient and proxy-rated assessments examined together.

Construct validity
Our analysis confirmed our a priori hypothesis regarding PCOC SAS and phase, with higher PCOC SAS total scores observed for unstable and deteriorating phases, as compared to total scores for stable and terminal phases. This was also observed for each individual PCOC SAS item. Higher percentages of assessments revealed more instances of patients being distress free when in stable (21.5%) and terminal (54.4%) phases, as compared to unstable (13.8%) and deteriorating (14.4%) phases. Our hypothesis that those with a lower AKPS would have lower PCOC SAS scores was not observed as those with higher and lower AKPS scores reported less distress. A U-curve for the percentage of assessments was found for the PCOC SAS item scores (excluding the 'other' item) when compared with the AKPS item scores ( Table 4).
Calculation of Spearman's rank-sum correlation coefficient to examine correlations between the PCPSS with PCOC SAS scores indicated a high correlation between the pain items on both measures (r = 0.76, p = < 0.001). As anticipated, mid-range correlations between the PCOC SAS total score (excluding pain) and the PCPSS 'other symptoms' score (r = 0.55, p = < 0.001), and the total scores of PCOC SAS and the PCPSS were found (r = 0.59, p = < 0.001), with rounding conventions applied.
Reliability. Moderate and substantial levels of inter-rater reliability were observed for all PCOC SAS items ( Table 5). The exception was the bowel item in the stable phase, and insomnia in the unstable and deteriorating phases. However, the percentage of exact (76.8%) and tolerance agreements (85%) for the bowel item was high in the stable phase, indicating substantial reproducibility. Similarly, the level of agreement was very good for the insomnia Table 4. PCOC SAS patient-rated assessments: Median total score distress-free total scores (%), and distress-free item scores by palliative care phase and functional status (AKPS).

PLOS ONE
item, with an acceptable exact (71.3%) and tolerance (75.9%) agreements. Of note, the 95% CI for the insomnia item was relatively larger than all other items (95% CI 0.08-0.48). The sample size for this item was 100 patients (Table 5). In relation to test-retest (intra-rater) reliability, moderate to substantial agreement on all items resulted. The range of the 95% CI for the insomnia item remained similar in breadth to the inter-rater reliability, however there was a larger number of exact and tolerance agreements ( Table 6). Interpretability. Testing of the measure showed sufficient interpretability. The Flesch reading ease was 86.2 (grade six level USA education i.e., 11−12 years old). The Flesch-Kincaid grade level was 4.6. Acceptable levels of missing data (i.e., �4%) was found on all items, except the insomnia item.
Acceptability. After removal of the screening response option of 0, evaluation of the distribution of the PCOC SAS scores indicated that the values were near the midpoint of the scale, especially for pain, fatigue and breathing. The distribution was skewed. No floor or ceiling effects were found.
Sensitivity. We found that that PCOC SAS could discriminate between individual symptom distress scores by phase. In relation to diagnosis, we observed that patients diagnosed with lung cancer or COPD had higher scores of distress for breathing as compared to all other patients diagnosed with cancer. Patients with GIT cancers were found to report higher levels of distress for the nausea item, when compared to all patients diagnosed with cancer. No other statistically significant differences were found for the disease group comparisons (Table 7). There were more reports of distress related to fatigue in patients in unstable and deteriorating phases (Table 4).

PLOS ONE
The PCOC Symptom Assessment Scale (SAS): A valid measure for routine use in palliative and end-of-life care

Discussion
Our analysis confirms that PCOC SAS is a valid and reliable patient-reported outcome measure with sufficient interpretability for use as part of routine clinical care. It is sensitive enough to detect clinically relevant changes in patients with lung cancer, COPD and GIT cancers. It can be understood and comprehended by a 12-year old. Importantly, our findings were derived from information where PCOC SAS was used as part of routine clinical care with patients receiving palliative care. This applied health services research approach serves to strengthen the conclusions reported in our paper, which highlight the strong utility, feasibility One of the findings in our study that warrants further discussion is the levels of agreement between instances of patient reports. Moderate to substantial levels of agreement were observed for all PCOC SAS items. The exceptions were the bowel item in the stable phase, and insomnia in unstable and deteriorating phases. That acknowledged, the percentage of exact and tolerance agreements for these items were high, especially when considering the relatively large number of response options in PCOC SAS (11 response options). As the 95% CI for the insomnia item was larger than all other items and the sample size for this smaller, the item should be examined again with a larger sample size.
Missing data levels for PCOC SAS items were low, ranging from 0.8% for pain through to 2.0% for appetite. These levels are well below the level �4% usually deemed acceptable (i.e., �4%) [15]. They compare favourably to other measures in palliative care that assess subjective states [7]. The only exception to this was the insomnia item where very large volumes of data were missing (26.3%). Further investigation with services revealed that these missing values were due to an IT software usage issue, rather than a reflection of this item being missed as part of routine care or being of lower quality. Unlike other validation studies, our study was reliant on information collected as part of routine care, meaning these findings highlight the excellent extent of missing data for the PCOC SAS in routine care.
An additional novel finding was that national outcome measurement programs can be used to help develop measures for application with patients with advanced disease. We demonstrate that the testing of test-retest reliability is feasible when the concept of episode of care along with palliative care phase are combined to identify a period of stability in the patient. This is a useful finding as previously it has been suggested that the use of a 24-or 48-hour period of reassessment for test-retest reliability may not be possible as palliative patients may change too much in this time period [7]. Our study shows that examining test-retest in routine practice is possible.
An unanticipated finding was the inverse relationship between performance status and distress in patients with advanced disease. We anticipated that those with lower AKPS scores would have higher reports of distress, as compared to those with higher AKPS scores. However, we found that reports of distress in patients with advanced disease are lower when they are able to function without impairment or with a great deal of impairment. This finding, whilst different to our hypothesis, is plausible, and many different theories may account for this unexpected finding. For example, a response shift may have occurred in these patients. That is, the patients may have adjusted their internal standard by which they measure their own distress. They may have changed the way they valued what was being measured or they may have even redefined their concept of distress as their illness continued to advance [19,20]. An alternative explanation is that our finding may reflect the positive impact that services may have on helping patients achieving a response shift or symptom relief [21].
A major strength of our study is the large volume of evaluable data retrieved from routine clinical care involving 80% of specialist palliative care providers across a country, which arguably helps reduce selection bias. This means, for example, that patients with cognitive challenges were still included in the study, rather than deemed ineligible in a research study, for example. However, at the same time, the unexpected finding regarding the inverse relationship between performance status and distress, and the missing data in relation to the insomnia item are illustrative of the limitations inherent with this approach. Nevertheless, even despite skewness in distributions of the SAS total score and individual items, the statistical tests applied have resulted in robust results. The large non-cancer population, the representation of services from inpatient and community settings (both private and public providers), and the inclusion of data from younger age groups in our study are further strengths. The applied nature of our study design while a strength also resulted in a number of the quality checks being employed only after all of the data had been submitted by the services, and this resulted in the need to further investigate missing values for the insomnia item in the national outcomes program and in future studies.

Conclusion
PCOC SAS is a valid and reliable patient-reported measure able to be used by patients with advanced disease. The attributes of PCOC SAS allow it to be used as part of routine clinical care, including with younger patients, those with a diagnosis other than cancer, across settings, with individual services (e.g., tertiary providers through to smaller community providers), and as part of a national palliative care outcome programs. Given these attributes, PCOC SAS is distinguished from other scales that aim to measure what affects patients with advanced illness. It is recommended for use in palliative care.