Theoretical foundation of the Family Focused Mental Health Practice Questionnaire

The Family Focused Mental Health Practice Questionnaire (FFMHPQ) [30] was developed against the backdrop of increasing recommendations for FFP; the broad variation in practice with families when a parent has a mental illness; and so the need for an agreed minimum skill set for FFP. The questionnaire was originally designed to measure mental health professionals’ FFP working within adult services in Australia. Initially, Maybery et al developed the items for the FFMHPQ in parallel with a systematic review of the literature and with detailed input from the Victorian Families where a Parent has a Mental Illness (FaPMI) and the Co-coordinators and Vichamps project [31]. The review highlighted workplace policy and supports, worker skill and knowledge, and service user factors as central to FFP. These focal domains guided the initial generation of 100 items. These items were then subjected to rigorous review, rewrite and re- review, in Delphi focus groups with FaPMI coordinators. Subsequently, 14 subscales were developed, comprising a total of 45 items, which are scored on a seven-point Likert scale (ranging from strongly disagree = 1 to strongly agree = 7). Five subscales are said to measure professionals’ family focused clinical practices and activities such as providing parenting support, referring family members to services and collaboration with other professionals to meet the needs of the family. The remaining subscales are said to measure organisational influences that can impact these activities such as workplace policies and support, and workload as well as issues related to service user engagement. Principal components analysis of the original scale in an Australian study [30], revealed a sixteen-factor solution, however, there was poor internal consistency in three of the sixteen subscales. In addition, some of the remaining factors consisted of two items, indicating they may have been unstable [32].

Since its inception, the FFMHPQ has been adapted for use within a number of professional contexts internationally, particularly as the components of FFP are recognised and utilised across many sectors responding to issues associated with PMI. For example [33], adapted the measure for the early educational sector. In their study of preschool teachers and childcare professionals in Australia [33], removed eight of the original sixteen factors based on the research team’s decision that they were not appropriate for teachers. Five of the remaining subscales possessed Cronbach’s alpha values below 0.5, so consequently items were removed to improve reliability. Furthermore, in recognition of the need to acknowledge different language, culture and policy differences across countries, the FFMHPQ has been adapted for use in Ireland with mental health nurses [8] and translated into other languages including in Thailand [34] and in Norway [35, 36].

While none of these authors undertook psychometric evaluation, exploratory factor analysis (EFA) was undertaken as part of a study in NI with health visitors [37] and in Japan with mental health professionals [38]. While Leonard et al. [37] found a two-factor solution related to professional and organisational influences on FFP, Ueno et al. [38] found a 13-factor solution. Family focused practice is characterized by various activities on a continuum [20] and organizational and professional factors that enable and hinder it [8, 39]. The type of service and sector will influence professionals’ understanding and interpretation of FFP and how and to what extent professionals use it in their practice [22]. Hence, the psychometric properties of the scale may vary in accordance with the particular profession and service context in which it is used depending on how FFP has been conceptualized, operationalized and embedded in practice.

Maybery et al. [30] recommended that future research using the FFMHPQ should examine and attempt to replicate the psychometric properties of the measure as well as attempt to expand its component structure to include additional important items and factors. Ueno et al. [38] also recommended that further research be conducted to examine FFP within different services, sectors and professionals to determine if there are differences that may impact FFP. Based on previous adaption and validation studies of the FFMHPQ, the aim of this study is to test the psychometric properties and factorial structure of the scale in a population of health and social care (HSC) professionals across NI. Our objective was to explore if family focused practices are influenced by both the individual professional and their core overlapping practices as well as the organizational structures which surround them. To our knowledge this is the first study that has evaluated the psychometric properties of the FFMHPQ when used within two distinct sectors, namely adult mental health and children’s services. Results from this psychometric evaluation of the FFMHPQ can be used to inform future practice benchmarks and organizational developments.

Design

A cross sectional research design was implemented for the current study, with data collected using a survey approach. The survey consisted of three sections. Section one collected demographic data, section two included items from the FFMHPQ which is designed to measure professionals’ FFP, and section three included items which aimed to capture, through a number of open-ended questions, professionals’ experience of working with parents. Data pertaining to section two of the survey will be the focus of analysis within the current study.

Measure

The FFMHPQ was developed by Maybery et al. [30] within the Australian context and further refined by Grant [39] within the Irish context. Within the current study, professionals responded to 21 subscales (which included 66 items) using a seven-point Likert Scale (ranging from strongly disagree to strongly agree). Subscales were designed to measure different family focused activities and behaviours (e.g. assessing the impact of PMI on the child, providing support to parents, carers and children), and factors that impact these (i.e. workplace support, policy and procedures, inter professional practice and professional development). A low score in a particular subscale suggests a reduced family focus practice in this area, with a high score suggesting increased family focus [30]. Psychometric information of the subscales is detailed in the works of Maybery et al. [30]. The measure was reported as having excellent content and construct validity and good internal subscale reliability.

As the FFMHPQ was devised for use in the Australian context, with a variety of professional disciplines (e.g. psychologists, psychiatric nurses, social workers), it required minor adaption and testing in the NI context particularly for health and social care (HSC) professionals practicing within adult mental health and children’s services. Accordingly, the term ‘consumer parent’ was changed to ‘service user’ [8]. Further, in conjunction with developers of the original instrument, research team and advisory group, and in response to the emerging literature on PMI, FFP and organisational developments in FFP in NI, three additional subscales (containing 10 additional items) were included within the current survey. These new subscales aimed to measure professionals’ understanding of The Family Model [17] and child protection protocols, and their interventions to reduce the impact of the parenting role on parental mental health. These new subscales are further detailed in Table 3 (Factor 4,7 & 9).

The validity of the FFMHPQ subscales outside the Australian adult mental health service context was also established. Initially an advisory panel assessed the items in the FFMHPQ subscales for their content validity. Panel members were selected for their expertise in FFP and PMI. All the items to be included were deemed relevant and therefore retained. The final survey including the FFMHPQ was then piloted in one organisation with ten HSC professionals (5 from children’s services and 5 from adult services) not included in the main study to evaluate the clarity of the questions and their layout. The main changes made to the survey involved further refinement to the structure and language used particularly in relation to section three of the survey.

Participants & procedure

Dissemination of the survey among HSC professionals across all five HSC Trusts in NI was achieved in two main ways, (1) online (via Survey Monkey) and (2) hard copy completion. Team managers across adult mental health and children’s services were contacted via email and asked to circulate the information sheet and link to the online survey. The option for hard copy completion was also offered for those who preferred this method. Blank surveys were posted to requested teams using recorded delivery methods and later collected by a member of the research team. A database was created in SPSS for data entry of hard copy surveys; this database was later amalgamated with the online database of completed online surveys. Once both datasets had been merged, correct value range was checked. Additionally, every 25th hard copy survey was audited, and the data compared with the SPSS input data in order to ensure quality and consistency of manually entered data; these checks indicated that the data entry was reliable.

We sought to include a wide range of HSC Professionals working across adult and children’s services with families where a parent has a mental illness. The survey respondents broadly mirror the relevant workforce which has been the focus of Think Family NI initiatives. The survey was distributed to Approx. 3585 HSC professionals within adult mental health and children’s services across the five HSC Trusts. The minimum number of HSC professionals needed to complete a survey (n = 878) was determined by various factors, including the size of the population to which results are generalizable to, the results of previous research and particularly findings from previous use of the FFMHPQ in different populations and countries and the overall purpose of the current study, which was to compare two groups of HSC professionals with regard to their FFP. Hence, a two sample comparison of means was used to estimate the overall sample size.

We ensured that the characteristics of respondents reflected the population of HSC professionals who fulfilled the inclusion criteria. To promote maximum variation and to secure sample access, a principal investigator (PI) for each Trust was identified along with an independent point of contact for the study.

A total of 1088 survey questionnaires were returned by HSC professionals giving a response rate of 30%. However, 119 of these were ineligible based on study inclusion criteria; 48 surveys completed by trainees and support workers, and 71 surveys completed by professionals in ineligible service areas (e.g. disability services) were excluded. Due to significant missing information, 101 cases were also removed from the dataset as more than 90% of the survey had not been completed and would not be suitable for inclusion in final analysis. The final sample comprised of 868 HSC professionals, a response rate of 24.2%. The total sample of HSC professionals was derived from all five HSC Trusts and included professionals from both adult mental health (n = 493) and children’s social care services (n = 316) (Missing information regarding service area = 59). The largest number of responses were obtained from community mental health teams (28%), followed by children’s services family intervention teams (18.1%), acute mental health and addictions inpatient services (9.3%), initial intake teams for children’s services (Gateway Teams) (9.3%), community addictions teams (6.5%), 16yrs plus teams for adolescents (5.3%), crisis resolution home treatment (4.4%), and single point of access (0.9%). Given the variety of titles and terms attributed to different services across each Trust, the survey offered professionals the option to note their service area under a specialist mental health service or other category (15.2%). Such services included for example unscheduled care, Cognitive Behavioural Therapy (CBT), and those working within family centres.

A range of professional disciplines across these service areas participated. The most common were Social Workers (n = 473, 54.5%) followed by Nurses (n = 293, 33.8%). Other professionals, included Allied Health professionals (n = 44, 5.1%), Psychiatrists (n = 33, 3.8%), Psychologists (n = 12, 1.4%) and Other, for example, CBT Therapist (n = 13, 1.5).

Ethical approval was obtained from the Office for Research Ethics Committees Northern Ireland and Research Governance permission was obtained from the five HSC Trusts which provide statutory HSC services across NI (REC Reference 16/NI/0079). Participants gave informed consent; they were informed of the details of the study in online explanatory statements. Implied consent was obtained through participation in the completion of the online or hard copy, anonymous questionnaire.

Statistical analysis

We used Exploratory Factor Analysis (EFA) to investigate the dimensions underlying the pattern of observed responses in the FFMHPQ. Exploratory Factor Analysis was run on the 66 items considering them as ordered categorical items. For this purpose, EFA parameters were estimated using a diagonally Weighted Least Square matrix with Missing Values (WLMSV) estimator, specifically designed for ordered data. We used a Geomin rotation, i.e. an oblique rotation, which assumed correlations between the underlying dimensions. The use of the WLMSV estimator has the advantage of estimating parameters using all the information provided by respondents, even if information is incomplete (i.e. respondents do not answer all the questions). Approaches like this one, based on Full Information Maximum Likelihood (FIML), have been established to be more reliable and providing less biased results compared to traditional approaches that drop cases with missing data, as long as the reasons that can explain missing data can be assumed to be unrelated to the missing data themselves. Of note, 32 cases did not answer the items on which EFA analyses were based. Thus, results are based on n = 836 respondents that provided complete or incomplete data in the questionnaire.

The choice of the final models was based on the eigenvalues for the sample correlation matrix: factors were retained if the eigenvalue of factors was above 1. Furthermore, we estimated the model fit indices to check these provided adequate fit. These included the Comparative Fit Index (CFI) and the Tucker-Lewis Index (TLI): authors indicate values above 0.90 of these indices signal good model fit [40]. Other indices considered were the Root Mean Square of Error Approximation (RMSEA): a value <0.05 is considered to indicate a “close fit”, while values <0.08 indicate adequate fit. Finally, the Standardized Root Mean Square Residual (SRMR) was also considered: values < 0.08 are considered to indicate adequate model fit. EFA was run using Mplus 7 [41].

The results of the EFA models were inspected and considered alongside theoretical considerations [9, 30, 38, 42] to develop a more parsimonious model that could explain the variation in participants’ responses. The fit of this model was tested using Confirmatory Factor Analysis (CFA): CFA is a measurement model whereby the associations between observed items and underlying factors are explicitly modelled to allow items to load only on specific factors (i.e. factors are associated specifically with some items, but not with others). Associations between factors (or lack thereof) can also be modelled, alongside other parameters (e.g. associations between residual variances). For the sake of parsimony, when developing the measurement model through CFA, we retained fewer items in the models, taking into account results from the EFA and theoretical considerations in the selection of the items (i.e. retaining items that had substantive meaning according to our understanding of FFMHPQ).

The CFA was run considering the items as categorical ordered variables, similarly to our procedure in EFA. For this reason, we used a WLMSV estimator. Statistics considered in estimating model fit were, once again, the CFI, the RMSEA, and the SRMR. Models were estimated using Mplus 7 on all n = 835 participants who provided complete and incomplete responses to the items included in the final model.

Exploratory Factor Analysis

Initial review of the scree plot suggested that Factors from 1 to 16 displayed eigenvalues above 1 therefore solutions with 16 factors or less were considered plausible solutions. As can be seen from Fig 1, a sudden shift between the 12 and 13 factor solution was evident, therefore the 12-factor solution was also considered as a plausible option and retained for further inspection.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Scree plot of Exploratory Factor Analysis.

The horizontal axis reports the factors from factor 1 to factor 25. The vertical axis reports the estimated eigenvalues associated with each factor. Eigenvalue = 1 is indicated by the dotted line.

https://doi.org/10.1371/journal.pone.0285835.g001

Similarly, the solution with 14 factors was the first to provide adequate fit indices (including TLI) and was also retained for inspection. The fit of different solutions inspected are reported in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Fit indices of solutions 12 to 16.

https://doi.org/10.1371/journal.pone.0285835.t001

After inspection of these solutions, we considered the solution with 14 factors as the solution that provided the best balance between statistical fit and interpretability of the solution based on evidence based literature [22, 30, 38, 43]. The model with 14 factors displayed adequate model fit, as reported in Table 1, and the item loadings provided meaningful and coherent interpretations of the underlying factors, which were in line with our substantive understanding of FFP. Factors seemed to represent some of the previously identified constructs of FFP such as skills and knowledge of professionals, workplace support and inter-professional practices [22]. Table 2 reports the items that were most strongly related with each factor.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Factor loadings by items and factors.

Loadings with * are significant at p = .05.

https://doi.org/10.1371/journal.pone.0285835.t002

Some correlations between the factors extracted were significant: these correlations are reported in S1 Table. Relationships between subscales measuring professionals’ skills and knowledge (Factors 1, 2, 7, 8, 9), professional practice/ behavioural subscales (Factors 4, 5, 6), and higher order level subscales that describe organization/workplace factors that influence this (Factors 3, 10, 11, 12,) were observed.

Confirmatory Factor Analysis (CFA)

The 14-factors model described in the previous section was the basis from which we derived a comprehensive more parsimonious model that included 12 factors. In what follows we describe the steps that led to this model. Items were retained based on recommended criteria that item loadings should be above.32, which represent over 10% of item variance explained by the underlying factor. Further, factors with fewer than three items or those items that cross loaded were assigned to the factor determined as a stronger associate with the exception of item 41R (I am not able to determine the level of importance that parents who have mental illness place on their children maintaining strong relationships with others outside the family) which was retained in factors 2 and 4, both of which address child related issues. We considered this as a more parsimonious solution that could adequately represent the variability in participants’ responses particularly as factors 4 and 8 reflect distinctive knowledge regarding either the parent or the child which may also be linked to differences in professional focus and service remit (i.e. Adult mental health and Children services). The items retained in the analyses and the underlying factors, together with the factors’ labels and their interpretations are reported in Table 3. In Table 4 we report the fit indices of this 12-factor solution. The model demonstrated adequate fit according to the CFI and the RMSEA. S2 Table displays the factor loadings of the items with the respective underlying factor.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. 12 factor model including items and factor labels and definitions.

https://doi.org/10.1371/journal.pone.0285835.t003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Model fit parameters of the 12-factor solution used in CFA.

https://doi.org/10.1371/journal.pone.0285835.t004

All the items included in the model had generally strong relationships with the underlying factor and were significant at p < .001., as reported in S3 Table. Furthermore, the factors showed patterns of correlations that were consistent with substantive knowledge and theories. These correlations are reported in Table 5. Note that some correlations were negative and significant (e.g. correlations between Factor 1 Training and Factor 7 Understanding The Family Model = -0.322; Factor 1 Training and Factor 11 Service Availability = -0.321). We suggest that this may be reflecting a relationship between those indicating a need/willingness for further family focused training and those who indicated they had less understanding of The Family Model and that those who felt they needed more training also perceived that there was a lack of services to refer parents and children to.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Correlations between factors of 12 factor model.

https://doi.org/10.1371/journal.pone.0285835.t005

A high number of remaining relationships were positive and strong (r >.32), with 10% of shared variance between these factors. Subscales measuring professionals’ skills and knowledge were positively correlated across one another (e.g. Skills and knowledge to support children WITH Connectedness; Child Protection WITH connectedness). Professional practice/ behavioural subscales were also positively correlated with these individual level subscales (e.g. Skills and knowledge to support parent with their parenting WITH Connectedness; Referral WITH Connectedness; Understanding The Family Model WITH Psycho Education and Referral; Skills and knowledge to support children WITH Skill and Knowledge to support parent and their parenting, Referral and Psycho education; Child protection WITH Skill and Knowledge to support parent and their parenting) as well as across one another (e.g. Referral WITH Skill and Knowledge to support parent and their parenting; Psycho education WITH Referral).

Higher order level subscales that describe organisation/workplace factors were also positively correlated with individual professionals’ skills and knowledge factors (Understanding The Family Model WITH Workplace support; Engagement issues WITH Skills and knowledge to support children; Time and workload WITH Skills and knowledge to support children) and professional practice/ behavioural factors (e.g. Referral WITH Workplace support; Psycho education WITH Workplace support; Time and workload WITH Psycho education) as well as across one another (Time and Workload WITH Workplace support, Service availability WITH Workplace support and Time and Workload; Engagement issues WITH Time and Workload). These findings are reflective of proposed models of family focused practice which suggest that specific family focus practises are influenced by both the individual professional and their core overlapping practices as well as the organisational structures which surround them [20, 22, 44, 45].

Limitations

Although the professional composition of respondents was similar to those in other studies [30, 38], the sample was drawn from two different sectors making it difficult to directly compare findings with previous research. The FFMHPQ is a self-report questionnaire from the professionals’ perspectives and this may not be a fully accurate reflection of actual practice. It may also differ from perspectives of service users, families and managers. Observational research might be conducted to provide additional data regarding the dimensions of FFP and could include the perspectives of family members and managers. While the FFMHPQ had documented validity and reliability in the Australian context [30], in the current study two of the subscales, psychoeducation and child protection, had Cronbach’s alpha coefficients below 0.60, hence care should be taken when using these. In addition, in future research, efforts could be made to improve their reliability. The nature and specificity of the items within these two subscales could be revised in order to better reflect professional’s experience in relation to the concepts being measured.