Development and validation of an instrument in job evaluation factors of physicians in public hospitals in Beijing, China

Background Job evaluation has been widely used to establish a foundation for internal equity and other human resource functions. The United Kingdom adopts the National Health Service (NHS) Job Evaluation scheme to determine the pay bands for most NHS staff and ensure equal pay for work of equal value. The challenges of recruiting and retaining physicians in Chinese public hospitals have heightened the need for a reliable job evaluation system to ensure the equality of physician compensation. The aim of this study was to construct job evaluation factors of physicians in Chinese public hospitals based on the NHS Job Evaluation scheme and to examine the reliability and validity of the established system.


Introduction
A compensation plan is recognized as an essential method of attracting, retaining and motivating employees in an organization. It is important to develop an effective compensation programme by adopting an objective and systematic approach due to the complexity and subjectivity of compensation [1]. Job evaluation has been commonly used to establish a foundation for developing a fair wage structure and sustaining internal equity [2]. Specifically, job evaluation refers to a method for systematically determining the relative internal value of different jobs in an organization [2][3][4]. It can be used to design a fair payroll structure by comparing the relative similarities and differences in the content and the value of jobs. In addition to compensation management, job evaluation also serves as the basis for other human resource functions, such as recruitment, training, appraisal and career management [5].
There are several methods employed in job evaluation, including ranking, classification, factor comparison, job components and point systems [1]. The point-factor approach has been the most widespread method due to its comparatively high accuracy and reasonable cost [6,7]. The point-factor approach evaluates each position's value through a set of compensable factors. These factors refer to the specific characteristics of the positions that contribute to the fulfilment of an organization's strategy and objectives, and they are typically categorized as skills, responsibilities, effort and working conditions [4,8,9]. In this method, there is a scale defined by a number of levels for each of the factors, and each level is associated with a certain point value. The jobs are rated on all factors, and wage rates are assigned to the jobs based on the sum of all the factor scores [10].
Job evaluation, and in particular the point-factor method, has been widely used by a considerable number of companies and governments [11]. It is estimated that job evaluation has a major influence on the wages of half of American jobholders. For example, Mercer's International Position Evaluation system with point-factor rating methodology has been used by multinational organizations to harmonize their compensation programmes [12]. The General Schedule developed by the U.S. Office of Personnel Management is the predominant pay scale for federal employees, including professional positions in healthcare organizations [13].
Job evaluation also has widespread usage in the field of healthcare. The National Health Service (NHS) Job Evaluation (JE) scheme has been the most significant change in human resource management in healthcare systems in the past decades [11]. The NHS JE scheme was developed by the Department of Health in the United Kingdom in 2003 and has been updated to the seventh edition [14]. It uses a common set of terms to describe jobs and produce national job profiles that are classified into the following eight professional groups: administrative services, allied health professionals, emergency services, health science services, nursing and midwifery, personal social services, support services and professional managers. The system provides a fair and clear method for determining the pay bands for all NHS staff, with the exception of doctors, dentists, apprentices and some senior managers, who have separate pay contracts [15]. The Review Body on Doctors' and Dentists' Remuneration (DDRB) brings an independent and expert view on rates of pay for medical and dental workforce whose salaries are based on years of experiences, merit awards, overtime and out-of-office hour payment [16]. Given that the internal relativities between medical and dental career paths is probably a controversial area, it is recommended that the relativities across different paths should be confirmed by a more detailed points factor job evaluation exercise, such as NHS JE system and Hay system [17]. NHS JE scheme focuses on the need to maintain equality and fairness and improve morale through better pay and overall quality. Previous research has assessed the quality of job evaluations and has mostly emphasized the issues of reliability and validity [1,10]. In addition, a number of studies have conducted a local evaluation of several healthcare jobs by implementing the NHS JE system to examine the appropriateness of the system in countries outside the United Kingdom [11,18].
Given the challenges of recruiting and retaining physicians and the fact that the public health sector is the main healthcare provider, physician compensation in public hospitals has been one of the core issues of China's health system reforms [19]. Currently, Chinese public hospitals implement the Post-Performance Payment system to remunerate physicians, which consists of the basic salary, benefits and bonuses [20,21]. Following a unified standard, the basic salary is determined by physicians' educational level, professional titles and years of practice. Benefits mainly consist of mandated benefits and supplemental welfare, such as medical insurance, endowment insurance, transportation subsidies and so on. The bonuses account for the majority of the total earnings and are wholly or partly bound to the revenues of the department [20]. Current system fails to consider other essential factors, such as the varying risks, responsibilities, skills and knowledge across different posts, which results in internal inequity and job dissatisfaction [22,23]. Moreover, physicians' profit driven incentive is attributable to the bonuses linked to departmental revenues [20]. Job evaluation, especially the NHS JE system, is a promising method for addressing the internal inequity. The system that the basic pay makes up the majority of the total earnings for NHS staff also provides lessons for the physician compensation in Chinese public hospitals. However, all positions within an organization have their own characteristics. It is necessary to develop a tailor-made system to ensure that the system reflects local working practices. Furthermore, the selection of evaluation factors is a preliminary step in the design process of a point-factor job evaluation system. It is common to confirm the evaluation factors on the basis of the designer's experience with and intentions for the management level, which causes a lack of theoretical support [24]. Therefore, to make the factors more reasonable and scientific, this study aims to develop the Chinese version of physician job evaluation (CN-PJE) system by building on the NHS JE system and adopting a questionnaire survey and to examine the reliability and validity of CN-PJE.

Sample
In this study, we used a multistage sampling design to select the samples. In the first stage, we used cluster sampling to choose one tertiary public hospital within each district of Beijing, which resulted in six hospitals being included in our survey. Each of the participating hospitals had 800 to 1500 clinic beds and 1200 to 3500 employees. In the second stage, 5% of physicians were sampled within each of the participating hospitals using stratified random sampling and physicians were stratified based on the profession. A total of 900 physicians participated in the questionnaire survey.

Questionnaire development
The selection of job evaluation factors for the questionnaire was based on a literature review, focused interviews with physicians and the Delphi method. First, through a literature analysis, we retained the 16 factors from the NHS JE system and consolidated another eight factors from four job evaluation models designed by governments or consulting firms [12,13,25,26]. Second, we identified an additional three factors through focused interviews with 20 physicians. Table 1 shows the lists of sources of job evaluation factors for the questionnaire.
We screened and determined the factors using the two-round Delphi method with a panel of 15 experts. These experts included senior attending physicians and experts on hospital management and human resources. All of the experts were professors and had at least a Master's degree. Most of the experts were 40 to 60 years old and had more than 20 years of work experience. We received 13 valid responses in the first round of the questionnaire and 12 valid responses in the second round. The result showed that the expert authority coefficient for all factors in the two rounds was higher than 0.8, suggesting that the result of the experts' evaluation had high reliability. The Kendall coordination coefficient for all factors in the two rounds was statistically As shown in S1 and S2 Tables, the final version of the questionnaire contained 25 factors. The questionnaire asked participants to rate the importance of each factor on a 5-point Likert scale, ranging from "very important" (5 points) to "very unimportant" (1 point).

Reliability and validity of the questionnaire
The questionnaire was pretested to assess its reliability and validity. A total of 50 physicians from a tertiary general hospital in Beijing participated in the preliminary survey. A total of 48 questionnaires were returned, with 45 valid responses. In the retest survey, 30 out of the 50 physicians accepted the retest administration within a two-week time span. The results showed that the overall Cronbach's alpha of the questionnaire was 0.90, which implies satisfactory reliability. The test-retest reliability coefficient was 0.87, suggesting good retest reliability.
With the data from the preliminary survey, we applied Bartlett's Test of Sphericity and the Kaiser-Meyer-Olkin (KMO) test to measure how suited the data are for factor analysis. The results showed that the Bartlett's Test of Sphericity was significant at p< 0.05, and the KMO value was 0.879, which indicated that the data were appropriate for the purpose of factor analysis. We then used a principal component analysis to obtain the initial factor solutions. Five principal components with Eigenvalues greater than one were extracted, and they cumulatively explained 60.42% of the variance. Hence, the results indicated that the questionnaire had good structural validity.

Data processing and statistical analysis
After receiving the completed questionnaires, a preprocessing step was applied to remove incomplete or invalid data. The exclusion criteria were as follows: 1) there were more than three factors unanswered; 2) all the factors were answered the same; and 3) the answers displayed an obvious pattern. In addition, after all the data had been entered, 5% of the data were drawn for rechecking to ensure the accuracy of data entry.
We performed a principal component analysis with a varimax rotation for feature extraction and dimensionality reduction. The factors with Eigenvalues greater than one were retained, and the factors with factor loadings less than 0.5 were removed from the analysis. The reliability of the job evaluation system was assessed by calculating Cronbach's alpha and analysing the correlation between factors from the same and different dimensions. The validity was assessed via a structural validity test on the job evaluation factors. The data were processed with Microsoft Excel 2015 and analysed with SPSS 21.0.

Ethics approval and consent to participate
This study has received ethics approval from Tsinghua Shenzhen International Graduate School Medical Ethics Committee. The questionnaire surveys were anonymous and voluntary, and written informed consent had been obtained from all study participants and experts.

Characteristics of respondents
In this survey, a total of 900 questionnaires were distributed, and 759 were completed, which resulted in a response rate of 84.3%. A total of 693 questionnaires were regarded as valid, with a validity rate of 91.3%. Table 2 shows the characteristics of the study samples. Overall, 51.9% of the respondents were male, and 48.9% were aged between 30 and 39 years old. In terms of education, 51.7% had a Master's degree. Attending physicians made up 47.8% of the sample. The majority of the respondents were general staff (70.7%) and had an annual salary of between 10,000 and 20,000 dollars (34.4%). In addition, most of the respondents worked in the surgery department (33.8%), followed by the internal medicine department (25.8%). There was no significant difference in the demographic characteristics between the exploratory samples and the confirmatory samples (p>0.05), which indicated that the responses had good consistency.

Descriptive statistics of the job evaluation factor questionnaire
As shown in Table 3, the average scores of the 25 factors ranged from 3.8 to 4.7, with a mean value of 4.2. The factor that received the highest score was Professional Knowledge (4.7±0.60),

Factor analysis and dimensions
Bartlett's Test of Sphericity was significant at p< 0.05, and the KMO value was 0.939, which supported the appropriateness of the principal component analysis technique. The principal components with Eigenvalues greater than one were retained. Five principal components were extracted from the principal component analysis and were defined as the five dimensions of the job evaluation system. The five components cumulatively explained 66.26% of the total variance, and the result was acceptable. A varimax rotation was used to transform the components into factors that were more clearly interpretable. Factors with factor loadings greater than 0.5 were extracted. All 25 factors had factor loadings above 0.5, and thus, all were retained in this study. Considering the content of the factors under each dimension, we named the five dimensions Task Characteristics (four factors), Knowledge (five factors), Responsibility (five factors), Effort/Environment (four factors) and Skills (seven factors). Table 4 demonstrates the factor loadings for each factor. The factor loadings of the factors in some dimensions were all above

Reliability and validity analysis of the CN-PJE
The Cronbach's alpha coefficients of the five dimensions ranged from 0.841 to 0.909, which indicated a high level of internal consistency. The Cronbach's alphas were then calculated for each factor under the dimensions. As shown in Table 5, the removal of any factor would result in a lower Cronbach's alpha. In terms of the corrected factor-total correlation, all correlation coefficients met the requirements, which suggested that the system's factors and dimensions had good reliability. Moreover, the correlation coefficients between factors under the same dimension were all greater than 0.376, which indicated that the factors had good consistency, as shown in S3 Table.  Table 6 displays the inter-correlations of the five dimensions. The correlation coefficients between the dimensions were greater than 0.488, suggesting good consistency between the dimensions. Knowledge and Skills (r = 0.718) were the most correlated, while Knowledge and Responsibility (r = 0.488) were the least correlated.
The above results of the factor analysis show that the system has good structural validity. The content validity is well established by building on the NHS JE system and other well- established job evaluation systems with a two-round Delphi method. Therefore, our results show that the CN-PJE has both good reliability and validity.

Discussion
In this study, we developed an instrument in job evaluation factors of physicians in public hospitals in Beijing, China. The CN-PJE was built on the NHS JE system with a two-round Delphi method and was further developed by a questionnaire survey to adjust for the differences in culture and healthcare delivery systems between these two countries. The CN-PJE consists of 25 factors scattered in five dimensions. It has high reliability, with Cronbach's alphas all above 0.8. The results of the factor analysis show that the CN-PJE has sound structural validity. Therefore, the CN-PJE has both sound reliability and validity.

The feature of CN-PJE
The survey results indicate that the CN-PJE can reflect the characteristics of physician posts. First, physicians are usually confronted with a number of challenges, including complicated clinical practices, varied clinical demands of patients, high risks, and heavy workloads [27]. These characteristics are reflected by the dimensions of Task Characteristics (four factors) and Effort/Environment (four factors). These factors should play an important role in determining a physician's salary. Second, a physician is a knowledge-intensive position with high requirements for knowledge and skills [28,29]. These features are reflected by the dimensions of Knowledge (five factors) and Skills (seven factors). Third, physicians need to shoulder multiple responsibilities in hospitals [30], which can be captured by the Responsibility dimension (five factors). In addition to patient care, physicians also engage in management responsibilities, which involve policy development and implementation, financial and physical resources, human resources and information resources. Responsibility for research and development is an integral part of the work of physicians, especially for those in tertiary hospitals. Usually, research achievements can affect the salary of physicians in Chinese public hospitals. Furthermore, more responsibilities require more skills. Hospitals also put great value on physicians' skills in problem solving, social interaction, administration and innovation. The survey suggests that physicians also attach importance to these skills and responsibilities. The factors we developed in this study conform to the latest trends in job evaluation. To adapt to changes in the internal and external environments of hospitals, the job evaluation factors must reflect the organizational strategy and pay attention to the performance, contribution and development of employees [4]. Furthermore, job evaluation systems pay attention not only to fairness but also to flexibility [31]. These trends are embodied in this study. In conclusion, the CN-PJE can reasonably represent the relative values of physicians.

The comparative analysis of two job evaluation systems
There are similarities and differences between the NHS JE and the CN-PJE, as shown in Table 7. Compared with the NHS JE system (five dimensions and 16 factors), the CN-PJE consists of similar dimensions but more factors (five dimensions and 25 factors). The factors of the Effort/Environment dimension are exactly the same, and the factors of Responsibility are almost the same as those of the NHS JE system. The results suggest that these dimensions and factors are reasonable.
In terms of differences, these two systems apply to different posts in healthcare organizations. The NHS JE scheme is used to determine the pay bands for all staff employed in the NHS, with the exception of doctors, dentists and some senior managers, who have separate pay contracts [14,15,32]. In contrast, the job evaluation factors in this study are specifically developed for physicians. Although the NHS JE system does not apply to physicians, it provides a rational framework for this study because most evaluation factors are common to all job positions. Moreover, significant effort has been made in this study to take into account the different aspects of physician posts so that the evaluation factors are adopted to the particular characteristics of physicians.
There are some differences in the factors of some dimensions. First, there are different numbers of factors under the Knowledge dimension. In the NHS JE system, there is only one factor under the Knowledge dimension to measure all the forms of knowledge required to fulfil the job responsibilities. This factor is the most heavily weighted factor (24%) and usually can make a difference between one pay band and the next [14]. In our study, we identified five factors to measure different forms of knowledge, specifically professional knowledge, experience, physical skills, awareness of quality and safety, and knowledge updates. Previous studies have suggested that having only one factor to measure all forms of knowledge might result in inaccurate and inconsistent outcomes [11,18]. Indeed, in addition to abundant professional knowledge, strong practical skills and rich experience [28], physicians need to have an awareness of quality and safety, given that patient safety is one of the most important components of health care delivery. In addition, physicians also need to constantly improve themselves through learning and updating new knowledge and techniques. Therefore, it is essential to define factors to measure each of these areas of knowledge. Second, this study identified three new factors in the Task Characteristics dimension. Task complexity and Task relevance are mentioned by many other models, including the General Schedule of the U.S. Office of Personnel Management [13], Hay's Job Evaluation model [25] and the International Position Evaluation model [12]. The amount of working time and duty periods vary from one post to another. Consequently, it is important to take into account the temporal Awareness of quality and safety Newly added factor characteristics. As suggested by our survey results, these new factors are highly accepted by physicians. In brief, these five factors can fully reflect the post characteristics of physicians. Third, under the Skills dimension, Emergency response skills and Innovation skills are factors that are not included in the NHS JE system but are identified in our system. These factors can clearly reflect the skills required for physicians to address complicated medical services and the diversified clinical demands of patients. Training is from the Knowledge dimension, and Responsibility for patient/client care is from the Skills dimension of the NHS JE system. These two factors are the basis of other skills. Overall, the factors in this study are significantly correlated and can fully capture the characteristics of physicians.

The application and implication of CN-PJE
The point-factor job evaluation is a systematic method for realizing internal equity within public organizations [11,33]. The current Post-Performance Payment system in Chinese public hospitals fails to consider internal relativities across different posts and results in internal inequity, job dissatisfaction and physicians' profit driven incentive [20,22,23]. The CN-PJE system can be used in salary plans and to ensure internal equity for physicians. As the CN-PJE system is built on survey data from physicians, its design can better reflect what values physicians consider to be fair and will thus be better accepted by physicians. The CN-PJE has been adopted by two public hospitals in Beijing and have been used in their internal compensation programmes. Therefore, physician job evaluation can provide an important basis for the development of physician compensation plans. In addition, it can be combined with a variety of performance appraisal methods, such as the balanced scorecard and key performance indicators, to develop a reasonable salary system for physicians [34,35]. Furthermore, based on China's explorations, the methods adopted by this study will be applicable to the localization of NHS JE and the establishment of job evaluation factors for other positions and in other countries.

Limitations
Several limitations of this study should be mentioned. First, the samples for this study had a limited geographical range. As a result, this study can only represent the situation in Beijing and it can be viewed as a preliminary study. Further studies conducted in China's different regions will reflect China's whole situation and allow greater generalization. Second, the established CN-PJE has only been used in two hospitals in Beijing. The applicability of the system needs further verification through practice. Finally, this study was conducted in tertiary hospitals. Future studies can test the CN-PJE in other different levels of hospitals.

Conclusions
This study developed job evaluation factors for physicians in public hospitals in Beijing, China. The CN-PJE system is an instrument with fair reliability and validity, consisting of five dimensions and 25 factors. These factors reflect the position characteristics of physicians in a comparatively complete and scientific manner. This system can provide an important basis for developing physician compensation plans and ensuring equal pay for work of equal value. The methods adopted by this study will be applicable to the establishment of job evaluation factors for other positions and other countries.
Supporting information S1