Initial conditions

First, the LHTAU (the agent) and the decision maker who is asking for a technology assessment (the principal) are located in the same organization (i.e. a health care facility). Therefore, the LHTAU is not structurally or institutionally independent of the decision maker. Second, some of the technologies that the LHTAU is asked to evaluate are already in use in the facility, while others are not. This initial condition is crucial to our theoretical prediction, because it is the condition that directly makes our research hypothesis testable. Third, the LHTAU produces a report following a voluntary demand formulated by a decision maker who wishes to receive a recommendation regarding the relevance of using a health technology. As a consequence, the existence and survival of the LHTAU directly depend on its ability to stimulate the demand for its evaluation services within the facility. To stimulate the demand for its evaluation services, the evaluation unit needs to produce HTA reports that include recommendations sufficiently aligned with the preferences of the decision maker. Otherwise, the perceived costs of demanding a technology assessment would be higher, and the demand for the service would ultimately decline. In other words, the LHTAU is structurally incentivized to formulate recommendations that meet the decision maker's utility. The LHTAUs considered in this study can make decision makers reveal their preferences through conversations during the production phase or by asking them to comment on preliminary versions of their reports.

We can consider decision makers as having a multi-attribute utility function, meaning that a given HTA can procure their satisfaction for two main reasons, that is, by increasing their knowledge about a technology and by providing them with arguments to legitimize or delegitimize its use. Like the decision maker (the principal), the LHTAU (the agent) can also be seen as having a multi-attribute utility function. We posit that this utility function is composed of two main attributes, each corresponding to a reputational asset that is needed to survive as a local HTA unit. First, an HTA mandate provides the agent with the opportunity to increase its scientific reputation. In effect, to stimulate the demand for its evaluation services, the evaluation unit needs to produce HTA reports that are seen as coherent and rigorous. For example, in an HTA report that recommends favorably the use of a given technology, the authors of the report must have concluded that the technology is effective based on a systematic review of existing research evidence. Second, an HTA mandate provides the agent with the opportunity to increase its reputation of relevance or usefulness, a reputational asset that should be distinguished from the scientific reputation. A scientifically sound HTA that concludes that a technology currently in use in a facility is significantly outperformed by an alternative one could be perceived by a decision maker as a constraint given that the replacement of the technology is likely to be unacceptable in a given unit for various reasons such as when the technology is an important share of the practitioners’ revenues. Therefore, through their assessment reports, LHTAUs play their scientific and relevance reputation, two attributes that can be antagonistic.

The difference between the utility function of the agent and of the principal can theoretically be detrimental to one party over the other. This would be the case, for example, in a situation where the principal asks for an HTA to legitimize the use of a technology, thus inciting through different communication signals the principal to be aware of this preference, but in trying to satisfy this principal’s preference, the agent may contribute to the reduction of its scientific reputation, an important attribute of its utility function. An LHTAU with a utility function that includes only one attribute—say scientific reputation (thus excluding the reputation of relevance)—could produce HTAs that would be detrimental to a decision maker who wishes to use the HTA recommendations to legitimize a technology, as in this case, the agent would not be willing to make a compromise that satisfies the strategic utility of the principal. In sum, the demand for HTAs within the health-care facility is possible in a context where the LHTAU has a dual-attribute utility function that makes it act to maintain both its scientific and relevance reputation.

Empirical and theoretical regularities

In order to have a scientific explanation, these initial conditions should be combined with empirical or theoretical regularities.[7] First, there can be no communication or transmission of information between an agent and a principal when their preferences toward a state of the world differ too much.[9] This regularity has been tested multiple times in laboratory experiments.[4,10–13] Second, a principal will not delegate work to an agent with preferences that are too different from his own.[14] Third, overtime, workers in hierarchical organizations such as health care facilities tend to develop pride towards their practices[15] and to resist changes promoted by outsiders.[16] Finally, as already mentioned, health technology assessments have been shown to be used symbolically, that is to legitimize or delegitimize the use of health technologies.[17–21] /

Data

We created a database of health technologies using the following inclusion criteria:

• assessed by the main LHTAU of the following university-affiliated health care centers: McGill, CHU de Montreal (CHUM), CHU de Québec and CHU de Sherbrooke;
• assessed in a formal, full-scale, health technology assessment report (excluding mini-HTA reports and knowledge syntheses) published on the LHTAU's web site from the date of creation of the evaluation unit until May 2016;
• assessed following an explicit demand from one or more individuals working inside the health care center as indicated in the HTA report (excluding technologies assessed following a demand from the outside);
• assessed in an HTA report that includes explicit recommendations regarding the relevance of using the technology;
• assessing the technology was an explicit objective indicated in the HTA report.

According to a recent mapping of resources in the Province of Quebec[22], 19 health care institutions have a health technology evaluation unit. For reasons of budgetary nature, it was not possible to study the reports of all 19 LHTAUs. We then chose to analyze the reports of the main LHTAU of each territory attached to one of the four faculties of medicine in Quebec, that is, the LHTAU of the Quebec City university health center (attached to Laval University), the Montreal university health center (attached to the University of Montreal), the McGill University Health Center, and the LHTAU of the Sherbrooke’s university health center.

The independence of the observations (i.e. the technologies) was ensured by excluding three technologies that have been assessed independently by two LHTAUs. When a given technology was assessed in two or more reports produced by the same LHTAU, we selected the most recent and up-to-date assessment.

Data extraction and coding

The following information was extracted from the HTA reports by two independent coders (PL and AR) (verified by one auditor (MO)):

• name of the technology;
• type of technology (preventive, diagnosis, therapeutic or organizational);}
• name of the internal demander(s);
• name of the LHTAU;
• technology assessed in a report that also includes an assessment of one or more other technologies (yes or no);
• use status (technology already in use in the facility (yes or no));
• global efficacy judgment (authors believe that based on their review of the research evidence, the technology is effective, not effective or not enough supporting evidence—the last two categories were merged);
• direction of the recommendation regarding the relevance of using the technology (unfavorable or favorable).

The percentage of agreement between the two independent coders was 94.4 percent for the main explanatory variable (i.e. use status at the time of the evaluation) and 97.2 percent for the outcome variable (i.e. direction of the recommendation). Coding examples for the outcome variable are presented in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Coding examples for the outcome variable.

https://doi.org/10.1371/journal.pone.0185183.t001

As for the global efficacy judgment variable, we coded the sentences in the reports where the assessors summarize their interpretation of the research evidence that they found in their systematic literature review regarding the extent to which the technology succeeds in achieving its aim. Although all LHTAUs seem to use the same broad analytical approaches to determine efficacy (e.g. systematic review of RCTs), we noticed that the parameters used to form their final judgment of the available evidence are not standardized, meaning that they vary from one HTA report to the others. For example, the paucity of evidence on how to use a given technology (e.g. data on dosage) has been used to support the claim that there is not enough supporting evidence in favor of a technology, while in another report, the authors explicitly stressed the same gap in the evidence, but did not refrain from concluding that enough evidence exists in support of the technology.

Each technology in the database was classified in one of the following types: preventive, diagnosis, therapeutic or organizational. A preventive technology is a technology that aims at reducing the likelihood of adverse health outcomes (e.g. Safety devices to reduce needlestick injuries). Diagnosis technologies are those designed and used to identify the presence and nature of a health condition (e.g. Video Capsule Endoscopy Systems). Technologies aiming at treating a health condition were classified as therapeutic (e.g. Transcatheter aortic valve implantation). Finally, technologies that were classified as “organizational” are those that aim at improving one or more organizational or systemic aspects of health care (e.g. comprehensive policy for the antiviral treatment of hepatitis C patients, computerized prescription medical entry software, etc.).

Data analysis

All data analyses were performed with Stata v12. The Stata file including the variables described below is available at: http://doi.org/10.3886/E100955V1. The research hypothesis was tested by estimating a logistic regression equation with the direction of the recommendation (i.e. unfavorable vs. favorable) as the outcome variable and the following variables as correlates:

• use status (0 = not in use in the facility at the time of the evaluation; 1 = in use in the facility at the time of the evaluation);
• global efficacy judgment (0 = authors believed that, based on their review of the research evidence, the technology is ineffective or there is not enough supporting evidence; 1 = the technology is considered effective);
• type of technology (one dummy per type of technology—therapeutic as the reference category);
• the LHTAU (one dummy per unit—McGill as the reference category);}
• technology assessed in a report that also includes an assessment of one or more other technologies (0 = no; 1 = yes).

We also estimated a logistic regression equation with the variable 'use status' as the outcome variable to examine the characteristics of the technologies that were in use in the facility at the time of the evaluation as compared to the technologies that were not in use. Rate ratios (RR) and their corresponding 95 percent confidence interval were calculated by using 'oddsrisk', a Stata module to convert Logistic Odds Ratios to Risk Ratios.[23] Each RR presented in the paper corresponds to the effect of one predictor of interest on the outcome variable. Each RR was calculated based on the prevalence rate of the predictor of interest (binary (1/0)).

Sample characteristics

From the 108 health technologies included in our database, 63.9 percent (69) were recommended favorably by the LHTAU, while the use of 36.1 percent (39) were not recommended. Six of the 108 technologies were not explicitly assessed for their efficacy. From the 102 technologies that were assessed for efficacy, 61.8 percent (63) were judged as globally effective based on the review of existing research evidence, while 38.2 percent (39) were judged as either ineffective or as lacking proof of efficacy. A little more than half of the 108 health technologies were in use within the health care facility at the time of the evaluation (53.7 percent; n = 58), while 46.3 percent (50) were not in use. The majority of the technologies (59.3 percent; n = 64) were the only technology assessed in the HTA report.

The McGill HTA unit is the one that evaluated the largest number of technologies (44.4 percent; n = 48), followed by the CHU de Quebec (25 percent; n = 27), the CHU de Sherbrooke (17.6 percent; n = 19) and the CHUM (Montreal) (13 percent; n = 14). More than the half of the 108 technologies (53.7 percent; n = 58) were therapeutic, 18.5 percent (20) were preventive, 16.7 percent (18) were diagnosis technologies, and only 11.1 percent (12) were of the organizational type (e.g. new way of organizing services).

Technologies' use status at the time of the evaluation

The technologies that were in use in the facility at the time of the evaluation were not more (or less) likely to be judged as effective by the assessors than the technologies that were not in use (RR = 1.41; 95 percent CI = 0.86–1.85). As compared with technologies that were not in use in the facility at the time of the evaluation, the technologies that were in use were more likely to be assessed in a report that also includes an assessment of one or more other technologies (RR = 1.68; 95 percent CI = 1.08–2.10). The technologies that were in use in the facility at the time of the evaluation were less likely to have been evaluated by the CHU de Sherbrooke's unit than by the McGill unit (RR = 0.30; 95 percent CI = 0.08–0.83) and of being of the organizational type rather than of the therapeutic one (RR = 0.17; 95 percent CI = 0.02–0.89)

Are recommended technologies judged as more effective than unrecommended ones?

The answer to this question is clearly yes. More precisely, while controlling for the correlates described in the data analysis subsection and by using 'global efficacy judgment' as the primary predictor in the oddsrisk model, the technologies that were judged as effective by the assessors based on the review of existing evidence were much more likely to have been recommended favorably than those which were judged as ineffective or as lacking supporting evidence (RR = 3.28; 95 percent CI = 2.64–3.48).

Do the assessors' recommendations vary according to the evaluation unit and the type of technology?

The logistic regression results show that, holding other things constant, one evaluation unit (i.e. the CHU de Sherbrooke) is more likely than the McGill one to recommend favorably the use of the technologies it assessed (RR = 1.72; 95 percent CI = 1.47–1.75). This evaluation unit has recommended favorably 16 of the 19 technologies it assessed (84.2 percent), while the positive recommendation rates for the other units were 63 percent (CHU de Quebec), 58.3 percent (McGill) and 57.1 percent (CHU de Montreal or CHUM).

As for the type of technology assessed, the regression results show that, holding other things constant, preventive technologies were less likely than the therapeutic ones to be recommended favorably (RR = 0.39; 95 percent CI = 0.09–0.98). A closer look at the raw data shows that the use of 72.4 percent of the 58 therapeutic technologies in the database was recommended favorably, as compared to 61.1 percent for the diagnosis technologies, 55 percent for the preventive technologies, and 41.7 percent for the technologies of the organizational type (although the chi-square test is not statistically significant: 5.15; p = 0.16, maybe due to the lack of statistical power).

Should our research hypothesis be rejected?

The answer to this question is no, at least based on our data. Based on the principal-agent theory, we formulated the hypothesis that health technology assessment units located in health care facilities tend to be more optimistic toward technologies that are in use in their organization than toward technologies that are not. Controlling for the correlates described in the data analysis subsection and using the variable reflecting the use status as the predictor of interest in the oddsrisk model, technologies that were in use in the facility at the time of the evaluation are 62 percent more likely than technologies that were not in use to be recommended favorably by the assessors (RR = 1.62; 95 percent CI = 1.06–1.88).

The raw data show that the use of 75.9 percent (44) of the 58 technologies that were in use in the facility at the time of the evaluation was recommended favorably, while the use of 50 percent (25) of the 50 technologies that were not in use was recommended favorably (chi-square = 7.78; p = 0.005).

It should be recalled that the association between the use status of the technology within the facility at the time of the evaluation, and on the other hand the judgment in terms of efficacy is not statistically significant (chi-quare = 1.38, p = 0.24). Descriptive analyses also show that the use of 67.7 percent of the 28 technologies that the evaluators found to be effective, but which were not in use within the institution at the time of the evaluation, was favorably recommended, while 97.1 percent of the 35 technologies considered effective that were currently in use within the facility were favorably recommended, a difference of almost 30 percentage points.

Moreover, six of the 108 technologies have not been evaluated for their effectiveness. These six technologies were in use within the facility at the time of the evaluation and assessed in a report that also includes an assessment of one or more other technologies. The evaluators, without an explicit consideration of their efficacy, favorably recommended the use of five of them.