Meta-Review of the Quantity and Quality of Evidence for Knee Arthroplasty Devices

Introduction Some cardiovascular devices are licensed based on limited evidence, potentially exposing patients to devices that are not safe or effective. Research is needed to ascertain if the same is true of other types of medical devices. Knee arthroplasty is a widely-used surgical procedure yet implant failures are not uncommon. The purpose of this study was to characterize available evidence on the safety and effectiveness of knee implants. Methods A review of primary studies included in health technology assessments (HTA) on total (TKA) and unicompartmental knee arthroplasty (UKA) was conducted. MEDLINE, EMBASE, CINAHL, Cochrane Library and Biotechnology & BioEngineering Abstracts were searched from 2005 to 2014, plus journal tables of contents and 32 HTA web sites. Patients were aged 18 and older who underwent primary TKA or UKA assessed in cohort or randomized controlled studies. Summary statistics were used to report study characteristics. Results A total of 265 eligible primary studies published between 1986 and 2014 involving 59,217 patients were identified in 10 HTAs (2 low, 7 moderate, 1 high risk of bias). Most evaluated TKA (198, 74.5%). The quality of evidence in primary studies was limited. Most studies were industry-funded (23.8%) or offered no declaration of funding or conflict of interest (44.9%); based on uncontrolled single cohorts (58.5%), enrolled fewer than 100 patients (66.4%), and followed patients for 2 years or less (UKA: single cohort 29.8%, comparative cohort 16.7%, randomized trial 25.0%; TKA: single cohort 25.0%, comparative cohort 31.4%, randomized trial 48.6%). Furthermore, most devices were evaluated in only one study (55.3% TKA implants, 61.1% UKA implants). Conclusions Patients, physicians, hospitals and payers rely on poor-quality evidence to support decisions about knee implants. Further research is needed to explore how decisions about the use of devices are currently made, and how the evidence base for device safety and effectiveness can be strengthened.


Introduction
Medical decision-making is meant to be informed by the best available evidence, clinical judgment, and patient values and preferences. However it appears that what constitutes evidence may differ between drug and non-drug technologies. While pharmaceutical products must undergo years of rigorous testing, analysis of evidence for high-risk cardiovascular devices approved by the United States Food and Drug Administration found that the quantity and quality of pre-and post-market studies was lacking, potentially exposing patients to devices that were not safe and effective [1,2]. The same was true of metal-on-metal total hip replacement, which was associated with high revision rates, and subsequent analysis of explanted components found that they had been modified from the manufacturer's specifications [3].
Before advocating for broad changes in the policies and processes of pre-and/or post-market surveillance, further studies are needed to ascertain if the same is true of other types of medical devices. Knee arthroplasty is among the most common and effective procedure currently performed [4], and is expected to increase in frequency [5]. However, surgical complications and implant failures are not uncommon, with ten-year revision rates of 6.2% for total (TKA), and 16.5% for unicompartmental knee arthroplasty (UKA) [6,7].
No research has fully described the evidence that payers, hospitals, physicians and patients must rely on when making decisions about knee arthroplasty. It was hypothesized that, similar to studies of cardiovascular devices [1][2][3], evidence on the safety and effectiveness of knee implants may be limited due to issues of randomization, blinding, and the expense of measuring long-term outcomes [5]. The purpose of this study was to characterize the nature of the available evidence regarding the safety and effectiveness of knee arthroplasty devices. Specifically, this study sought to describe limitations of studies that evaluated knee implants.

Approach
This study described the limitations of studies that evaluated knee arthroplasty devices; it did not seek to assess if knee implants are clinically effective as that research has been done [4]. A meta-review was conducted of primary studies included in health technology assessments (HTAs) of knee arthroplasty devices. HTA is defined as the systematic evaluation of properties, effects and/or impacts of health technologies and interventions including intended and unintended consequences [6]. We used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) criteria (S1 Checklist) [7]. A protocol for this review was not registered. Institutional review board approval was not necessary.
then to screen primary studies included in those reviews. Patients included adults aged 18 and older from any country who underwent knee arthroplasty for any indication. The intervention of interest was primary total (TKA) or unicompartmental knee arthroplasty (UKA). Comparisons included single cohort studies evaluating a device, either before and after, or only after surgery; or comparative cohort studies or randomized trials comparing patients before and after, or only after receiving different types of devices. Studies varied in the outcomes they reported. To include studies that evaluated the safety and effectiveness of devices while favouring inclusion, eligible studies reported at least two of the following most frequently reported outcomes: complications (surgical or device-specific),revision rate (absolute number or device survival), or functional outcomes (e.g. pain, health status, quality of life, ability to complete physical tasks, satisfaction) assessed either by clinicians or patients using standardized instruments. Searches were limited to English language. Publications in the form of editorials, protocols, abstracts, or proceedings were not eligible. Studies were not eligible if they focused on evaluating the effectiveness of a surgical approach (i.e. minimally invasive, computer-aided) or technique (e.g. posterior, lateral or anterior approach; type of incision, sutures, instrumentation, bone cement), or on rehabilitation interventions or quality of life following surgery. This review focused on HTAs because HTAs are a form of evidence that is readily available to the majority of health care professionals to inform real-time decisions about which devices to purchase and use; and include primary studies of pre-and post-market evaluation upon which regulatory licensing decisions are made. Studies based on registry data were excluded. Although such studies provide useful data due to the large number of included patients, they do so only after a considerable period of time during which devices were licensed and used in many patients.

Searching and screening
Knee arthroplasty HTAs were identified in MEDLINE, EMBASE, CINAHL, Health Technology Assessment database in the Cochrane Library, and Biotechnology & BioEngineering Abstracts. These were searched on January 14, 2015 from 2005 to 2014 inclusive. The search strategy (S1 Table) was purposefully broad to be as inclusive as possible. We also searched the tables of contents of Health Technology Assessment and the International Journal of Technology Assessment in Health Care, and 35 web sites of HTA agencies (S2 Table). Titles and abstracts were independently screened by three reviewers. All items selected by at least one reviewer were retrieved. Then the primary studies included in HTAs were screened. If two or more primary studies were based on the same cohort of patients, the single most recent or complete study was eligible and the outcomes it reported were included.

Data extraction
Data were extracted from primary studies on author, country, year published, HTA source, arthroplasty type (UKA, TKA) and device (model, company). To identify limitations of the studies that evaluated knee implants, data were extracted on study design, number of patients included in final analyses, years of follow-up and conflict of interest (independent, industry funded, undetermined). We did not extract outcome data; the effectiveness of knee arthroplasty has already been established (4). ARG and two trained research assistants independently pilot-tested data extraction on the same three articles and compared findings through two iterations at which time data extraction was congruent. Two research assistants extracted data from remaining studies. ARG independently checked data to resolve discrepancies or other issues.

Data analysis
The methodological quality of HTAs was assessed using the Assessing the Methodological Quality of Systematic Reviews (AMSTAR) instrument [8]. Each study was scored for the presence of 11 elements, and the total score was categorized as high (0 to 4), moderate (5 to 8) and low (9 to 11) risk of bias. Summary statistics were used to describe the number of studies by country, year of publication, and type of implant. The methodological quality of primary studies was described with summary statistics for study design, number of participants, length of follow-up and potential conflicts of interest.

HTA characteristics
Of the 10 HTAs, 2 had a low risk of bias, 7 had a moderate risk of bias and 1 had a high risk of bias (Table 1, based on S3 Table). Two HTAs were issued in Canada and the United States, and one each in Austria, Australia, Belgium, Italy, Netherlands, and the United Kingdom. HTAs included a median of 15.5 studies (range 3 to 115).

Primary study characteristics
The 265 primary studies were published between 1986 and 2014 by authors from 28 countries.
The majority of studies were published by authors in the United States (101/265, 38.1%). Among the 265 studies, 63 (23.8%) declared industry funding, 83 (31.3%) declared independent funding, and 119 (44.9%) offered no explicit conflict of interest statement or acknowledgement of funding.

Devices evaluated
The majority of devices were evaluated in very few studies (Table 2). For example, 26 of 47 (55.3%) TKA implants and 11 of 18 (61.1%) UKA implants were each evaluated in a single study.

Participants
Across 265 studies, 59,217 patients were evaluated (Table 3). Notably, several studies failed to report the number of patients who received knee implants. Overall, most studies had 100 or fewer participants. By study design, 61.6% of patients were assessed in SC with a median (range) of 105.5 (14.

Discussion
While hundreds of studies on knee arthroplasty devices were identified, there is little reliable data on the effectiveness and safety of most types of knee implants. Of 265 eligible primary studies, the findings of 70% of the primary studies that were industry-or undeclared sponsorship should be interpreted with some caution. The quality of evidence in primary studies was limited. Most studies were based on uncontrolled single cohorts, enrolled fewer than 100 patients, and followed patients for 2 years or less. Furthermore, most devices were evaluated in only one study. If safety or effectiveness of devices is a key concern, decisions regarding the choice of medical devices appear to be largely unsupported by reliable evidence. Similar findings were identified in other assessments of syntheses and of primary studies. Sharma et al. [20] assessed the methodological quality of 77 meta-analyses in joint arthroplasty. Among these5 (6%) had extensive flaws, 34 (44%) had major flaws, 30 (39%) had minor flaws, and 8 (10%) had minimal flaws; the quality of 14 meta-analyses based on TKA was not reported. Nieuwenhuijse et al. [21] conducted a systematic review to appraise the evidence base for orthopedic devices including high flexion TKA. Among 56 studies describing 52 cohorts, study quality was judged to be low or moderate in over 60% of 56 studies describing 52 cohorts. However, our review was a more comprehensive assessment of the quality of primary studies on knee implants than either of these studies. Our meta-review included fewer syntheses of knee arthroplasty than the Sharma et al. study [20], but it included more detail about the quality of the primary studies. Our meta-review also included many more primary studies than the Nieuwenhuijse et al. review [21].
Our study has several strengths. We used rigorous meta-review methodology, and applied stringent eligibility criteria to retrieve the highest quality of evidence available. We may not have identified all eligible studies based on the search strategy employed, and because registry studies and non-English language studies were excluded. Notably, there was little overlap of primary studies across HTAs, in part due to the fact that the HTAs differed in the span of years    Explicit-the number of participating patients was clearly reports Unclear-studies used two or more devices but did not report the division of patients/knees between these thus the number reported here represents the total number of patients (i.e. 50 knees were implanted with either Device X or Device Y) Min/Max-studies reported either a minimum or maximum number of participants; the number reported here reflects the number of participants reported by studies that stated either a minimum or maximum doi:10.1371/journal.pone.0163032.t002 they covered. Primary studies varied in the consistency and completeness of information they reported so it was difficult to extract and summarize data. Given that included studies were published as early as 1986, some of the devices evaluated in included studies may no longer be used. The Balliol Collaboration issued recommendations for improving the evidence base for surgical innovations [22]. However, in the current market approval process, uncontrolled clinical studies may suffice as evidence for device effectiveness and safety, so there is little incentive for manufacturers to undertake additional or rigorous studies of a more costly nature [23]. Furthermore, many medical devices are not marketed for long before they are replaced by newer versions, and therefore fail to undergo sufficient, long-term evaluation [24]. Given the rapid rate of new medical device development and marketing [25], and tensions between systemlevel funding policies and organizational purchasing decisions [26], future research should investigate how to generate, synthesize and share evidence on the safety and effectiveness of medical devices in a manner that balances innovation and safety. Others have suggested that improved regulation (pre-market) and professional society oversight (post-market) strategies are both needed to optimize patient safety [21].
Although users of health technologies are expected to use evidence to guide decisions about the use of medical devices, our study of knee arthroplasty implants-among the most commonly used implantable devices in major surgical procedures-suggests that little high-quality evidence actually exists. Our study raises serious questions about the nature of clinical evidence supporting the safety and effectiveness of implants used for knee arthroplasty Supporting Information S1 Checklist. PRISMA Checklist. (DOC) S1