https://orcid.org/0000-0002-0782-5721
Figures
Abstract
Context
The use of clinical-practice guidelines is a suggested method for improving health outcomes by the earlier provision of necessary and effective medical interventions.
Objective
To quantify the influence of adherence to guideline-recommended interventions in the first week of treatment for an initial low back pain (LBP) injury on lost workdays.
Methods
In a retrospective cohort of California’s workers’ compensation claims data from May 2009 to May 2018, 41 diagnostic and treatment interventions were abstracted from the medical claims for workers with acute LBP injuries and compared with guideline recommendations. Lost workdays within 1-year post-injury were compared by guideline adherence using quantile regressions.
Results
Of the 59,656 workers who met the study inclusion criteria, 66.1% were male and the average (SD) age was 41 (12) years. The median number (IQR) of lost workdays was 27 (6–146) days. In the first week of treatment, 14.2% of workers received only recommended interventions, 14.6% received only non-recommended interventions, and 51.1% received both recommended and non-recommended interventions. Opioid prescriptions fell 86% from 2009 to 2018. Workers who received only guideline-recommended interventions experienced significantly fewer lost workdays (11.5 days; 95% CI: -13.9, -9.1), a 29.3% reduction, than workers who received only non-recommended interventions. The percentage of workers receiving only recommended interventions increased from 10.3% to 18.2% over the 9 years.
Conclusion and relevance
When workers received guideline-recommended interventions, they typically returned to work in fewer days. The majority of workers received at least one non-recommended intervention, demonstrating the need for adherence to guideline recommendations. Fewer lost workdays and improved quality care are outcomes that strongly benefit injured workers.
Citation: Gaspar FW, Thiese MS, Wizner K, Hegmann K (2021) Guideline adherence and lost workdays for acute low back pain in the California workers’ compensation system. PLoS ONE 16(6): e0253268. https://doi.org/10.1371/journal.pone.0253268
Editor: Satya Surbhi, University of Tennessee Health Science Center, UNITED STATES
Received: October 15, 2020; Accepted: June 1, 2021; Published: June 17, 2021
Copyright: © 2021 Gaspar et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Study data were provided to authors under a data use agreement by California’s Department of Industrial Relations (DIR), which prohibits the authors from making the data set publicly available. Interested researchers may contact DIR (wcis@dir.ca.gov) to apply to gain access to the Workers Compensation Information System database in the same way the authors obtained the data.
Funding: This work was self-funded by ReedGroup. The funder provided support in the form of salaries for authors FG and KW, but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the ‘author contributions’ section. KH and MT were not paid for their contribution.
Competing interests: FG and KW were employees of ReedGroup Ltd during the study, which provided salaries for authors FG and KW. KH is the Editor and MT the lead researcher for the American College of Occupational and Environmental Medicine’s (ACOEM) Practice Guidelines used in this study. Drs. Hegmann and Thiese are compensated to develop the ACOEM Guidelines. ReedGroup, Ltd. are the sole proprietors of the guidelines. ReedGroup, Ltd did not have any role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
Introduction
Overtreatment and low-value care cost the U.S. healthcare system between $75.7 and $101.2 billion annually [1]. Despite the associated high cost, unnecessary or ineffective care appear to be on the rise [2]. One strategy to promote quality, value-based care is applying evidence-based medicine (EBM) to help guide treatment decisions. EBM integrates medical research with clinical expertise and patient values to support decision making based on the best available evidence [3].
In the U.S., state workers’ compensation (WC) systems have developed or adopted treatment guidelines to promote evidence-based care for occupational injuries. The most common occupational injury is back strain [4], and occupational stressors are thought to contribute to low back pain (LBP) [5, 6]. Considerable differences exist between evidence-based recommendations and current clinical practice for the treatment of LBP [7, 8]. An increasing number of studies in the literature suggest that following EBM guidelines improves outcomes and/or decreases costs [9–11]. For example, Owens et al. (2019) found that medical and total claim costs, which included payments for lost time (i.e., indemnity), decreased by $353 and $586, respectively, per unit of compliance with LBP guidelines in Workers’ Compensation Fund of Utah claims [11].
For occupational injuries and diseases, the majority of costs are related to a loss of functional capacity, including the inability to work [12]. Work absences not only reduce workers’ immediate earnings but may also increase the risk of future unemployment and loss of income [13]. In addition, both the employer and society experience negative effects from lost time, including reduced productivity and economic losses [14]. Therefore, the reduction of lost workdays and the promotion of effective, value-based care is an important area to investigate.
The aim of this study was to determine the influence of adherence to guideline recommendations on lost workdays for workers with acute LBP claims in California’s WC system.
Materials and methods
Study population
Workers’ Compensation claims reported to the California’s Department of Industrial Relations Workers’ Compensation Information System (WCIS) with a date of injury between May 2009 and May 2018 were used in this study. These dates were chosen to explore 9-year trends and capture at least one year of follow-up after the claim started. The data come from the 4010 and 5010 billing systems and include claims from employers with at least 150 employees, as smaller employers are not required to report to the WCIS. The data provided met the criteria for a limited-use data set under HIPAA and this study was not evaluated by an institutional review board because it did not meet the definition of human subjects research [15].
The study population consisted of workers with a lost time claim whose first report of injury indicated that the body part of injury was for the “low back area (lumbar and lumbosacral)” or the “lumbar and/or sacral vertebrae (vertebrae NOC trunk)” (Fig 1). Only workers with at least one medical visit were included. To focus on the treatment of uncomplicated LBP claims, claims were removed if they had an inpatient admission during the claim or a “red flag” diagnosis that may indicate treatment outside of guideline recommendations, such as fracture, cancer, infection, aortic aneurysm, and paralysis (S1 Table). Claims were excluded with stenosis diagnoses if they also had an accompanying surgical procedure (i.e., decompression, fusion, or adhesiolysis). Claims were also excluded if there had been a prior LBP claim to remove complicated clinical presentations due to LBP recurrences.
To confirm the LBP diagnosis, all primary medical diagnoses recorded within a week of the first medical visit were abstracted and at least 50% of the diagnoses had to be associated with acute LBP. Examples of back pain diagnoses not considered include pain in thoracic region, radicular pain, and spondylosis with myelopathy. Diagnoses included are listed in S2 Table.
Guideline treatments
The American College of Occupational and Environmental Medicine (ACOEM)’s Low Back Disorders Practice Guideline was used as the source of guideline recommendations [16]. ACOEM’s treatment guidelines have been adopted in California’s Medical Treatment Utilization Schedule (MTUS), which determines what is reasonable and necessary medical care [17, 18]. Forty-one distinct treatments and diagnostic interventions within a week of the first medical visit were studied. The one-week treatment timeframe was a priori set to study, as the first appointment sets patient expectations and implementation of in/effective treatments begins, which are theorized to influence and set the trajectory of the LBP claim. Only treatments that could be reliably captured in a medical billing system were analyzed. Common Procedure Terminology® and Healthcare Common Procedure Coding System codes were abstracted from the WCIS system to study diagnostic interventions and treatment procedures. National Drug Codes for the medications and topical creams were abstracted from the WCIS system and grouped using IBM Micromedex® RED BOOK® to evaluate the pharmaceutical interventions.
Workers were categorized into four distinct groups to study guideline adherence: 1) worker received only non-recommended interventions; 2) worker received only recommended interventions; 3) worker received recommended and non-recommended interventions; and 4) worker received either no intervention or other medical interventions. Other medical interventions could include treatment and diagnostic interventions that have no recommendation in the guideline. No recommendations are assigned when there is a lack of evidence and the balance of benefits, harms, and costs cannot be determined [19].
Outcomes
Lost workdays were calculated as the number of workdays between the payment start and end dates for temporary total disability. For individuals receiving lump sum indemnity payments, the lump sum was divided by their daily wage. In California, injured workers receive two-thirds of their gross wages as payment, with minimum and maximum allowed weekly payments [20]. To calculate the temporary total disability daily payment per worker, the workers’ daily wages were calculated as their yearly income divided by the number of workdays in a typical year (261) and multiplied by two-thirds. If the weekly payments were below the minimum or above the maximum allowed weekly payments, they were adjusted up or down, respectively.
Statistical analysis
Frequency and percent were calculated for all categorical variables used. Differences in the percent of workers receiving a type of medical intervention over time were tested using logistic regression models, with year of injury as the independent variable. Normality, homoscedasticity, and linearity of the continuous outcome variable, lost workdays, was assessed. Multiple variable quantile regression, an extension of linear regression, was used to test the influence of guideline adherence on lost workdays. In addition to the advantage of interpretability, quantile regressions were employed due to the right skewness of lost workdays. Covariates used in the multiple variable models included age (years), whether medical claims were present in 4010 billing system (yes/no), gender (male/female), whether the workers’ home was in a rural location (yes/no), yearly income (categorized), employment industry (categorized), employment status (regular employment vs. other), whether the worker had a previous WC claim (yes/no), time from injury to first medical visit (numeric days), medical complexity (numeric variables), year of injury, and the presence of at least one comorbidity (yes/no). Covariates were selected a priori and included if they were theoretical confounders or predictors of the outcome.
The industry of the patient’s employer was categorized using the North American Industry Classification System (NAICS) sectors, with any Standard Industrial Classification (SIC) codes mapped to 2017 NAICS codes using U.S. Census Bureau crosswalk tables [21]. When income was not available (11.4% of workers), it was imputed using the median income of the workers’ Standard Occupational Classification (SOC) or, if unavailable, the NAICS sector [22]. The workers’ SOC was determined by mapping the workers’ job description to the O*NET 24.2 database’s job descriptions [23]. Rurality of the worker was determined by mapping the worker’s zip code to rurality indicators by the Centers for Medicare & Medicaid Services [24]. Medical complexity was derived by counting the number of distinct medical visits and diagnoses in the first week of treatment, with higher counts indicating more complex cases. Comorbidities recorded in the first week were grouped using Quan et al.’s (2005) coding algorithm [25]. In addition, whether a worker smoked was determined using nicotine dependence diagnosis codes. Due to the limited capturing of comorbidities and smoking status, a single binary variable was developed to indicate if at least one comorbidity was present or if the worker smoked.
Missing variables (maximum percent missing = 16.5% for employment status) were imputed using the observed empirical distributions. As a sensitivity analysis, the quantile regression models were rerun to include 5,640 workers who had a primary LBP diagnosis that was considered unspecific and more general than the diagnoses used for the study population (S2 Table). Examples of unspecific LBP diagnosis include spondylosis of unspecified site, Schmorl’s nodes of unspecified region, and unspecified thoracic, thoracolumbar and lumbosacral intervertebral disc disorder. The sensitivity analysis was used to test the influence of our medical code inclusion criteria, as medical billers may use these unspecific codes when billing for LBP. A second sensitivity analysis was performed to test the influence of treatment bias using propensity score weights in the multiple variable quantile regression. Propensity score weights were calculated using multinomial predicted probabilities, where the dependent variable was treatment recommendation categories and the independent variables were a subset of the full model covariates hypothesized to be associated with treatment allocation. These independent variables included age, gender, rurality, number of comorbidities, and year of injury.
Due to the large sample size, statistical significance was a priori set at a p-value of < 0.01. Data management was performed in SQL Server 2012 and data cleaning and analyses were performed in R Version 3.6.1 [26].
Results
A total of 59,656 workers met the study inclusion criteria and had an average (SD) age of 41 (12) years (Table 1). The workers tended to be male (66.1%), from an urban area (92.9%), and had regular employment (66.7%). The median yearly income was $32,404. The top three industry sectors were retail trade (15.2%), manufacturing (11.4%), and health care and social assistance (11.1%). The median number (interquartile range [IQR]) of lost workdays was 27 (6–146).
In the first week of treatment, 14.2% of workers received only recommended interventions, 14.6% received only non-recommended interventions, and 51.1% received both recommended and non-recommended interventions. Fewer than half of the workers received multiple (>1) recommended interventions (43.7%) or multiple non-recommended interventions (23.7%). The most common recommended interventions included prescriptions for nonsteroidal anti-inflammatory drugs (NSAIDs) (44.2%) and muscle relaxants (35.2%) (Table 2). The most common non-recommended interventions included x-rays (49.6%) and prescriptions for opioids (19.6%). Between the years 2009 and 2018, the percentage of workers receiving only recommended interventions increased from 10.3% to 18.2%, whereas the percentage receiving only non-recommended treatments increased at a slower rate from 12.1% to 15.4%.
For the most frequent interventions, there were statistically significant trends over the study years, some increasing and some decreasing (p-value < 0.01, Fig 2). Interventions that increased through the study period included ketorolac injections (strongly recommended), acetaminophen (recommended), manual therapy/massage (recommended), physical or occupational therapy (recommended), and x-rays (moderately not recommended). Interventions that decreased through the study years included NSAIDs (strongly recommended), muscle relaxants (moderately recommended), manipulation (recommended), lidocaine and topical creams (no recommendation), infrared therapy (no recommendation), electrical stimulation (not recommended), and opioids (strongly not recommended). Opioids had the largest decrease, with 28.9% of workers receiving a prescription in 2009 but only 4.1% receiving a prescription in 2018, an 86% reduction.
Workers receiving an x-ray had slightly longer lost workdays (median = 28 days) than workers not receiving an x-ray (median = 27 days) (Kolmogorov-Smirnov test p-value <0.001). Opioid prescriptions had more of an influence on lost workdays, as workers prescribed an opioid were typically out of work four days longer than workers not prescribed opioids (medians = 30 vs. 26 days) (Kolmogorov-Smirnov test p-value <0.001).
In multiple variable quantile regression models, workers who received only recommended interventions experienced 11.5 fewer lost workdays (95% CI: -13.9, -9.1), a 29.3% reduction, than workers who received only non-recommended interventions (Table 3 and S3 Table). Workers who received both recommended and non-recommended interventions experienced 7.9 fewer lost workdays (95% CI: -10.3, -5.5) than workers who received only non-recommended interventions. In addition, workers who received either no intervention or other medical interventions experienced more favorable outcomes than those who received non-recommended interventions (Beta = -7.7, 95% CI: -10.3, -5.1). The sensitivity analysis that included workers with primary LBP diagnoses that were less specific to acute LBP did not meaningfully change the results (S4 Table). In addition, propensity score weights in the quantile regressions did not meaningfully change the results (S5 Table).
Discussion
Workers receiving only recommended interventions incurred 11.5 fewer lost workdays, a 29.3% reduction, compared with those who received only non-recommended interventions in this large, retrospective cohort study. The majority of workers received at least one medical intervention that was not recommended by the ACOEM Low Back Disorders Practice Guideline. The percentage of workers receiving only recommended interventions increased from 10.3% to 18.2%. Opioid prescriptions fell 86% from 2009 to 2018. X-rays were a common non-recommended diagnostic intervention that increased in frequency throughout the study period.
According to the Institute of Medicine, the delay for incorporation of EBM into clinical practice approaches 17 years [27]. Identifying, digesting, and integrating research findings into clinical practice can be overwhelming for most clinicians, given that approximately 2 million papers are published annually and the number of published papers grows exponentially at 8–9% per year [28]. Even when limiting a research literature review to randomized control trials, a clinician needs to read approximately 48 papers a day to stay current with the literature [29]. The trends in changes in rates of prescriptions for specific treatments over nine years appear to support a long time horizon for the incorporation of practice changes. Reliance on high quality EBM treatment guidelines is important and will likely become more necessary in the future.
To address the disconnect between evidence and practice, it has been proposed that applying clinical decision-support systems and tools to bring guidelines to the point-of-care improves healthcare processes, such as facilitating preventive services, ordering clinical studies, and prescribing therapies [30]. In this spirit, as part of the Protecting Access to Medicare Act, the Centers for Medicare & Medicaid Services will require practitioners, or staff acting on their behalf, to consult a clinical decision support mechanism when ordering advanced diagnostic imaging services for Medicare beneficiaries starting on January 1, 2021. In 2018, the California Department of Industrial Relations began providing web-based access to ACOEM’s guidelines for all California workers’ compensation system providers. The expectation is that giving clinicians increased access to guidelines will better align treatment decisions between providers, workers, employers, and payers.
Changes in California’s workers’ compensation MTUS, including the adoption of an evidence-based drug formulary based on ACOEM’s guidelines (effective date January 1, 2018), were not considered in the analysis. Through the study period, four versions of ACOEM’s guidelines were published; however, the recommendations, especially for the most common treatments and diagnostic interventions, have not significantly through the study period (S6 Table). For purposes of workers’ compensation reimbursement, a significant change is to move from recommended to not recommended, or vice versa, as other changes (e.g., from Recommended C to Recommended B) do not change approvals and payments for services or treatments. For example, x-rays and opioids for acute low back pain, the two most common not recommended diagnostic intervention and treatment found in this study, have been not recommended in every version since 2008.
In the state of California, ACOEM’s guidelines are the presumptively correct guidelines that physicians must adhere to in treatment and diagnostic intervention decisions. Therefore, we did not consider the recommendations from other guidelines including the American College of Physicians (ACP) or the United Kingdoms’ National Institute of Health and Care Excellence (NICE) guidelines [31, 32]. In general, ACOEM guidelines tended to match NICE’s recommendations better than ACP’s guideline recommendations (summarized in S7 Table). For example, ACOEM and NICE do not recommend acupuncture for treatment of acute low back pain, whereas ACP does recommend acupuncture. The differences in recommendations between guideline may be important to consider when extrapolating the results of this study to other populations that may follow different treatment guidelines.
The strengths of this research include the use of a large, population-based integrated database to match medical and lost workdays information. The use of medical billing information allowed the investigation of 41 medical interventions; however, this study could not investigate interventions that were not tracked by the claims system, including over-the-counter NSAID use, bed rest, and modified duty. The coding also does not allow for detailed assessment of specific types of exercise, rather it is captured at the level of physical or occupational therapy. Confounders were identified and statistically controlled using multivariable regression. These included two medical complexity variables used to control for disease severity; however, additional metrics including standardized pain scales were not available. Additionally, temporality of the treatments is demonstrated by these data; therefore, the potentially causal link between recommended treatments and fewer lost workdays is plausible. Reverse causation in this instance is unlikely. Some variables known to influence lost workdays including race or psychosocial factors were not available and could not be controlled for in our analysis. Finally, the workers’ job requirements were not known, which may have influenced their ability to return to work and the total number of lost workdays, although this seems likely to have been non-differentially distributed and thus a possible bias towards the null.
Conclusions
When workers received guideline-recommended interventions, they typically returned to work in fewer days. The majority of workers received at least one non-recommended intervention, demonstrating the need for adherence to guideline recommendations. Fewer lost workdays and improved quality care are outcomes that strongly benefit injured workers.
Supporting information
S1 Table. Red flag diagnoses used to exclude claims.
Red flag diagnoses may support medical interventions outside of guideline recommendations and thus were a priori excluded.
https://doi.org/10.1371/journal.pone.0253268.s001
(DOCX)
S2 Table. Medical codes used to define acute low back pain (LBP) cases.
Primary LBP codes are medical codes used to define the study population, whereas the unspecific LBP codes were used in the sensitivity analysis to explore the influence of the LBP definition.
https://doi.org/10.1371/journal.pone.0253268.s002
(DOCX)
S3 Table. Full results of the quantile regression model testing the influence of receiving only recommended, only non-recommended, and both recommended and non-recommended treatments on lost workdays.
https://doi.org/10.1371/journal.pone.0253268.s003
(DOCX)
S4 Table. Sensitivity analysis including workers whose primary diagnosis was unspecific to the acute LPP diagnoses used in the main study population (n = 65,296) testing the influence of receiving only recommended, only non-recommended, and both recommended and non-recommended treatments on lost workdays.
https://doi.org/10.1371/journal.pone.0253268.s004
(DOCX)
S5 Table. Sensitivity analysis including propensity score weights within the quantile regression testing the influence of receiving only recommended, only non-recommended, and both recommended and non-recommended treatments on lost workdays.
https://doi.org/10.1371/journal.pone.0253268.s005
(DOCX)
S6 Table. ACOEM Guidelines’ acute low back pain recommendation changes over time for common treatments in study.
https://doi.org/10.1371/journal.pone.0253268.s006
(DOCX)
S7 Table. Comparisons between select recommendation statements of the ACOEM, American College of Physicians (ACP) and United Kingdoms’ National Institute of Health and Care Excellence (NICE) guidelines for acute low back pain.
https://doi.org/10.1371/journal.pone.0253268.s007
(DOCX)
References
- 1. Shrank WH, Rogstad TL, Parekh N. Waste in the US Health Care System Estimated Costs and Potential for Savings. 2019;40202.
- 2. Brownlee S, Chalkidou K, Doust J, Elshaug AG, Glasziou P, Heath I, et al. Evidence for overuse of medical services around the world. Lancet. 2017 Jul;390(10090):156–68. pmid:28077234
- 3.
Straus S, Paul G, Richardson SW, Haynes BR. Evidence-Based Medicine—5th Edition. Elsevier. Philadelphia; 2018.
- 4.
WorkSafeBC. Statistics 2018.
- 5. Punnett L, Prüss-Üstün A, Nelson DI, Fingerhuf MA, Leigh J, Tak SW, et al. Estimating the global burden of low back pain attributable to combined occupational exposures. American Journal of Industrial Medicine. 2005. pmid:16299708
- 6. Driscoll T, Jacklyn G, Orchard J, Passmore E, Vos T, Freedman G, et al. The global burden of occupationally related low back pain: Estimates from the Global Burden of Disease 2010 study. Ann Rheum Dis. 2014. pmid:24665117
- 7. Kim LH, Vail D, Azad TD, Bentley JP, Zhang Y, Ho AL, et al. Expenditures and Health Care Utilization Among Adults With Newly Diagnosed Low Back and Lower Extremity Pain. JAMA Netw open. 2019;2(5):e193676. pmid:31074820
- 8. Foster NE, Anema JR, Cherkin D, Chou R, Cohen SP, Gross DP, et al. Prevention and treatment of low back pain: evidence, challenges, and promising directions. Lancet (London, England). 2018 Jun;391(10137):2368–83.
- 9. Gaspar FW, Kownacki R, Zaidel CS, Conlon CF, Hegmann KT. Reducing Disability Durations and Medical Costs for Patients With a Carpal Tunnel Release Surgery Through the Use of Opioid Prescribing Guidelines. J Occup Environ Med. 2017;1. pmid:28937443
- 10. Demian C, Barron B, Almudevar A. Effects of the New York State Workers Compensation Board Medical Treatment Guidelines on Return to Work. J Occup Environ Med. 2018;60(7):617–21. pmid:29280771
- 11. Owens JD, Hegmann KT, Thiese MS, Phillips AL. Impacts of Adherence to Evidence-Based Medicine Guidelines for the Management of Acute Low Back Pain on Costs of Worker’s Compensation Claims. J Occup Environ Med. 2019;61(6):1. pmid:31167221
- 12. Leigh JP. Economic burden of occupational injury and illness in the United States. Milbank Q. 2011 Dec;89(4):728–72. pmid:22188353
- 13. Helgesson M, Johansson B, Nordqvist T, Lundberg I, Vingård E. Sickness absence at a young age and later sickness absence, disability pension, death, unemployment and income in native Swedes and immigrants. Eur J Public Health. 2015.
- 14. Schulte PA, Pana-Cryan R, Schnorr T, Schill AL, Guerin R, Felknor S, et al. An approach to assess the burden of work-related injury, disease, and distress. Am J Public Health. 2017;107(7):1051–7. pmid:28520495
- 15.
U.S. Department of Health and Human Services. Protection of Human Subjects, Title 45 Code of Federal Regulations Part 46. United States of America; 2009.
- 16.
American College of Occupational and Environmental Medicine. Practice Guidelines for Low Back Disorders. MDGuidelines Reedgr ltd. 2018.
- 17.
State of California Department of Industrial Relations. Medical Treatment Utilization Schedule. 2019.
- 18.
Department of Industrial Relations. Medical Treatment Utilization Schedule—Drug Formulary. 2019.
- 19. Harris JS, Weiss MS, Haas NS, Hegmann KT, Holland JP, Kistner F, et al. Methodology for ACOEM’s Occupational Medicine Practice Guidelines-2017 Revision. J Occup Environ Med. 2017. pmid:28891890
- 20.
Department of Industrial Relations. Temporary Disability Benefits. Chapter 5. 2016. 21–25 p.
- 21.
United States Census Bureau. Concordances.
- 22.
Burea of Labor Statistics. Occupational Employment Statistics. 2017.
- 23.
National Center for O*NET Development. O*NET 24.2 Database. O*NET Resource Center. 2020.
- 24.
Centers for Medicare & Medicaid Services. Zip Code to Carrier Locality File—Revised 11/19/2019.
- 25. Quan H, Sundararajan V, Halfon P, Fong A. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 Administrative Data. 2005;43(11).
- 26.
R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria; 2019.
- 27. Morris ZS, wooding S, Grant J. The answer is 17 years, what is the question: Understanding time lags in translational research. J R Soc Med. 2011 Dec;104(12):510–20. pmid:22179294
- 28. Bornmann L, Mutz R. Growth Rates of Modern Science: A Bibliometric Analysis Based on the Number of Publications and. J Assoc Inf Sci Technol Technol. 2015;66(11):2215–22.
- 29.
Corlan AD. Medline trend: automated yearly statistics of PubMed results for any query. 2004.
- 30. Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, et al. Effect of Clinical Decision-Support Systems: A Systematic Review. Ann Intern Med. 2012 Jul;157(1):29. pmid:22751758
- 31. Qaseem A, Wilt TJ, McLean RM, Forciea MA. Noninvasive treatments for acute, subacute, and chronic low back pain: A clinical practice guideline from the American College of Physicians. Annals of Internal Medicine. 2017.
- 32.
National Institute for Health and Care Excellence. Low back pain and sciatica in over 16s: assessment and management. 2016; https://www.nice.org.uk/guidance/ng59