Citation: Minelli C, Baio G (2015) Value of Information: A Tool to Improve Research Prioritization and Reduce Waste. PLoS Med 12(9): e1001882. doi:10.1371/journal.pmed.1001882
Published: September 29, 2015
Copyright: © 2015 Minelli, Baio. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Funding: No funding was received for this work.
Competing interests: CM is senior investigator in the ALEC Study funded by the European Union's Horizon 2020 Research and Innovation programme under grant agreement No. 633212. GB is partially funded by a research grant sponsored by Mapi, a consultancy company working in the area of health economic evaluation. CM is a member of the PLOS Medicine editorial board.
Abbreviations: COPD, chronic obstructive pulmonary disease; NICE, National Institute for Health and Care Excellence; QALY, Quality-Adjusted Life Year; VOI, Value of Information
Provenance: Commissioned; not externally peer reviewed
At a time when the scale of investments has raised justifiable concerns about the ability of ongoing research to fulfill expectations , the long-run sustainability of research programs will depend on demonstration of value for money. Yet, there has been remarkably little recognition of the need to formally assess research value for money in funding allocation by national governments, funding agencies, and research institutions.
Currently, research priorities are mostly decided using subjective approaches based on consensus among experts, decision makers, and other stakeholders, which tend to lack transparency and may be unduly influenced by special interest groups. More objective measures have been developed based either on the burden of disease or on variations in clinical practice . Prioritization of diseases with the highest burden (morbidity, mortality, or aggregate societal costs) is useful in selecting general areas of neglect , but does not help identify what research should be undertaken within these areas. Prioritizing research in areas of disagreement in clinical practice can help practitioners decide between different clinical strategies and, by clarifying what is best practice, reduce variations. In the “clinical variations” method, priorities are defined based on welfare losses due to disagreement , with a cost-effectiveness element added in the “payback from research” method (“payback” referring to future savings as a result of the research investment) [5,6]. Setting research priorities based on variations in clinical practice, however, may not be ideal. Scientific uncertainty is not the only cause of clinical variations, which can also be due to poor implementation of research findings. Arguably, only the first should be addressed through additional research, and the second should be dealt with using more efficient means to promote good practice .
A decision-theoretic tool, known as “Value of Information” (VOI) [8,9], has been proposed to tackle the complexities of research prioritization in a more comprehensive way. Despite having been promoted and used for over a decade by the National Institute for Health and Care Excellence (NICE) in the United Kingdom , VOI is still relatively unknown to the medical scientific community.
What Is VOI and How Does it Work?
The VOI approach consists of a set of analytic tools that can be used to assess the value of acquiring additional evidence to inform a clinical (or public health) decision [8,9]. VOI quantifies the net benefit from the improvement of population health expected from additional research against the cost of implementation. Within this framework, the value of a study is the extent to which it reduces uncertainty on a particular topic, thus potentially reducing the errors in decision-making that would have been made, had less definitive evidence been used instead.
Fig 1 schematically describes the VOI approach. The starting point is a “statistical model” that estimates all relevant unknown quantities. For example, if decision-making is about implementation of a cancer screening program, relevant parameters would include sensitivity and specificity of the screening test, cancer prevalence, and health benefits (relative to mortality, morbidity, and quality of life) and costs associated with the clinical pathway with and without the proposed program. These quantities are linked by complex relationships and informed by composite sources of evidence, from published studies or resources directly available to the researcher. Evidence is synthesized using a probabilistic model, often developed within a Bayesian framework .
The results of the statistical model are fed into the “economic model,” which builds suitable population summaries for benefits and costs. In the example of cancer screening, we would compute the overall cost for both clinical pathways (with and without screening) by multiplying the average cost per person associated with the resources utilized (e.g., screening, diagnostic procedures, and treatments) by the expected number of users. Quality-Adjusted Life Years (QALYs), which combine changes in quantity and quality of life associated with an intervention, are often used as an average measure of clinical benefits. In our example, we would compute overall benefits of the program by multiplying QALYs gained through a successful diagnosis by the number of successful diagnoses, but we may also need to compute QALYs lost as a result of false-positive findings.
The output of the economic model provides the basis for the “decision analysis,” which applies a set of rules to determine the best course of action given current evidence. Any intervention (screening or treatment) is considered cost-effective if its cost-per-QALY does not exceed a pre-specified willingness-to-pay threshold, for example, £20,000–30,000 in the UK .
All parameters in the statistical model are subject to uncertainty. “Uncertainty analysis” (technically a probabilistic sensitivity analysis) assesses how much the limited knowledge about the parameters can impact the results of the decision analysis and quantifies the expected economic return of obtaining new evidence before committing to a decision. In the VOI approach, this is the crucial step informing “research prioritization.” VOI analysis may, for example, point to additional research on test sensitivity as the highest priority to reduce errors in the decision of whether to implement a screening program.
VOI analysis can be applied both to identify the crucial parameters within the appraisal of a single intervention and to compare different interventions. This kind of comparison is particularly important when allocating research funding across different topics and medical areas to maximize value for money and reduce waste.
Challenges and Opportunities
The comprehensiveness of VOI comes at the price of a complex analytic framework based on a number of modelling choices and assumptions that may influence the results. An important choice is the perspective from which costs are to be considered. For example, costs related to productivity losses are a crucial aspect when taking a societal perspective and dealing with diseases affecting a working-age population. Productivity costs will often show large uncertainty, and a societal perspective may well point to the need to acquire this information as a priority . In contrast, data on productivity costs are irrelevant from a health system perspective, in which the only costs considered are those related to health care. The results of VOI analyses also depend on the choice of health metrics. Although QALYs are far from perfect and have been criticized for relying on strong theoretical assumptions about consistency in people’s preferences , they represent a convenient common currency that allows comparison of health gains between different interventions and across different diseases.
VOI has classically been used for prioritization of research on health care screening and treatment interventions [14–18], but it can be equally useful in other areas, for example, risk prediction research. We are investigating its application within the Ageing Lungs in European Cohorts study (http://www.alecstudy.org), a European Commission–funded project aimed at developing a predictive risk score for chronic obstructive pulmonary disease (COPD). Although COPD is well known to be a smoking-related disease , there is evidence that other factors may contribute to a substantial proportion of its worldwide burden . However, in addressing COPD in those who are not exposed to tobacco smoke, it is difficult to decide which other lifestyle, environmental, clinical, or genetic potential predictors should receive research priority. VOI can address this issue by evaluating the relative value for money of different research strategies according to their expected contribution in reducing the uncertainty in COPD prediction and thus minimizing prediction errors.
VOI has been used in combination with other methods at different stages of the prioritization process [21,22], but the flexible Bayesian approach  also allows incorporation, directly within the VOI modelling framework, of the information on which other methods are based. Such information could include disease burden and variations in clinical practice, as well as experts’ opinions on parameters for which empirical evidence is limited or not available. Similarly, patients’ preferences could be formally incorporated in VOI analyses. For example, recognition of the importance of patients’ engagement in setting research agendas has motivated initiatives such as the “priority setting partnerships” created by the James Lind Alliance, in which patients and clinicians collaborate to identify research priorities (http://www.lindalliance.org).
While VOI has been increasingly employed by regulatory agencies to inform decisions about adoption and reimbursement of treatments, its uptake by institutions prioritizing and commissioning research has been much more limited, although arguably the need to improve credibility and transparency of decisions is equally important for the latter . Although VOI analyses are based on complex and computationally demanding modelling, statistical packages have been made freely available and fast methods recently developed to reduce the computational burden [23–25]. Dissemination outside the fields of health policy and health economics has also recently improved, and there are currently no real barriers to wider uptake of VOI in research prioritization.
Wrote the first draft of the manuscript: CM GB. Contributed to the writing of the manuscript: CM GB. Agree with the manuscript’s results and conclusions: CM GB.
- 1. Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gulmezoglu AM, et al. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383(9912):156–65. doi: 10.1016/S0140-6736(13)62229-1. pmid:24411644
- 2. Fleurence RL, Torgerson DJ. Setting priorities for research. Health policy. 2004;69(1):1–10. pmid:15484602 doi: 10.1016/j.healthpol.2003.11.002
- 3. Michaud CM, Murray CJ, Bloom BR. Burden of disease—implications for future research. Jama. 2001;285(5):535–9. pmid:11176854 doi: 10.1001/jama.285.5.535
- 4. Phelps CE, Parente ST. Priority setting in medical technology and medical practice assessment. Medical care. 1990;28(8):703–23. pmid:2117223 doi: 10.1097/00005650-199008000-00004
- 5. Buxton M, Hanney S. How can payback from health services research be assessed? Journal of health services research & policy. 1996;1(1):35–43. pmid:10180843
- 6. Davies L, Drummond M, Papanikolaou P. Prioritizing investments in health technology assessment. Can we assess potential value for money? International journal of technology assessment in health care. 2000;16(1):73–91. pmid:10815355 doi: 10.1017/s0266462300016172
- 7. Claxton KP, Sculpher MJ. Using value of information analysis to prioritise health research: some lessons from recent UK experience. PharmacoEconomics. 2006;24(11):1055–68. pmid:17067191 doi: 10.2165/00019053-200624110-00003
- 8. Claxton K, Posnett J. An economic approach to clinical trial design and research priority-setting. Health economics. 1996;5(6):513–24. pmid:9003938 doi: 10.1002/(sici)1099-1050(199611)5:6<513::aid-hec237>3.0.co;2-9
- 9. Howard RA. Information Value Theory. IEEE Transactions on Systems Science and Cybernetics; 1966. doi: 10.1109/tssc.1966.300074
- 10. Spiegelhalter DJ, Abrams KR, Myles JP. Bayesian Approaches to Clinical Trials and Health-Care Evaluation. West Sussex: Wiley; 2004.
- 11. Guide to the methods of technology appraisal. NICE; 2013. https://www.nice.org.uk/article/pmg9/resources/non-guidance-guide-to-the-methods-of-technology-appraisal-2013-pdf.
- 12. Mohseninejad L, van Baal PH, van den Berg M, Buskens E, Feenstra T. Value of information analysis from a societal perspective: a case study in prevention of major depression. Value in health: the journal of the International Society for Pharmacoeconomics and Outcomes Research. 2013;16(4):490–7. doi: 10.1016/j.jval.2012.12.007. pmid:23796282
- 13. Holmes D. Report triggers quibbles over QALYs, a staple of health metrics. Nature medicine. 2013;19(3):248. doi: 10.1038/nm0313-248. pmid:23467219
- 14. Bojke L, Claxton K, Sculpher MJ, Palmer S. Identifying research priorities: the value of information associated with repeat screening for age-related macular degeneration. Medical decision making: an international journal of the Society for Medical Decision Making. 2008;28(1):33–43. doi: 10.1177/0272989X07309638. . pmid:18263560
- 15. Dong H, Coyle D, Buxton M. Value of information analysis for a new technology: computer-assisted total knee replacement. International journal of technology assessment in health care. 2007;23(3):337–42. pmid:17579936 doi: 10.1017/s0266462307070419
- 16. Micieli A, Bennell MC, Pham B, Krahn M, Singh SM, Wijeysundera HC. Identifying future research priorities using value of information analyses: left atrial appendage occlusion devices in atrial fibrillation. Journal of the American Heart Association. 2014;3(5):e001031. doi: 10.1161/JAHA.114.001031. pmid:25227405
- 17. Tuffaha HW, Gordon LG, Scuffham PA. Value of information analysis in oncology: the value of evidence and evidence of value. Journal of oncology practice / American Society of Clinical Oncology. 2014;10(2):e55–62. doi: 10.1200/JOP.2013.001108. pmid:24194511
- 18. Woods B, Veenstra D, Hawkins N. Prioritizing pharmacogenetic research: a value of information analysis of CYP2D6 testing to guide breast cancer treatment. Value in health: the journal of the International Society for Pharmacoeconomics and Outcomes Research. 2011;14(8):989–1001. doi: 10.1016/j.jval.2011.05.048. pmid:22152167
- 19. Global Strategy for the Diagnosis, Management and Prevention of COPD. Global Initiative for Chronic Obstructive Lung Disease (GOLD); 2015. http://www.goldcopd.org/uploads/users/files/GOLD_Report_2015_Apr2.pdf.
- 20. Salvi SS, Barnes PJ. Chronic obstructive pulmonary disease in non-smokers. Lancet. 2009;374(9691):733–43. doi: 10.1016/S0140-6736(09)61303-9. pmid:19716966
- 21. Carlson JJ, Thariani R, Roth J, Gralow J, Henry NL, Esmail L, et al. Value-of-information analysis within a stakeholder-driven research prioritization process in a US setting: an application in cancer genomics. Medical decision making: an international journal of the Society for Medical Decision Making. 2013;33(4):463–71. doi: 10.1177/0272989X13484388. pmid:23635833
- 22. Wald HL, Leykum LK, Mattison ML, Vasilevskis EE, Meltzer DO. Road map to a patient-centered research agenda at the intersection of hospital medicine and geriatric medicine. Journal of general internal medicine. 2014;29(6):926–31. doi: 10.1007/s11606-014-2777-1. pmid:24557516
- 23. Heath A MI, Baio G. A Review of Methods for the Analysis of the Expected Value of Information. Pre-publication manuscript arXiv:150702513 [statAP]; 2015. http://arxiv.org/pdf/1507.02513v1.pdf.
- 24. Strong M, Oakley JE, Brennan A. Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach. Medical decision making: an international journal of the Society for Medical Decision Making. 2014;34(3):311–26. doi: 10.1177/0272989X13505910. pmid:24246566
- 25. Welton NJ, Thom HH. Value of Information: We've Got Speed, What More Do We Need? Medical decision making: an international journal of the Society for Medical Decision Making. 2015;35(5):564–6. doi: 10.1177/0272989X15579164. pmid:25840903