Figures
Abstract
Background
Health systems need to prioritize their services, ensuring efficiency and equitable health provision allocation and access. Alongside, health technology assessment (HTA) seeks to systematically evaluate various aspects of health technologies to be used by policy- and decision-makers. In the present study, we aim to identify strengths, weaknesses, opportunities, and threats in developing an HTA in Iran.
Method
This qualitative study was conducted using 45 semi-structured interviews from September 2020 to March 2021. Participants were selected from key individuals involved in health and other health-related sectors. Based on the objectives of the study, we used purposive sampling (snowball sampling) to select individuals. The range of length of the interviews was between 45 to 75 minutes. Four authors of the present study carefully reviewed the transcripts of interviews. Meanwhile, the data were coded on the four domains of strengths, weaknesses, opportunities, and threats (SWOT). Transcribed interviews were then entered into the software and analyzed. Data management was performed using MAXQDA software, and also analyzed using directed content analysis.
Results
Participants identified eleven strengths for HTA in Iran, namely the establishment of an administrative unit for HTA within the Ministry of Health and Medical Education (MOHME); university-level courses and degrees for HTA; adapted approach of HTA models to the Iranian context; HTA is mentioned as a priority on the agenda in upstream documents and government strategic plans. On the other hand, sixteen weaknesses in developing HTA in Iran were identified: unavailability of a well-defined organizational position for using HTA graduates; HTA advantages and its basic concept are unfamiliar to many managers and decision-makers; weak inter-sectoral collaboration in HTA-related research and key stakeholders; and, failure to use HTA in primary health care. Also, participants identified opportunities for HTA development in Iran: support from the political side for reducing national health expenditures; commitment and planning to achieve universal health coverage (on behalf of the government and parliament); improved communication among all stakeholders engaged in the health system; decentralization and regionalization of decisions; and capacity building to use HTA in organizations outside the MOHME. High inflation and bad economic situation; poor transparency in decisions; lack of support from insurance companies; lack of sufficient data to conduct HTA research; rapid change of managers in the health system; and economic sanctions against Iran are threats to the developmental path of HTA in Iran.
Citation: Behzadifar M, Ghanbari MK, Azari S, Bakhtiari A, Rahimi S, Ehsanzadeh SJ, et al. (2023) A SWOT analysis of the development of health technology assessment in Iran. PLoS ONE 18(3): e0283663. https://doi.org/10.1371/journal.pone.0283663
Editor: Ziqiang Zeng, Sichuan University, CHINA
Received: April 29, 2022; Accepted: March 14, 2023; Published: March 30, 2023
Copyright: © 2023 Behzadifar et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting information files.
Funding: The author(s) received no financial support for the research, authorship, and/or publication of this article.
Competing interests: The authors have declared that no competing interests exist.
Abbreviations: SWOT, Strengths, weaknesses, opportunities, threats; HTA, Health technology assessment; COREQ, Consolidated criteria for reporting qualitative research; UHC, Universal health coverage; LMIC, Low- and middle-income countries; SDGs, Sustainable development goals; MOHME, Ministry of health and medical education; PHC, Primary health care; FDA, Food and drug administration
Introduction
Healthcare systems need to prioritize health services, ensuring efficient and equitable access and allocation of health provisions. This is fundamental in order to improve healthcare performance to promote and enhance health at the community/population level [1]. In recent decades, with globally changing political and social conditions, health systems have undergone extensive reforms to cope with complexities and new emerging situations and improve health, which represents a societal onus [2]. On the other hand, available resources for health systems are limited, and in order to provide appropriate health services, decisions must be made about the interventions required, and how to organize the activities of the health systems to meet societal needs [3]. Also, communities expect to receive high-quality services based on modern equipment in an affordable and sustainable way [4].
Health technology assessment (HTA) is a process that seeks to systematically evaluate various aspects of health technologies to be used by policy- and decision-makers in making appropriate, evidence-based, and effective decisions about a given technology [5]. HTA is intended to provide information on various policies and decisions in the health system and is a strong foundation in the field of health sector research [6]. Health policymakers are looking to devise and implement effective interventions, and, as a result, HTA has been increasingly used worldwide. In various health circumstances, HTA is a very effective tool for decision-making that helps prioritize interventions and resource allocation, and assess evidence using a consultative process with the cooperation of various relevant stakeholders [7]. HTA findings can help overcome uncertainties and facilitate decision-making in a transparent way based on evidence that is accurate, timely, and usable [8].
Different countries, based on their political commitment to achieve the goals of universal health coverage (UHC), need to decide on various interventions using currently available evidence [9]. HTA can be utilized as a tool to set priorities, and is an effective way to ensure achieving goals. Low- and middle-income countries (LMICs) are implementing various UHC-related programs as part of their efforts to achieve sustainable development goals (SDGs) by 2030 [10]. The lack of sustainable funding, human resources, and effective research are some of the challenges in moving toward UHC in these countries; also, the institutionalization of HTA has become a daunting task [11]. Therefore, the use of HTA in these countries has become increasingly important due to the efforts and need to properly inform UHC-related decision-making processes regarding the selection of health technologies and effective interventions based on available resources [12]. There is also a wide range of technologies and interventions in the healthcare sector that should be based on the optimal use of resources [13]. The use of HTA can improve the process of making appropriate decisions, as well as the design and implementation of health system policies [14, 15].
The Iranian healthcare system
The public sector has the largest share in the delivery of health services and provisions in Iran [16]. The Ministry of Health and Medical Education (MOHME) is the main custodian of health and, in addition to training the human resources needed for health, is responsible for providing health services. Additionally, the government and insurance companies play a key role in financing the health system [17]. About 24,000 health centers across the country are responsible for providing primary healthcare (PHC) [18]. The PHC network is one of the most valuable achievements of the health system in Iran which has considerably improved the level of health in the country [19].
The MOHME is responsible for the identification of health needs, the implementation of health policies, and the delivery of healthcare provisions through an integrated network of medical, social, and educational services, aimed at reducing the burden of disease, and improving public and environmental health, as well as enhancing occupational health [20]. The MOHME oversees the activities of medical research institutions and units, monitoring research progress and outcomes, and validating and funding the programs of medical universities [21]. Moreover, the MOHME coordinates the activities of the PHC network, with a particular focus on maternal, child, and oral health. Finally, the MOHME is in charge of public assistance along with the development of the country’s health insurance system, which is one of the most important functions of any health system worldwide [22]. In Iran, both the public and private sectors are responsible for health insurance. Social security is the largest and most extensive insurance coverage program that provides insurance premiums through the participation of individuals. Military forces in Iran have also their own insurance program [23]. Finally, the Iranian Food and Drug Administration (FDA) is another governmental organization under the MOHME [20].
HTA in Iran
Based on the decision of health policy-makers in Iran, HTA-related activities started in 2007 in the form of a Secretariat. The initial stages of its formation took place with the participation and support of researchers in order to receive requests concerning health technology evaluation and submission of reports [24]. From the beginning of 2010 and under the supervision of the deputy of treatments and the office of technology evaluation, standardization, and health tariffs, the HTA office resumed its activities with a new structure [25]. Health policymakers in Iran decided to train human resources with HTA skills, hence the Master’s degree program in HTA was established. Currently, four medical universities are responsible for educating students in the HTA field. Due to the role and position of policies related to drugs and other health products in Iran, the HTA and the economic management study group have been formed within the FDA [25].
We conducted the present study to identify the strengths, weaknesses, opportunities, and threats of the HTA program development and implementation in Iran. The results of this study can be a potential roadmap for health policy- and decision-makers in Iran.
Methods
Ethical approval
This qualitative study was conducted from September 2020 to March 2021. The study was approved by the ethical committee at Lorestan University of Medical Sciences with the following approval code: IR.LUMS.REC.1399.112. All methods were performed in accordance with the relevant guidelines and regulations. Also, we conducted this study in compliance with the Declaration of Helsinki. In addition, participants were informed of the purpose of the study, and were asked to consent to be audio recorded. All participants gave written informed consent before the start of the interview.
Research framework
Strengths, weaknesses, opportunities, and threats were identified and analyzed using the SWOT analytical approach [26]. SWOT can help policy- and decision-makers in different sectors plan and implement various policies and programs as well as decision processes [27]. SWOT is a suitable tool for identifying internal (strengths and weaknesses) and external factors (opportunities and threats) that can affect the policy or program under study, and facilitate analysis of the subject and decisions [28]. The purpose of using SWOT is to show the positive forces that are involved in a policy or program, and the problems and challenges that need to be identified and implemented in the future [29]. This tool enables participants to share their views using a structured process. In this method, all features and relationships between internal and external factors of a phenomenon are considered [30].
In the existing scholarly literature, various studies related to HTA have been conducted using the SWOT framework. For instance, in a study conducted in Ukraine, SWOT was used to identify the strengths, weaknesses, opportunities, and threats of a proper creation and implementation of an HTA ecosystem in the country. The findings of this study showed that the establishment of a professional HTA organization based on the appraisal of the available body of evidence and the use of pharmacoeconomics is a strength, and the lack of a legal framework for establishing communication between HTA and stakeholders is a weakness. Creating a regulatory framework and developing plans for HTA communication with key actors was considered one of the most important opportunities [31].
In another study performed in Estonia, SWOT was used to assess a program aimed at developing HTA. The findings of this study showed that there are 16 strengths, 17 weaknesses, 17 opportunities, and 10 threats to establish HTA in this country. Some of the strengths included: providing access to information sources that others do not have and the multi-disciplinary nature of HTA. Mass media not being sufficiently involved and methodologies still evolving were deemed as weaknesses. Health reforms enabling the implementation of HTA-related activities and demand for transparency in decision-making were considered as opportunities, whilst depending on funding and unreasonably high expectations of HTA were seen as threats [32].
In Turkey, the SWOT framework was utilized to evaluate the development of an HTA program. The establishment of curated databases and information networks was considered a strength. Poor access to data or poor data quality, and traditional "expert-based" decision-making were identified as weaknesses. Increasing demand for the use of evidence and for transparency in decision-making was seen as an opportunity for the development of HTA in Turkey. The lack of dedicated funding was considered one of the most important threats in this study [30].
Interview development
Based on the SWOT framework, a questionnaire including four questions was prepared. As a pilot, to meet the validity and reliability criteria, the texts of the interviews were sent out to five experts (see S1 Appendix). They approved the questions, which were used in the interviews.
Participants
In-depth semi-structured interviews were used. Based on the objectives of the study, purposive sampling (snowball sampling) was used to select individuals. To maximize the data collected, as many participants as possible were interviewed.
Participants were selected from key individuals in the health and other health-related sectors. Individuals with 12 or more years of work experience and familiarity with HTA were eligible to participate. Some people did not participate in the study due to being too busy, not having time for interviews, and lack of interest. A total of 45 participants were interviewed. The overview of the participants is also presented in Table 1. Each participant was assigned a code. This study was conducted based on the location of the participants in different cities in Iran. The interviewers were selected to avoid any conflict of interest.
Data collection
Before the start of the interview, an invitation email was sent by M.B. and A.B. to the participants, in which the objectives of the study and related upstream documents were contained. Later the questionnaire was sent. All the interviews were conducted and recorded by an experienced and educated team consisting of three people (M.B., A.B., and M.K.G.), who have Ph.D. in the health policy and health management fields and have several years of experience related to health sector issues, qualitative research and HTA. They are currently working in health-related research centers, and had no previous relationship with the participants.
M.B., A.B., and M.K.G. at the beginning of the interviews, explained the details of the interview and the reason for choosing the participant. Due to the COVID-19 pandemic, they had to have the interviews on Skype. All the interviews were conducted in a non-English language (Farsi) and, considering that the interviews were conducted through Skype, participants were interviewed based on their preference at their workplace or home. With the consent of the participants, interviews were recorded through field notes and participants’ voices. The interviews lasted between 45 and 75 minutes with an average duration of 55 minutes and continued until data saturation was reached. No repeat interview was carried out.
Data analysis
After each interview, their text was transcribed word-for-word by hand and then typed. All interview transcripts were sent to the participants to confirm their accuracy and to ensure credibility and their feedback was collected. Regarding data saturation, a group discussion was conducted to achieve consistency among three researchers. As the original text of the interviews was in non-English language, their text was translated by a native author. If someone would like to have access to the original text of the interviews, they should contact the corresponding author.
For content analysis, Graneheim and Lundman’s strategy was used to analyze texts [33]. The steps of this strategy included: collecting the text of the interviews by the three researchers who conducted the interviews, writing the text of the interviews, initial coding of the texts, collecting the initial codes, and subdividing them according to the SWOT categories based on a careful comparative assessment in terms of similarities and differences. The data was, then, given to two researchers to review and agree on the method of coding. We finally prepared subsets based on the SWOT categories.
The transcripts of interviews were scrutinized by four authors (A.B., S.A, N.S, and S.M.) of the present study. Later, the data were coded according to the four domains of SWOT (strengths, weaknesses, opportunities, and threats). Data processing was performed using the MAXQDA software. In the next step, transcribed interviews were entered into the software. Also, all the steps including collecting, analyzing, coding, and placing based on the four SWOT items were reviewed by an external observer. Directed content analysis was also used for data analysis.
Managing research bias
To improve the reliability of the results and reduce bias in data analysis, we conducted standard assessments such as data credibility, transferability, dependability, and confirmability. After collecting and analyzing the data, we sent out to all the authors via email a copy of the data analysis and classification of the interviews based on the SWOT framework. Also, we consulted with five experts to increase the validity of the findings and their interpretations. All the experts confirmed the findings. Meanwhile, to present and promote acceptable results in this study, a wide and diverse range of participants was considered.
The consolidated criteria for reporting qualitative research (COREQ) checklist were followed (see S2 Appendix) [34].
Results
Table 2 shows the analysis of the interviews regarding HTA development in Iran, based on the four SWOT items. Participants mentioned 11 strengths for HTA in Iran: i) establishment of an administrative unit for HTA within the MOHME; ii) specific HTA-related university-level courses and degrees; iii) HTA being frequently used in decision making by some policy-makers and managers; iv) presenting HTA and its applications at national workshops and transferring knowledge; v) adapted approach of HTA models to the Iranian context; vi) HTA being mentioned and prioritized on the agenda in upstream documents, governmental bills and strategic plans related to the health sector; vii) scientific production and publications on HTA; viii) focusing on the efficiency of the health system among decision-makers and managers; ix) tendency to make optimal use of limited health resources; x) a gradual shift in the healthcare system toward evidence-based decision-making; and, xi) Health Transformation Plan (HTP) and other reforms in the health system (as overviewed in Table 3).
Sixteen weaknesses in the development of HTA in Iran were also identified. Despite the training of human resources, there are no suitable organizational positions available for them, and the skills of many of these people are still inadequate. Many policymakers are unfamiliar with the basis and objectives of HTA and do not make much use of evidence based on HTA. Cross-sectoral collaboration is still inappropriate for the development and implementation of HTA. Also, cooperation with international organizations is not very noticeable. HTA-related research is not very popular in the health system and only focuses on its clinical aspects. The subthemes and quotations of the weaknesses of developing the HTA in Iran are presented in Table 4.
The participants also identified opportunities for HTA development in Iran. The subthemes and quotations of the opportunities for developing the HTA in Iran are shown in Table 5. These included: support from the political side for reducing national health expenditures, commitment on behalf of the government and parliament and planning to achieve UHC; improved communication among all stakeholders engaged in the health system; decentralization and regionalization of decisions; increasing sensitivity of society and stakeholders for transparency in decisions and the use of HTA-based technologies and demand for new technologies; capacity building to use HTA in organizations outside of the MOHME; increased bargaining and reduced political lobbying; and economic sanctions against Iran.
Finally, the subthemes and quotations of the threats to development of the HTA in Iran are presented in Table 6. These include high inflation and bad economic situation; the desire to use expensive equipment without doing HTA; conflicts of interest and poor transparency on conflicts of interest in the health system; poor transparency on decisions, which should leverage evidence-based medicine; lack of support from some sectors, including insurance companies; increased demand for rapid introduction of technologies and lack of evaluation and sufficient data to conduct HTA research; rapid change of managers and lobbying in the health system. Resistance and the disbelief of policy- and decision-makers to the body of evidence provided by HTA as well as economic sanctions were identified as threats.
Discussion
The existence of an HTA organizational unit in the MOHME has made it possible for policymakers to get acquainted with HTA activities. The MOHME attempted to plan a reliable, evidence-based decision-making system by establishing and making use of HTA [24]. Having units or organizations for HTA can enable the coordination of various activities, and facilitate their development and implementation [35]. Having such a structure determines how tasks can be categorized, segmented, coordinated, and which and how goals can be achieved in a better and faster fashion. In the Iranian health system, it was decided to structure its activities based on the findings and insights of HTA [6, 36]. Additionally, having an organizational structure for reporting and evaluating HTA is, indeed, crucial. In the absence of an HTA unit, only a committee of a few people would review the quality of HTA reports, and this would not provide opportunities for other groups and stakeholders to engage in in-depth reviews. This would also create serious problems and issues such as the lack of transparency. In a study conducted in 10 countries, the results showed that the presence of HTA agencies and units makes the processes of policies related to drug pricing and economic approaches more transparent, leaving less room to the application of administrative discretion in health policies [37].
It seems that in Iran, a larger institution needs to be established to provide clear and accurate evidence to decision-makers in addition to being in charge of HTA processes, including the opportunity to reconsider the technologies currently used in the health system as well as previous decisions, or provide evidence using HTA on technologies that have already entered the health system without HTA. One of the general policies related to HTA to increase awareness is to create different disciplines connected with HTA topics [38].
Studies from other countries show that low awareness of HTA and its benefits can lead to a limited understanding of HTA concepts and the evaluation of different interventions for the health system; also, their prioritization is not considered. For this reason, the lack of knowledge and basic understanding of HTA is one of the challenges of implementation of HTA in many countries [39]. Regarding HTA training, holding permanent workshops and intensive courses may play an important role. In many successful HTA departments, efforts have been made to increase officials’ skills through continuing education [40, 41].
Our study shows that one of the weaknesses of MOHME is the lack of awareness of its policy- and decision-makers about the basis of HTA and its valuable benefits. This is in line with other studies evaluating HTA in LMICs [30–32].
Although faced with the financial problems, the MOHME has attempted to use the mentoring capacities in the country to train skilled HTA-related manpower [6]. It seems that manpower training should be part of a structured HTA program [36]. This trend has been considered by policymakers in the Iranian health system, and so far, skilled manpower has started conducting HTA research. Also, multidisciplinary tools and methods must be developed to enhance evidence for decision-making, its use in evidence-based processes, and their transparency [42].
One of the problems in Iran is the lack of various skills among trained people. In Turkey, as in Iran, people are very well trained in HTA, but they do not have the necessary skills, and are not able to share their experiences [43]. Very good training in various fields of HTA is necessary, but it is not enough, and people should also apply their knowledge in a practical way so that their skills to perform HTA activities can increase. In order to solve the existing inefficiencies in Iran’s health system, the low skill level of trained people should be taken into consideration and HTA training should be dynamic. Moreover, this problem can be solved with face-to-face courses and regular conferences. Studies have shown that the lack of trained people and the existence of HTA low-skill professionals are the main obstacles to the use of HTA [44].
Using an HTA country-wide model locally run and informed by local, relevant data is fundamental in those countries that are heterogeneous in terms of wealth and size. In this regard, many countries draw on the experiences of developed and high-income countries in relation to HTA [45]. However, different contexts and settings that are country-specific should be considered for using HTA. For example, political, social, cultural differences, local, health, and economic data can affect it; therefore, HTA differences in different countries should be taken into account [8]. Paying attention to these differences can play an important role in the evaluation process of a given technology for decision-makers. In addition to using international evidence, it is recommended that local evidence should also be collected and considered in making decisions [46]. One of the activities related to HTA in Iran could be designing HTA processes based on the country’s conditions. Although these activities are not enough, they have been recently understood by several policy- and decision-makers.
In order to have HTA processes implemented in Iran, we should create a database of all the people who are trained in HTA and have different skill levels [47]. On the other hand, the establishment of an HTA center independent of the Ministry of Health can be effective in the development of HTA in Iran. The experience of some countries has shown that with the creation of such centers and assistance from international organizations in the form of meetings, conferences or symposiums, the development of HTA has accelerated. Collaboration between local and international researchers is effective for creating and supporting the institutionalization of HTA, and global partners can support the development of HTA and also raise its awareness in a country [48].
The use of evidence is a common practice in many health systems in recent years, and, in the Iranian health sector, interest in using evidence to make more appropriate decisions is growing. However, it is not yet pervasive in all areas; on the other hand, it seems that in the future decisions will be based on evidence. The use of HTA in Iran is expanding, and this is a promising event.
In many countries, policy-makers need to prioritize the use of health system budgets and try to make suitable investments in the health field. Also, they want these investments to have a real impact on health outcomes and improve health care delivery. In some countries, despite the policy-makers’ interest in using HTA, their willingness to use HTA is low [49]. In this regard, different actors and stakeholders should be included in the agenda, as it has been done in several countries [50].
The MOHME alone cannot provide all the skills that students need, and there must be cross- and multi-sectoral cooperation, especially with organizations that can improve students’ skills. Manpower also needs to increase skills to build credible evidence. Some countries, due to a lack of skilled manpower, have difficulties in building and implementing an HTA network, and, despite the availability of strong guidelines for using HTA, are unable to use it in their healthcare system [30, 51].
Different areas and subdisciplines/subfields can be considered in HTA, and this directly points to its multi-sector nature. The cultural, social, legal, ethical, economic, technical, and medical criteria of an intervention in HTA should be examined, requiring special skills and competences [52]. In the curriculum of medical and pharmacy programs, there are some courses on the clinical aspect of technology, but other HTA skills have not been considered [53]. Developing and poor countries should work to minimize the costs of inappropriate decisions in the health sector; in doing so, one of their problems is allocating sufficient economic-financial resources to HTA. Due to budget deficit, capacity building for HTA is faced with many problems.
Various medical, ethical, economic, and social implications must be considered in HTA. The views of other stakeholders and organizations should also be taken into account [46]. In addition, cross-sectoral cooperation should be considered for the development of evidence-based multicultural decision making, in which the views of almost all stakeholders have to be involved [54]. The MOHME in Iran is currently the most important actor in HTA, but alone it cannot provide all the context for HTA and other stakeholders must participate.
In connection with HTA, there are various stakeholders who can influence it depending on the political, social and economic situations. In Turkey, various policies have been implemented to attract the participation of different stakeholders, raising awareness, holding meetings and conferences to increase the orientation/attitudes of different stakeholders and their use of HTA for decision-making. There are stakeholders beyond the health sector who can make critical decisions that should be involved [30].
Meanwhile, there is no extensive cooperation to use all the potential of the financial capacity of the private sector in relation to health in Iran. The main activities of the private sector are related to equipment and medicines, and of course, HTA mechanisms have not been widely adopted by them [52]. The private sector has a strong desire to introduce various technologies into the health sector in Iran, and will also get many financial benefits, but the use of their financial resources for HTA research has been neglected. It can be an incentive to support and allocate funding for HTA if appropriate mechanisms are developed, requiring the private sector to submit HTA reports that are evaluated by policy-makers. Of course, all these mechanisms should be open and transparent so that they do not exploit HTA findings to their advantage [55].
The private sector is one of the most important users of HTA. Health policies and decisions play an important role in the adoption and use of new technologies. In a study, the findings showed that the private sector’s use of HTA in technology-related decisions was limited, and these results are consistent with our study. It is, indeed, necessary for the MOHME and other stakeholders to bridge the gap between the private sector and the government in order to increase the participation of the private sector [52].
To apply HTA, sufficient resources must be available. Without financial and human resources, as well as ongoing training programs, the quality of HTA processes might be affected [52]. Also, depending on the scope of HTA activities, researchers should be provided with appropriate funding. Various budget sources (public or private) should be allocated to HTA if the process of using HTA in a health system is to be supported by governmental regulations. Of course, with large private sector participation, a large portion of the HTA funding needs can be met. The use of HTA is a mandatory element for the use of new technologies, pricing, and refunding in the health system, and the private sector can, in whole or in part, accelerate this process [52].
In addition, HTA graduates do not have a good organizational position in the MOHME structure, and HTA is still unknown in most parts of it. Due to the lack of employment opportunities, graduates are forced to engage in activities that are not related to HTA, which means that their abilities have not been used [36]. Also, the lack of awareness has created many problems for the development of HTA in other countries [3, 56]. When a policy-maker has little knowledge of HTA, they cannot be expected to use it in their decisions. It seems that the development of HTA training is limited to the training of students, and still there is no suitable program for training managers.
Not only for HTA, but also in other parts of the health system, the lack of development of digital infrastructure and its use is a major weakness. Although some policies have been implemented to develop digital infrastructure, they are not enough, which has led to the lack of reliable and comprehensive data. In many cases, relevant data is missing or incomplete in connection with research activities that are part of the HTA ecosystem. The use of HTA requires adequate infrastructure and data to provide more logical and robust evidence [57, 58].
In some countries, data infrastructure is not suitable, and therefore many policy-makers use international data. This can affect decisions due to the differences between local and international data [58]. Information infrastructure and data facilities should be created in every country and serious attention should be paid to their development. Studies have shown that the existence of appropriate information infrastructure can help managers and policy-makers make clear decisions in using HTA [59]. Also, policy-makers need short and practical reports on how to use HTA findings [60]. There is still no consensus on the HTA reporting framework. Also, many policy- and decision-makers do not have much time to read long reports due to their busy schedules.
Political and organizational dimensions can also affect the use of health evidence. The allocation of resources and the availability of sufficient manpower cannot guarantee the use of HTA by policy-makers in their decisions; in this regard, a strong legal framework must be provided to build and strengthen capacity and demand for HTA [61]. This legal framework can also shape all HTA activities, and influence evidence-based decisions in all areas of health [62].
Widrig et al.’s study addressed the use of a given technology, and the analysis of legal issues that plays an important role in decision-making [61]. Findings showed that although legal issues are an important part of HTA, these issues are generally overlooked and should be included in HTA. Paying attention to the legal issues of HTA in Iran should also be taken into account; thus, preparing a framework can be useful in this regard [63].
Political support for the development of HTA can have important implications for evidence-based decision-making and resource allocation in health systems. Political support for HTA in Iran has been increasing, and the scope of decision-making based on strong, evidence-based, and verifiable criteria has clearly increased. Meanwhile, the use of HTA requires political willingness, mechanisms developed to use HTA, and appropriate legislation to reduce political resistance to HTA. HTA-related legal infrastructure is being established in Iran, and given its benefits, policy-makers are enacting appropriate laws to use HTA in various health sectors.
In 2014, the MOHME launched the HTP which was implemented with the aim of accelerating UHC. The implementation of this plan caused more policy-makers to pay attention to the health sector [18]. With the implementation of this plan, the role of evidence became more prominent paving the path toward using HTA.
Also, Turkey’s experience showed that with political changes and reforms in the health sector, the development and use of HTA increased [30, 64]. In one of the WHO reports, it is mentioned that there are HTA networks and institutions in the Asia Pacific region, and those few HTA agencies that operate in LMICs have problems due to lack of political support [65].
Given challenges such as Iran’s economic sanctions, declining revenues, and increasing health sector spending, policy-makers in various sectors, including the health sector, have implemented policies to manage resources, to get the most out of the health sector at the lowest possible cost [66]. In addition, decision-making in the Iranian health system is top-down; thus, the implementation of many policies is difficult due to the lack of attention to social, economic, cultural, and health differences in various provinces in Iran [67]. Therefore, in recent years, decision-making regarding some policies has been decentralized.
Changes in the government in Iran have provided the ground for HTA [17, 18]. Due to the lack of financial resources in the health sector, in the field of medicine and medical equipment, strict monitoring has been adopted for the introduction of new interventions. On the other hand, to expand their constituency, many members of Parliament lobby for various services and equipment to improve the health of the people in their constituency. In many cases, they receive equipment that is not necessary, and impose high costs on the MOHME. This is especially seen in connection with the import of various diagnostic devices. With the political changes in Iran, lobbying has also increased, and it seems that strong evidence must be provided for the use of various equipment, and the ground is also being prepared for the use of HTA.
While lobbyists cannot be prevented in Iran, policy-makers can be encouraged to use HTA in decision-making in order to use effectively health sector resources [68]. For example in France, with the development of HTA, the strong influence of political issues, and decision-making related to various interventions in health sector, the use of HTA has become a powerful lever in this country, strongly influencing decisions related to interventions that has led to the quality of healthcare provisions and public health [69].
Iran’s economic sanctions have caused high inflation and many economic problems for the people; also, the budget of many programs has been reduced [70]. Decreased funding means that decision-makers are not interested in using new technologies, and despite the existence of a new technology and HTA’s emphasis on using it, the health sector cannot have it.
On the other hand, the tendency to use expensive technology in Iran is expanding. Some individuals and companies import these technologies without considering HTA, which incurs high costs to the health sector and society. The greater danger is that they create a public demand for expensive and unnecessary technologies for the public sector that the public sector may not be able to use it [25].
Also, studies show that there is an increasing demand for the use of expensive and new technologies in the health sector, and policy-makers are not quick to approve them and limit their use [68]. There must be a road map to use new and expensive technologies, and HTA can be the best option. One of the major threats in all areas of health is the conflict of interest [68]. Especially in the field of medicine and equipment, this conflict causes several disadvantages. In Iran, too, many policy-makers have stakes in various health-related companies [71].
Also, strengthening HTA processes will make decisions become more transparent. Taking into account potential conflicts of interest in Iran’s health system has not received enough attention yet. In some cases, policy-makers have made decisions in their favor, and the uptake of technology innovations is directly related to their position or role. Various studies have shown that if the conflict of interest in decisions is reduced, HTA can become a more transparent and valuable option [68].
HTA is not just evaluation of technologies but can be used in various areas of health. The use of HTA in insurance and reimbursement issues is very important and it is applied in different countries [72]. However, in Iran, the use of HTA by insurers is not well accepted yet. Financial issues are one of the most important aspects of insurance. If insurers use HTA, they can also use its strengths in managing financial resources, reimbursement, and developing service packages [73].
Studies show that insurers have an important role to play in achieving UHC, and in this regard, they try to use HTA as a primary tool for preparing their lists of covered drugs and reimbursements [1, 74].
Although HTA and its role have been emphasized by decision-makers in various countries, there are still high-level executives who do not believe much in evidence-based approaches for the implementation of a program. In Iran, at different levels of managers, there are different views on the use of technologies and HTA [24]. Some policy- and decision-makers are against HTA and do not believe in it, potentially because of inadequate knowledge about HTA, or conflicts of interest [75].
Iran’s economic-financial resources for buying and utilizing expensive technologies are limited. Despite the achievements in improving the level of health, Iran still has a long way to reach a desirable state of health; therefore, Iran needs to use new technologies that help shortcut this path. The pricing logic of new technologies in Iran faces many challenges, and accordingly, health sector managers must make great efforts to obtain political support to enact HTA-related laws.
This study suffers from some limitations. We did not have access to all the experts in the field of HTA, so we could not get their opinions. Due to COVID-19, it was not possible to interview face-to-face and interact more appropriately with the participants. Another limitation of this study is that different points of view can be applied regarding the classification of items in the four SWOT categories, and an opportunity can be considered a strength by another person. Also, one person can see an item as a threat and another person as an opportunity. In this study, efforts were made to consider the most common opinions in such a way to reach a consensus.
Conclusion
HTA plays an important role in the healthcare sector of all countries; also, health systems have been increasingly using it to manage their resources. Findings of this study show that, based on the SWOT analysis, in the development of HTA in Iran, four dimensions of SWOT should be considered. Findings also show that Iran has a long way to take for the development of a cutting-edge HTA. A wide range of strengths, weaknesses, opportunities, and threats were identified for a proper development of HTA that should be addressed by policy- and decision-makers.
Supporting information
S1 Appendix. The four questions of the questionnaire based on the SWOT framework, which was used during the interview.
https://doi.org/10.1371/journal.pone.0283663.s001
(DOCX)
S2 Appendix. Consolidated criteria for reporting qualitative studies (COREQ): 32-item checklist.
https://doi.org/10.1371/journal.pone.0283663.s002
(DOCX)
References
- 1. Uzochukwu BSC, Okeke C, O’Brien N, Ruiz F, Sombie I, Hollingworth S. Health technology assessment and priority setting for universal health coverage: a qualitative study of stakeholders’ capacity, needs, policy areas of demand and perspectives in Nigeria. Global Health. 2020;16(1):58. pmid:32641066
- 2. Leelahavarong P, Doungthipsirikul S, Kumluang S, Poonchai A, Kittiratchakool N, Chinnacom D, et al. Health Technology Assessment in Thailand: Institutionalization and Contribution to Healthcare Decision Making: Review of Literature. Int J Technol Assess Health Care. 2019;35(6):467–73. pmid:31190670
- 3. Hollingworth S, Fenny AP, Yu SY, Ruiz F, Chalkidou K. Health technology assessment in sub-Saharan Africa: a descriptive analysis and narrative synthesis. Cost Eff Resour Alloc. 2021;19(1):39. pmid:34233710
- 4. Kriza C, Hanass-Hancock J, Odame EA, Deghaye N, Aman R, Wahlster P, et al. A systematic review of health technology assessment tools in sub-Saharan Africa: methodological issues and implications. Health Res Policy Syst. 2014;12:66. pmid:25466570
- 5. Alkhaldi M, Al Basuoni A, Matos M, Tanner M, Ahmed S. Health Technology Assessment in High, Middle, and Low-income Countries: New Systematic and Interdisciplinary Approach For Sound Informed-policy Making: Research Protocole. Risk Manag Healthc Policy. 2021;14:2757–70. pmid:34234591
- 6. Almomani E, Alabbadi I, Fasseeh A, Al-Qutob R, Al-Sharu E, Hayek N, et al. Implementation Road Map of Health Technology Assessment in Middle-Income Countries: The Case of Jordan. Value Health Reg Issues. 2021;25:126–34. pmid:34015521
- 7. Pichon-Riviere A, Augustovski F, García Martí S, Alfie V, Sampietro-Colom L. The link between health technology assessment and decision making for the allocation of health resources in Latin America. Int J Technol Assess Health Care. 2020;36(2):173–8. pmid:32312340
- 8. Csanádi M, Inotai A, Oleshchuk O, Lebega O, Alexandra B, Piniazhko O, et al. Health Technology Assessment Implementation in Ukraine: Current Status and Future Perspectives. Int J Technol Assess Health Care. 2019;35(5):393–400. pmid:31583985
- 9. Chalkidou K, Marten R, Cutler D, Culyer T, Smith R, Teerawattananon Y, et al. Health technology assessment in universal health coverage. Lancet. 2013;382(9910):e48–9. pmid:24360390
- 10. Hollingworth S, Gyansa-Lutterodt M, Dsane-Selby L, Nonvignon J, Lopert R, Gad M, et al. Implementing health technology assessment in Ghana to support universal health coverage: building relationships that focus on people, policy, and process. Int J Technol Assess Health Care. 2020;36(1):8–11. pmid:31775950
- 11. Sharma M, Teerawattananon Y, Luz A, Li R, Rattanavipapong W, Dabak S. Institutionalizing Evidence-Informed Priority Setting for Universal Health Coverage: Lessons From Indonesia. Inquiry. 2020;57:46958020924920. pmid:32513029
- 12. Oraro-Lawrence T, Wyss K. Policy levers and priority-setting in universal health coverage: a qualitative analysis of healthcare financing agenda setting in Kenya. BMC Health Serv Res. 2020;20(1):182. pmid:32143629
- 13. Kamae I, Thwaites R, Hamada A, Fernandez JL. Health technology assessment in Japan: a work in progress. J Med Econ. 2020;23(4):317–22. pmid:31944134
- 14. Ritrovato M, Faggiano FC, Tedesco G, Derrico P. Decision-Oriented Health Technology Assessment: One Step Forward in Supporting the Decision-Making Process in Hospitals. Value Health. 2015.18(4):505–11. pmid:26091605
- 15. Rosselli D, Quirland-Lazo C, Csanádi M, Ruiz de Castilla EM, González NC, Valdés J, et al. HTA Implementation in Latin American Countries: Comparison of Current and Preferred Status. Value Health Reg Issues. 2017;14:20–7. pmid:29254537
- 16. Doshmangir L, Moshiri E, Farzadfar F. Seven Decades of Primary Healthcare during Various Development Plans in Iran: A Historical Review. Arch Iran Med. 2020;23(5):338–52. pmid:32383619
- 17. Doshmangir L, Bazyar M, Majdzadeh R, Takian A. So Near, So Far: Four Decades of Health Policy Reforms in Iran, Achievements and Challenges. Arch Iran Med. 2019;22(10):592–605. pmid:31679362
- 18. Olyaeemanesh A, Behzadifar M, Mousavinejhad N, Behzadifar M, Heydarvand S, Azari S, et al. Iran’s Health System Transformation Plan: A SWOT analysis. Med J Islam Repub Iran. 2018;32:39. pmid:30159290
- 19. Letafat M, Beyranvand T, Aryankhesal A, Behzadifar M, Behzadifar M. Universal Health Coverage (UHC) in Iran. Iran J Public Health. 2018;47(7):1061–2. pmid:30182014
- 20. GBD 2019 Iran Collaborators. Health system performance in Iran: a systematic analysis for the Global Burden of Disease Study 2019. Lancet. 2022 399(10335):1625–45. pmid:35397236
- 21. Doshmangir L, Bazyar M, Rashidian A, Gordeev VS. Iran health insurance system in transition: equity concerns and steps to achieve universal health coverage. Int J Equity Health. 2021;20(1):37. pmid:33446202
- 22. Behzadifar M, Ghanbari MK, Ravaghi H, Bakhtiari A, Shahabi S, Doshmangir L, et al. Health policy analysis in Eastern Mediterranean region using a health policy triangle framework: Historical and ethical insights from a systematic review. J Prev Med Hyg. 2022;63(2):E351–E73. pmid:35968073
- 23. Mohamadi E, Takian A, Olyaeemanesh A, Rashidian A, Hassanzadeh A, Razavi M, et al. Health insurance benefit package in Iran: a qualitative policy process analysis. BMC Health Serv Res. 2020;20(1):722. pmid:32762695
- 24. Doaee S, Olyaeemanesh A, Emami S, Mobinizadeh M, Abooee P, Nejati M, et al. Development and implementation of health technology assessment: a policy study. Iran J Public Health. 2013;42(Supple1):50–4. pmid:23865016
- 25. Arab-Zozani M, Sokhanvar M, Kakemam E, Didehban T, Hassanipour S. History of Health Technology Assessment in Iran. Int J Technol Assess Health Care. 2020;36(1):34–9. pmid:31928553
- 26. Christiansen T. A SWOT analysis of the organization and financing of the Danish health care system. Health Policy. 2002;59(2):99–106. pmid:11755992
- 27. Manzini F, Diehl EE, Farias MR, Dos Santos RI, Soares L, Rech N, et al. Analysis of a Blended, In-Service, Continuing Education Course in a Public Health System: Lessons for Education Providers and Healthcare Managers. Front Public Health. 2020;8:561238. pmid:33324594
- 28. Zon H, Pavlova M, Groot W. Decentralization and health resources transfer to local governments in Burkina Faso: A SWOT analysis among health care decision makers. Health Sci Rep. 2019;2(6):e119. pmid:31236487
- 29. Buttigieg SC, Schuetz M, Bezzina F. Value Chains of Public and Private Health-care Services in a Small EU Island State: A SWOT Analysis. Front Public Health. 2016;4:201. pmid:27683658
- 30. Kahveci R, Meads C. Analysis of strengths, weaknesses, opportunities, and threats in the development of a health technology assessment program in Turkey. Int J Technol Assess Health Care. 2008;24(2):235–40. pmid:18400128
- 31. Piniazhko O, Dumenko T, Malyshevska I, Dudlei M, Romanenko I, Dvoieglazova M, et al. PNS211 SWOT analysis for creation of HTA ecosystem in Ukraine. Value Health. 2019;22:S796.
- 32. Gibis B, Artiles J, Corabian P, Meiesaar K, Koppel A, Jacobs P, et al. Application of strengths, weaknesses, opportunities and threats analysis in the development of a health technology assessment program. Health Policy. 2001;58(1):27–35. pmid:11518600
- 33. Graneheim UH, Lundman B. Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness. Nurse Educ Today. 2004;24(2):105–12. pmid:14769454
- 34. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57. pmid:17872937
- 35. Yazdizadeh B, Mohtasham F, Velayati A. Impact assessment of Iran’s health technology assessment programme. Health Res Policy Syst. 2018;16(1):15. pmid:29471838
- 36. Fasseeh A, Karam R, Jameleddine M, George M, Kristensen FB, Al-Rabayah AA, et al. Implementation of Health Technology Assessment in the Middle East and North Africa: Comparison Between the Current and Preferred Status. Front Pharmacol. 2020;11:15. pmid:32153393
- 37. Mazumder D, Kapoor A, Gwatkin N, Medeiros C. A quantitative analysis of ten Health Technology Assessment (HTA) agencies for attributes driving transparency of the organization. Value in Health. 2015;18:A855.
- 38. Kaló Z, Landa K, Doležal T, Vokó Z. Transferability of National Institute for Health and Clinical Excellence recommendations for pharmaceutical therapies in oncology to Central-Eastern European countries. Eur J Cancer Care (Engl). 2021;21(4):442–9.
- 39. Mueller D. Addressing the challenges of implementing a Health Technology Assessment Policy Framework in South Africa. Int J Technol Assess Health Care. 2020;13:1–6. pmid:32787995
- 40. Tantivess S, Chalkidou K, Tritasavit N, Teerawattananon Y. Health Technology Assessment capacity development in low- and middle-income countries: Experiences from the international units of HITAP and NICE. F1000Res. 2017;6:2119. pmid:29333249
- 41. Pichler F, Oortwijn W, Ruether A, Trowman R. Defining capacity building in the context of HTA: a proposal by the HTAi Scientific Development and Capacity Building Committee. Int J Technol Assess Health Care. 2019;35(5):362–6. pmid:31506132
- 42. Doherty JE, Wilkinson T, Edoka I, Hofman K. Strengthening expertise for health technology assessment and priority-setting in Africa. Glob Health Action. 2017;10(1):1370194. pmid:29035166
- 43. Ozturk K, Karadayı B, Şener O. Stakeholders’ perceptions of health technology assessment in Turkey. Int J Technol Assess Health Care. 2018;34(1):97–104. pmid:29151380
- 44. Serrano-Aguilar P, Asua-Batarrita J, Molina-López MT, Espallargues M, Pons-Rafols J, García-Armesto S, et al. The Spanish Network of Agencies for Health Technology Assessment and Services of the National Health System (RedETS). Int J Technol Assess Health Care. 2019;35(3):176–80. pmid:31006412
- 45. Hass B, Pooley J, Feuring M, Suvarna V, Harrington AE. Health technology assessment and its role in the future development of the Indian healthcare sector. Perspect Clin Res. 2012;3(2):66–72. pmid:22701823
- 46. Kido K, Matsumaru N, Tsukamoto K. Health Technology Assessment in Japan: A Pharmaceutical Industry Perspective. Ther Innov Regul Sci. 2019;53(4):472–80. pmid:30157674
- 47. Cavazza M, Jommi C. Stakeholders involvement by HTA Organisations: why is so different? Health Policy. 2012;105(2–3):236–45. pmid:22364715
- 48. Vokó Z, Cheung KL, Józwiak-Hagymásy J, Wolfenstetter S, Jones T, Muñoz C, et al. EQUIPT Study Group. Similarities and differences between stakeholders’ opinions on using Health Technology Assessment (HTA) information across five European countries: results from the EQUIPT survey. Health Res Policy Syst. 2016;14(1):38.
- 49. Paolucci F, Redekop K, Fouda A, Fiorentini G. Decision Making and Priority Setting: The Evolving Path Towards Universal Health Coverage. Appl Health Econ Health Policy. 2017;15(6):697–706. pmid:28871512
- 50. Mitton C, Seixas BV, Peacock S, Burgess M, Bryan S. Health Technology Assessment as Part of a Broader Process for Priority Setting and Resource Allocation. Appl Health Econ Health Policy. 2019;17(5):573–6. pmid:31161365
- 51. Guzvic V, Catic T, Kostic M. Health technology assessment in CentraL-Eastern and South Europe countries: Bosnia and Herzegovina. Int J Technol Assess Health Care. 2017;33(3):390–5. pmid:28434417
- 52. Trosman JR, Van Bebber SL, Phillips KA. Health technology assessment and private payers’s coverage of personalized medicine. Am J Manag Care. 2011: 17 Suppl 5 Developing::SP53–60.
- 53. Dams F, Gonzalez Rodriguez JL, Cheung KL, Wijnen BFM, Hiligsmann M. Relevance of barriers and facilitators in the use of health technology assessment in Colombia. J Med Econ. 2018;21(5):510–7. pmid:29513062
- 54. Chen Y, He Y, Chi X, Wei Y, Shi L. Development of health technology assessment in China: New challenges. Biosci Trends. 2018;12(2):102–8. pmid:29760354
- 55. Rubinstein A, Pichon-Riviere A, Augustovski F. Development and implementation of health technology assessment in Argentina: two steps forward and one step back. Int J Technol Assess Health Care. 2009;25 Suppl 1:260–9. pmid:19527545
- 56. Cheung KL, Evers SMAA, de Vries H, Hiligsmann M. Most important barriers and facilitators regarding the use of health technology assessment. Int J Technol Assess Health Care. 2017;33(2):183–91. pmid:28849758
- 57. Goeree R, Levin L, Chandra K, Bowen JM, Blackhouse G, Tarride JE, et al. Health technology assessment and primary data collection for reducing uncertainty in decision making. J Am Coll Radiol. 2009;6(5):332–42. pmid:19394574
- 58. Raftery J, Roderick P, Stevens A. Potential use of routine databases in health technology assessment. Health Technol Assess. 2005;9(20):1–92, iii–iv. pmid:15899148
- 59. Treadwell JR, Erinoff E, Coates V. How electronic clinical data can improve health technology assessment. EGEMS (Wash DC). 2013;1(2):1028. pmid:25848573
- 60. Draborg E, Gyrd-Hansen D, Poulsen PB, Horder M. International comparison of the definition and the practical application of health technology assessment. Int J Technol Assess Health Care. 2005;21(1):89–95. pmid:15736519
- 61. Widrig D, Tag B. HTA and its legal issues: a framework for identifying legal issues in health technology assessment. Int J Technol Assess Health Care. 2014;30(6):587–94. pmid:25816824
- 62. Sihvo S, Ikonen T, Mäkelä M. Implementing health technology assessment-based recommendations in Finland: managed uptake of medical methods. Int J Technol Assess Health Care. 2017;33(4):430–43. pmid:28703090
- 63. Stühlinger V. Health Technology Assessment (HTA) and Access Policies. Eur J Health Law. 2020;27(3):274–89. pmid:33652403
- 64. Dundar M, Uzak AS, Karabulut Y. Healthcare in overview of Turkey. EPMA J. 2010;1(4):587–94. pmid:23199111
- 65.
Factors Conducive to the Development of Health Technology Assessment in Asia 2015 [updated 25 September 2022. https://apo.who.int/publications/i/item/9789290617341.
- 66. Akbarialiabad H, Rastegar A, Bastani B. How Sanctions Have Impacted Iranian Healthcare Sector: A Brief Review. Arch Iran Med. 2021;24(1):58–63. pmid:33588569
- 67. Rahimi SA, Alizadeh M, Légaré F. Shared decision making in Iran: Current and future trends. Z Evid Fortbild Qual Gesundhwes. 2017 123–124:52–5. pmid:28549749
- 68. Frybourg S, Remuzat C, Kornfeld Å, Toumi M. Conflict of interest in Health Technology Assessment decisions: case law in France and impact on reimbursement decisions. J Mark Access Health Policy. 2015;3(1):25682. pmid:29785249
- 69. Weill C, Banta D. Development of health technology assessment in France. Int J Technol Assess Health Care. 2009:108–11. pmid:19534838
- 70. Olyaeemanesh A, Majdzadeh R. Health technology assessment: A necessity in post-sanctions Iran while implementing the health transformation plan. Med J Islam Repub Iran. 2016;30:436. pmid:28210601
- 71. Sarrafkia A, Nedjat S, Majdzadeh R. Future of Health Technology Assessment System in Iran: Necessity of Reinforcement in Both Scenarios of Continuation or Relief of Sanctions. Health Tech Asmnt Act. 2020;4(2):e6227.
- 72. Oortwijn W, Mathijssen J, Banta D. The role of health technology assessment on pharmaceutical reimbursement in selected middle-income countries. Health Policy. 2010;95(2–3):174–84. pmid:20074829
- 73. Kamae I. Value-based approaches to healthcare systems and pharmacoeconomics requirements in Asia: South Korea, Taiwan, Thailand and Japan. Pharmacoeconomics. 2010;28(10):831–8. pmid:20831290
- 74. Scintee SG, Ciutan M. Development of health technology assessment IN Romania. Int J Technol Assess Health Care. 2017;33(3):371–5. pmid:28436346
- 75. Vukovic V, Favaretti C, Ricciardi W, de Waure C. Health technology assessment evidence on e-health/m-health technologies: evaluating the transparency and thoroughness. Int J Technol Assess Health Care. 2018;34(1):87–96. pmid:29455685